Dataset (magpie API)

java.lang.Object
- magpie.data.Dataset

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Citable, Commandable, Options, Printable, Savable

Direct Known Subclasses:

MultiPropertyDataset
```
public class Dataset
extends java.lang.Object
implements java.io.Serializable, java.lang.Cloneable, Printable, Savable, Options, Commandable, Citable
```
Provides a basic storage container for data-mining tasks. Must be filled with entries that are subclasses of BaseEntry
To implement a new Dataset, you first need to create an extension of BaseEntry that represents the new kind of data. Then, you need to overload the following operations:
- getEntry(int) - Might be useful to overload with an operation that returns entry type associated with this model
- addEntry(java.lang.String) - Call the constructor to the associated entry type
- calculateAttributes() - Compute any new attributes for for class
- emptyClone() - Create clone of dataset. Cloning entries is handled in clone(), which does not need to be modified.
Usage: *No options to set*
Implemented Commands:
add <entries...> - Add entries to a dataset
entries...: Strings describing entries to be added
add $<dataset> [-force] - Add entries from another dataset
dataset: Dataset to be merged with this one
-force: Optional: Whether to force merge if attributes / classes / properties are different
If attributes, classes, or properties are different, attributes and class values in new entries (i.e., those from the other dataset) will be deleted and properties will be merged
<output> = clone [-empty] - Create a copy of this dataset
-empty: Do not copy entries from dataset into clone
combine $<dataset> - Add all entries from another dataset
dataset: Dataset to merge with this one. It will remain unchanged.
duplicates <resolver> [<resolver options>] - Eliminate duplicates within a dataset
resolver: Name of BaseDuplicateResolver used to handle duplicates
resolver options: Any options for the resolver
filter <include|exclude> <method> [<options...>] - Run dataset through a filter
include|exclude: Whether to include/exclude only entries that pass the filter
method: Filtering method. Name of a BaseDatasetFilter
options...: Options for the filter
generate <method> [>options<] - Generate new entries
method: Name of a BaseEntryGenerator.
options: Any options for the entry generator
match $<dataset> <num> - Find most similar entries in this dataset
dataset: Dataset containing entries to be matched
num: Number of closest entries to print
Prints the most similar entries in this dataset to those in the dataset passed as the argument.
modify <method> [<options>] - Modify the dataset
method: How to modify dataset. Name of a BaseDatasetModifier.
options: Any options for the dataset
import <filename> [<options...>] - Import data by reading a file
filename: Name of file to import data from
options...: Any options used when parsing this dataset (specific to type of Dataset)
subtract $<data> - Subtract another dataset from this one
data: Data to be subtracted from this dataset
After this operation, this dataset will only contain entries not in data
rank <number> <maximum|minimum> <measured|predicted> <method> [<options>] - Print the top ranked entries based by some measure
number: Number of top entries to print
maximum|minimum: Whether to print entries with the largest or smallest objection function
measured|predicted: Whether to use the measured or predicted values when calculation
method: Object function used to rank entries. Name of a BaseEntryRanker
options...: Any options for the objective function
<output> = split <number|fraction> - Randomly select and remove entries from dataset
number|fraction: Either the fraction or number of entries to be removed
output: New dataset containing randomly selected entries that were in this dataset
<output> = subset <number|fraction> - Generate a random subset from this dataset
number|fraction: Either the fraction or number of entries to select
output: New dataset containing random selection from this dataset
attributes - Print all attributes
attributes clear - Clear all attribute data
attributes expanders add <method> [<options...>] - Add an attribute expander to be run after generating attributes
method: How to expand attributes. Name of a BaseAttributeExpander
options...: Any options for the expansion method These expanders are designed to create new attributes based on existing ones.
attributes expanders clear - Clear the current list of attribute expanders
attributes expanders run - Run the currently-defined list of attribute expanders
attributes generators add <method> [<options...>] - Add a new attribute generator to list of generators
method: New generation method. Name of a BaseAttributeGenerator
options...: Any options for the generator method These expanders are designed to create new attributes tailored for a specific application.
attributes generators run - Run the currently-defined list of attribute expanders
attributes generators clear - Clear the current list of attribute generators
attributes generate - Generate attributes for each entry
attributes rank <number> <method> [<options...>] - Rank attributes based on predictive power
number: Number of top attributes to print
method: Method used to rank attributes. Name of a BaseAttributeEvaluator
options...: Options for the evaluation method.
Implemented Print Commands:
details - Print details about this class
dist - Print distribution of entries between known classes
description - Print description of this dataset
Implemented Save Formats: TBD
csv - Comma-separated value format.
The value of each attribute and the measured class variable, if defined.
arff - Weka's ARFF format.
Requires that a measured value is available for the class variable of each entry.
stats - Writes predicted and measured class variables.
This is intended to allow an external program to evaluate model performance.
template - Save an empty clone of the dataset using serialization
json - Save dataset into JSON format
See Also:

Serialized Form

Field Summary

Fields
Modifier and Type	Field and Description
`protected java.util.ArrayList<java.lang.String>`	`AttributeName` Names of attributes that describe each entry
`protected java.util.ArrayList<BaseEntry>`	`Entries` Internal array that stores entries
`protected java.util.List<BaseAttributeExpander>`	`Expanders` Tools to generate new attributes based on existing ones
`protected java.util.List<BaseAttributeGenerator>`	`Generators` Tools to generate special-purpose attributes

Constructor Summary

Constructors
Constructor and Description

Dataset()
Generate a blank dataset

Constructors
Constructor and Description
`Dataset()` Generate a blank dataset

Method Summary

All Methods Static Methods Instance Methods Concrete Methods
Modifier and Type	Method and Description
`java.lang.String`	`about()` Prints a simple status message about this object
`void`	`addAttribute(java.lang.String name, double[] values)` Add in a new attribute.
`void`	`addAttributeExpander(BaseAttributeExpander expander)` Add a new tool to expand the number of attributes for this dataset.
`void`	`addAttributeGenerator(BaseAttributeGenerator generator)` Add a new tool to generate additional attributes
`void`	`addAttributes(java.util.List<java.lang.String> names)` Add new attributes.
`void`	`addEntries(java.util.Collection<? extends BaseEntry> entries)` Add many entries to a the data set
`void`	`addEntries(Dataset otherDataset, boolean forceMerge)` Add entries from another dataset.
`void`	`addEntry(BaseEntry e)` Add an entry.
`BaseEntry`	`addEntry(java.lang.String input)` A new entry by parsing an input string.
`protected void`	`calculateAttributes()` Compute attributes that are specific to this class.
`void`	`clearAttributeExpanders()` Reset the list of attribute expanders
`void`	`clearAttributeGenerators()` Reset the list of attribute generators
`void`	`clearAttributes()` Clear attribute data.
`void`	`clearData()` Clear out all entries
`Dataset`	`clone()` Creates a new instance of this dataset, and clones of each entry.
`void`	`combine(java.util.Collection<Dataset> d)` Combine the data structure with a collection of other data structures.
`void`	`combine(Dataset d)` Combine the data structure with another.
`void`	`combine(Dataset[] d)` Combine the data structure with an array of other Datasets.
`boolean`	`containsEntry(BaseEntry Entry)` Determine whether a dataset contains a certain entry
`Dataset`	`createTemplate()` Create a template designed to be saved using serialization
`void`	`deleteMeasuredClasses()` Delete all measured classes from entries
`void`	`deletePredictedClasses()` Delete all predicted classes from entries
`Dataset`	`emptyClone()` Creates a new instance with the same class and attribute names, but without any entries.
`protected void`	`finalizeGeneration()` Run after generating attributes.
`void`	`generateAttributes()` Generate attributes for this dataset
`double[][]`	`getAttributeArray()` Output the attributes of each entry into an array
`java.util.List<BaseAttributeExpander>`	`getAttributeExpanders()` Get a copy of the list of currently-employed attribute expanders.
`java.util.List<BaseAttributeGenerator>`	`getAttributeGenerators()` Get a copy of the list of currently-employed attribute generators.
`int`	`getAttributeIndex(java.lang.String Name)` Get index of a certain attribute
`java.lang.String`	`getAttributeName(int index)` Get name of a specific attribute
`java.lang.String[]`	`getAttributeNames()`
`java.util.List<org.apache.commons.lang3.tuple.Pair<java.lang.String,Citation>>`	`getCitations()` Return a list of citations for this object and any underlying objects.
`java.lang.String`	`getClassName(int value)` Get the name of a certain class (for data with multiple possible classes)
`java.lang.String[]`	`getClassNames()`
`double[][]`	`getClassProbabilityArray()` Get an array of class probabilities
`double[]`	`getDistribution()` Get the distribution of entries between known classes
`int[]`	`getDistributionCount()` Get the distribution of entries between known classes
`java.util.Map<BaseEntry,java.util.List<BaseEntry>>`	`getDuplicates()` Get a map of an example of a duplicate entry to all duplicates.
`java.util.List<BaseEntry>`	`getEntries()` Return copy of list of entries.
`java.util.List<BaseEntry>`	`getEntriesWriteAccess()` Get the internal list of entries from this dataset.
`BaseEntry`	`getEntry(int index)` Retrieve a single entry from the dataset
`double[][]`	`getEntryArray()` Output the attributes and class of each entry
`double[]`	`getMeasuredClassArray()` Output an array of the measured classes for each entry.
`double[]`	`getPredictedClassArray()` Get the predicted class for each entry
`Dataset`	`getRandomSplit(double fraction)` Split off a certain number of entries into a separate dataset.
`Dataset`	`getRandomSplit(double fraction, long seed, boolean stratified)` Split off a certain number of entries into a separate dataset.
`Dataset`	`getRandomSplit(int number)` Split off a certain number of entries into a separate dataset.
`Dataset`	`getRandomSplit(int number, long seed, boolean stratified)` Split off a certain number of entries into a separate dataset.
`Dataset`	`getRandomSubset(double fraction)` Generate a random subset of the original data, which is left intact
`Dataset`	`getRandomSubset(int number)` Generate a random subset of the original data, which is left intact
`double[]`	`getSingleAttributeArray(int Attribute)` Output a single attribute for each entry
`Dataset`	`getSubset(int[] indicies)` Get a specific list of entries from the dataset.
`Dataset`	`getTrainingExamples()` Get a dataset that only contains entries with a measured class variable.
`java.util.Map<BaseEntry,java.util.List<BaseEntry>>`	`getUniqueEntries()` Get a map of all unique entries, and their duplicates.
`void`	`importText(java.lang.String filename, java.lang.Object[] options)` Imports data from a text file.
`static Dataset`	`loadState(java.lang.String filename)` Read the state from file using serialization
`java.util.List<BaseEntry>`	`matchEntries(BaseEntry entry, int n)` Find entries in this dataset with attributes closest to a user-provided entry.
`int`	`NAttributes()`
`int`	`NClasses()`
`int`	`NEntries()`
`Dataset[]`	`partition(int[] labels)` Given a list of labels, separate Dataset into multiple subsets
`Dataset[]`	`partition(int[] labels, int number)` Given a list of labels, separate Dataset into multiple subsets (some may be empty)
`java.lang.String`	`printAttributeDescription(boolean htmlFormat)` Print out description of attributes.
`java.lang.String`	`printCommand(java.util.List<java.lang.String> Command)` Handles more complicated printing commands.
`java.lang.String`	`printDescription(boolean htmlFormat)` Print out name of dataset and what attributes are generated.
`java.lang.String`	`printDistribution()` Print out the distribution of entries in the known classes
`java.lang.String`	`printEntries(int[] list)` Print out data regarding a list of entries.
`java.lang.String`	`printEntryDescription(boolean htmlFormat)` Print out what the entries to this dataset are.
`java.lang.String`	`printUsage()` Print out required format for options.
`void`	`removeDuplicates()` Remove all duplicate entries without any selection strategy
`void`	`resolveDuplicates(BaseDuplicateResolver resolver)` Resolve duplicates according to some strategy
`void`	`restoreAttributes(weka.core.Instances weka)` Restore attribute data to each entry.
`protected java.lang.Object`	`runAttributeCommand(java.util.List<java.lang.Object> Command)` Run commands related to attributes of each entry.
`void`	`runAttributeExpanders()` Expand the list of attributes using the currently-set list of attribute expanders.
`protected void`	`runAttributeExpansionCommand(java.util.List<java.lang.Object> Command)` Run commands relating to expanding the attribute pool.
`protected void`	`runAttributeGeneratorCommand(java.util.List<java.lang.Object> Command)` Run commands relating to generating new attributes.
`void`	`runAttributeGenerators()` Expand the list of attributes using the currently-set list of attribute generators.
`java.lang.Object`	`runCommand(java.util.List<java.lang.Object> command)` Process some command described by a list of Objects.
`java.lang.String`	`saveCommand(java.lang.String Basename, java.lang.String Command)` Handles complicated saving commands.
`void`	`saveState(java.lang.String filename)` Save the state of this object using serialization
`void`	`saveTemplate(java.lang.String filename)` Save a template of this dataset to disk using
`void`	`setAttributeNames(java.util.List<java.lang.String> attributeNames)` Set the names of each attributes.
`void`	`setClassNames(java.lang.String[] newClassNames)` Set name of class variable (or possible classes)
`void`	`setClassProbabilities(double[][] predictions)` Set class probabilities for each entry
`void`	`setMeasuredClasses(double[] measurements)` Set measured class for each entry, given an array of measurements
`void`	`setOptions(java.util.List<java.lang.Object> Options)` Set any options for this object.
`void`	`setPredictedClasses(double[] predictions)` Set predicted class for each entry, given an array of predictions
`Dataset[]`	`splitForThreading(int NThreads)` Split for threading purposes.
`Dataset[]`	`splitIntoFolds(int folds)` Split the dataset into multiple folds for cross-validation, empties the original test set
`Dataset[]`	`splitIntoFolds(int folds, long seed)` Split the dataset into multiple folds for cross-validation, empties the original test set
`void`	`subtract(Dataset data)` Remove all entries that are in another dataset from this dataset
`org.json.JSONObject`	`toJSON()` Print dataset to JSON format
`java.lang.String`	`toString()`
`weka.core.Instances`	`transferToWeka(boolean useClass, boolean useDiscreteClass)` Convert to Weka Instances object, delete attribute information in each entry.

Methods inherited from class java.lang.Object
equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

- Field Detail
  - AttributeName
```
protected java.util.ArrayList<java.lang.String> AttributeName
```
    Names of attributes that describe each entry
  - Entries
```
protected java.util.ArrayList<BaseEntry> Entries
```
    Internal array that stores entries
  - Expanders
```
protected java.util.List<BaseAttributeExpander> Expanders
```
    Tools to generate new attributes based on existing ones
  - Generators
```
protected java.util.List<BaseAttributeGenerator> Generators
```
    Tools to generate special-purpose attributes
- Constructor Detail
  - Dataset
```
public Dataset()
```
    Generate a blank dataset
- Method Detail
  - loadState
```
public static Dataset loadState(java.lang.String filename)
                         throws java.lang.Exception
```
    Read the state from file using serialization
    
    Parameters:
    
    filename - Filename for input
    
    Returns:
    
    Dataset stored in that file
    
    Throws:
    
    java.lang.Exception
  - setOptions
```
public void setOptions(java.util.List<java.lang.Object> Options)
                throws java.lang.Exception
```
    Description copied from interface: Options
    
    Set any options for this object. If parsing the options fails, return an Exception with the usage as its message.
    
    Specified by:
    
    setOptions in interface Options
    
    Parameters:
    
    Options - Array of options as Objects - can be null
    
    Throws:
    
    java.lang.Exception - if problem with inputs
  - printUsage
```
public java.lang.String printUsage()
```
    Description copied from interface: Options
    
    Print out required format for options. For consistency, should start with "Usage: ".
    
    Specified by:
    
    printUsage in interface Options
    
    Returns:
    
    Usage for this function
  - clone
```
public Dataset clone()
```
    Creates a new instance of this dataset, and clones of each entry.
    
    Overrides:
    
    clone in class java.lang.Object
    
    Returns:
    
    Clone of this dataset
  - emptyClone
```
public Dataset emptyClone()
```
    Creates a new instance with the same class and attribute names, but without any entries.
    
    Returns:
    
    Dataset with same properties, no entries
  - createTemplate
```
public Dataset createTemplate()
```
    Create a template designed to be saved using serialization
    
    Returns:
    
    An empty clone of the dataset, with any additional changes made
  - getTrainingExamples
```
public Dataset getTrainingExamples()
```
    Get a dataset that only contains entries with a measured class variable.
    
    Returns:
    
    Dataset with all entries that have a measured class
  - clearData
```
public void clearData()
```
    Clear out all entries
  - clearAttributes
```
public void clearAttributes()
```
    Clear attribute data. Used when it is necessary to conserve memory
  - addAttributeExpander
```
public void addAttributeExpander(BaseAttributeExpander expander)
```
    Add a new tool to expand the number of attributes for this dataset.
    
    Parameters:
    
    expander - New expander
  - clearAttributeExpanders
```
public void clearAttributeExpanders()
```
    Reset the list of attribute expanders
  - getAttributeExpanders
```
public java.util.List<BaseAttributeExpander> getAttributeExpanders()
```
    Get a copy of the list of currently-employed attribute expanders.
    
    Returns:
    
    List of attribute expanders
    
    See Also:
    
    addAttributeExpander(magpie.attributes.expanders.BaseAttributeExpander)
  - runAttributeExpanders
```
public void runAttributeExpanders()
```
    Expand the list of attributes using the currently-set list of attribute expanders.
    
    See Also:
    
    getAttributeExpanders()
  - addAttributeGenerator
```
public void addAttributeGenerator(BaseAttributeGenerator generator)
```
    Add a new tool to generate additional attributes
    
    Parameters:
    
    generator - New generator
  - clearAttributeGenerators
```
public void clearAttributeGenerators()
```
    Reset the list of attribute generators
  - getAttributeGenerators
```
public java.util.List<BaseAttributeGenerator> getAttributeGenerators()
```
    Get a copy of the list of currently-employed attribute generators.
    
    Returns:
    
    List of attribute expanders
    
    See Also:
    
    addAttributeGenerator(BaseAttributeGenerator)
  - runAttributeGenerators
```
public void runAttributeGenerators()
                            throws java.lang.Exception
```
    Expand the list of attributes using the currently-set list of attribute generators.
    
    Throws:
    
    java.lang.Exception
    
    See Also:
    
    getAttributeGenerators()
  - generateAttributes
```
public final void generateAttributes()
                              throws java.lang.Exception
```
    Generate attributes for this dataset
    
    Throws:
    
    java.lang.Exception - If any error is encountered
  - calculateAttributes
```
protected void calculateAttributes()
                            throws java.lang.Exception
```
    Compute attributes that are specific to this class.
    
    Throws:
    
    java.lang.Exception - If attribute calculation fails
  - getAttributeNames
```
public java.lang.String[] getAttributeNames()
```
    Returns:
    
    Names of all attributes
  - setAttributeNames
```
public void setAttributeNames(java.util.List<java.lang.String> attributeNames)
```
    Set the names of each attributes.
    NOTE: This will not effect the number of attributes of each entry. Make sure to update those if needed!
    
    Parameters:
    
    attributeNames - New names for each attribute
  - getAttributeName
```
public java.lang.String getAttributeName(int index)
```
    Get name of a specific attribute
    
    Parameters:
    
    index - Attribute number
    
    Returns:
    
    Name of that attribute
  - getAttributeIndex
```
public int getAttributeIndex(java.lang.String Name)
```
    Get index of a certain attribute
    
    Parameters:
    
    Name - Name of desired attribute
    
    Returns:
    
    Index of that attribute (-1 if it does not exist)
  - importText
```
public void importText(java.lang.String filename,
                       java.lang.Object[] options)
                throws java.lang.Exception
```
    Imports data from a text file. Expected format for file:
    Attribute1Name, Attribute2Name, ..., AttributeNName, Class
    Attribute1, Attribute2, ..., AttributeN, ClassVariable
    
    Parameters:
    
    filename - Path to data file
    
    options - Any options used to control import
    
    Throws:
    
    java.lang.Exception - If text import fails
  - getClassNames
```
public java.lang.String[] getClassNames()
```
    Returns:
    
    Names of possible classes for class variable
  - setClassNames
```
public void setClassNames(java.lang.String[] newClassNames)
```
    Set name of class variable (or possible classes)
    
    Parameters:
    
    newClassNames - New name(s) to use
  - getClassName
```
public java.lang.String getClassName(int value)
```
    Get the name of a certain class (for data with multiple possible classes)
    
    Parameters:
    
    value - Value of class variable
    
    Returns:
    
    Name of that class
  - addAttribute
```
public void addAttribute(java.lang.String name,
                         double[] values)
```
    Add in a new attribute. Places at end of list
    
    Parameters:
    
    name - Name to be added
    
    values - Value of attribute for each entry
  - addAttributes
```
public void addAttributes(java.util.List<java.lang.String> names)
```
    Add new attributes. If you use this operation, you must add attributes to each new entry manually.
    
    Parameters:
    
    names - Names of new attributes
    
    See Also:
    
    BaseEntry.addAttributes(double[])
  - NAttributes
```
public int NAttributes()
```
    Returns:
    
    Number of features describing each entry
  - NClasses
```
public int NClasses()
```
    Returns:
    
    Number of possible (discrete) values for class variable. 1 means variable is continuous
  - NEntries
```
public int NEntries()
```
    Returns:
    
    Number of entries in Dataset
  - addEntry
```
public void addEntry(BaseEntry e)
```
    Add an entry. You may need to run generateAttributes()
    
    Parameters:
    
    e - Entry to be added
  - addEntry
```
public BaseEntry addEntry(java.lang.String input)
                   throws java.lang.Exception
```
    A new entry by parsing an input string. After using this operation, it may be necessary to recalculate attributes.
    
    Parameters:
    
    input - String describing the entry
    
    Returns:
    
    Entry representing the parsed string.
    
    Throws:
    
    java.lang.Exception - If conversion fails
  - addEntries
```
public void addEntries(java.util.Collection<? extends BaseEntry> entries)
```
    Add many entries to a the data set
    
    Parameters:
    
    entries - Any collection type of entries
  - addEntries
```
public void addEntries(Dataset otherDataset,
                       boolean forceMerge)
                throws java.lang.Exception
```
    Add entries from another dataset. If other dataset has different attributes or class attributes, you can force a merge between these datasets by deleting the attributes and classes from the other dataset.
    Entries will be cloned before adding them to this dataset.
    
    Parameters:
    
    otherDataset - Dataset to be added to this one
    
    forceMerge - Whether to force merge if attributes / class are different.
    
    Throws:
    
    java.lang.Exception - If datasets have different classes or attributes, and forceMerge is false.
  - removeDuplicates
```
public void removeDuplicates()
```
    Remove all duplicate entries without any selection strategy
  - resolveDuplicates
```
public void resolveDuplicates(BaseDuplicateResolver resolver)
```
    Resolve duplicates according to some strategy
    
    Parameters:
    
    resolver - Duplicate resolution strategy
  - getUniqueEntries
```
public java.util.Map<BaseEntry,java.util.List<BaseEntry>> getUniqueEntries()
```
    Get a map of all unique entries, and their duplicates.
    
    Returns:
    
    Map of entry to all duplicate entries
  - getDuplicates
```
public java.util.Map<BaseEntry,java.util.List<BaseEntry>> getDuplicates()
```
    Get a map of an example of a duplicate entry to all duplicates.
    
    Returns:
    
    Map of duplicates
  - containsEntry
```
public boolean containsEntry(BaseEntry Entry)
```
    Determine whether a dataset contains a certain entry
    
    Parameters:
    
    Entry - Entry to be tested
    
    Returns:
    
    Whether the dataset contains Entry
  - matchEntries
```
public java.util.List<BaseEntry> matchEntries(BaseEntry entry,
                                              int n)
```
    Find entries in this dataset with attributes closest to a user-provided entry. Measures distance using the L2 norm.
    
    Parameters:
    
    entry - Entry to be matched
    
    n - Number of top entries to list
    
    Returns:
    
    List of top entries
  - combine
```
public void combine(Dataset d)
```
    Combine the data structure with another. Does not alter the other dataset. Assumes that the two dataset have the same attributes, and classes.
    Entries are not cloned when during combination.
    If you are looking to add entries from another dataset, also consider using addEntries(magpie.data.Dataset, boolean). This function will clone each entry of the dataset so that the two datasets can be treated as completely independent after the merge.
    
    Parameters:
    
    d - Dataset to be added
  - combine
```
public void combine(Dataset[] d)
```
    Combine the data structure with an array of other Datasets. Does not alter input Datasets.
    
    Parameters:
    
    d - Array of Datasets
  - combine
```
public void combine(java.util.Collection<Dataset> d)
```
    Combine the data structure with a collection of other data structures. Does not alter input arrays
    
    Parameters:
    
    d - Collection of Datasets
  - subtract
```
public void subtract(Dataset data)
```
    Remove all entries that are in another dataset from this dataset
    
    Parameters:
    
    data - Second dataset
  - getEntry
```
public BaseEntry getEntry(int index)
```
    Retrieve a single entry from the dataset
    
    Parameters:
    
    index - Index of entry
    
    Returns:
    
    Specified entry
  - getEntries
```
public java.util.List<BaseEntry> getEntries()
```
    Return copy of list of entries. Does not allow user to delete entries from the dataset.
    
    Returns:
    
    Collection of entries
  - getEntriesWriteAccess
```
public java.util.List<BaseEntry> getEntriesWriteAccess()
```
    Get the internal list of entries from this dataset. User can modify anything about the internal dataset (ex: sort, delete entries)
    
    Returns:
    
    Internal list of entries
  - partition
```
public Dataset[] partition(int[] labels)
```
    Given a list of labels, separate Dataset into multiple subsets
    
    Parameters:
    
    labels - Label defining in which subset to label an entry
    
    Returns:
    
    Array of subsets of length max(label) + 1, where each member, i, contains entries with label[i] == i.
  - partition
```
public Dataset[] partition(int[] labels,
                           int number)
```
    Given a list of labels, separate Dataset into multiple subsets (some may be empty)
    
    Parameters:
    
    labels - Label defining in which subset to label an entry
    
    number - Number subsets to produce, must be greater than max(labels)
    
    Returns:
    
    Array of subsets of length max(label), where each member, i, contains entries with label[i] == i.
  - getSubset
```
public Dataset getSubset(int[] indicies)
```
    Get a specific list of entries from the dataset. These entries are not removed from the original dataset
    
    Parameters:
    
    indicies - List of entry IDs to be removed
    
    Returns:
    
    A new dataset containing only the specified entries
  - getRandomSplit
```
public Dataset getRandomSplit(int number)
```
    Split off a certain number of entries into a separate dataset. Deletes those entries from the original set.
    Does not specifically ensure that split has same distribution of classes as this dataset
    
    Parameters:
    
    number - Fraction of entries from original set to move
    
    Returns:
    
    Dataset containing a subset of entries
    
    See Also:
    
    getRandomSplit(int, long, boolean)
  - getRandomSplit
```
public Dataset getRandomSplit(int number,
                              long seed,
                              boolean stratified)
```
    Split off a certain number of entries into a separate dataset. Deletes those entries from the original set
    
    Parameters:
    
    number - Number of entries in new set
    
    seed - Seed used for random number generator
    
    stratified - Ensure split has the same distribution of classes as the
    
    Returns:
    
    Dataset containing a subset of entries
  - getRandomSplit
```
public Dataset getRandomSplit(double fraction)
```
    Split off a certain number of entries into a separate dataset. Deletes those entries from the original set.
    Does not specifically ensure that split has same distribution of classes as this dataset
    
    Parameters:
    
    fraction - Fraction of entries from original set to move
    
    Returns:
    
    Dataset containing a subset of entries
    
    See Also:
    
    getRandomSplit(double, long, boolean)
  - getRandomSplit
```
public Dataset getRandomSplit(double fraction,
                              long seed,
                              boolean stratified)
```
    Split off a certain number of entries into a separate dataset. Deletes those entries from the original set
    
    Parameters:
    
    fraction - Fraction of entries of original set to move to new set
    
    seed - Seed used for random number generator
    
    stratified - Ensure split has the same distribution of classes as the host dataset
    
    Returns:
    
    Dataset containing a subset of entries
  - getRandomSubset
```
public Dataset getRandomSubset(int number)
```
    Generate a random subset of the original data, which is left intact
    
    Parameters:
    
    number - Number of entries to move over
    
    Returns:
    
    Dataset containing a subset of entries
  - getRandomSubset
```
public Dataset getRandomSubset(double fraction)
```
    Generate a random subset of the original data, which is left intact
    
    Parameters:
    
    fraction - Fraction of entries used in new set
    
    Returns:
    
    Dataset containing a subset of entries
  - splitForThreading
```
public Dataset[] splitForThreading(int NThreads)
```
    Split for threading purposes. Does not worry about randomization, thread n contains all entries where [entry ID] % NThreads == 0.
    Example: 4 threads, thread 1 has entry #1, 5, 9, ...
    The entries in this object are untouched.
    
    Parameters:
    
    NThreads - Number of subsets to create
    
    Returns:
    
    Array of equally-sized Dataset objects
  - splitIntoFolds
```
public Dataset[] splitIntoFolds(int folds)
```
    Split the dataset into multiple folds for cross-validation, empties the original test set
    
    Parameters:
    
    folds - Number of folds
    
    Returns:
    
    Independent data sets
  - splitIntoFolds
```
public Dataset[] splitIntoFolds(int folds,
                                long seed)
```
    Split the dataset into multiple folds for cross-validation, empties the original test set
    
    Parameters:
    
    folds - Number of folds
    
    seed - Random seed
    
    Returns:
    
    Independent data sets
  - transferToWeka
```
public weka.core.Instances transferToWeka(boolean useClass,
                                          boolean useDiscreteClass)
```
    Convert to Weka Instances object, delete attribute information in each entry. This can be used to conserve memory when using Weka.
    
    Parameters:
    
    useClass - Whether to output class data. Note: If there is no measured class data and useDiscreteClass is true, value will be set to Zero. This allows the Instances to contain information about how many classes are available regardless of whether this Dataset contains any measurements.
    
    useDiscreteClass - Whether to treat class variable as discrete
    
    Returns:
    
    Dataset in Weka format
    
    See Also:
    
    restoreAttributes(weka.core.Instances)
  - restoreAttributes
```
public void restoreAttributes(weka.core.Instances weka)
                       throws java.lang.Exception
```
    Restore attribute data to each entry.
    
    Parameters:
    
    weka - Weka object containing attribute information. Assumes last variable has class variable
    
    Throws:
    
    java.lang.Exception
    
    See Also:
    
    transferToWeka(boolean, boolean)
  - getEntryArray
```
public double[][] getEntryArray()
```
    Output the attributes and class of each entry
    
    Returns:
    
    Array where the last column is the measured class variable (0 if no measured)
  - getAttributeArray
```
public double[][] getAttributeArray()
```
    Output the attributes of each entry into an array
    
    Returns:
    
    Array of attributes
  - getSingleAttributeArray
```
public double[] getSingleAttributeArray(int Attribute)
```
    Output a single attribute for each entry
    
    Parameters:
    
    Attribute - Which Attribute to output
    
    Returns:
    
    Array of attribute values
  - getMeasuredClassArray
```
public double[] getMeasuredClassArray()
```
    Output an array of the measured classes for each entry. Assumes that each
    
    Returns:
    
    1D double array containing measured classes
    
    Throws:
    
    java.lang.RuntimeException - If an entry is missing a measured value
  - getPredictedClassArray
```
public double[] getPredictedClassArray()
```
    Get the predicted class for each entry
    
    Returns:
    
    1D double array containing measured classes
  - getClassProbabilityArray
```
public double[][] getClassProbabilityArray()
```
    Get an array of class probabilities
    
    Returns:
    
    Probabilities of each entry being in each class
  - deleteMeasuredClasses
```
public void deleteMeasuredClasses()
```
    Delete all measured classes from entries
  - deletePredictedClasses
```
public void deletePredictedClasses()
```
    Delete all predicted classes from entries
  - setPredictedClasses
```
public void setPredictedClasses(double[] predictions)
```
    Set predicted class for each entry, given an array of predictions
    
    Parameters:
    
    predictions - Predictions in the same order as generated by getFeatures
  - setMeasuredClasses
```
public void setMeasuredClasses(double[] measurements)
```
    Set measured class for each entry, given an array of measurements
    
    Parameters:
    
    measurements - Measurements in the same order as generated by getFeatures
  - setClassProbabilities
```
public void setClassProbabilities(double[][] predictions)
```
    Set class probabilities for each entry
    
    Parameters:
    
    predictions - Probabilities in the same order as generated by getFeatures
  - about
```
public java.lang.String about()
```
    Description copied from interface: Printable
    
    Prints a simple status message about this object
    
    Specified by:
    
    about in interface Printable
    
    Returns:
    
    Short status message
  - printDescription
```
public java.lang.String printDescription(boolean htmlFormat)
```
    Print out name of dataset and what attributes are generated.
    
    Specified by:
    
    printDescription in interface Printable
    
    Parameters:
    
    htmlFormat - Whether to print in HTML format
    
    Returns:
    
    String describing this dataset
    
    See Also:
    
    #printModel()
  - getCitations
```
public java.util.List<org.apache.commons.lang3.tuple.Pair<java.lang.String,Citation>> getCitations()
```
    Description copied from interface: Citable
    
    Return a list of citations for this object and any underlying objects.
    
    Specified by:
    
    getCitations in interface Citable
    
    Returns:
    
    List of pairs: [Reason for Citation], [Citation]
  - printAttributeDescription
```
public java.lang.String printAttributeDescription(boolean htmlFormat)
```
    Print out description of attributes.
    Implementation Guide
    Subclasses should describe what kind of attributes are generated by default. If it uses a separate generator class, those are captured by the printDescription(boolean) section.
    
    Parameters:
    
    htmlFormat - Whether to print in HTML format
    
    Returns:
    
    Description of the attributes or an empty string ("") if there are no default attributes.
  - printEntryDescription
```
public java.lang.String printEntryDescription(boolean htmlFormat)
```
    Print out what the entries to this dataset are.
    
    Parameters:
    
    htmlFormat - Whether to print in HTML format
    
    Returns:
    
    Description of each entry (i.e., what kind of data is this).
  - toString
```
public java.lang.String toString()
```
    Overrides:
    
    toString in class java.lang.Object
  - toJSON
```
public org.json.JSONObject toJSON()
```
    Print dataset to JSON format
    
    Returns:
    
    JSONObject representing dataset.
  - getDistributionCount
```
public int[] getDistributionCount()
```
    Get the distribution of entries between known classes
    
    Returns:
    
    Number of entries of each class
  - getDistribution
```
public double[] getDistribution()
```
    Get the distribution of entries between known classes
    
    Returns:
    
    Fraction of entries of each class
  - printDistribution
```
public java.lang.String printDistribution()
```
    Print out the distribution of entries in the known classes
    
    Returns:
    
    Distribution as a String
  - printEntries
```
public java.lang.String printEntries(int[] list)
```
    Print out data regarding a list of entries. Format:
    ID, Entry, Measured Class, Predicted Class, Class Probabilities
    
    Parameters:
    
    list - ID numbers of entries to be printed.
    
    Returns:
    
    Desired information as a String
  - saveState
```
public void saveState(java.lang.String filename)
               throws java.io.IOException
```
    Save the state of this object using serialization
    
    Parameters:
    
    filename - Path to output file
    
    Throws:
    
    java.io.IOException
  - saveTemplate
```
public void saveTemplate(java.lang.String filename)
                  throws java.io.IOException
```
    Save a template of this dataset to disk using
    
    Parameters:
    
    filename - Path to output file
    
    Throws:
    
    java.io.IOException
  - printCommand
```
public java.lang.String printCommand(java.util.List<java.lang.String> Command)
                              throws java.lang.Exception
```
    Description copied from interface: Printable
    
    Handles more complicated printing commands.
    
    Specified by:
    
    printCommand in interface Printable
    
    Parameters:
    
    Command - Command specifying what to print
    
    Returns:
    
    String formatted as requested
    
    Throws:
    
    java.lang.Exception - If command not understood
  - saveCommand
```
public java.lang.String saveCommand(java.lang.String Basename,
                                    java.lang.String Command)
                             throws java.lang.Exception
```
    Description copied from interface: Savable
    
    Handles complicated saving commands.
    Dev Note: Make sure to add save format to Javadoc. See Dataset as an example. Required format:
    <save><p><b>format<b> - Description
    <br>Optional room to talk more about format </save>
    
    Specified by:
    
    saveCommand in interface Savable
    
    Parameters:
    
    Basename - Name of file without extension
    
    Command - Command specifying format in which to print
    
    Returns:
    
    Filename Path to output file
    
    Throws:
    
    java.lang.Exception - If command not understood
  - runCommand
```
public java.lang.Object runCommand(java.util.List<java.lang.Object> command)
                            throws java.lang.Exception
```
    Description copied from interface: Commandable
    
    Process some command described by a list of Objects.
    
    Specified by:
    
    runCommand in interface Commandable
    
    Parameters:
    
    command - Command as a list of objects
    
    Returns:
    
    Any output (null if no output generated by command)
    
    Throws:
    
    java.lang.Exception - If something goes wrong
  - runAttributeCommand
```
protected java.lang.Object runAttributeCommand(java.util.List<java.lang.Object> Command)
                                        throws java.lang.Exception
```
    Run commands related to attributes of each entry. Starts with the action to perform on the attributes
    
    Parameters:
    
    Command - Operation to be run on/about attributes
    
    Returns:
    
    Any output (null if nothing is created)
    
    Throws:
    
    java.lang.Exception - On any error
  - runAttributeExpansionCommand
```
protected void runAttributeExpansionCommand(java.util.List<java.lang.Object> Command)
                                     throws java.lang.Exception
```
    Run commands relating to expanding the attribute pool.
    
    Parameters:
    
    Command - Attribute expansion command (e.g., "run")
    
    Throws:
    
    java.lang.Exception
  - runAttributeGeneratorCommand
```
protected void runAttributeGeneratorCommand(java.util.List<java.lang.Object> Command)
                                     throws java.lang.Exception
```
    Run commands relating to generating new attributes.
    
    Parameters:
    
    Command - Attribute expansion command (e.g., "run")
    
    Throws:
    
    java.lang.Exception
  - finalizeGeneration
```
protected void finalizeGeneration()
```
    Run after generating attributes. Performs some operations to reduce the amount of memory used.

Class Dataset

Field Summary

Constructor Summary

Method Summary

Methods inherited from class java.lang.Object

Field Detail

AttributeName

Entries

Expanders

Generators

Constructor Detail

Dataset

Method Detail

loadState

setOptions

printUsage

clone

emptyClone

createTemplate

getTrainingExamples

clearData

clearAttributes

addAttributeExpander

clearAttributeExpanders

getAttributeExpanders

runAttributeExpanders

addAttributeGenerator

clearAttributeGenerators

getAttributeGenerators

runAttributeGenerators

generateAttributes

calculateAttributes

getAttributeNames

setAttributeNames

getAttributeName

getAttributeIndex

importText

getClassNames

setClassNames

getClassName

addAttribute

addAttributes

NAttributes

NClasses

NEntries

addEntry

addEntry

addEntries

addEntries

removeDuplicates

resolveDuplicates

getUniqueEntries

getDuplicates

containsEntry

matchEntries

combine

combine

combine

subtract

getEntry

getEntries

getEntriesWriteAccess

partition

partition

getSubset

getRandomSplit

getRandomSplit

getRandomSplit

getRandomSplit

getRandomSubset

getRandomSubset

splitForThreading

splitIntoFolds

splitIntoFolds

transferToWeka

restoreAttributes

getEntryArray

getAttributeArray

getSingleAttributeArray

getMeasuredClassArray