magpie.statistics.performance

Class TargetRegressionStatistics

java.lang.Object

magpie.statistics.performance.BaseStatistics

magpie.statistics.performance.RegressionStatistics

magpie.statistics.performance.TargetRegressionStatistics

All Implemented Interfaces:

java.io.Serializable, java.lang.Cloneable, Commandable, Options, Printable, Savable

public class TargetRegressionStatistics
extends RegressionStatistics

Statistics for the ability of a model to find entries with a class near a target value. Basically, all this does is report statistics of abs( class - target ) for each entry.

Window Size Analysis

This class can perform an analysis how many entries are predicted to have a class variable be less than a certain distance from the target value, and how many of those actually truly fall within some specified distance of the target. Consider this as a way of testing how well this model can be used as a way of screening candidate materials for possessing a certain level of a target property.In order to run this analysis, specify a maximum window size and a tolerance window for successful predictions. This chart can be printed out using the "window" command.

Candidate Analysis

Similarly, one can test then number of entries truly within the acceptance tolerance when the a certain number of closest candidates are selected. For a range of numbers of candidates up to the maximum number of candidates, the average distance of the entries from the target value and fraction correctly identified to be within a certain tolerance are calculated. These results will be calculated if the maximum number of candidates and tolerance are set. Print using the "rank" command;

Usage: <target> [-accept <Acceptance Tolerance>] [-window <Max Window Size>] [-cands <Max Number Candidates>]
Target: Desired value of class variable
Acceptance Tolerance: If the actual class variable is within this value of the target, consider the prediction a success
Maximum Window Size: Set this and Acceptance Tolerance to analyze accuracy as a function of tolerance window size
Max Number Candidates: Set this and acceptance tolerance in order to study accuracy as a function of number of candidates selected

Implemented Print Commands:

rank - Prints how the performance of best-performing entries according to model predictions

window - Prints statistics regarding this model’s ability to act as a filter

See Also:

Serialized Form

Field Summary

Fields

Modifier and Type	Field and Description
protected int	MaxCandidates Maximum number of candidates for ranking analysis
protected double	MaxWindowSize Maximum tolerance during window size analysis
protected double	Target Target value of class variable
protected double	Tolerance Tolerance for successful prediction

Fields inherited from class magpie.statistics.performance.RegressionStatistics

MAE, MeasuredMAD, MeasuredMedian, MeasuredRange, MeasuredStdDev, MRE, R, Rho, RMSE, Tau

Fields inherited from class magpie.statistics.performance.BaseStatistics

Measured, NumberTested, Predicted, ROC, ROC_AUC

Constructor Summary

Constructors

Constructor and Description
TargetRegressionStatistics()

Method Summary

Modifier and Type	Method and Description
protected void	evaluate_protected(Dataset Results) Internal method for acutally computing results.
java.lang.String	printCandidateAnalysis() Print results of the candidate ranking analysis.
java.lang.String	printCommand(java.util.List<java.lang.String> Command) Handles more complicated printing commands.
java.lang.String	printUsage() Print out required format for options.
java.lang.String	printWindowAnalysis() Print results of the tolerance size analysis.
protected void	runCandidateAnalysis(double[] measured, double[] predicted) Perform a candidate selection analysis (described in documentation for this class as a whole).
protected void	runWindowAnalysis(double[] measured, double[] predicted) Perform a window-size analysis.
void	setMaxCandidates(int MaxCandidates) Set the maximum number of candidates for an analysis of how the number of candidates affects prediction accuracy.
void	setMaxWindowSize(double MaxWindowSize) Set the maximum window size for an analysis of how tolerance window size affects prediction accuracy.
void	setNSteps(int NSteps) Set the number of steps to use when calculating model performance for ranking or filtering.
void	setOptions(java.util.List<java.lang.Object> OptionsObj) Set any options for this object.
void	setTarget(double Target) Define target value of class variable.
void	setTolerance(double Tolerance) Define the error threshold that is considered a successful prediction.

Methods inherited from class magpie.statistics.performance.RegressionStatistics

clone, getStatistics, getStatistics, printBaselineStats, toString

Methods inherited from class magpie.statistics.performance.BaseStatistics

about, evaluate, getROCCurve, getStatisticsNoNaNs, printDescription, printROCCurve, runCommand, saveCommand, savePerformanceData

Methods inherited from class java.lang.Object

equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait

Field Detail

Target

protected double Target

Target value of class variable

Tolerance

protected double Tolerance

Tolerance for successful prediction

MaxWindowSize

protected double MaxWindowSize

Maximum tolerance during window size analysis

MaxCandidates

protected int MaxCandidates

Maximum number of candidates for ranking analysis

Constructor Detail

TargetRegressionStatistics

public TargetRegressionStatistics()

Method Detail

setOptions

public void setOptions(java.util.List<java.lang.Object> OptionsObj)
                throws java.lang.Exception

Description copied from interface: Options

Set any options for this object. If parsing the options fails, return an Exception with the usage as its message.

Specified by:

setOptions in interface Options

Overrides:

setOptions in class BaseStatistics

Parameters:

OptionsObj - Array of options as Objects - can be null

Throws:

java.lang.Exception - if problem with inputs

printUsage

public java.lang.String printUsage()

Description copied from interface: Options

Print out required format for options. For consistency, should start with "Usage: ".

Specified by:

printUsage in interface Options

Overrides:

printUsage in class BaseStatistics

Returns:

Usage for this function

setNSteps

public void setNSteps(int NSteps)

Set the number of steps to use when calculating model performance for ranking or filtering.

Parameters:

NSteps - Number of step increments (default: 20)

setTarget

public void setTarget(double Target)

Define target value of class variable.

Parameters:

Target - Desired target value

setTolerance

public void setTolerance(double Tolerance)

Define the error threshold that is considered a successful prediction. For example, if the target value is 2.5 and anything between 2.0 and 3.0 is OK, use 0.5. Set to -1 if you do not want to use any kind of tolerance analysis

Parameters:

Tolerance - Desired tolerance

setMaxCandidates

public void setMaxCandidates(int MaxCandidates)

Set the maximum number of candidates for an analysis of how the number of candidates affects prediction accuracy. Note that one must also specify the desired tolerance.

Parameters:

MaxCandidates - Desired maximum number of candidates

setMaxWindowSize

public void setMaxWindowSize(double MaxWindowSize)

Set the maximum window size for an analysis of how tolerance window size affects prediction accuracy. Note that one must also specify the desired tolerance.

Parameters:

MaxWindowSize - Desired maximum window size

evaluate_protected

protected void evaluate_protected(Dataset Results)

Description copied from class: BaseStatistics

Internal method for acutally computing results. BaseStatistics.evaluate(Dataset) contains some error-checking code

Overrides:

evaluate_protected in class RegressionStatistics

Parameters:

Results - Dataset to be evaluated

runCandidateAnalysis

protected void runCandidateAnalysis(double[] measured,
                                    double[] predicted)

Perform a candidate selection analysis (described in documentation for this class as a whole). Basically, its just the accuracy of the model as a function of number of candidates selected.

Parameters:

measured - Measured distance of class variable from target

predicted - Predicted distance of class variable from target

runWindowAnalysis

protected void runWindowAnalysis(double[] measured,
                                 double[] predicted)

Perform a window-size analysis. Basically, this test the ability of a model to find candidates within a certain window if all materials within a specified tolerance are selected.

Parameters:

measured - Measured distance of class variable from target

predicted - Predicted distance of class variable from target

printCandidateAnalysis

public java.lang.String printCandidateAnalysis()

Print results of the candidate ranking analysis.

Returns:

A table containing for each row: Number of candidates selected, average distance, hit count

printWindowAnalysis

public java.lang.String printWindowAnalysis()

Print results of the tolerance size analysis.

Returns:

A table containing for each row: Window size, # inside window, Fraction correct

printCommand

public java.lang.String printCommand(java.util.List<java.lang.String> Command)
                              throws java.lang.Exception

Description copied from interface: Printable

Handles more complicated printing commands.

Specified by:

printCommand in interface Printable

Overrides:

printCommand in class BaseStatistics

Parameters:

Command - Command specifying what to print

Returns:

String formatted as requested

Throws:

java.lang.Exception - If command not understood