edu.stanford.nlp.util
Class EquivalenceClassEval

java.lang.Object
  edu.stanford.nlp.util.EquivalenceClassEval

public class EquivalenceClassEval
extends Object

A class for calculating precision and recall statistics based on comparisons between two Collections. Allows flexible specification of:

• The criterion by which to evaluate whether two Objects are equivalent for purposes of precision and recall calculation (specified by an EquivalenceClassEval.EqualityChecker instance)
• The criterion by which Objects are grouped into equivalence classes for purposes of calculating subclass precision and recall (specified by an EquivalenceClasser instance)
• Evaluation is set-based or bag-based (by default, it is set-based). For example, if a gold collection has {a,a,b} and a guess collection has {a,b}, then recall is 100% in set-based evaluation, but is 66.67% in bag-based evaluation.

Note that for set-based evaluation, sets are always constructed using object equality, NOT equality on the basis of an EquivalenceClassEval.EqualityChecker if one is given. If set-based evaluation were conducted on the basis of an EqualityChecker, then there would be indeterminacy when it did not subsume the EquivalenceClasser, if one was given. For example, if objects of the form X:y were equivalence-classed by the left criterion and evaluated for equality on the right, then set-based evaluation based on the equality checker would be indeterminate for a collection of {A:a,B:a} because it would be unclear whether to use the first or second element of the collection.

Author:

Roger Levy

Nested Class Summary

static interface	EquivalenceClassEval.EqualityChecker A strategy-type interface for specifying an equality criterion for pairs of Objects.
static interface	EquivalenceClassEval.Factory

Field Summary

static EquivalenceClassEval.EqualityChecker	DEFAULT_CHECKER A default equality checker that uses Object.equals(java.lang.Object) to determine equality.
static EquivalenceClasser	NULL_EQUIVALENCE_CLASSER Maps all objects to the equivalence class null

Constructor Summary

EquivalenceClassEval()
Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes

EquivalenceClassEval(EquivalenceClasser eq)
Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval(EquivalenceClasser eq, EquivalenceClassEval.EqualityChecker e)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval(EquivalenceClasser eq, EquivalenceClassEval.EqualityChecker e, String summaryName)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval(EquivalenceClasser eq, String name)
Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval(EquivalenceClassEval.EqualityChecker e)
Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.

Method Summary

void	display() Displays the cumulative results of the evaluation to System.out.
void	display(PrintWriter pw) Displays the cumulative results of the evaluation.
void	displayLast() Displays the results of the previous Collection pair evaluation to System.out.
void	displayLast(PrintWriter pw) Displays the results of the previous Collection pair evaluation.
void	eval(Collection guesses, Collection golds) Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.
void	eval(Collection guesses, Collection golds, PrintWriter pw)
static Counter	f1(Counter precision, Counter recall)
static double	f1(double precision, double recall)
double	f1(Object key)
EquivalenceClassEval.Factory	factory() returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this
Counter	lastF1()
double	lastF1(Object key)
Counter	lastNumGolds()
double	lastNumGolds(Object key)
Counter	lastNumGoldsCorrect()
Counter	lastNumGuessed()
double	lastNumGuessed(Object key)
Counter	lastNumGuessedCorrect()
Counter	lastPrecision()
double	lastPrecision(Object key)
Counter	lastRecall()
double	lastRecall(Object key)
static void	main(String[] args)
double	precision(Object key)
double	recall(Object key)
void	setBagEval(boolean bagEval) If bagEval is set to true, then multiple instances of the same item will not be merged.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

NULL_EQUIVALENCE_CLASSER

public static final EquivalenceClasser NULL_EQUIVALENCE_CLASSER

Maps all objects to the equivalence class null

DEFAULT_CHECKER

public static final EquivalenceClassEval.EqualityChecker DEFAULT_CHECKER

A default equality checker that uses Object.equals(java.lang.Object) to determine equality.

Constructor Detail

EquivalenceClassEval

public EquivalenceClassEval()

Specifies a default EquivalenceClassEval, using Object.equals(java.lang.Object) as equality criterion and grouping all items into the "null" equivalence class for reporting purposes

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser eq)

Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClassEval.EqualityChecker e)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items into a single equivalence class for reporting statistics.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser eq,
                            String name)

Specifies an EquivalenceClassEval using Object.equals(java.lang.Object) as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser eq,
                            EquivalenceClassEval.EqualityChecker e)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

EquivalenceClassEval

public EquivalenceClassEval(EquivalenceClasser eq,
                            EquivalenceClassEval.EqualityChecker e,
                            String summaryName)

Specifies an EquivalenceClassEval using the Eval.EqualityChecker argument as equality criterion and grouping all items according to the EquivalenceClasser argument.

Method Detail

setBagEval

public void setBagEval(boolean bagEval)

If bagEval is set to true, then multiple instances of the same item will not be merged. For example, gold (a,a,b) against guess (a,b) will be scored as 100% precision and 66.67% recall. It is false by default.

eval

public void eval(Collection guesses,
                 Collection golds)

Adds a round of evaluation between guesses and golds Collections to the tabulated statistics of the evaluation.

eval

public void eval(Collection guesses,
                 Collection golds,
                 PrintWriter pw)

Parameters:

guesses - Collection of guessed objects

golds - Collection of gold-standard objects

pw - PrintWriter to print eval stats

display

public void display()

Displays the cumulative results of the evaluation to System.out.

display

public void display(PrintWriter pw)

Displays the cumulative results of the evaluation.

displayLast

public void displayLast()

Displays the results of the previous Collection pair evaluation to System.out.

displayLast

public void displayLast(PrintWriter pw)

Displays the results of the previous Collection pair evaluation.

precision

public double precision(Object key)

recall

public double recall(Object key)

lastPrecision

public double lastPrecision(Object key)

lastPrecision

public Counter lastPrecision()

lastRecall

public double lastRecall(Object key)

lastRecall

public Counter lastRecall()

lastNumGuessed

public double lastNumGuessed(Object key)

lastNumGuessed

public Counter lastNumGuessed()

lastNumGuessedCorrect

public Counter lastNumGuessedCorrect()

lastNumGolds

public double lastNumGolds(Object key)

lastNumGolds

public Counter lastNumGolds()

lastNumGoldsCorrect

public Counter lastNumGoldsCorrect()

f1

public double f1(Object key)

lastF1

public double lastF1(Object key)

lastF1

public Counter lastF1()

f1

public static double f1(double precision,
                        double recall)

f1

public static Counter f1(Counter precision,
                         Counter recall)

main

public static void main(String[] args)

factory

public EquivalenceClassEval.Factory factory()

returns a new EquivalenceClassEval.Factory instance that vends new EquivalenceClassEval instances with settings like this