PPLRE Evaluation - Snowball
This page describes the results of PPLRE Snowball's Algorithm on the PPLRE Evaluation Task as reported by the PPLRE Automated Evaluation System.
Overview
The evaluation is currently under way. Some preliminary results below:
070303 Data (v1.2 PerfEvaluator)
Performance results on the 070303 data (Run-4 seeds=100) based on PPLRE PerfEvaluator v1.2. The PO and PL confidence threshold values that optimized F-Measure where PO=0.6 and PL=0.1(or less).
| Predicted.Positive | Predicted Negative | |
| Actual.Positive | TP=12 | FN=54 |
| Actual.Negative | FP=6 | TN=145 |
070309 Data (v1.2 PerfEvaluator)
Performance results on the 070309 data (Run-3 seeds=190) based on PPLRE PerfEvaluator v1.2. The PO and PL confidence threshold values that optimized F-Measure where PO=0.64 and PL=0.4.
| Predicted.Positive | Predicted Negative | |
| Actual.Positive | TP=6 | FN=59 |
| Actual.Negative | FP=2 | TN=145 |
070303 Data (v1.1 PerfEvaluator)
Performance results on the 070303 data (Run-4 seeds=100) based on PPLRE PerfEvaluator v1.2. The combined PO/PL confidence threshold value that optimized F-Measure was PO=0.8.
| Predicted.Positive | Predicted Negative | |
| Actual.Positive | TP=12 | FN=53 |
| Actual.Negative | FP=8 | TN=153 |
= Precision vs. Confidence (for a binary relation prediction)
Here is data on the correlation between Precision and Confidence. The data comes from the selected PO run use to analysis the performance of 070303. Notice that the precision ascends at first and the descends after ~0.70 confidence. The ascension at the begining is unexpected. It maybe due to random effects.
TP FP Precision
0.75 0 1 0
0.75 1 1 0.5
0.75 1 2 0.333333333
0.75 1 3 0.25
0.75 2 3 0.4
0.75 2 4 0.333333333
0.75 2 5 0.285714286
0.75 3 5 0.375
0.75 3 6 0.333333333
0.75 4 6 0.4
0.75 4 7 0.363636364
0.75 5 7 0.416666667
0.75 6 7 0.461538462
0.75 6 8 0.428571429
0.75 6 9 0.4
0.74 7 9 0.4375
0.74 8 9 0.470588235
0.74 8 10 0.444444444
0.74 8 11 0.421052632
0.73 9 11 0.45
0.73 10 11 0.476190476
0.73 11 11 0.5
0.72 12 11 0.52173913
0.72 13 11 0.541666667
0.72 14 11 0.56
0.72 14 12 0.538461538
0.72 14 13 0.518518519
0.72 14 14 0.5
0.71 15 14 0.517241379
0.71 16 14 0.533333333
0.71 17 14 0.548387097
0.70 17 15 0.53125
0.70 18 15 0.545454545
0.70 18 16 0.529411765
0.68 19 16 0.542857143
0.67 19 17 0.527777778
0.66 19 18 0.513513514
0.64 19 19 0.5
0.64 19 20 0.487179487
0.63 19 21 0.475
0.63 19 22 0.463414634
0.63 20 22 0.476190476
0.60 21 22 0.488372093
0.60 21 23 0.477272727
0.60 21 24 0.466666667
0.59 21 25 0.456521739
0.54 22 25 0.468085106
0.53 23 25 0.479166667
0.51 24 25 0.489795918
0.51 24 26 0.48
0.48 24 27 0.470588235
0.48 24 28 0.461538462
0.47 24 29 0.452830189
0.46 24 30 0.444444444
0.46 24 31 0.436363636
0.44 24 32 0.428571429
0.44 24 33 0.421052632
0.40 24 34 0.413793103
0.40 24 35 0.406779661
0.39 24 36 0.4
0.39 24 37 0.393442623
0.38 24 38 0.387096774
0.37 24 39 0.380952381
0.37 24 40 0.375
0.37 24 41 0.369230769
0.36 24 42 0.363636364
0.36 25 42 0.373134328
0.32 25 43 0.367647059
0.28 26 43 0.376811594
070223 Evaluation
PO Relation Extraction Performance
http://www.gabormelli.com/images/Snowball_Performance_070223_PO.gif
PL Relation Extraction Performance
http://www.gabormelli.com/images/Snowball_Performance_070223_PL.gif
