PPLRE Evaluation - Snowball

From GM-RKB
Jump to: navigation, search

This page describes the results of PPLRE Snowball's Algorithm on the PPLRE Evaluation Task as reported by the PPLRE Automated Evaluation System.


Overview

The evaluation is currently under way. Some preliminary results below:


070303 Data (v1.2 PerfEvaluator)

Performance results on the 070303 data (Run-4 seeds=100) based on PPLRE PerfEvaluator v1.2. The PO and PL confidence threshold values that optimized F-Measure where PO=0.6 and PL=0.1(or less).

Predicted.PositivePredicted Negative
Actual.Positive TP=12 FN=54
Actual.Negative FP=6 TN=145

070309 Data (v1.2 PerfEvaluator)

Performance results on the 070309 data (Run-3 seeds=190) based on PPLRE PerfEvaluator v1.2. The PO and PL confidence threshold values that optimized F-Measure where PO=0.64 and PL=0.4.

Predicted.PositivePredicted Negative
Actual.Positive TP=6 FN=59
Actual.Negative FP=2 TN=145

070303 Data (v1.1 PerfEvaluator)

Performance results on the 070303 data (Run-4 seeds=100) based on PPLRE PerfEvaluator v1.2. The combined PO/PL confidence threshold value that optimized F-Measure was PO=0.8.

Predicted.PositivePredicted Negative
Actual.Positive TP=12 FN=53
Actual.Negative FP=8 TN=153


= Precision vs. Confidence (for a binary relation prediction)

Here is data on the correlation between Precision and Confidence. The data comes from the selected PO run use to analysis the performance of 070303. Notice that the precision ascends at first and the descends after ~0.70 confidence. The ascension at the begining is unexpected. It maybe due to random effects.

TP FP Precision

0.75 	0	1	0
0.75 	1	1	0.5
0.75 	1	2	0.333333333
0.75 	1	3	0.25
0.75 	2	3	0.4
0.75 	2	4	0.333333333
0.75 	2	5	0.285714286
0.75 	3	5	0.375
0.75 	3	6	0.333333333
0.75 	4	6	0.4
0.75 	4	7	0.363636364
0.75 	5	7	0.416666667
0.75 	6	7	0.461538462
0.75 	6	8	0.428571429
0.75 	6	9	0.4
0.74 	7	9	0.4375
0.74 	8	9	0.470588235
0.74 	8	10	0.444444444
0.74 	8	11	0.421052632
0.73 	9	11	0.45
0.73 	10	11	0.476190476
0.73 	11	11	0.5
0.72 	12	11	0.52173913
0.72 	13	11	0.541666667
0.72 	14	11	0.56
0.72 	14	12	0.538461538
0.72 	14	13	0.518518519
0.72 	14	14	0.5
0.71 	15	14	0.517241379
0.71 	16	14	0.533333333
0.71 	17	14	0.548387097
0.70 	17	15	0.53125
0.70 	18	15	0.545454545
0.70 	18	16	0.529411765
0.68 	19	16	0.542857143
0.67 	19	17	0.527777778
0.66 	19	18	0.513513514
0.64 	19	19	0.5
0.64 	19	20	0.487179487
0.63 	19	21	0.475
0.63 	19	22	0.463414634
0.63 	20	22	0.476190476
0.60 	21	22	0.488372093
0.60 	21	23	0.477272727
0.60 	21	24	0.466666667
0.59 	21	25	0.456521739
0.54 	22	25	0.468085106
0.53 	23	25	0.479166667
0.51 	24	25	0.489795918
0.51 	24	26	0.48
0.48 	24	27	0.470588235
0.48 	24	28	0.461538462
0.47 	24	29	0.452830189
0.46 	24	30	0.444444444
0.46 	24	31	0.436363636
0.44 	24	32	0.428571429
0.44 	24	33	0.421052632
0.40 	24	34	0.413793103
0.40 	24	35	0.406779661
0.39 	24	36	0.4
0.39 	24	37	0.393442623
0.38 	24	38	0.387096774
0.37 	24	39	0.380952381
0.37 	24	40	0.375
0.37 	24	41	0.369230769
0.36 	24	42	0.363636364
0.36 	25	42	0.373134328
0.32 	25	43	0.367647059
0.28 	26	43	0.376811594


070223 Evaluation

PO Relation Extraction Performance

http://www.gabormelli.com/images/Snowball_Performance_070223_PO.gif


PL Relation Extraction Performance

http://www.gabormelli.com/images/Snowball_Performance_070223_PL.gif