PPLRE Evaluation - Ensemble
A quick analysis of an ad hoc Ensemble Algorithm was performed using the ~ March 23 evaluation data of the PPLRE Project's three current relation recognition algorithm: ZParser, Snowball, Coocurrence. The analysis was of the relative performance with respect to the overlap in their True Positive and False Positive predictions.
Here is a summary of the number of shared TPs and FPs. For example, the first record shows that all three algorithms attained a TP for some test record; similary two algorithm's attained a FP on the same record. The fact that 1) there were several records in which all three algorithms were correct and that 2) no record foiled all three algorithms into a TP, suggests that an ensemble would be beneficial.
(the values 1.03 and 0.97 were manual modifications of a "1" result in order to facilitate the use of the data in a visualization. If left as "1" then the two lines overlap and it becomes difficult to see that one stops earlier than the other.)
Shared TPs Shared FPs
1 3 2
2 3 2
3 3 2
4 3 2
5 3 1.03
6 3 1.03
7 3 1.03
8 3 1.03
9 2 1.03
10 2 1.03
11 2 1.03
12 2 1.03
13 2 1.03
14 2 1.03
15 2 1.03
16 2 1.03
17 2 1.03
18 0.97 1.03
19 0.97 1.03
20 0.97 1.03
21 0.97 1.03
22 0.97 1.03
23 0.97 1.03
24 0.97 1.03
25 0.97 1.03
26 0.97 1.03
27 0.97 1.03
28 0.97 1.03
29 0.97 1.03
30 0.97 1.03
31 0.97 1.03
32 0.97 1.03
33 0.97 1.03
34 0.97 1.03
35 0.97 1.03
36 0.97 1.03
37 0.97 1.03
38 0.97 1.03
39 0.97 1.03
40 0.97 1.03
41 0.97 1.03
42 0.97 1.03
43 1.03
44 1.03
45 1.03
46 1.03
47 1.03
48 1.03
49 1.03
50 1.03
51 1.03
52 1.03
53 1.03
54 1.03
55 1.03
56 1.03
57 1.03