PPLRE Evaluation - Ensemble

From GM-RKB
Jump to: navigation, search

A quick analysis of an ad hoc Ensemble Algorithm was performed using the ~ March 23 evaluation data of the PPLRE Project's three current relation recognition algorithm: ZParser, Snowball, Coocurrence. The analysis was of the relative performance with respect to the overlap in their True Positive and False Positive predictions.

Shared TPs and FPs

Here is a summary of the number of shared TPs and FPs. For example, the first record shows that all three algorithms attained a TP for some test record; similary two algorithm's attained a FP on the same record. The fact that 1) there were several records in which all three algorithms were correct and that 2) no record foiled all three algorithms into a TP, suggests that an ensemble would be beneficial.
(the values 1.03 and 0.97 were manual modifications of a "1" result in order to facilitate the use of the data in a visualization. If left as "1" then the two lines overlap and it becomes difficult to see that one stops earlier than the other.)

Shared TPs Shared FPs 1 3 2 2 3 2 3 3 2 4 3 2 5 3 1.03 6 3 1.03 7 3 1.03 8 3 1.03 9 2 1.03 10 2 1.03 11 2 1.03 12 2 1.03 13 2 1.03 14 2 1.03 15 2 1.03 16 2 1.03 17 2 1.03 18 0.97 1.03 19 0.97 1.03 20 0.97 1.03 21 0.97 1.03 22 0.97 1.03 23 0.97 1.03 24 0.97 1.03 25 0.97 1.03 26 0.97 1.03 27 0.97 1.03 28 0.97 1.03 29 0.97 1.03 30 0.97 1.03 31 0.97 1.03 32 0.97 1.03 33 0.97 1.03 34 0.97 1.03 35 0.97 1.03 36 0.97 1.03 37 0.97 1.03 38 0.97 1.03 39 0.97 1.03 40 0.97 1.03 41 0.97 1.03 42 0.97 1.03 43 1.03 44 1.03 45 1.03 46 1.03 47 1.03 48 1.03 49 1.03 50 1.03 51 1.03 52 1.03 53 1.03 54 1.03 55 1.03 56 1.03 57 1.03