PPLRE Evaluation - Cooccurrence

From GM-RKB
Jump to: navigation, search

This page describes the results of PPLRE RR Algorithm - Cooccurrence's Performance on the PPLRE Evaluation Task as reported by the PPLRE Automated Evaluation System.


Overview

The evaluation is currently under way. Some preliminary results below:


Algorithm version v2.3 on test set v1.3.1


Optimizations

A round of optimizations were performed. One proviso is that the optimization was performed against the test set and not the train set. The reason for this is that v1.3.1 of the train set was not ready.

The optimization options included:

  • Number of Organism concepts (in a sentence)
  • Number of Protein concepts (in a sentence)
  • Number of Location concepts (in a sentence)
  • The presence of a UMLS [Spatial_Concept] concep (in the sentence).
  • The presence of a UMLS [Laboratory_Procedure] concept (in the sentence).
  • The type of protein name.

general optimizations

The following settings were found to be beneficial, w.r.t to both Precision and F-Score:

  • Organism count per sentence: 1 (e.g. two cases of E. coli in one sentence count as two instances)
  • (logical) Location count per sentence: 1 (e.g. two cases of extracellular in one sentence cound as one instance).
  • Restrict to sentences with a [Spacial_Concept]: not beneficial.
  • Restrict to sentences with a [Laboratory_Procedure]: not beneficial.
  • The protein name: beneficial (at least three chars & one upper case character OR a composite name with a space between words).

contingent optimizations

The following settings were found to trade off precision for f-score.

  • Number of protein concepts per sentence

RunIDProteinsTPFPFNTNPRFFP2P2F2
0318210447_31560115235012639.5%23.1%29.1%3629.4%25.9%
0318210939_32609227663811229.0%41.5%34.2%8923.3%29.8%
0318211040_851330833510726.5%46.2%33.7%11820.3%28.2%
0318211202_2012434903110627.4%52.3%36.0%13719.9%28.8%
0318211251_2733534953110526.4%52.3%35.1%14718.8%27.6%


Algorithm version v2.3t on test set v1.3.1

  • Preliminary experiments into two sentence passages over one sentence passages suggested that performance generally drops.

RunIDProteinsTPFPFNTNPRFFP2P2F2
0319000619_153751227631270.0690.0310.043490.0390.034
0319000834_16806218120471010.1300.2770.1772410.0690.111
0319000442_1464442914736940.1650.4460.2412830.0930.154