PPLRE Evaluation - Cooccurrence

This page describes the results of PPLRE RR Algorithm - Cooccurrence's Performance on the PPLRE Evaluation Task as reported by the PPLRE Automated Evaluation System.

---

Overview

The evaluation is currently under way. Some preliminary results below:

---

Algorithm version v2.3 on test set v1.3.1

---

Optimizations

A round of optimizations were performed. One proviso is that the optimization was performed against the test set and not the train set. The reason for this is that v1.3.1 of the train set was not ready.

The optimization options included:

• Number of Organism concepts (in a sentence)
• Number of Protein concepts (in a sentence)
• Number of Location concepts (in a sentence)
• The presence of a UMLS [Spatial_Concept] concep (in the sentence).
• The presence of a UMLS [Laboratory_Procedure] concept (in the sentence).
• The type of protein name.

general optimizations

The following settings were found to be beneficial, w.r.t to both Precision and F-Score:

• Organism count per sentence: 1 (e.g. two cases of E. coli in one sentence count as two instances)
• (logical) Location count per sentence: 1 (e.g. two cases of extracellular in one sentence cound as one instance).
• Restrict to sentences with a [Spacial_Concept]: not beneficial.
• Restrict to sentences with a [Laboratory_Procedure]: not beneficial.
• The protein name: beneficial (at least three chars & one upper case character OR a composite name with a space between words).

contingent optimizations

The following settings were found to trade off precision for f-score.

• Number of protein concepts per sentence

RunID	Proteins	TP	FP	FN	TN	P	R	F	FP2	P2	F2
0318210447_31560	1	15	23	50	126	39.5%	23.1%	29.1%	36	29.4%	25.9%
0318210939_32609	2	27	66	38	112	29.0%	41.5%	34.2%	89	23.3%	29.8%
0318211040_851	3	30	83	35	107	26.5%	46.2%	33.7%	118	20.3%	28.2%
0318211202_2012	4	34	90	31	106	27.4%	52.3%	36.0%	137	19.9%	28.8%
0318211251_2733	5	34	95	31	105	26.4%	52.3%	35.1%	147	18.8%	27.6%

---

Algorithm version v2.3t on test set v1.3.1

• Preliminary experiments into two sentence passages over one sentence passages suggested that performance generally drops.

RunID	Proteins	TP	FP	FN	TN	P	R	F	FP2	P2	F2
0319000619_15375	1	2	27	63	127	0.069	0.031	0.043	49	0.039	0.034
0319000834_16806	2	18	120	47	101	0.130	0.277	0.177	241	0.069	0.111
0319000442_14644	4	29	147	36	94	0.165	0.446	0.241	283	0.093	0.154