ACE-2004
Jump to navigation
Jump to search
See: ACE Benchmark Task, ACE Program.
References
ACE-2004 Overview
| ENTITY TYPE | ENTITY SUBTYPE |
| Person (PER) | (none) |
| Organization (ORG) | Government, Commercial, Educational, Non-Profit, Other |
| Location (LOC) | Address, Boundary, Celestial, Land-Region-Natural, Region-Local, Region-Subnational, Region-Naitonal, Region-International, Water-Body, Other |
| Geo-Political Entity (GPE) | |
| Facility (FAC) | |
| Vehicle (VEH) | |
| Weapon (WEA) |
http://www.nist.gov/speech/tests/ace/ace04/doc/ace04-evalplan-v7.pdf
| RELATION TYPE | RELATION SUBTYPE |
| Physical (PHYS) | Located, Near*, Part-whole |
| Personal / Social (PER-SOC) | Business*, Family*, Other* |
| Employment / Membership / Subsidiary (EMP-ORG) | Employ-Executive, Employ-Staff, Employ-Undetermined, Member-of-Group, Partner*, Subsidiary, Other* |
| Agent-Artifact (ART) | User-or-Owner, Invesntor-or-Manufacturer, Other. |
| PER/ORG Affiliation (OTHER-AFF) | Ethnic, Ideology, Other |
| GPE Affiliation (GPE-AFF) | Citizen-or-Resident, Based-in, Other |
| Discouver (DISC) | (none) |
- http://www.nist.gov/speech/tests/ace/ace04/
- (ACE RDC Guidelines, 2004) ⇒ ACE. (2004). “Annotation Guidelines for Relation Detection and Characterization (RDC)." Version 4.3.2 - 20040401
- http://www.ldc.upenn.edu/Catalog/CatalogEntry.jsp?catalogId=LDC2005T09
- http://www.nist.gov/speech/tests/ace/ace04/doc/ace04-evalplan-v7.pdf
- Relation Detection and Recognition (RDR) Task
- The RDR task is to infer ACE-defined relations from the source language and to recognize and output selected attributes and information about these relations, includeing information about their mentions. A major part of correctly recognizting relations is correctly recognizing the arguments (entities) that are related by the relation. Thefore good EDR performance is important to achieving god RDR performance. The Value formula for RDR (for single relation) is defined as the product of an inherent relation value and the sum of the values of the two entities that are the relation arguments:
- Relation Detection and Recognition (RDR) Task
Value(sys_relation) = Relation_Value(sys_relation) - SUM Argument_Value(sys_argument a)
where the Relation_Value/ is a functino of relation type and the Argument_Value is the value of the entity arguement as compluter for EDR scorting. Refer to appendix A for a complete description of the RDR Value formula.
- Relation Mention Detection (RMD)
- As with entities, the RMD task complements the RDR task by disregarding the co-reference issue. In essense, the relation mentions are treated as though each mention were the mention of a unique relation that has only a single mention and is thus distinct and separate from all other relation mentions. This treatment allows the relation mentions to be evaluated directly, as relations, using the Value formula for RDR. Thus the mechanics of RMD scoring are identical to those of RDR. There is however, a significant difference in computing the contribution of the relation artuments to the value of a relation mention. This is namely that the contribution of an argument is computed only for the single entity mention that is referenced in the relation mention output.
- Relation Mention Detection (RMD)
- (Jiang and Zhai, 2007)
- "The candidate relations were generated by considering all pairs of enties that occur in the same sentence. We obtained 48,625 candidate relation instances in total, among which 4,296 instances were positive."
- (Zhang et al., 2006)
- "The ACE 2004 data contains 451 documents and 5,702 relation instances. It redefines 7 entity types, 7 major relation types and 23 subtypes."
ACE-2004 Reported Results
| Paper | Method | 7Mt-P | 7Mt-R | 7Mt-F | 23St-P | 23St-R | 23St-F |
| ZZJZ07 | Composite kernel (cntxt. sens.) | 82.2 | 70.2 | 75.8 | 70.3 | 62.2 | 66.0 |
| ZZJZ07 | Conv. tree kernel (cntxt.sense.) | 81.1 | 66.7 | 73.2 | 68.8 | 60.3 | 64.3 |
| JZ07 | MaxEnt(Seq+Syn-H3+H4) | 74.6 | 71.3 | 72.9 | |||
| ZZSZ06 | Composite kernel 2 (poly exp) | 76.1 | 68.4 | 72.1 | 68.6 | 59.3 | 63.6 |
| JZ07 | MaxEnt(Seq+Syn) | 73.7 | 69.4 | 71.5 | |||
| JZ07 | MaxEnt(Syn(uni+bi+tri)) | 72.6 | 68.8 | 70.7 | |||
| ZZSZ06 | Composite kernel 1 (lin comb) | 73.5 | 67.0 | 70.1 | |||
| ZG05 | Composite feature kernel | 69.2 | 70.5 | 70.4 | |||
| JZ07 | MaxEnt(Seq(uni+bi+tri)) | 71.7 | 65.3 | 68.3 | |||
| JZ07 | MaxEnt(Seq(uni+bi)) | 66.2 | 70.1 | 68.1 | |||
| ZZSZ06 | Tree kernel only | 72.5 | 56.7 | 63.6 | |||
| JZ07 | MaxEnt(Seq(uni)) | 64.7 | 61.4 | 63.0 | |||
| ZZSZ06 | Entity kernel only | 75.1 | 42.7 | 54.4 |
- 7Mt ⇒ 7 Major types
- 23St ⇒ 23 Subtypes
- P ⇒ Precision.
- R ⇒ Recall.
- F ⇒ F-Measure.
- ZZJZ07 ⇒ (ZhouZJZ, 2007)
- ZZSZ06 ⇒ (Zhang et al., 2006)
- JZ07 ⇒ (Jiang and Zhai, 2007)
- HHN06 ⇒ (HassanHN, 2006
- ZG05 ⇒ (Zhao and Grishman, 2005)