Ontology Alignment Evaluation Initiative - OAEI-2011.5 CampaignOAEI
Evaluation campaign is over! Preliminary results here

Ontology Alignment Evaluation Initiative

2011.5 Campaign

The increasing number of methods available for schema matching/ontology integration necessitate to establish a consensus for evaluation of these methods. Since 2004, OAEI organizes evaluation campaigns aiming at evaluating ontology matching technologies.

This year, we will execute an OAEI 2011.5 evaluation campaign, fully running on the SEALS platform and coordinated with the Second SEALS evaluation campaigns. The results will be reported at the 2nd iWEST workshop of the 9th Extended Semantic Web Conference (ESWC 2012) and will be integrated to those of OAEI 2012.

Problems

The goal of this campaign is to provide more automation to the evaluation and more direct feedback to the participants. As in OAEI 2011, participants in this modality must follow the specific instructions for participation.

Anatomy
The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.
The dataset has been used already in OAEI 2011. For this track, we will only run systems that have not been participating in OAEI 2011 as well as systems that have been modified between OAEI 2011 and OAEI 2011.5. The measured results will be integrated in an extended OAEI 2011 results presentation.
Conference
The goal of this track is to find all correct correspondences within a collection of ontologies describing the domain of organising conferences (the domain being well understandable for every researcher). Additionally, 'interesting correspondences' are also welcome. Results will be evaluated automatically against reference alignment and by data-mining and logical reasoning techniques. Sample of correspondences and 'interesting correspondences' will be evaluated manually.
The dataset has been used already in OAEI 2011. For this track, we will only run systems that have not been participating in OAEI 2011 as well as systems that have been modified between OAEI 2011 and OAEI 2011.5. The measured results will be integrated in an extended OAEI 2011 results presentation.
Multifarm (NEW DATASET)
This dataset is composed of a subset of the Conference dataset, translated in eight different languages (Chinese, Czech, Dutch, French, German, Portuguese, Russian, and Spanish) and the corresponding alignments between these ontologies. Based on these test cases, it is possible to evaluate and compare the performance of matching approaches with a special focus on multilingualism.
Matching Large Biomedical Ontologies (NEW DATASET)
This track consists of finding alignments between the Foundational Model of Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI). These ontologies are semantically rich and contain tens of thousands of classes. Note that for the OAEI 2011.5 only the case FMA-NCI will be evaluated. UMLS Metathesaurus has been selected as the basis for the track reference alignments.
Scalability tests: new generated benchmarks (NEW DATASETS)
In order to test the ability of matchers to deal with data sets of increasing number of elements, we generated new benchmark datasets, in the same models as the previous benchmarks, from seed ontologies with different sizes.

Evaluation process

The evaluation process follows the same pattern:

  1. Participants wrap their tools as a SEALS platform package and register them to the SEALS portal;
  2. Participants can test their tools with the SEALS client on the dataset provided with reference alignments (until March 15th);
  3. Organisers run the evaluation on the SEALS platform from the tools registered in the platform and with both blind and published datasets;
  4. Results are (automatically) available on the SEALS portal.

The standard evaluation measures will be precision and recall computed against the reference alignments. For the matter of aggregation of the measures we will use weighted harmonic means (weight being the size of reference alignment). Precision/recall graphs (a.k.a. precision at n) will also be computed, so it is advised that participants provide their results with a weight to each correspondence they found.

Schedule

January 15th
datasets available.
March 15th Deadline Extension: New Deadline March 18th
participants send final versions of their tools.
March - April
evaluation is executed and results are analyzed.
May 27-31
final results are reported at ESWC 2012.
Later on
results to be recomputed and integrated for OAEI 2012.

Tools and material

Here are some tools that may help participants.

SEALS platform

The instruction for using the SEALS platform are available here.

Processing tools

Participants may use the Alignment API for generating and manipulating their alignments (in particular for computing evaluation of results).