Ontology Alignment Evaluation Initiative - OAEI 2012 Campaign

Ontology Alignment Evaluation Initiative

2012 Campaign

The increasing number of methods available for schema matching/ontology integration necessitate to establish a consensus for evaluation of these methods. Since 2004, OAEI organizes evaluation campaigns aiming at evaluating ontology matching technologies.

This year, we will execute an OAEI 2012 evaluation campaign, fully running on the SEALS platform, with the exception of the Instance track. The results will be reported at the Ontology matching workshop of the 11th International Semantic Web Conference (ISWC 2012). It also integrates results from OAEI 2011.5, such results will be replayed, eventually with newly submitted matchers, and integrated in detailed result reports.

The overall process of participation including how to accomplish the bundling of your tool is described here.

Problems

The OAEI 2012 campaign will once again confront ontology matchers to ontology and data sources to be matched. Following the OAEI 2011.5 campaign, it is now possible to automate evaluation to a large extent. This year many new test sets are available.

benchmark

Like in previous campaigns, a systematic benchmark series has to be matched. The goal of this benchmark series is to identify the areas in which each alignment algorithm is strong and weak. The test is not anymore based on the very same dataset that has been used from 2004 to 2010. We are now able to generate undisclosed tests with the same structure. They provide strongly comparable results and allow for testing scalability.

anatomy

The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.

conference

The goal of this track is to find alignments within a collection of ontologies describing the domain of organising conferences (the domain being well understandable for every researcher). Additionally, 'complex correspondences' are also very welcome. Results will be evaluated automatically against reference alignments and potentially by data-mining and logical reasoning techniques. Sample of correspondences and 'complex correspondences' will be evaluated manually.

Multifarm

This dataset is composed of a subset of the Conference dataset, translated in eight different languages (Chinese, Czech, Dutch, French, German, Portuguese, Russian, and Spanish) and the corresponding alignments between these ontologies. Based on these test cases, it is possible to evaluate and compare the performance of matching approaches with a special focus on multilingualism.

Library

The library track is a real-word task to match the STW and the TheSoz social science thesauri in SKOS. The goal of this track is to find whether the matchers can handle these lightweight ontologies including a huge amount of concepts and additional descriptions. Results will be avaluated both against a reference alignment and through manual scrutiny of alignments.

Large Biomedical Ontologies (largebio)

This track consists of finding alignments between the Foundational Model of Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI). These ontologies are semantically rich and contain tens of thousands of classes. UMLS Metathesaurus has been selected as the basis for the track reference alignments.

Instance matching (im)

The instance data matching track aims at evaluating tools able to identify similar instances among different RDF and OWL datasets. It features Web datasets, as well as a generated benchmark. Instance matching at OAEI 2012 is focused on RDF and OWL data in the context of the Semantic Web. Participants will be asked to execute their algorithms against various datasets and their results will be evaluated by comparing them with a pre-defined reference alignment. Participating systems are free to use any combination of matching techniques and background knowledge. Results, in the alignment format, will be evaluated according to standard precision and recall metrics. This year there are two tasks:

Interlinking New-York Times Data

Participants are requested to re-build the links among the NYT dataset itself, and to the external data sources DBPedia, Geonames and Freebase. This data set is also supplied in random segments for cross-validation of matching systems that use training data.

Synthetic Freebase data

Participants should match synthetic data generated from Freebase, in the same style as for the benchmark task.

Modalities

There are two types of evaluation modalities:

• Those tests evaluated with the SEALS platform;
• Those evaluated differently (the Instance matching tasks).

Please note that a tool that participates in one of the tracks conducted in SEALS modality, will be evaluated with respect to all of the other tracks in SEALS modality even though the tool might be specialized for some specific kind of matching problems. We know that this can be a problem for some systems that have specifically been developed for, e.g., matching biomedical ontologies; but this point can still be emphasized in the specific results paper that you have to write about your system in case the results generated for some specific track are not good at all.

SEALS evaluation process

Following the successful OAEI 2011.5 campaign, many tests will be evaluated under the SEALS platform. The evaluation process is detailed here, and in general it follows the same pattern than that one of OAEI 2011.5:

1. Participants wrap their tools as a SEALS platform package and register them to the SEALS portal;
2. Participants can test their tools with the SEALS client on the data-sets provided with reference alignments by each track organizer (until August 31th). The ids of those data-sets are given in each track web page;
3. Organizers run the evaluation on the SEALS platform from the tools registered in the platform and with both blind and published datasets;
4. For some tracks, results are (automatically) available on the SEALS portal.

Instance matching evaluation process

Contrary to the SEALS evaluation process, the Instance matching track requires participants to submit the alignments produced by their system instead of their system itself. Participants will be asked to execute their instance matching algorithms on the provided datasets, and submit the resulting alignments by sending them to the contact address mentioned on the Instance matching website. Results should be in the form of RDF Alignments. For the evaluation of instance matching systems that use training data, there is a specific partitioned data set available for 10-fold cross validation.

The standard evaluation measures will be precision and recall computed against the reference alignments. For the matter of aggregation of the measures we will use weighted harmonic means (weight being the size of reference alignment). Precision/recall graphs (a.k.a. precision at n) will also be computed, so it is advised that participants provide their results with a weight to each correspondence they found.

Schedule

Dates are subject to change.

June 15th

datasets available (for presceening).

July 1st

datasets are frozen.

September 1st

participants send final versions of their tools in the case of SEALS tracks, or their produced alignments in the case of the Instance matching track.

September 22nd

evaluation is executed and results are analyzed.

October 15th

final paper due.

November 11th

Ontology matching workshop.

Presentation

From the results of the experiments the participants are expected to provide the organisers with a paper to be published in the proceedings of the Ontology matching workshop. The paper must be no more than 8 pages long and formatted using the LNCS Style. To ensure easy comparability among the participants it has to follow the given outline. A package with LaTeX and Word templates is available here. The above mentioned paper must be sent in PDF format before October 16th to Jose-Luis . Aguirre (a) inria . fr with copy to pavel (a) dit . unitn . it.

Participants may also submit a longer version of their paper, with a length justified by its technical content, to be published online in the CEUR-WS collection and on the OAEI web site (this last paper will be due just before the workshop).

The outline of the paper is as below (see templates for more details):

1. Presentation of the system
   
   1. State, purpose, general statement
   2. Specific techniques used
   3. Adaptations made for the evaluation
   4. Link to the system and parameters file
   5. Link to the set of provided alignments (in align format)
2. Results
   
   • 2.x) a comment for each dataset performed
3. General comments
   (not necessaryly by putting the section below but preferably in this order).
   1. Comments on the results (strength and weaknesses)
   2. Discussions on the way to improve the proposed system
   3. Comments on the OAEI procedure (including comments on the SEALS evaluation, if relevant)
   4. Comments on the OAEI test cases
   5. Comments on the OAEI measures
   6. Proposed new measures
4. Conclusions
   
5. References
   

These papers are not peer-reviewed and are here to keep track of the participants and the description of matchers which took part in the campaign.

The results from both selected participants and organizers will be presented at the Seventh International Workshop on Ontology Matching collocated with ISWC 2012 taking place at Boston (USA) in November, 2012. We hope to see you there.