Ontology Alignment Evaluation Initiative - OAEI 2017 CampaignOAEI
OAEI 2017 results available here.
IBM Research sponsors a prize for the instance matching tracks

Ontology Alignment Evaluation Initiative

2017 Campaign

Since 2004, OAEI organises evaluation campaigns aiming at evaluating ontology matching technologies.

Problems

The OAEI 2017 campaign will once again confront ontology matchers to ontology and data sources to be matched. This year, the following test sets are available:

anatomy
The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.
conference
The goal of the track is to find alignments within a collection of ontologies describing the domain of organising conferences. Additionally, 'complex correspondences' are also very welcome. Alignments will be evaluated automatically against reference alignments also considering its uncertain version presented at ISWC 2014. Summary results along with detail performance results for each ontology pair (test case) and comparison with tools' performance from last years will be provided.
Multifarm
This dataset is composed of a subset of the Conference dataset, translated in nine different languages (Arabic, Chinese, Czech, Dutch, French, German, Portuguese, Russian, and Spanish) and the corresponding alignments between these ontologies. Based on these test cases, it is possible to evaluate and compare the performance of matching approaches with a special focus on multilingualism.
Interactive matching evaluation (interactive)
This track offers the possibility to compare different interactive matching tools which require user interaction. The goal is to show if user interaction can improve the matching results, which methods are most promising and how many interactions are necessary. All participating systems are evaluated using an oracle which bases on the reference alignment. Using the SEALS client, the matching system only needs to be slightly adapted to participate to this track.
Large Biomedical Ontologies (largebio)
This track consists of finding alignments between the Foundational Model of Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI). These ontologies are semantically rich and contain tens of thousands of classes. UMLS Metathesaurus has been selected as the basis for the track reference alignments.
Disease and Phenotype (phenotype)
The Pistoia Alliance Ontologies Mapping project team organises and sponsors this track based on a real use case where it is required to find alignments between disease and phenotype ontologies. Specifically, the selected ontologies are the Human Phenotype (HP) Ontology, the Mammalian Phenotype (MP) Ontology, the Human Disease Ontology (DOID), the Orphanet and Rare Diseases Ontology (ORDO), the Medical Subject Headings (MESH) ontology, and the Online Mendelian Inheritance in Man (OMIM) onbtology.
Process Model Matching (pm)
This track is a spinoff from the Process Model Matching Contest. It is concerned with the task of matching process models, originally represented in BPML. These models have been converted to an ontological representation. The resulting matching task is a special case of an interesting instance matching problem.
Instance Matching (im)
The Instance Matching Track aims at evaluating the performance of matching tools when the goal is to detect the degree of similarity between pairs of instances expressed in the form of OWL Aboxes. The track consists of different independent tasks and participants can submit results related to one, more, or even all the expected tasks.
HOBBIT Link Discovery (hobbit)
In this track two benchmark generators are proposed to deal with link discovery for spatial data where spatial data are represented as trajectories (i.e., sequences of longitude, latitude pairs). This new track is based on the HOBBIT platform and it requires to follow different intructions from the SEALS-based tracks (see details here).

Evaluation

This year we will start adopting the HOBBIT platform to conduct the evaluation of the HOBBIT Link Discovery track. Systems willing to participate in this track need to follow the instructions listed here.

The rest of OAEI 2017 tracks will continue the procedure of running using the SEALS infrastructure introduced in 2011. The overall process of participation including how to accomplish tool bundling using the SEALS pltaform is described here.

The results will be reported at the Ontology matching workshop of the 16th International Semantic Web Conference (ISWC 2017).

Evaluation rules

Participants will be evaluated with respect to all of the OAEI tracks even though the system might be specialized for some specific kind of matching problems. We know that this can be a problem for some systems that have specifically been developed for, e.g., matching biomedical ontologies; but this point can still be emphasized in the specific results paper about the system in case the results generated for some specific track are not good at all.

Please note that, a matcher may want to behave differently given what it is provided with as ontologies; however, this should not be based on features specific of the tracks (e.g., there is a specific string in the URL, or a specific class name) but on features of the ontologies (e.g., there are no instances or labels are in German). Check the OAEI rules here.

Systems that rely or are derived from other ontology matching systems should: (a) clearly state the system they rely on, and (b) what was changed from / added to the original system.

Evaluation process

Following the successful campaigns since 2011, most of the tests will be evaluated using the SEALS infrastructure. The evaluation process is detailed here, and in general it follows the same pattern as in past years:

  1. Participants register their tool;
  2. SEALS participants wrap their tools as a SEALS package;
  3. HOBBIT participants follow the instructions for the HOBBIT platform;
  4. SEALS participants can test their tools with the SEALS client on the data-sets provided with reference alignments by each track organizer. The ids of those data-sets are given in each track web page;
  5. HOBBIT participants can test their system online (see details here);
  6. Organizers run the evaluation with both blind and published datasets;

Visual support for the evaluation (optional use)

AlignmentCubes is an interactive visual environment which provides comparative exploration and evaluation of multiple ontology alignments at different level of detail. AlignmenCubes can support (a) developers during the process of developing and debugging alignment algorithms, (b) evaluators to make observations at different level of detail, and (c) data integrators to select and configure their tools as well as to develop and debug alignments. More information can be found here.

Schedule

June 1st
(preliminary) datasets available.
June 30th July 15th
datasets are frozen.
June 30th (still open)
participants register their tool (mandatory).
July 15th
submission is open, zipped SEALS packages (e.g., LogMap.zip) can be submitted using this form (requires a google account and a valid email).
July 31st
participants submit preliminary wrapped versions (zip file) of their tools (mandatory).
August 31st
participants submit final versions of their tools (zip file). SEALS tracks.
September 15th
participants submit final versions of their tools. HOBBIT track.
September 30th
evaluation is executed and results are analyzed. SEALS and HOBBIT tracks.
September 30thOctober 10th
Preliminary version of system papers due. Submit PDF paper (e.g., LogMap_prelim.pdf) using this form (requires a google account and a valid email).
October 21st
Ontology matching workshop.
November 15th
Final version of system papers due. Submit PDF (e.g., LogMap_final.pdf) paper using this form (requires a google account and a valid email).

Presentation

From the results of the experiments, participants are expected to provide the organisers with a paper to be published in the proceedings of the Ontology matching workshop. The paper should be no more than 8 pages long and formatted using the LNCS Style. To ensure easy comparability among the participants it has to follow the given outline.

The outline of the paper is as below (see templates for more details):

  1. Presentation of the system
    1. State, purpose, general statement
    2. Specific techniques used
    3. Adaptations made for the evaluation
    4. Link to the system and parameters file
    5. Link to the set of provided alignments (in align format)
  2. Results
    • 2.x) a comment for each dataset performed
  3. General comments
    (not necessaryly by putting the section below but preferably in this order).
    1. Comments on the results (strength and weaknesses)
    2. Discussions on the way to improve the proposed system
    3. Comments on the OAEI procedure (including comments on the SEALS evaluation, if relevant)
    4. Comments on the OAEI test cases
    5. Comments on the OAEI measures
    6. Proposed new measures
  4. Conclusions
  5. References

These papers are not peer-reviewed and are here to keep track of the participants and the description of matchers which took part in the campaign.

The results from both selected participants and organizers were presented at the International Workshop on Ontology Matching collocated with ISWC 2017 taking place at Wien (AT) in October 21st, 2017.

http://oaei.ontologymatching.org/2017/