Ontology Alignment Evaluation Initiative - OAEI 2017.5 Campaign

Ontology Alignment Evaluation Initiative

2017.5 Campaign

Since 2004, OAEI organises evaluation campaigns aiming at evaluating ontology matching technologies.

As in 2012, we are running a pre-evaluation campaign to smooth the transition to the new evaluation platform, the HOBBIT platform. We are also having a special issue devoted to the participants and evaluation of the OAEI 2017.5 in the Knowledge Engineering Review journal. As usual, participants will also be invited to present their results during the Ontology Matching workshop 2018.

To ease communication between participants and track organizers, we will have two OAEI contact points: Ernesto Jimenez-Ruiz (ernesto [.] jimenez [.] ruiz [at] gmail [.] com) and Daniel Faria (daniel [.] faria [.] 81 [at] gmail [.] com).

Problems

The OAEI 2017.5 campaign will once again confront ontology matchers to ontology and data sources to be matched. Please carefully follow the new instructions for participation. The following test sets are available (to be completed):

anatomy

The anatomy real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.

conference

The goal of the track is to find alignments within a collection of ontologies describing the domain of organising conferences. Additionally, 'complex correspondences' are also very welcome. Alignments will be evaluated automatically against reference alignments also considering its uncertain version presented at ISWC 2014. Summary results along with detail performance results for each ontology pair (test case) and comparison with tools' performance from last years will be provided.

Large Biomedical Ontologies (largebio)

This track consists of finding alignments between the Foundational Model of Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI). These ontologies are semantically rich and contain tens of thousands of classes. UMLS Metathesaurus has been selected as the basis for the track reference alignments.

Instance Matching (im)

The Instance Matching Track aims at evaluating the performance of matching tools when the goal is to detect the degree of similarity between pairs of instances expressed in the form of OWL Aboxes.

Link Discovery (link)

In this track two benchmark generators are proposed to deal with link discovery for spatial data where spatial data are represented as trajectories (i.e., sequences of longitude, latitude pairs). This track was the first OAEI track based on the HOBBIT platform.

Preparation phase

OAEI track organisers are required to migrate the SEALS datasets following the HOBBIT benchmark specification.

The following benchmark definitions can be used as reference:

• The OAEI 2017.5 Largebio Benchmark. Main contact: Ernesto Jimenez Ruiz: ernesto [dot] jimenez [dot] ruiz [at] gmail [dot] com
• The OAEI 2017 Spatial Benchmark. Main contact: Nina Saveta: jsaveta [at] gmail [dot] com

Execution phase

The OAEI 2017.5 pre-campaign will fully rely on the HOBBIT platform, thus system developers are required to follow different instructions with respect to the SEALS-based wrapping.

Systems willing to participate in the OAEI 2017.5 need to follow the following instructions:

• Ontology matching tasks. See example of LogMap wrapping for HOBBIT. So far implements the interfaces for Conference, Anatomy and LArgebio tracks. Main contact: Ernesto Jimenez Ruiz: ernesto [dot] jimenez [dot] ruiz [at] gmail [dot] com
• Link discovery task. See instructions here. Main contact: Nina Saveta: jsaveta [at] gmail [dot] com

Alternatively, Sven Hertling (sven [at] informatik [dot] uni-mannheim [dot] de) also provide support to migrate tools from SEALS to HOBBIT. Check details here and/or get in touch with him.

Please check the OAEI rules here before participating. In addition, systems that rely or are derived from other ontology matching systems should: (a) clearly state the system they rely on, and (b) what was changed from / added to the original system.

Evaluation phase

Evaluation will be run under the HOBBIT platform.

We aim at evaluating participants will respect to all of the OAEI tracks even though the system might be specialized for some specific kind of matching problems. We know that this can be a problem for some systems that have specifically been developed for, e.g., matching biomedical ontologies; but this point can still be emphasized in the specific results paper about the system in case the results generated for some specific track are not good at all.

The results will be reported at the Ontology matching workshop, which will be collocated with the 17th International Semantic Web Conference (ISWC 2018).

Visual support for the evaluation (optional use): AlignmentCubes is an interactive visual environment which provides comparative exploration and evaluation of multiple ontology alignments at different level of detail. AlignmenCubes can support (a) developers during the process of developing and debugging alignment algorithms, (b) evaluators to make observations at different level of detail, and (c) data integrators to select and configure their tools as well as to develop and debug alignments. More information can be found here.

Schedule

January

Preliminary datasets are available.

March 1 (extended)

System registration using this form.

March 9 (extended)

Final datasets available in HOBBIT.

March 30 (extended)

Deadline for tool submission using HOBBIT instructions.

April 27 (extended)

Evaluation completed.

http://oaei.ontologymatching.org/2017.5/