Since 2004, OAEI organises evaluation campaigns aiming at evaluating ontology matching technologies.
As in 2012, we are running a pre-evaluation campaign to smooth the transition to the new evaluation platform, the HOBBIT platform. We are also having a special issue devoted to the participants and evaluation of the OAEI 2017.5 in the Knowledge Engineering Review journal. As usual, participants will also be invited to present their results during the Ontology Matching workshop 2018.
To ease communication between participants and track organizers, we will have two OAEI contact points: Ernesto Jimenez-Ruiz (ernesto [.] jimenez [.] ruiz [at] gmail [.] com) and Daniel Faria (daniel [.] faria [.] 81 [at] gmail [.] com).
The OAEI 2017.5 campaign will once again confront ontology matchers to ontology and data sources to be matched. Please carefully follow the new instructions for participation. The following test sets are available (to be completed):
OAEI track organisers are required to migrate the SEALS datasets following the HOBBIT benchmark specification.
The following benchmark definitions can be used as reference:
The OAEI 2017.5 pre-campaign will fully rely on the HOBBIT platform, thus system developers are required to follow different instructions with respect to the SEALS-based wrapping.
Systems willing to participate in the OAEI 2017.5 need to follow the following instructions:
Alternatively, Sven Hertling (sven [at] informatik [dot] uni-mannheim [dot] de) also provide support to migrate tools from SEALS to HOBBIT. Check details here and/or get in touch with him.
Please check the OAEI rules here before participating. In addition, systems that rely or are derived from other ontology matching systems should: (a) clearly state the system they rely on, and (b) what was changed from / added to the original system.
Evaluation will be run under the HOBBIT platform.
We aim at evaluating participants will respect to all of the OAEI tracks even though the system might be specialized for some specific kind of matching problems. We know that this can be a problem for some systems that have specifically been developed for, e.g., matching biomedical ontologies; but this point can still be emphasized in the specific results paper about the system in case the results generated for some specific track are not good at all.
The results will be reported at the Ontology matching workshop, which will be collocated with the 17th International Semantic Web Conference (ISWC 2018).
Visual support for the evaluation (optional use): AlignmentCubes is an interactive visual environment which provides comparative exploration and evaluation of multiple ontology alignments at different level of detail. AlignmenCubes can support (a) developers during the process of developing and debugging alignment algorithms, (b) evaluators to make observations at different level of detail, and (c) data integrators to select and configure their tools as well as to develop and debug alignments. More information can be found here.