The OAEI 2016 campaign will once again confront ontology matchers to
ontology and data sources to be matched.
This year, the following test sets are available:
- benchmark
-
Like in previous campaigns, a systematic benchmark series has
to be matched. The goal of this benchmark series is to identify the areas in
which each alignment algorithm is strong and weak. The test is not
anymore based on the very same dataset that has been used from 2004
to 2010. We are now able to generate undisclosed tests with the same
structure. They provide strongly comparable results and allow for
testing scalability.
- anatomy
- The anatomy
real world case is about matching the Adult Mouse Anatomy (2744 classes) and the NCI Thesaurus (3304 classes) describing the human anatomy.
- conference
-
The goal of the track is to find alignments within a collection of ontologies describing the domain of organising conferences. Additionally, 'complex correspondences' are also very welcome. Alignments will be evaluated automatically against reference alignments also considering its uncertain version presented at ISWC 2014. Summary results along with detail performance results for each ontology pair (test case) and comparison with tools' performance from last years will be provided.
- Multifarm
-
This dataset is composed of a subset of the Conference dataset, translated in nine different languages (Arabic, Chinese, Czech, Dutch, French, German, Portuguese, Russian, and Spanish) and the corresponding alignments between these ontologies. Based on these test cases, it is possible to evaluate and compare the performance of matching approaches with a special focus on multilingualism.
- Interactive matching evaluation
(interactive)
-
This track offers the possibility to compare different interactive matching tools which require user interaction.
The goal is to show if user interaction can improve the matching results, which methods are most promising and how many
interactions are necessary. All participating systems are evaluated using an oracle which bases on the reference alignment.
Using the SEALS client, the matching system only needs to be slightly adapted to participate to this track.
- Large Biomedical Ontologies (largebio)
-
This track consists of finding alignments between the Foundational Model of
Anatomy (FMA), SNOMED CT, and the National Cancer Institute Thesaurus (NCI).
These ontologies are semantically rich and contain tens of thousands of
classes. UMLS Metathesaurus has been selected as the basis for the track
reference alignments.
- Disease and Phenotype (phenotype)
-
The Pistoia Alliance Ontologies Mapping project team organises and sponsors this
track based on a real use case where it is required to find alignments between
disease and phenotype ontologies. Specifically, the selected ontologies are the Human Phenotype Ontology (HPO), the Mammalian Phenotype Ontology (MP),
the Human Disease Ontology (DOID), and the Orphanet and Rare Diseases Ontology (ORDO).
- Process Model Matching (pm)
-
This track is a spinoff from the Process Model Matching Contest. It is concerned with the task of matching process models,
originally represented in BPML. These models have been converted to an ontological representation. The resulting matching
task is a special case of an interesting instance matching problem.
- Instance Matching (im)
-
The Instance Matching Track aims at evaluating the performance of matching tools when the goal is to detect the degree of
similarity between pairs of instances expressed in the form of OWL Aboxes.
The track constains the following three independent tasks, namely: SABINE, SYNTHETIC and DOREMUS.
Evaluation
OAEI 2016 will continue the procedure of running using the SEALS
infrastructure introduced in 2011. The results will be reported at
the Ontology matching
workshop of the 13th
International Semantic Web Conference (ISWC 2016).
The overall process of participation including how to accomplish tool
bundling is described here.
Participants will be evaluated with respect to
all of the OAEI tracks even though the system
might be specialized for some specific kind of matching problems.
We know that this can be a problem for some systems that have specifically been developed for, e.g., matching biomedical ontologies;
but this point can still be emphasized in the specific results paper about the system in case the results generated for some specific track are not good at all.
Please note that, a matcher may want to behave differently given what it is
provided with as ontologies; however, this should not be based on features
specific of the tracks (e.g., there is a specific string in the URL, or a specific
class name) but on features of the ontologies (e.g., there are no instances or
labels are in German). Check the OAEI rules here.
SEALS evaluation process
Following the successful campaigns since 2011, most of the tests will be evaluated using the SEALS infrastructure.
The evaluation process is detailed here, and in general it follows the same pattern as in past years:
- Participants register their tool;
- Participants wrap their tools as a SEALS package;
- Participants can test their tools with the SEALS client on the
data-sets provided with reference alignments by each track organizer. The ids of those data-sets are given in each track web page;
- Organizers run the evaluation with both blind and published
datasets;
June 1st- (preliminary) datasets available.
June 30th- datasets are frozen.
- June 30th
- participants register their tool (still open!).
July 15th- submission instructions sent to participants.
July 31st- participants submit preliminary wrapped versions of their tools (zip file).
August 19th August 31st- participants submit final versions of their tools (zip file).
September 15th September 23rd- evaluation is executed and results are analyzed.
September 30th- Preliminary version of system papers due (details will be sent after registration).
October 18th- Ontology matching workshop.
November 15th- Final version of system papers due (sharp).
Authors of (1) the best biomedical-themed papers,
(2) system papers with competitive results in the OAEI biomedical-themed tracks, and
(3) biomedical-themed dataset descriptions, will be invited to submit an extended version of their contributions to be considered
in a special issue of the Journal of Biomedical Semantics (JBMS).
Presentation
From the results of the experiments, participants are expected
to provide the organisers with a paper to be published in the proceedings
of the Ontology matching workshop.
The paper should be no more than 8 pages long and formatted using the
LNCS Style.
To ensure easy comparability among the participants it has to follow the given
outline.
The outline of the paper is as below (see templates for more details):
- Presentation of the system
- State, purpose, general statement
- Specific techniques used
- Adaptations made for the evaluation
- Link to the system and parameters file
- Link to the set of provided alignments (in align format)
- Results
- 2.x) a comment for each dataset performed
- General comments
(not necessaryly by putting the section below but preferably in
this order).
- Comments on the results (strength and weaknesses)
- Discussions on the way to improve the proposed system
- Comments on the OAEI procedure (including comments on the SEALS evaluation, if relevant)
- Comments on the OAEI test cases
- Comments on the OAEI measures
- Proposed new measures
- Conclusions
- References
These papers are not peer-reviewed and are here to keep track of the
participants and the description of matchers which took part in the
campaign.
The results from both selected participants and organizers were presented
at the International Workshop on Ontology
Matching collocated with ISWC 2016 taking
place at Kobe (JP) in October 18th, 2016.
