An increasing number of matchers are now capable of deriving mapping relations other than equivalence relations, such as subsumption, disjointness or named relations. This is a necessity given that we need to compute alignments between ontologies at different granularity levels or between ontologies that elaborate on non-equivalent elements. The evaluation of such mappings was addressed already in OAEI (2009) Oriented Matching track. Following these goals we wish to get a better insight into the types of non-equivalence mappings that state-of-the-art tools produce and report on existing methods for computing subsumption mappings. The track aims also to report on evaluation methods and measures for subsumption mappings, in conjunction to the computation of equivalence mappings. Targeting these goals, we have built new benchmark datasets that are described below. The track focuses on a gold standard based evaluation for two given datasets that has been derived from a) the OAEI benchmark dataset (all possible pairs of the real ontologies 301 to 304) and b) all pairs of ontologies in the Illinois course catalogs dataset.
Concerning the evaluation of equivalence and subsumption relations the track provides two datasets for the participants:
For each pair of ontologies we provide reference alignments for both equivalence and subsumption mappings and we aim to evaluate tools for their ability to identify both equivalences and subsumptions between classes. However, we aim at distinguishing between tools that (a) compute class equivalences and subsumptions, (b) compute class subsumptions without computing any equivalences and (c) compute class subsumptions and non-class equivalences. These can be done either by declaration of the participants or by providing justifications for the alignments: For subsumption mappings that cannot be inferred (e.g. by exploiting equivalence or subsumption relations between classes/properties), we have added an annotation property called 'Alignment_Argument' in the corresponded alignment cells of the refalign.rdf file. The value of this property is an argument upon the reference alignment. Each tool can assign a value to this property in relation to each mapping relation discovered that provides grounding for its decisions.
It is important to use our related namespace (http://ai-lab-webserver.aegean.gr/ai-lab/oaei2011/alignArguments.rdf) in order to annotate your alignments (similar to our refalign.rdf files provided)!!!
The reference alignments (refalign.rdf) have been constructed by manually creating a merged ontology for each pair of source ontologies as follows:
The datasets are accompanied with documentation of the specific changes made to the original datasets. The rationale behind these changes can be found a) in the documentation.txt file in each ontology pair folder, b) as an annotation property in the specification of the subsumption relations in the merged ontology, c) in the corresponded alignment cells of the refalign.rdf file.
Datasets can be downloaded directly from here.
We will use precision, recall and f-measures for the efficacy of tools to produce the mappings. Participants will have to provide an alignment containing both equivalent and subsumption relations, optionally (but desired) with corresponding arguments.
The alignArguments namespace extension in order to be able to annotate your alignments (similar to our refalign.rdf files provided) can be found at http://ai-lab-webserver.aegean.gr/ai-lab/oaei2011/alignArguments.rdf. Additional argumentation extensions, published as part of the on-line Alignment API documentation at http://alignapi.gforge.inria.fr/labels.html, may be used.
George Vouros, Konstantinos Kotis and Vassilis Spiliopoulos, University of the Aegean, Greece E-mails: georgev@aegean.gr, kotis@aegean.gr, 2billy@gmail.com