| The 2005 results are available here. |
The increasing number of methods available for schema matching/ontology integration suggests the need to establish a consensus for evaluation of these methods. There is now a coordinated international initiative to forge this consensus. After the two events organized in 2004 (namely, the Information Interpretation and Integration Conference (I3CON) and the EON Ontology Alignment Contest), this year we organize one unique evaluation of which the outcome will be presented at the Workshop on Integrating Ontologies held in conjunction with K-CAP 2005 at Banff (Canada) on October 2, 2005.
This year's campaign will consist of three parts: it will feature two real world blind tests (anatomy and directory) and a systematic benchmark test suite. By blind tests it is meant the result expected from the test is not known in advance. The evaluation organisers provide the participants with the pairs of ontologies to align as well as (in the case of the systematic benchmark suite only) expected results. The ontologies are described in OWL-DL and serialized in the RDF/XML format. The expected alignments are provided in a standard format expressed in RDF/XML and described in http://co4.inrialpes.fr/align/.
The anatomy real world case covers the domain of body anatomy and will consists of two ontologies with an approximate size of several 10k classes and several dozen of relations.
The directory real world case consists of alignming web sites directory (like open directory or Yahoo's). It is more than two thousand elementary tests.
Like for last year's EON contest, a systematic benchmark series has been produced. The goal of this benchmark series is to identify the areas in which each alignment algorithm is strong and weak. The test is based on one particular ontology dedicated to the very narrow domain of bibliography and a number of alternative ontologies of the same domain for which alignments are provided.
The evaluation will be processed in three successive steps.
The ontologies and alignments of the evaluation are provided in advance during the period between June 1st and July 1st. This gives potential participants the occasion to send observations, bug corrections, remarks and other test cases to the organizers. The goal of this primary period is to be sure that the delivered tests make sense to the participants. The feedback is important, so all participants should not hesitate to provide it. The tests will certainly change after this period, but only for ensuring a better participation to the tests. The final test base will be released on July 4th.
During the execution phase the participants will use their algorithms to automatically align the ontologies of both part. The participants should only use one algorithm and the same set of parameters for all tests. Of course, it is regular to select the set of parameters that provide the best results. Beside the parameters the input of the algorithms must be the two provided ontology to align and any general purpose resource available to everyone (that is no resourse especially designed for the test). In particular, the participants should not use the data (ontologies and results) from other test sets to help their algorithm. And cheating is not fair...
The participants will provide their alignment for each test in the Alignment format. The results will be provided in a zip file containing one directory per test (named after its number) and each directory containing one result file in the RDF/XML Alignment format with always the same name (e.g., participant.rdf). This should yield the following structure:
participant.zip +- benchmark | +- 101 | | +- participant.rdf | +- 102 | | +- participant.rdf | + ... +- anatomy | +- participant.rdf +- directory +- 1 | +- participant.rdf + ...
They will also provide a paper to be published in the proceedings and a link to their program and parameter set.
The only interesting alignments are those involving classes and properties of the given ontologies. So the alignments should not align individuals, nor entities from the external ontologies.
The organizers will evaluate the results of the algorithms used by the participants and provide comparisons on the basis of the provided alignments.
In order to ensure that it will be possible to process automatically the provided results, the participants are requested to provide (preliminary) results by August 15th. In the case of the real world ontologies only the organizers will do the evaluation with regard to the withheld alignments. An email with the location of the required zip file must be sent to the contact addresses below.
The standard evaluation measures will be precision and recall computed against the reference alignments. For the matter of aggregation of the measures we will use weighted harmonic means (weight being the size of reference alignment). Another improvement that might be used is the computation of precision/recall graphs.
Further, it is planned to introduce new measures addressing some limitations of precision and recall. These will be presented at the workshop discussion in order for the participants to provide feedback on the opportunity to use them in a further evaluation.
Finally, in an experimental way, we will attempt this year at reproducing your results. To that extent, we require a link to the tool and the set of parameters.
First publication of test cases: June 10th, 2005.
Comments due: anytime before July 1st, 2005
Final publication of test cases: July 4th, 2005.
Preliminary results due: August 15th, 2005.
Camera ready copies: September 2nd, 2005.
Workshop : October 2nd, 2005.
From the results of the experiments the participants are expected to provide the organisers with a paper to be published in the proceedings of the workshop. The paper must be at most 8 pages long and formatted according to the guidelines of the main conference (http://www.kcap05.org/). To ensure easy comparability among the participants it has to follow the given outline. A package with LaTeX and Word templates can be found here The above mentionned paper must be sent by September 2nd to Heiner Stuckenschmidt (heiner (à) cs . vu . nl) with copy to Jerome . Euzenat (à) inrialpes . fr.
The results from both, the participants and the organizers, will be presented at the Workshop on Integrating Ontologies at K-CAP 2005 taking place at Banff (Canada) on October, 2nd 2005. We hope to see you there.
The zip file containing all the data is here as well as the full description of the benchmarks.
Real world case on directories can be found here.
Real world case on anatomy can be found here.
The participants may use the Alignment API for generating and manipulating their alignments (in particular for computing evaluation of results).
Benjamin Ashpole (Lockheed Martin Advanced Technology Lab.)
Marc Ehrig (University of Karlsruhe)
Jérôme Euzenat (INRIA Rhône-Alpes)
Lewis Hart (Applied Minds)
Todd Hughes (Lockheed Martin Advanced Technology Labs)
Natasha Noy (Stanford University)
Heiner Stuckenschmidt (Vrije Universiteit Amsterdam)
Petko Valtchev (Université de Montréal, DIRO)