The increasing number of methods available for schema matching/ontology integration necessitate to establish a consensus for evaluation of these methods. Since 2004, OAEI organizes evaluation campaigns aiming at evaluating ontology matching technologies.
The OAEI 2010 campaign is associated to the ISWC Ontology matching and the Evaluation of semantic technologies workshops held in Shanghai, China in November, 2010.
The 2010 campaign introduces a new evaluation modality in association with the SEALS project. Its goal is to provide more automation to the evaluation and more direct feedback to the participants. The concerned datasets are benchmark, conference and anatomy. Participants in this modality must follow the specific instructions for participation.
We summarize below the variation between the results expected by these tests (all results are given in the Alignment format):
| test | seals | language | relations | confidence | Modalities | Language | Size (≈) |
| benchmarks | | OWL | = | [0 1] | open | EN | (36+61)^2*49 |
| anatomy | | OWL | = | [0 1] | blind | EN | 3k*3k |
| conference | | OWL-DL | =, <= | [0 1] | blind+open | EN | (20^2)*21 |
| directory | OWL | = | 1 | open | EN | 2854*6555+4000*10^2 | |
| imei | RDF | = | [0 1] | blind | EN | ||
| vlcr | SKOS +RDF | exactMatch | [0 1] | blind expert | DU+EN |
Each data set has a different evaluation process. They can be roughly divided into four groups:
For the tracks included in the new modality, namely benchmark, conference and anatomy, the participants must run their tools in the SEALS platform, following the instructions. For the other tracks, the participants must return their results to organisers.
However, the evaluation will be processed in the same three successive steps as before.
Ontologies are described in OWL-DL and serialized in the RDF/XML format. The expected alignments are provided in the Alignment format expressed in RDF/XML.
The ontologies and alignments of the evaluation are provided in advance during the period between June 1st and June 21st. This gives potential participants the occasion to send observations, bug corrections, remarks and other test cases to the organizers. The goal of this primary period is to be sure that the delivered tests make sense to the participants. The feedback is important, so all participants should not hesitate to provide it. The tests will certainly change after this period, but only for ensuring a better participation to the tests. The final test bases will be released on July 5th.
During the execution phase the participants will use their algorithms to automatically match the ontologies. Participants should only use one algorithm and the same set of parameters for all tests in all tracks. Of course, it is fair to select the set of parameters that provide the best results (for the tests where results are known). Beside the parameters the input of the algorithms must be the two provided ontology to match and any general purpose resource available to everyone (that is no resource especially designed for the test). In particular, participants should not use the data (ontologies and results) from other test sets to help their algorithm. And cheating is not fair...
The deadline for delivering final results is October 4th, sharp.
For old-style tracks, it is highly advised that participants send results before (preferably by August 30th) to the organisers so that they can check that they will be able to evaluate the results smoothly and can provide some feedback to participants.
Participants will provide their alignment for each test in the Alignment format. The results will be provided in a zip file containing one directory per test (named after its number) and each directory containing one result file in the RDF/XML Alignment format with always the same name (e.g., participant.rdf replacing "participant" by the name you want your system to appear in the results, limited to 6 alphanumeric characters). This should yield the following structure:
participant.zip +- benchmarks | +- 101 | | +- participant.rdf | +- 103 | | +- participant.rdf | + ... +- anatomy | +- 1 | | +- participant.rdf | +- 2 | | +- participant.rdf | +- ... +- directory | +- 1 | | +- participant.rdf | + ... + ...
For the SEALS modality, participants must register their tools by that time in the SEALS portal (see instructions) and must test their web service implementation in the SEALS platform (see instructions). Participants must guarantee that their tools generate alignments in the correct format (Alignment API).
All participants will also provide, for October 4th, a paper to be published in the proceedings
All participants are required to provide a link to their program and parameter set.
Apart from the instance matching track, the only interesting alignments are those involving classes and properties of the given ontologies. So these alignments should not align individuals, nor entities from the external ontologies.
The organizers will evaluate the results of the algorithms used by the participants and provide comparisons on the basis of the provided alignments.
In order to ensure that it will be possible to process automatically the provided results, participants are requested to provide (preliminary) results by August 30th (old-style tracks). In the case of blind tests only the organizers will do the evaluation with regard to the withheld alignments. In the case of double blind tests, the participants will provide a version of their system and the values of the parameters if any. An email with the location of the required zip files must be sent to the contact addresses below.
The standard evaluation measures will be precision and recall computed against the reference alignments. For the matter of aggregation of the measures we will use weighted harmonic means (weight being the size of reference alignment). Precision/recall graphs will also be computed, so it is advised that participants provide their results with a weight to each correspondence they found (participants can provide two alignment results: <name>.rdf for the selected alignment and <name>-full.rdf for the alignment with weights. Additionally, with the help of the SEALS platform, we will be able to measure runtime and alignment coherence for the anatomy and conference tracks.
From the results of the experiments the participants are expected to provide the organisers with a paper to be published in the proceedings of the Ontology matching workshop. The paper must be no more than 8 pages long and formatted using the LNCS Style. To ensure easy comparability among the participants it has to follow the given outline. A package with LaTeX and Word templates is made available here. The above mentionned paper must be sent in PDF format before October 4th to Cassia . Trojahn (a) inrialpes . fr with copy to pavel (a) dit . unitn . it.
Participants may also submit a longer version of their paper, with a length justified by its technical content, to be published online in the CEUR-WS collection and on the OAEI web site (this last paper will be due just before the workshops).
The outline of the paper is as below (see templates for more details):
These papers are not peer-reviewed and are here to keep track of the participations and the description of matchers which took part in the campaign.
The results from both selected participants and organizers will be presented at the Ontology matching workshop and at the Evaluation of semantic technologies workshop (for the tracks using the SEALS platform) at ISWC 2010 taking place at Shanghai (CN) in November, 2010. We hope to see you there.
Here are some tools that may help participants.
The instruction for using the SEALS platform are available here.
Participants may use the Alignment API for generating and manipulating their alignments (in particular for computing evaluation of results).
The participants may use various options if they need to convert SKOS vocabularies into OWL.
Vassilis Spiliopoulos pointed out to Altova transformer from OWL to N3 notation. This can be useful for some. This is a commercial tool with a 30 days free trial.