Ontology Alignment Evaluation Initiative - OAEI-2015 Campaign

Benchmark test

The goal of the benchmark test is to offer a set of tests which are wide in feature coverage, progressive and stable. It serves the purpose of evaluating the strength and weakness of matchers (by being progressive and wide coverage) and measuring the progress of matchers (by being stable and reusable over the years).

Expected results of matchers are alignments between the named and properties in the namespace of the ontologies. Instances are not in the reference alignments.

Data sets

The benchmark test library consists of data sets that are built from reference ontologies of different sizes and from different domains.

Since OAEI 2011, Benchmark test suites (or data sets) have been generated from seed ontologies [1]. This year, in addition to the bibliographic ontology, we have used one large ontology:

biblio
The bibliographic ontology described here has been the main reference ontology since the beginning of OAEI campaigns. The usual bibliography ontology which concerns bibliographic references and is inspired freely from BibTeX.
ifc
IFC4 is the release 4 of the buildingsmart Industry Foundation Classes in OWL. It has been modified so as to suppress the Ifc prefixes and converted to OWL/RDF.
energy
energyresource is an ontology representing energy information for Smart Home Systems developes at the Technische Universität Wien.

The following table summarizes the information about ontologies' sizes. classes+prop instances entities triples

Test set biblio energy
ontology size
classes+prop 97 523+110
instances 112 16
entities 209 723
triples 1332 9331

Testing your tool

It is not necessary anymore to download the data sets (it has always been better to get it on the web). The SEALS platform will provide the data sets.

All those data sets maintain the structure explained in the Example of a complete benchmark data set section, and testing with those data sets can be done by using the SEALS client. This client iterates over tests in a data set whose identifier is provided as a parameter. In all cases, the ontologies found in the data set directories are matched (either against the ontology found in 101/onto.rdf, or both ontologies to match are in the same directory). The resulting alignments must be outputted in the alignment format. They are placed in a local directory given also as a parameter to the client.

The identifiers of data sets for testing with the SEALS client are given below:

Biblio data set

IFC data set
  • Repository: http://repositories.seals-project.eu/tdrs/
  • Suite-ID: 2015benchmarks-ifc
  • Version-ID: 2015benchmarks-ifc-r1, 2015benchmarks-ifc-r2, 2015benchmarks-ifc-r3, 2015benchmarks-ifc-r4, 2015benchmarks-ifc-r5

This last test was replaced by:

energy data set

We encourage you to use the Alignment API for manipulating and generating your alignments, and, in particular, for computing evaluation of your results.

Contacts

Contact address is Jerome : Euzenat # inria : fr

References

[1] Jérôme Euzenat, Maria Roşoiu, Cássia Trojahn dos Santos. Ontology matching benchmarks: generation, stability, and discriminability, Journal of web semantics 21:30-48, 2013 [DOI:10.1016/j.websem.2013.05.002]