Since 2010, it is not necessary anymore to download the data set (it has always been better to get it on the web. The SEALS platform will provide the data sets. In the future, it is even planned that it will generate the data set on the fly.
The goal of the benchmark test library is to offer a set of tests which are wide in feature coverage, progressive and stable. It serves the purpose of evaluating the strength and weakness of matchers (by being progressive and wide coverage) and measuring the progress of matchers (by being stable and reusable over the years).
Appart from that, it is made of a set of 111 pairs of ontologies for which the participants have to return an alignment in the alignment format. These results are compared against reference alignments which are available from the beginning.
The domain of this first test is Bibliographic references. It is, of course, based on a subjective view of what must be a bibliographic ontology. There can be many different classifications of publications (based on area, quality, etc.). We choose the one common among scholars based on mean of publications; as many ontologies below (tests #301-304), it is reminiscent to BibTeX.
This reference ontology contains 33 named classes, 24 object properties, 40 data properties, 56 named individuals and 20 anonymous individuals.
The complete ontology is that of test #101.
The reference ontology is based on the one of the first EON Ontology Alignment Contest. It is improved by comprising a number of circular relations that were missing from the first test. In 2006, we have put the UTF-8 version of the tests as standard, the ISO-8859-1 being optional. In 2007, the tests are the same as in 2006.
The reference ontology is put in the context of the semantic web by using other external resources for expressing non bibliographic information. It takes advantage of FOAF (http://xmlns.com/foaf/0.1/) and iCalendar (http://www.w3.org/2002/12/cal/) for expressing the People, Organization and Event concepts. Here are the external reference used:
This reference ontology is a bit limited in the sense that it does not contain attachement to several classes.
Similarly the kind of proposed alignments is still limited: they only match named classes and properties, they mostly use the "=" relation with confidence of 1.
The ontologies are described in OWL-DL and serialized in the RDF/XML format.
The tests are systematically generated to as to start from some reference ontology and discarding a number of information in order to evaluate how the algorithm behave when this information is lacking.
Test numbering (almost) fully preserves the numbering of the first EON contest.
The table below summarize what has been retracted from the reference ontology. There are here 6 catagories of alteration:
| # | Name | Com | Hier | Inst | Prop | Class | Comment | |
| 101 | 0 | 0 | 0 | 0 | 0 | 0 | Reference alignment | |
| 102 | Irrelevant ontology | |||||||
| 103 | 0 | 0 | 0 | 0 | 0 | 0 | Language generalization | |
| 104 | 0 | 0 | 0 | 0 | 0 | 0 | Language restriction | |
| 201 | R | 0 | 0 | 0 | 0 | 0 | No names | |
| 202 | R | N | 0 | 0 | 0 | 0 | No names, no comments | |
| 203 | 0 | N | 0 | 0 | 0 | 0 | No comments (was missspelling) | |
| 204 | C | 0 | 0 | 0 | 0 | 0 | Naming conventions | |
| 205 | S | 0 | 0 | 0 | 0 | 0 | Synonyms | |
| 206 | F | F | 0 | 0 | 0 | 0 | Translation | |
| 207 | F | 0 | 0 | 0 | 0 | 0 | ||
| 208 | C | N | 0 | 0 | 0 | 0 | ||
| 209 | S | N | 0 | 0 | 0 | 0 | ||
| 210 | F | N | 0 | 0 | 0 | 0 | ||
| 221 | 0 | 0 | N | 0 | 0 | 0 | No specialisation | |
| 222 | 0 | 0 | F | 0 | 0 | 0 | Flatenned hierarchy | |
| 223 | 0 | 0 | E | 0 | 0 | 0 | Expanded hierarchy | |
| 224 | 0 | 0 | 0 | N | 0 | 0 | No instance | |
| 225 | 0 | 0 | 0 | 0 | R | 0 | No restrictions | |
| 226 | No datatypes | |||||||
| 227 | Unit difference | |||||||
| 228 | 0 | 0 | 0 | 0 | N | 0 | No properties | |
| 229 | Class vs instances | |||||||
| 230 | 0 | 0 | 0 | 0 | 0 | F | Flattened classes | |
| 231* | 0 | 0 | 0 | 0 | 0 | E | Expanded classes | |
| 232 | 0 | 0 | N | N | 0 | 0 | ||
| 233 | 0 | 0 | N | 0 | N | 0 | ||
| 236 | 0 | 0 | 0 | N | N | 0 | ||
| 237 | 0 | 0 | F | N | 0 | 0 | ||
| 238 | 0 | 0 | E | N | 0 | 0 | ||
| 239 | 0 | 0 | F | 0 | N | 0 | ||
| 240 | 0 | 0 | E | 0 | N | 0 | ||
| 241 | 0 | 0 | N | N | N | 0 | ||
| 246 | 0 | 0 | F | N | N | 0 | ||
| 247 | 0 | 0 | E | N | N | 0 | ||
| 248 | N | N | N | 0 | 0 | 0 | ||
| 249 | N | N | 0 | N | 0 | 0 | ||
| 250 | N | N | 0 | 0 | N | 0 | ||
| 251 | N | N | F | 0 | 0 | 0 | ||
| 252 | N | N | E | 0 | 0 | 0 | ||
| 253 | N | N | N | N | 0 | 0 | ||
| 254 | N | N | N | 0 | N | 0 | ||
| 257 | N | N | 0 | N | N | 0 | ||
| 258 | N | N | F | N | 0 | 0 | ||
| 259 | N | N | E | N | 0 | 0 | ||
| 260 | N | N | F | 0 | N | 0 | ||
| 261 | N | N | E | 0 | N | 0 | ||
| 262 | N | N | N | N | N | 0 | ||
| 265 | N | N | F | N | N | 0 | ||
| 266 | N | N | E | N | N | 0 | ||
| 301 | Real: BibTeX/MIT | |||||||
| 302 | Real: BibTeX/UMBC | |||||||
| 303 | Real: Karlsruhe | |||||||
| 304 | Real: INRIA | |||||||
These transformations generate the set of tests displayed below with
their derivation relations. The upper the test, the easier it is
supposed to be.
Below are provided the set of tests with expected alignments in the Alignment format described here. Each ontology is to be aligned with the reference ontology (i.e., that of test #101).
The only interesting alignments are those involving classes and properties of the given ontologies. So the alignments should not align individuals, nor entities from the external ontologies.
There is some chance that the final test be improved by adding entity expansion and reduction. It is also possible that there will be a lot of more instances in each ontology.
This test compares the ontology to itself.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This test compares the ontology to a totally irrelevant one. We ussed the food ontology given in the OWL guide (verbatim).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
NOTE: The onto.rdf file is provided for compatibility purposes. However, it is possible to run the test directly on the true file, i.e., http://www.w3.org/TR/2003/PR-owl-guide-20031209/wine. Indeed, wine imports food which imports wine and some parsers do not realize that onto.rdf is the same ontology as wine.
This test compares the ontology with its generalisation in OWL Lite (i.e., unavailable constraints are replaced by the more general available). The generalization basically removes owl:unionOf and owl:oneOf and the Property types (owl:TransitiveProperty).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This test compares the ontology with its restriction in OWL Lite (where unavailable constraints have been discarded).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Each label or identifier is replaced by a random one.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Each label or identifier is replaced by a random one. Comments (rdfs:comment and dc:description) have been suppressed as well.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
A random, but consistent, typo generator should be applied to labels and comments.
Not available in this test (if you know how to do it, contact me).
Different naming conventions (Uppercasing, underscore, dash, etc.) are used for labels. Comments have been suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Labels are replaced by synonyms. Comments have been suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
The complete ontology is translated to another language than english (French in the current case, but other languages would be fine).
Ontology : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
Alignment : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
NOTE: You can use alternatively the ISO-Latin-1 (ISO-8859-1) version of the tests by renaming them after their UTF-8 version.
Each label or identifieris translated to another language than english (French in the current case, but other languages would be fine).
Ontology : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
Alignment : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
NOTE: You can use alternatively the ISO-Latin-1 (ISO-8859-1) version of the tests by renaming them after their UTF-8 version.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
Alignment : [RDF/XML]
[RDF/XML in ISO-8859-1]
[HTML]
NOTE: You can use alternatively the ISO-Latin-1 (ISO-8859-1) version of the tests by renaming them after their UTF-8 version.
All subclass assertions to named classes are suppressed.
Ontology : [RD/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: compile inheritance)
A hierarchy still exists but has been strictly reduced.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
The alignment here contains relations which are not "=" but "<".
Numerous intermediate classes are introduced within the hierarchy.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
All individuals have been suppressed from the ontology.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
All local restrictions on properties have been suppressed from the ontology.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: no property nor global restrictions on properties)
In this test all datatypes are converted to xsd:string.
Not available in this test
(Measurable) values are expressed in different datatypes.
Not available in this test
Properties and relations between objects have been completely suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: leave the properties in instances)
Some classes have become instances.
Not available in this test.
Some components of classes are expanded in the class structure (e.g., year, month, day attributes instead of date).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Here one limitation of the proposed format is that it does not cover alignments such as: journalName = name o journal.
Some classes are spreaded over several classes.
Not available in this test.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Note that the 3xx tests are only here for comparability with previous years. We know that the reference alignments for these tests are not perfect (especially because the ontologies sometimes contain flaws).
For a computer scientist, BibTeX is the starting point for a useful bibliographic ontology. It is of wide use and relatively well thought out. This ontology can be found at and is documented in BibTex in OWL.
This is a test of comparing our test ontology with an actual ontology, simpler and closer to the initial BibTeX ontology. The alignment result contains some inclusion (<) alignment relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This ontology is very similar to the previous one, even closer to the genuine BibTeX, with different extensions and naming conventions. It can be found at http://ebiquity.umbc.edu.
The alignment result also contains some inclusion (<) alignment relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This is a test of comparing our test ontology with an actual ontology which contains more items than the actual items used in the current ontology.
The Karlsruhe ontology (http://www.aifb.uni-karlsruhe.de/ontology) is used in the Ontoweb portal. It is a refinement from other ontologies such as (KA)2. As such it does not only defines bibliographic items but many other items.
The alignment contains < as well as > relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This is a test of comparing our test ontology with an actual ontology which is not equivalent but quite close (it can be though of as a previous version).
This INRIA ontology (fr.inrialpes.exmo.rdf.bib.owl) has been designed by Antoine Zimmermann from the BibTeX in OWL ontology and our Bibliographic XML DTD. Its goal was to gather easily a number of RDF items. These items were BibTeX entries found on the web and transformed in RDF according to this ontology.
The actual hierarchy of this ontology contains classes which are subclasses of several other classes.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
The release notes of the previous test versions used in the OAEI-2010 can be found here.
The test set is made of a set of directories (one per test) containing an ontology (onto.rdf) in OWL. Participants must match each of these ontologies with the ontology found in 101/onto.rdf and output the resulting alignment, in the alignment format, in a file participant.rdf in the same directory (participant is replaced by the name of the system: no more than 8 alphanumeric characters, lowercase).
The test set also contains the reference alignments against which the results will be evaluated. It is, of course, forbiden to use any for there results for performing the matching task.
The resulting alignments in the same directory structure can be zipped (please, zipped) and sent to the organisers.
The schedule is that of http://oaei.ontologymatching.org/2010/.
A zip file contains all the data about the test. That is:
We would like to find a simple converter from OWL/XML/RDF to N3 (and maybe RDFS).
You can use the Alignment API for manipulating and generating your alignments (in particular for computing evaluation of your results).
Many resources have been used for setting up this test:
Various people helped testing or suggested improvements and tests:
Contact address is Cassia : Trojahn # inrialpes : fr