The 2005 benchmark test is available HERE.

Ontology Alignment Evaluation Initiative

Test library

Reference ontology

The reference ontology is based on the one of the first EON Ontology Alignment Contest. It is improved by comprising a number of circular relations that were missing from the first test.

The domain of this first test is Bibliographic references. It is, of course, based on a subjective view of what must be a bibliographic ontology. There can be many different classifications of publications (based on area, quality, etc.). We choose the one common among scholars based on mean of publications; as many ontologies below (tests #301-304), it is reminiscent to BibTeX.

This reference ontology contains 33 named classes, 39 object properties, 20 data properties, 56 named individuals and 20 anonymous individuals.

The complete ontology is that of test #101.

The reference ontology is put in the context of the semantic web by using other external resources for expressing non bibliographic information. It takes advantage of FOAF (http://xmlns.com/foaf/0.1/) and iCalendar (http://www.w3.org/2002/12/cal/) for expressing the People, Organization and Event concepts. Here are the external reference used:

http://www.w3.org/2002/12/cal/#:Vevent (defined in http://www.w3.org/2002/12/cal/ical.n3 and supposedly in http://www.w3.org/2002/12/cal/ical.rdf)
http://xmlns.com/foaf/0.1/#:Person (defined in http://xmlns.com/foaf/0.1/index.rdf)
http://xmlns.com/foaf/0.1/#:Organization (defined in http://xmlns.com/foaf/0.1/index.rdf)

This reference ontology is a bit limited in the sense that it does not contain attachement to several classes.

Similarly the kind of proposed alignments is still limited: they only match named classes and properties, they mostly use the "=" relation with confidence of 1.

The ontologies are described in OWL-DL and serialized in the RDF/XML format. I'd be happy to translate them into N3 if someone have some translation program.

Tests at a glance

The tests are systematically generated to as to start from some reference ontology and discarding a number of information in order to evaluate how the algorithm behave when this information is lacking.

Test numbering (almost) fully preserves the numbering of the first EON contest.

The table below summarize what has been retracted from the reference ontology. There are here 6 catagories of alteration:

Name: Name of entities that can be replaced by (R/N) random strings, (S)ynonyms, (N)ame with different conventions, (F) strings in another language than english.
Comments: Comments can be (N) suppressed or (F) translated in another language.
Specialization Hierarchy: can be (N) suppressed, (E)xpansed or (F)lattened.
Instances: can be (N) suppressed
Properties: can be (N) suppressed or (R) having the restrictions on classes discarded.
Classes: can be (E)xpanded, i.e., relaced by several classes or (F)latened.

#	Name	Com	Hier	Inst	Prop	Class	Comment
101	0	0	0	0	0	0	Reference alignment
102							Irrelevant ontology
103	0	0	0	0	0	0	Language generalization
104	0	0	0	0	0	0	Language restriction
201	R	0	0	0	0	0	No names
202	R	N	0	0	0	0	No names, no comments
203	0	N	0	0	0	0	No comments (was missspelling)
204	C	0	0	0	0	0	Naming conventions
205	S	0	0	0	0	0	Synonyms
206	F	F	0	0	0	0	Translation
207	F	0	0	0	0	0
208	C	N	0	0	0	0
209	S	N	0	0	0	0
210	F	N	0	0	0	0
221	0	0	N	0	0	0	No specialisation
222	0	0	F	0	0	0	Flatenned hierarchy
223	0	0	E	0	0	0	Expanded hierarchy
224	0	0	0	N	0	0	No instance
225	0	0	0	0	R	0	No restrictions
226							No datatypes
227							Unit difference
228	0	0	0	0	N	0	No properties
229							Class vs instances
230	0	0	0	0	0	F	Flattened classes
231*	0	0	0	0	0	E	Expanded classes
232	0	0	N	N	0	0
233	0	0	N	0	N	0
236	0	0	0	N	N	0
237	0	0	F	N	0	0
238	0	0	E	N	0	0
239	0	0	F	0	N	0
240	0	0	E	0	N	0
241	0	0	N	N	N	0
246	0	0	F	N	N	0
247	0	0	E	N	N	0
248	N	N	N	0	0	0
249	N	N	0	N	0	0
250	N	N	0	0	N	0
251	N	N	F	0	0	0
252	N	N	E	0	0	0
253	N	N	N	N	0	0
254	N	N	N	0	N	0
257	N	N	0	N	N	0
258	N	N	F	N	0	0
259	N	N	E	N	0	0
260	N	N	F	0	N	0
261	N	N	E	0	N	0
262	N	N	N	N	N	0
265	N	N	F	N	N	0
266	N	N	E	N	N	0
301							Real: BibTeX/MIT
302							Real: BibTeX/UMBC
303							Real: Karlsruhe
304							Real: INRIA

Alignments

Below are provided the set of tests with expected alignments in the alignment format described at http://co4.inrialpes.fr/align/. Each ontology is to be aligned with the reference ontology (i.e., that of test #101.

The only interesting alignments are those involving classes and properties of the given ontologies. So the alignments should not align individuals, nor entities from the external ontologies.

There is some chance that the final test be significantly harder with most of the tests duplicated so that it does not suffice to compare labels of entities.

101) Concept test: Id

This test compares the ontology to itself.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

102) Concept test: ?

This test compares the ontology to a totally irrelevant one. We ussed the food ontology given in the OWL guide (verbatim).

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

103) Concept test: Language generalisation

This test compares the ontology with its generalisation in OWL Lite (i.e., unavailable constraints are replaced by the more general available). The generalization basically removes owl:unionOf and owl:oneOf and the Property types (owl:TransitiveProperty).

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

104) Concept test: Language restriction

This test compares the ontology with its restriction in OWL Lite (where unavailable constraints have been discarded).

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

201) Systematic: No names

Each label or identifier is replaced by a random one.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

202) Systematic: No names, no comment

Each label or identifier is replaced by a random one. Comments (rdfs:comment and dc:description) have been suppressed as well.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

203) Systematic: Misspelling

A random, but consistent, typo generator should be applied to labels and comments.

Not available in this test (if you know how to do it, contact me).

204) Systematic: Naming conventions

Different naming conventions (Uppercasing, underscore, dash, etc.) are used for labels. Comments have been suppressed.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

205) Systematic: Synonyms

Labels are replaced by synonyms. Comments have been suppressed.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

206) Systematic: Foreign names

The complete ontology is translated to another language than english (French in the current case, but other languages would be fine).

Ontology : [RDF/XML][RDF/XML in UTF-8] [HTML]
Alignment : [RDF/XML] [RDF/XML in UTF-8] [HTML]

NOTE: Some parsers do not like ISO-Latin in tags. You are authorized to use the Unicode (UTF-8) version instead (just replace onto.rdf by onto2.rdf).

207) Systematic:

Each label or identifieris translated to another language than english (French in the current case, but other languages would be fine).

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

208) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

209) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

210) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

221) Systematic: No hierarchy

All subclass assertions to named classes are suppressed.

Ontology : [RD/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: compile inheritance)

222) Systematic: Flattened hierarchy

A hierarchy still exists but has been strictly reduced.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

The alignment here contains relations which are not "=" but "<".

223) Systematic: Expanded hierarchy

Numerous intermediate classes are introduced within the hierarchy.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

224) Systematic: No instances

All individuals have been suppressed from the ontology.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

225) Systematic: No restrictions

All local restrictions on properties have been suppressed from the ontology.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: no property nor global restrictions on properties)

226) Systematic: No datatypes

In this test all datatypes are converted to xsd:string.

Not available in this test

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

227) Systematic: Unit differences

(Measurable) values are expressed in different datatypes.

Not available in this test

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

228) Systematic: No properties

Properties and relations between objects have been completely suppressed.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: leave the properties in instances)

229) Systematic: Class vs instances

Some classes have become instances.

Not available in this test.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

230) Systematic: Flattening entities

Some components of classes are expanded in the class structure (e.g., year, month, day attributes instead of date).

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

Here one limitation of the proposed format is that it does not cover alignments such as: journalName = name o journal.

231) Systematic: Multiplying entities

Some classes are spreaded over several classes.

Not available in this test.

232) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

233) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

236) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

237) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

238) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

239) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

240) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

241) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

246) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

247) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

248) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

249) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

250) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

251) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

252) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

253) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

254) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

257) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

258) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

259) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

260) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

261) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

262) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

265) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

266) Systematic:

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

301) Real ontology: BibTeX/MIT

For a computer scientist, BibTeX is the starting point for a useful bibliographic ontology. It is of wide use and relatively well thought out. This ontology can be found at and is documented in BibTex in OWL.

This is a test of comparing our test ontology with an actual ontology, simpler and closer to the initial BibTeX ontology. The alignment result contains some inclusion (<) alignment relations.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

302) Real ontology: BibTeX/UMBC

This ontology is very similar to the previous one, even closer to the genuine BibTeX, with different extensions and naming conventions. It can be found at http://ebiquity.umbc.edu.

The alignment result also contains some inclusion (<) alignment relations.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

303) Real ontology: Karlsruhe

This is a test of comparing our test ontology with an actual ontology which contains more items than the actual items used in the current ontology.

The Karlsruhe ontology (http://www.aifb.uni-karlsruhe.de/ontology) is used in the Ontoweb portal. It is a refinement from other ontologies such as (KA)². As such it does not only defines bibliographic items but many other items.

The alignment contains < as well as > relations.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

304) Real ontology: INRIA

This is a test of comparing our test ontology with an actual ontology which is not equivalent but quite close (it can be though of as a previous version).

This INRIA ontology (fr.inrialpes.exmo.rdf.bib.owl) has been designed by Antoine Zimmermann from the BibTeX in OWL ontology and our Bibliographic XML DTD. Its goal was to gather easily a number of RDF items. These items were BibTeX entries found on the web and transformed in RDF according to this ontology.

The actual hierarchy of this ontology contains classes which are subclasses of several other classes.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

Tools

A zip file contains all the material about the contest. That is:

This web page;
All ontology/alignment pairs in OWL/RDF/XML;
The prior known alignments in RDF/XML;
All ontology/alignment pairs in HTML;
XSLT stylesheet from ontologies and alignment to HTML;
The provided expected alignments in RDF/XML (for the systematic benchmark);
A directory with some reference ontologies (iCal, Foaf);

We would like to find a simple converter from OWL/XML/RDF to N3 (and maybe RDFS).

You can use the Alignment API for manipulating and generating your alignments (in particular for computing evaluation of your results).

A Ant build.xml file is also provided that can be used for generating the HTML pages. It will be enhanced for computing the characteristics of the results.

Results

The fully detailled results, as provided from the competitors files and according to the contests rules can be found here.

Acknowledgements

Many resources have been used for setting up this test (I must put links):

BibTeX
UMBC BibTeX
MIT BibTex
Java BibTeX parser
Karlsruhe ontology and bibtex to RDF translator that we adapted
INRIA's adaptation of the ontologies and the translator above by Antoine Zimmermann.
Lockheed evaluation format.
WonderWeb/Manchester's OWL species validator

Various people helped testing or suggested improvements and tests:

York Sure and Oscar Corcho for welcoming this at their EON workshop;
Heiner Stuckenschmidt helped providing several sorts of tests;
Liz Palmer proposed variation tests;
Natasha Noy and David Loup have been very helpful in noticing and correcting bugs.

Contacts

Contact address is Jerome . Euzenat (�) inrialpes . fr

http://oaei.inrialpes.fr/tests

$Id: index.html,v 1.4 2005/06/10 14:51:37 euzenat Exp $