|
New test set can be found here: here
previous set
and Natasha Noy's corrections are
still available You will find the list of main modifications Here Thanks |
The increasing number of methods available for schema matching/ontology integration suggests the need to establish a consensus for evaluation of these methods. There is now a coordinated international initiative to forge this consensus through two events in 2004:
The goal of this contest is to provide a set of tests on which to assess the merits of ontology alignment algorithms and tools and to compare them.
This is the first such contest, whose specific goal is to evaluate the merits of this test. We are aware that other tests are possible such as testing alignment between huge unchanged ontologies, or testing a battery of ontologies independently designed on the same topic. We feel that each kind of test has its own merit and should be performed.
The contest organisers provide the participants with a complete test base, including couples of ontologies to align as well as expected results. The test is based on one particular ontology dedicated to a very narrow domain and a number of alternative ontologies of the same domain for wich alignments are provided. The ontologies are provided in OWL. The expected alignments are provided in a standard format expressed in RDF/XML and described in http://co4.inrialpes.fr/align/. From the ontology to compare, the competitors are able to produce their alignments in the same format. They can also compute a number of measures on their results. They can use the ontology provided in http://www.atl.external.lmco.com/projects/ontology/ for giving their results.
The participants will provide the organisers with a paper along the following plan (at most 10 pages):
The above mentionned paper must be sent by September 15th to Jerome.Euzenat (à) inrialpes.fr with copy to ocorcho (à) isoco.com and sure (à) aifb.uni-karlsruhe.de.
The ontologies and alignments of the contest are provided in advance during the period between June 21th and July 4th. This is the occasion for potential participants to send observations, bug corrections, remarks and other test cases.
The goal of this primary period is to be sure that the delivered tests make sense to the participants. So do not hesitate to provide feed back.
The tests will certainly change after this period, but only for ensuring a better participation to the tests.
Do not hesitate, however, to contact us if you think that something is incorrect or should be changed in these tests.
First publication of test cases: June 21th.
Comments due: July 4th
Final publication of test cases: July 9th.
Papers due: Sep. 15th.
Camera ready copies: Oct. 1st.
Workshop : Nov 8th.
The domain of this first test is Bibliographic references. It is, of course, based on a subjective view of what must be a bibliographic ontology. There can be many different classifications of publications (based on area, quality, etc.). We choose the one common among scholars based on mean of publications; as many ontologies below (tests #301-304), it is reminiscent to BibTeX.
This reference ontology contains 33 named classes, 39 object properties, 20 data properties, 56 named individuals and 20 anonymous individuals.
However, it is also based on various experiments on designing and using ontologies for that domain. These various ontologies are provided and used as an alignment test (tests #301-304).
The complete ontology is that of test #101.
The reference ontology is put in the context of the semantic web by using other external resources for expressing non bibliographic information. It takes advantage of FOAF (http://xmlns.com/foaf/0.1/) and iCalendar (http://www.w3.org/2002/12/cal/) for expressing the People, Organization and Event concepts. Here are the external reference used:
This reference ontology is a bit limited in the sense that it does not contain circular definitions nor attachement to several classes.
Similarly the kind of proposed alignments is also limited: they only match named classes and properties, they mostly use the "=" relation with confidence of 1.
The ontologies are described in OWL-DL and serialized in the RDF/XML format. I'd be happy to translate them into N3 if someone have some translation program.
Below are provided the set of tests with expected alignments in the alignment format described at http://co4.inrialpes.fr/align/. Each ontology is to be aligned with the reference ontology (i.e., that of test #101.
The only interesting alignments are those involving classes and properties of the given ontologies. So the alignments should not align individuals, nor entities from the external ontologies.
There is some chance that the final test be significantly harder with most of the tests duplicated so that it does not suffice to compare labels of entities.
This test compares the ontology to itself.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This test compares the ontology to a totally irrelevant one. We ussed the food ontology given in the OWL guide (verbatim).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This test compares the ontology with its generalisation in OWL Lite (i.e., unavailable constraints are replaced by the more general available). The generalization basically removes owl:unionOf and owl:oneOf and the Property types (owl:TransitiveProperty).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This test compares the ontology with its restriction in OWL Lite (where unavailable constraints have been discarded).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Each label or identifier is replaced by a random one.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Each label or identifier is replaced by a random one. Comments (rdfs:comment and dc:description) have been suppressed as well.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
A random, but consistent, typo generator should be applied to labels and comments.
Not available in this test (if you know how to do it, contact me).
Different naming conventions (Uppercasing, underscore, dash, etc.) are used for labels. Comments have been suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Labels are replaced by synonyms. Comments have been suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
The complete ontology is translated to another language than english (French in the current case, but other languages would be fine).
Ontology : [RDF/XML][RDF/XML in UTF-8]
[HTML]
Alignment : [RDF/XML]
[RDF/XML in UTF-8]
[HTML]
NOTE: Some parsers do not like ISO-Latin in tags. You are authorized to use the Unicode (UTF-8) version instead (just replace onto.rdf by onto2.rdf).
All subclass assertions to named classes are suppressed.
Ontology : [RD/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: compile inheritance)
A hierarchy still exists but has been strictly reduced.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
The alignment here contains relations which are not "=" but "<".
Numerous intermediate classes are introduced within the hierarchy.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
All individuals have been suppressed from the ontology.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
All local restrictions on properties have been suppressed from the ontology.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: no property nor global restrictions on properties)
In this test all datatypes are converted to xsd:string.
Not available in this test
(Measurable) values are expressed in different datatypes.
Not available in this test
Properties and relations between objects have been completely suppressed.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
(variation: leave the properties in instances)
Some classes have become instances.
Not available in this test.
Some components of classes are expanded in the class structure (e.g., year, month, day attributes instead of date).
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
Here one limitation of the proposed format is that it does not cover alignments such as: journalName = name o journal.
Some classes are spreaded over several classes.
Not available in this test.
For a computer scientist, BibTeX is the starting point for a useful bibliographic ontology. It is of wide use and relatively well thought out. This ontology can be found at and is documented in BibTex in OWL.
This is a test of comparing our test ontology with an actual ontology, simpler and closer to the initial BibTeX ontology. The alignment result contains some inclusion (<) alignment relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This ontology is very similar to the previous one, even closer to the genuine BibTeX, with different extensions and naming conventions. It can be found at http://ebiquity.umbc.edu.
The alignment result also contains some inclusion (<) alignment relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This is a test of comparing our test ontology with an actual ontology which contains more items than the actual items used in the current ontology.
The Karlsruhe ontology (http://www.aifb.uni-karlsruhe.de/ontology) is used in the Ontoweb portal. It is a refinement from other ontologies such as (KA)2. As such it does not only defines bibliographic items but many other items.
The alignment contains < as well as > relations.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
This is a test of comparing our test ontology with an actual ontology which is not equivalent but quite close (it can be though of as a previous version).
This INRIA ontology (fr.inrialpes.exmo.rdf.bib.owl) has been designed by Antoine Zimmermann from the BibTeX in OWL ontology and our Bibliographic XML DTD. Its goal was to gather easily a number of RDF items. These items were BibTeX entries found on the web and transformed in RDF according to this ontology.
The actual hierarchy of this ontology contains classes which are subclasses of several other classes.
Ontology : [RDF/XML]
[HTML]
Alignment : [RDF/XML]
[HTML]
A zip file contains all the material about the contest. That is:
We would like to find a simple converter from OWL/XML/RDF to N3 (and maybe RDFS).
You can use the Alignment API for manipulating and generating your alignments (in particular for computing evaluation of your results).
A Ant build.xml file is also provided that can be used for generating the HTML pages. It will be enhanced for computing the characteristics of the results.
Many resources have been used for setting up this test (I must put links):
Various people helped testing or suggested improvements and tests:
Contact address is Jerome . Euzenat (À) inrialpes . fr