Ontology Alignment Evaluation Initiative - OAEI-2011 CampaignOAEI

Since 2010, it is not necessary anymore to download the data set (it has always been better to get it on the web. The SEALS platform will provide the data sets. In the future, it is even planned that it will generate the data set on the fly.

Benchmark2 test library

The goal of the benchmark2 test library is to offer a set of tests which are wide in feature coverage, progressive and stable. It serves the purpose of evaluating the strength and weakness of matchers (by being progressive and wide coverage) and measuring the progress of matchers (by being stable and reusable over the years).

Apart from that, it is made of a set of 103 pairs of ontologies for which the participants have to return an alignment in the alignment format. These results are compared against reference alignments which are available from the beginning.

Data sets

This year, we will use new systematically generated benchmarks, based on other ontologies than the bibliographic one. Participants can test their tools using the previous benchmark as well as a new generated test set, named Benchmark2. Both datasets are publicity available here and under the SEALS platform. However, the final evaluation will be based on a second newly generated benchmark, not available during the test period (we have already checked that results remain consistent across benchmarks).

Reference ontology

The Ekaw conference ontology

The Ekaw ontology, one of the ontologies from the conference track, has been used as reference ontology for generating the Benchmark 2 dataset. It contains 74 classes and 33 object properties.

Tests at a glance

The tests are systematically generated to as to start from some reference ontology and discarding a number of information in order to evaluate how the algorithm behave when this information is lacking.

Test numbering (almost) fully preserves the numbering of the first EON contest.

The table below summarize what has been retracted from the reference ontology. There are here 6 categories of alteration:

Name
Name of entities that can be replaced by (R/N) random strings, (N)ame with different conventions.
Comments
Comments can be (N) suppressed or (F) translated in another language.
Specialization Hierarchy
can be (N) suppressed, (E)xpansed or (F)lattened.
Instances
can be (N) suppressed
Properties
can be (N) suppressed or (R) having the restrictions on classes discarded.
Classes
can be (E)xpanded, i.e., relaced by several classes or (F)latened.

#NameComHierInstPropClassComment
101000000Reference alignment
201R00000No names
202RN0000No names, no comments
22100N000No specialisation
22200F000Flatenned hierarchy
22300E000Expanded hierarchy
224000N00No instance
2250000R0No restrictions
2280000N0No properties
23200NN00
23300N0N0
236000NN0
23700FN00
23800EN00
23900F0N0
24000E0N0
24100NNN0
24600FNN0
24700ENN0
248NNN000
249NN0N00
250NN00N0
251NNF000
252NNE000
253NNNN00
254NNN0N0
257NN0NN0
258NNFN00
259NNEN00
260NNF0N0
261NNE0N0
262NNNNN0
265NNFNN0
266NNENN0

One can observe that a subset of the modifications from original benchmark has been applied for generating the benchmark2 dataset. Some tests have been discarded while other ones have been added:

Alignments

Below are provided the set of tests with expected alignments in the Alignment format described here. Each ontology is to be aligned with the reference ontology (i.e., that of test #101).

The only interesting alignments are those involving classes and properties of the given ontologies. So the alignments should not align individuals, nor entities from the external ontologies.

There is some chance that the final test be improved by adding entity expansion and reduction. It is also possible that there will be a lot of more instances in each ontology.


101) Concept test: Id

This test compares the ontology to itself.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


201[-2-4-6-8]) Systematic: No names

Each label or identifier is replaced by a random one.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


221) Systematic: No hierarchy

All subclass assertions to named classes are suppressed.

Ontology : [RD/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: compile inheritance)


222) Systematic: Flattened hierarchy

A hierarchy still exists but has been strictly reduced.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

The alignment here contains relations which are not "=" but "<".


223) Systematic: Expanded hierarchy

Numerous intermediate classes are introduced within the hierarchy.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


224) Systematic: No instances

All individuals have been suppressed from the ontology.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


225) Systematic: No restrictions

All local restrictions on properties have been suppressed from the ontology.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: no property nor global restrictions on properties)


226) Systematic: No datatypes

In this test all datatypes are converted to xsd:string.

Not available in this test


227) Systematic: Unit differences

(Measurable) values are expressed in different datatypes.

Not available in this test


228) Systematic: No properties

Properties and relations between objects have been completely suppressed.

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]

(variation: leave the properties in instances)


229) Systematic: Class vs instances

Some classes have become instances.

Not available in this test.


232) Systematic: no hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


233) Systematic: no hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


236) Systematic: no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


237) Systematic: flattened hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


238) Systematic: expanded hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


239) Systematic: flattened hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


240) Systematic: expanded hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


241) Systematic: no hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


246) Systematic: flattened hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


247) Systematic: expanded hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


248[-2-4-6-8]) Systematic: scrambled labels + no comments + no hierarchy

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


249[-2-4-6-8]) Systematic: scrambled labels + no comments + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


250[-2-4-6-8]) Systematic: scrambled labels + no comments + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


251[-2-4-6-8]) Systematic: scrambled labels + no comments + flattened hierarchy

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


252[-2-4-6-8]) Systematic: scrambled labels + no comments + expanded hierarchy

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


253[-2-4-6-8]) Systematic: scrambled labels + no comments + no hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


254[-2-4-6-8]) Systematic: scrambled labels + no comments + no hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


257[-2-4-6-8]) Systematic: scrambled labels + no comments + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


258[-2-4-6-8]) Systematic: scrambled labels + no comments + flattened hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


259[-2-4-6-8]) Systematic: scrambled labels + no comments + expanded hierarchy + no instance

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


260[-2-4-6-8]) Systematic: scrambled labels + no comments + flattened hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


261[-2-4-6-8]) Systematic: scrambled labels + no comments + expanded hierarchy + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


262[-2-4-6-8]) Systematic: scrambled labels + no comments + no hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


265) Systematic: scrambled labels + no comments + flattened hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


266) Systematic: scrambled labels + no comments + expanded hierarchy + no instance + no property

Ontology : [RDF/XML] [HTML]
Alignment : [RDF/XML] [HTML]


Versions

The release notes of the test versions can be found here.

Modalities

The test set is made of a set of directories (one per test) containing an ontology (onto.owl) in OWL. Participants must match each of these ontologies with the ontology found in 101/onto.owl and output the resulting alignment, in the alignment format, in a file participant.rdf in the same directory (participant is replaced by the name of the system: no more than 8 alphanumeric characters, lowercase).

The test set also contains the reference alignments against which the results will be evaluated. It is, of course, forbidden to use any for there results for performing the matching task.

The resulting alignments in the same directory structure can be zipped (please, zipped) and sent to the organisers.

Schedule

The schedule is that of http://oaei.ontologymatching.org/2011/.

Tools

A zip file contains all the data about the benchmark2 dataset. That is:

We would like to find a simple converter from OWL/XML/RDF to N3 (and maybe RDFS).

You can use the Alignment API for manipulating and generating your alignments (in particular for computing evaluation of your results).

Contacts

Contact address is Cassia : Trojahn # inrialpes : fr

$Id: index.html,v 1.29 2011/05/31 16:11:53 euzenat Exp $