The focus of this task is to evaluate performance of existing alignment tools in real world taxonomy integration scenario. Our aim is to show whether ontology alignment tools can effectively be applied to integration of "shallow ontologies".
The evaluation dataset was extracted from Google, Yahoo and Looksmart web directories. The specific characteristics of the dataset are:
This implies that the task will be challenging from a technological point of view, but there is guidance for tuning matching approach that needs to be taken into account.
The matching tasks are represented by pairs of OWL ontologies, where classification relation is modeled as OWL subClassOf construct. Therefore all OWL ontologies are taxonomies (i.e., they contain only classes (without Object and Data properties) connected with subclass relation. The dataset can be downloaded from here. The matching tasks are numbered from 1 to 2265. Thus, for example, the first matching task is to find a mapping between 1/source.owl and 1/target.owl and to output it to a file called 1/yourname.rdf.
The task is to find an alignment between classes in the ontologies. In order to find the alignment any information in the two models can be used. In addition, it is allowed to use background knowledge, that has not specifically been created for the alignment tasks (ie no hand-made mappings between parts of the ontologies). Admissible background knowledge are "Oracles" such as WordNet, Cyc, UMLS, etc. Further, results must not be tuned manually by for instance removing obviously wrong mappings.
Results of the alignment should be represented using the common format for alignments and send to yatskevi@dit.unitn.it for evaluation by September 1st.
The final results of the evaluation are here.