This year we plan to evaluate results of participants with following evaluation methods:
Subset of all alignments will be evaluated against reference alignment. Therefore we will provide participants with traditional evaluation measures such as precision and recall. This year we will also provide a variant of semantic precision an recall obtained by the tool from the University of Mannheim. This enables us to measure precision recall including subsumptions in an appropriate way.
Evaluation will be ready at the time of Ontology Matching workshop 2009.
The number of all distinct correspondences is always quite high number, therefore we will take advantage of sampling. For this kind of evaluation we will follow the method of "Stratified Random Sampling" described in [4]. We will divide correspondeces of each participant to three subpopulations (strata) according to confidence measures. For each stratum we randomly chose certain number of correspondences for manual labelling.
Each individual correspondence was assigned a label (correct, incorrect, unclear, interesting incorrect, interesting correct).
As a result, this method provides an approximation of precision for each stratum (precision as proportion of correct correspondences given Bernoulli distribution). Moreover we will compute approximated precision in the entire population from the approximated precisions of the strata. Additionally, based on the assumption that this adheres to binomial distribution we computed margin of errors (confidence of 95%) for the approximated precision for each system. These computations are based on equations from [4], section "Stratified Random Sampling".
Evaluation will be ready at the time of Ontology Matching workshop 2009.
Data Mining technique enables us to discover non-trivial findings about systems of participants. These findings will be answers to so-called analytic questions, such as:
We will try to answer abovementioned and similar analytic question. Those analytic questions will also be dealing with so-called mapping patterns [2] and newly also with correspondence patterns [1].
For the purpose of this kind of evaluation, we will use the LISp-Miner tool. Particularly, we will use the 4ft-Miner procedure that mines association rules. This kind of evaluation was first tried two years ago [2].
Evaluation will be ready in November, 2009.
This method will be done by Christian Meilicke and Heiner Stuckenschmidt from Computer Science Institure at University Mannheim, Germany. In this kind of evaluation, ontologies will be merged on the base of correspondences submitted by participants. Subsequently, the incoherence of the mappings will be measured based on the incoherence of the merged ontology. This method is related to [3].
Contact addresses are Ondřej Šváb-Zamazal (ondrej.zamazal at vse dot cz) and Vojtěch Svátek (svatek at vse dot cz).
[1] Scharffe F., Euzenat J., Ding Y., Fensel,D. Correspondence patterns for ontology mediation. OM-2007 at ISWC-2007.
[2] Šváb O., Svátek V., Stuckenschmidt H.: A Study in Empirical and Casuistic Analysis of Ontology Mapping Results. ESWC-2007. Abstract Draft paper (final version available via SpringerLink)
[3] Meilicke C., Stuckenschmidt H. Incoherence as a basis for measuring the quality of ontology mappings. OM-2008 at ISWC 2008.
[4] van Hage W.R., Isaac A., Aleksovski Z. Sample evaluation of ontology matching systems. EON-2007, Busan, Korea, 2007.
Initial location of this page: http://nb.vse.cz/~svabo/oaei2009/eval1.html