Ontology Alignment Evaluation Initiative - OAEI-2021 CampaignOAEI OAEI

Complex track - Hydrography subtrack - Evaluation

There are three alignment systems enrolled in the complex track, which are AMD, AMLC, and AROA. Besides these three systems, we also evaluated other alignment systems. The systems were run on the Hydrography dataset, and output alignments were evaluated as described below.

Evaluation Methodology

In this subtrack, the alignments are automatically evaluated over the Hydrography dataset. In order to assess the quality of a mapping, there are two dimensions that we can look into. We can first evaluate if a mapping contains the correct entities that should be involved based on the reference alignment. Another dimension is the relationship between these entities, e.g., equivalence and subsumption. Based on this, we break the evaluation procedure down into three subtasks, which are entity identification, relationship identification, and full complex identification.

  1. Entity Identification

    For each entity in the source ontology, the alignment system is asked to list all of the entities in the target ontologies that are related to it in some way.

    For example:

    owl:equivalentClasses(ont1:A1 owl:intersectionOf(ont2:B1 owl:someValuesFrom(ont2:B2 ont2:B3))

    The goal in this task is to find the most relevant entities in the ont2 to the class ont1:A1. In this case, the best output would be ont2:B1, ont2:B2, and ont2:B3.

    The result is evaluated based on precision, recall, and f-measure.

  2. Relationship Identification

    For each alignmnet, the system should then endeavor to find the concrete relationships, such as equivalence, subsumption, intersection, value restriction, and so on, that hold between the entities. In terms of the example above, an alignment system needs to eventually determine that the relationship between the two sides is equivalence. Table 1 shows the different similarity that we used in the evaluation for different situations. We do not penalize the incorrect relationship by giving a ZERO value because that would completely neglect the entity identification outputs without considering whether it is a reaonable results or a completely incorrect one.

    Table 1. Similarity for Relationship Identification

    Found Relation Correct Relation Similarity Comment
    = = 1.0 correct relation
    1.0 correct relation
    1.0 correct relation
    = 0.8 return less information, but correct
    = 0.8 return less information, but correct
    = 0.6 return more information, but incorrect
    = 0.6 return more information, but incorrect
    0.3 incorrect relation
    0.3 incorrect relation
  3. Full Complex Alignment Identification

    This task is a combination of the former two steps. We multiply the results from the entity identification by the similarity of the relations as the relaxed precision, recall, and f-measure. To be accurate, it could also have been better aggregated with other aggregation functions rather than multiplication. [1]

    relaxed_precision = entity_precision * similarity of relationshio

    relaxed_recall = entity_recall * similarity of relationshio

    relaxed_f-measure = 2 * relaxed_precision * relaxed_recall/ (relaxed_precision + relaxed_recall)

Results

The output alignments as well as the detailed results of the systems over the Hydrography dataset are downloadable here.

Table 2. The Performance of All Alignment Systems on the Hydrography Benchmark

Systems (1:1) (1:n) (m:n) Cree-SWO Hydro3-SWO HydrOntology_native-SWO HydrOntology_translated-SWO Total
relaxed precision relaxed recall relaxed f-measure relaxed precision relaxed recall relaxed f-measure relaxed precision relaxed recall relaxed f-measure relaxed precision relaxed recall relaxed f-measure relaxed precision relaxed recall relaxed f-measure
reference alignment 113 69 15 - - - - - - - - - - - - - - -
AML 10 0 0 - - - 0.74 0.16 0.26 - - - 0.32 0.03 0.05 0.49 0.04 0.08
LogMap 10 0 0 - - - 0.87 0.16 0.27 - - - 0.53 0.04 0.08 0.67 0.05 0.10
LogMapLite 11 0 0 - - - 0.92 0.14 0.24 - - - 0.53 0.04 0.08 0.66 0.06 0.10

Discussion

There are three systems that can produce the alignment on Hydrography dataset. Among these found alignments, all correspondences from LogMap and LogMapLite are 1-to-1 simple equivalence. Only AMLC can produce complex alignments. AMLC correctly found two 1:n complex correspondences which are stating a class in source ontology is equivalent to union of two classes in the target ontology. For example, class DamWeir in the Hydro3 ontology equals to the union of class Dam and Weir in SWO ontology. The relaxed precision of most of the systems are still relatively high, which means that the traditional ontology alignment systems can handle the simple relations in real-world ontologies as well, since the Hydrography benchmark is created based on a real-world case. But, it is not surprising that the low recall reflects that the current ontology alignment systems are not capable of identifying more complex relations, which we hope that it will be improved in the next future years.

References

[1] Cheatham, Michelle, Varanka, Dalia, Arauz, Fatima and Zhou, Lu. "Alignment of Surface Water Ontologies - A comparison of manual and automated approaches", Journal of Geographical Systems, 2019

[2] Marc Ehrig, and Jérôme Euzenat. "Relaxed precision and recall for ontology matching." K-CAP 2005 Workshop on Integrating Ontologies, Banff, Canada, 2005.