Ontology Alignment Evaluation Initiative - OAEI-2025 Campaign

Complex track - Evaluation

General description

The complex track aims at evaluating the systems which can generate complex correspondences. The detailed description of each dataset can be found at the OAEI Complex track page

We had only one participant: Matcha system. Matcha delivered simple equivalences, complex correspondences, and subsumptions. Within this track only complex correspondences have been evaluated. With regard to complex correspondences no TPs were identified. All complex correspondences contained intersections (of classes and also of classes and properties). While the intersection of classes is an important construct, EDOAL does not allow for directly intersecting a class and a property. Instead, class restrictions should be applied, such as AttributeDomainRestriction.

Alignments from Matcha:

• matcha_conference_cmt-conference.rdf
• matcha_conference_conference-ekaw.rdf
• matcha_conference_cmt-ekaw.rdf

Other datasets

The following results relate to seven datasets: Populated Conference, Hydrography, GeoLink, Populated GeoLink, Populated Enslaved, Taxon, and Biomedical. All datasets and references can be found at https://github.com/liseda-lab/complex-OM-benchmark.

This year we had the participation of two systems: CMatch and Matcha. CMatch produces simple and complex 1:n mappings, and Matcha produces simple, subsumption and complex 1:n mappings. Matcha did not format mappings according to the expected EDOAL format. The results from three systems that competed previously are presented as baselines: AMLC, AROA, and CANARD (using alignments from OAEI 2020).

All alignments and evaluation results can be consulted here.

Graph Edit Distance (GED) evaluation

This evaluation converts both mapping and reference into graphs and calculated the edit distance between them. This metric evaluates simple 1:1 mappings and complex 1:n mappings, regardless of relation. Best results in each row by F-measure are in bold.

Matcha achieved the best F-measure overall, possibly explained by the fact that it produced more simple mappings than the other systems which improved the overall performance. Right behind is CANARD, which produces very consistent results and is less computationally heavy than Matcha, since Matcha uses language models in its strategy. CMatch achieves solid precision results but performs worse in terms of recall, resulting in lower values of F-measure. It is interesting to note that AROA outperforms all other systems in the single task it participated. Only Matcha participated in the complex multi-ontology matching task of the biomedical domain, so it will be interesting to see how other systems tackle this new challenge in the future.

Tree Edit Distance (TED)

This metric compares the matcher proposed alignment with the reference alignment using Tree Edit Distance (TED) and finds the most similar pairs of correspondences using an assignment algorithm. The metric also follows desired properties for complex matching evaluation based on reference alignments.

Class evaluation

In this evaluation, the classes used in each mapping were evaluated against the classes used by the reference. This metric evaluates all mappings: simple 1:1, complex 1:n, and complex n:m. False positives are not presented in this summarised table.

In general, the results mostly match the results obtained by the other evaluations. All systems obtain a fair amount of fully correct class sets in the tasks they participate. It is interesting to note that CANARD is the only system that consistently achieves more class sets in the "contains" category, while the categories of "contained" and "overlap" achieve lower counts across all systems. Counting all "correct" hits, AMLC achieves 0 across 5 tasks, AROA achieves 18 in 1 task, CANARD achieves 90 across 9 tasks, CMatch achieves 17 across 7 tasks, and Matcha achieves 100 across 10 tasks (discarding the biomedical tasks where only Matcha competed).

Looking at the overall results, Matcha and CANARD obtain mostly comparable results.Matcha achieves less "correct" and "contains" class sets in the conference tasks but also less "incorrect", with the inverse being true for the GeoLink and Populated Enslaved tasks where it finds more "incorrect" and mostly equal counts of the other categories.

Organizers

• Cassia Trojahn (IRIT, Toulouse, France), cassia [.] trojahn [at] irit [.] fr
• Ondřej Zamazal (Prague University of Economics and Business), ondrej [.] zamazal [at] vse [.] cz
• Marta Silva (University of Lisbon), mcdsilva [at] fc [.] ul [.] pt
• Guilheme Sousa (IRIT, Toulouse, France), Guilherme [.] Santos [-] Sousa [at] irit.fr

References

[1] Ondřej Zamazal, Vojtěch Svátek. The Ten-Year OntoFarm and its Fertilization within the Onto-Sphere. Web Semantics: Science, Services and Agents on the World Wide Web, 43, 46-53. 2017.

[2] Élodie Thiéblin, Ollivier Haemmerlé, Nathalie Hernandez, Cassia Trojahn. Task-Oriented Complex Ontology Alignment: Two Alignment Evaluation Sets. In : European Semantic Web Conference. Springer, Cham, 655-670, 2018.

[3] Élodie Thiéblin, Fabien Amarger, Nathalie Hernandez, Catherine Roussey, Cassia Trojahn. Cross-querying LOD datasets using complex alignments: an application to agronomic taxa. In: Research Conference on Metadata and Semantics Research. Springer, Cham, 25-37, 2017.

[4] Lu Zhou, Michelle Cheatham, Adila Krisnadhi, Pascal Hitzler. A Complex Alignment Benchamark: GeoLink Dataset. In: International Semantic Web Conference. Springer, Proceedings, Part II, pp. 273-288, 2018.

[5] Marc Ehrig, and Jérôme Euzenat. "Relaxed precision and recall for ontology matching." K-CAP 2005 Workshop on Integrating Ontologies, Banff, Canada, 2005.

[6] Élodie Thiéblin. Do competency questions for alignment help fostering complex correspondences?. In EKAW Doctoral Consortium, 2018.

[7] Élodie Thiéblin, Fabien Amarger, Ollivier Haemmerlé, Nathalie Hernandez, Cassia Trojahn. Rewriting SELECT SPARQL queries from 1:n complex correspondences. In: Ontology Matching, pp. 49-60, 2016.

[8] Silva, M. C., Faria, D., & Pesquita, C. (2024). Complex Multi-Ontology Alignment Through Geometric Operations on Language Embeddings. In ECAI 2024 (pp. 1333–1340). IOS Press. https://doi.org/10.3233/FAIA240632

[9] Aric A. Hagberg, Daniel A. Schult and Pieter J. Swart, “Exploring network structure, dynamics, and function using NetworkX”, in Proceedings of the 7th Python in Science Conference (SciPy2008), Gäel Varoquaux, Travis Vaught, and Jarrod Millman (Eds), (Pasadena, CA USA), pp. 11–15, Aug 2008

[10] Silva, M. C., Faria, D., & Pesquita, C. (2025). CMOMgen: Complex Multi-Ontology Alignment via Pattern-Guided In-Context Learning (No. arXiv:2510.21656). arXiv. https://doi.org/10.48550/arXiv.2510.21656

[11] Gargano MA et al., The Human Phenotype Ontology in 2024: phenotypes around the world. Nucleic Acids Res. 2024 Jan 5;52(D1):D1333-D1346. doi: 10.1093/nar/gkad1005. PMID: 37953324; PMCID: PMC10767975., https://doi.org/10.1093/nar/gkad1005