The complex track aims at evaluating the systems which can generate both simple and complex correspondences.
This track contains 7 datasets from 5 different domains: Conference, Populated Conference, Hydrography, GeoLink, Populated GeoLink, Populated Enslaved and Taxon.
The detailed description of each dataset can be found at the OAEI Complex track page
The complex correspondences generated by the systems were manually compared to the ones of the provided consensus alignment.
For this evaluation, only equivalence correspondences were considered and the confidence of the correspondence were not be taken into account.
The detailed results for this track are accessible in Conference results
In this subtrack, the alignments are automatically evaluated over a populated version of the Conference dataset.
The dataset as well as the evaluation systems are available at https://framagit.org/IRIT_UT2J/conference-dataset-population.
Two metrics are computed: a Coverage score and a Precision score.
The systems were run:
Then the best set of alignments output by each systems gives the final score for this track. For example, AML performed better on the Original Conference dataset than on the Populated version so the first sets of alignments are kept.
This sub-track has not been evaluated in 2021. It will come back in 2022.
In this subtack, in order to explain the performance of alignment systems, we break the evaluation down into three subtasks: Entity Identification, Relationship Identification, and Full Complex alignment Identification. The alignments generated for the final results have been evaluated using relaxed precision and recall.
The detailed results for this track are accessible in Hydrography results
The evaluation of GeoLink benchmark applies the same methods of evaluating Hydrography benchmark. The evaluation of the systems are performed by computing relaxed precision and recall for final results.
The detailed results for this track are accessible in GeoLink results
The evaluation of Populated GeoLink benchmark applies the same methods of evaluating Hydrography benchmark. The evaluation of the systems are performed by computing relaxed precision and recall for final results.
The detailed results for this track are accessible in Populated GeoLink results
The evaluation of Populated Enslaved benchmark applies the same methods of evaluating Hydrography benchmark. The evaluation of the systems are performed by computing relaxed precision and recall for final results.
The detailed results for this track are accessible in Populated Enslaved results
Even though the ontologies of the Taxon dataset have a common scope (plant taxonomy), they are unevenly populated. For this reason, the automatic evaluation system can not be applied to this dataset.
The detailed results for this track are accessible in Taxon results
[1] Ondřej Zamazal, Vojtěch Svátek. The Ten-Year OntoFarm and its Fertilization within the Onto-Sphere. Web Semantics: Science, Services and Agents on the World Wide Web, 43, 46-53. 2017.
[2] Élodie Thiéblin, Ollivier Haemmerlé, Nathalie Hernandez, Cassia Trojahn. Task-Oriented Complex Ontology Alignment: Two Alignment Evaluation Sets. In : European Semantic Web Conference. Springer, Cham, 655-670, 2020.
[3] Élodie Thiéblin, Fabien Amarger, Nathalie Hernandez, Catherine Roussey, Cassia Trojahn. Cross-querying LOD datasets using complex alignments: an application to agronomic taxa. In: Research Conference on Metadata and Semantics Research. Springer, Cham, 25-37, 2017.
[4] Lu Zhou, Michelle Cheatham, Adila Krisnadhi, Pascal Hitzler. A Complex Alignment Benchamark: GeoLink Dataset. In: International Semantic Web Conference. Springer, 2020.
[5] Marc Ehrig, and Jérôme Euzenat. "Relaxed precision and recall for ontology matching." K-CAP 2005 Workshop on Integrating Ontologies, Banff, Canada, 2005.
[6] Lu Zhou, Michelle Cheatham, Adila Krisnadhi, Pascal Hitzler. GeoLink DataSet: A Complex Alignment Benchmark from Real-world Ontology. In: Data Intellegence. Volume 2, Issue 3, Pages 353-378, MIT Press, 2020.
[7] Lu Zhou, Cogan Shimizu, Pascal Hitzler, Alicia M. Sheill, Seila Gonzalez Estrecha, Catherine Foley, Duncan Tarr, Dean Rehberger. The Enslaved Dataset: A Real-world Complex Ontology Alignment Benchmark using Wikibase. In: Conference on Information and Knowledge Management, ACM, 2020.