Ontology Alignment Evaluation Initiative - OAEI 2020 Campaign

MultiFarm Results for OAEI 2020

In this page, we report the results of the OAEI 2020 campaign for the MultiFarm track. The details on this data set can be found at the MultiFarm web page.

If you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.) do not hesitate to contact us (for the mail see below in the last paragraph on this page).

Experimental setting

We have conducted an evaluation based on the blind data set. This data set includes the matching tasks involving the edas and ekaw ontologies (resulting in 55 x 24 tasks). Participants were able to test their systems on the open subset of tasks, available via the SEALS repository. The open subset counts on 45 x 25 tasks and it does not include Italian translations.

We distinguish two types of matching tasks :

1. those tasks where two different ontologies (edas-ekaw, for instance) have been translated into two different languages;
2. those tasks where the same ontology (edas-edas) has been translated into two different languages.

As we could observe in previous evaluations, for the tasks of type (ii), good results are not directly related to the use of specific techniques for dealing with cross-lingual ontologies, but on the ability to exploit the fact that both ontologies have an identical structure.

Participants

This year, 6 systems have registered to participate in the MultiFarm track: AML, Lily, LogMap, LogMapLt, VeeAlign and Wiktionary. This number slightly increases with respect to the last campaign (5 in 2019, 6 in 2018, 8 in 2017, 7 in 2016, 5 in 2015, 3 in 2014, 7 in 2013, and 7 in 2012). The reader can refer to the OAEI papers for a detailed description of the strategies adopted by each system.

Evaluation results

Execution setting and runtime

The systems have been executed on a Ubuntu Linux machine configured with 8GB of RAM running under a Intel Core CPU 2.00GHz x4 processors. All measurements are based on a single run. As for each campaign, we observed large differences in the time required for a system to complete the 55 x 24 matching tasks: AML (170 minutes), Lily (453 minutes), LogMap (43 minutes) and Wiktionary (1290 minutes). These number are not comparable to those from last year given the fact that the SEALS repositories have moved to another server (with a different configuration). Furthermore, the concurrent access to the SEALS repositories during the evaluation period may have an impact in the time required for completing the tasks. These measurements are only indicative of the time the systems require for finishing the task in a common environment. Note a well that VeeAlign run in a different environment so runtime is not reported.

Overall results

The table below presents the aggregated results for the matching tasks. They have been computed using the Alignment API 4.6 and can slightly differ from those computed with the SEALS client. We haven't applied any threshold on the results. They are measured in terms of classical precision and recall. We do not report the results of non-specific systems here, as we could observe in the last campaigns that they can have intermediate results in tests of type ii) and poor performance in tests i). Lily has generated empty alignments so there are no results to be reported.

AML outperforms all other systems in terms of F-measure for task i) (same behaviour in the last campaigns). In terms of precision, Wiktionary is the system that generates the most precise alignments, followed by LogMap and AML. With respect to the task ii) LogMap has the overall best performance.

AML, LogMap and Wiktionary have participated last year. Unfortunately, we lose some one on the way (EVOCROS) but we compensate it with a newcomer (VeeAlign). Comparing the results from last year, in terms F-measure (cases of type i), AML maintains its overall performance (.45 in 2019, .46 in 2018, .46 in 2017, .45 in 2016 and .47 in 2015). The same could be observed for LogMap (.37 in 2019, .37 in 2018, .36 in 2017, and .37 in 2016). The performance in terms of f-measure of Wiktionary also remains stable (.31 in 2019).

Language specific results (type i)

Table below presents the results per pair of language, involving matching different ontologies (test cases of type i).

NaN: division per zero, likely due to empty alignment.

Conclusions

Unforntunetly, this track does not attract to many participants. In terms of performance, the F-measure for blind tests remains relatively stable across campaigns. As observed in several campaigns, still, all systems privilege precision in detriment to recall and the results are below the ones obtained for the Conference original dataset.

References

[1] Christian Meilicke, Raul Garcia-Castro, Fred Freitas, Willem Robert van Hage, Elena Montiel-Ponsoda, Ryan Ribeiro de Azevedo, Heiner Stuckenschmidt, Ondrej Svab-Zamazal, Vojtech Svatek, Andrei Tamilin, Cassia Trojahn, Shenghui Wang. MultiFarm: A Benchmark for Multilingual Ontology Matching. Accepted for publication at the Journal of Web Semantics.

An authors version of the paper can be found at the MultiFarm homepage, where the data set is described in details.

Contact

This track is organized by Beyza Yaman and Cassia Trojahn dos Santos. If you have any problems working with the ontologies, any questions or suggestions, feel free to write an email to beyza [_] yaman [at] hotmail [.] com and cassia [.] trojahn [at] irit [.] fr.