Ontology Alignment Evaluation Initiative - OAEI 2021 Campaign

MultiFarm Results for OAEI 2021

In this page, we report the results of the OAEI 2021 campaign for the MultiFarm track. The details on this data set can be found at the MultiFarm web page.

If you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.) do not hesitate to contact us (for the mail see below in the last paragraph on this page).

Experimental setting

We have conducted an evaluation based on the blind data set. This data set includes the matching tasks involving the edas and ekaw ontologies (resulting in 55 x 24 tasks). Participants were able to test their systems on the open subset of tasks, available via the SEALS repository. The open subset counts on 45 x 25 tasks and it does not include Italian translations.

We distinguish two types of matching tasks :

1. those tasks where two different ontologies (edas-ekaw, for instance) have been translated into two different languages;
2. those tasks where the same ontology (edas-edas) has been translated into two different languages.

As we could observe in previous evaluations, for the tasks of type (ii) which is similar ontologies, good results are not directly related to the use of specific techniques for dealing with cross-lingual ontologies, but on the ability to exploit the fact that both ontologies have an identical structure. This year, we report the results on different ontologies (i).

Participants

This year, 6 systems have registered to participate in the MultiFarm track: ALOD2Vec, AML, ATMatcher, LogMap, LogMapLt and Wiktionary. This numbers remains stable with respect to the last campaing (6 in 2020, 5 in 2019, 6 in 2018, 8 in 2017, 7 in 2016, 5 in 2015, 3 in 2014, 7 in 2013, and 7 in 2012). This year, we lost the participation of Lily and VeeAlign, but we received new participation from ALOD2Vec and ATMatcher. The reader can refer to the OAEI papers for a detailed description of the strategies adopted by each system.

Evaluation results

Execution setting and runtime

The systems have been executed on a Ubuntu Linux machine configured with 32GB of RAM running under a Intel Core CPU 2.00GHz x8 processors. All measurements are based on a single run. As for each campaign, we observed large differences in the time required for a system to complete the 55 x 24 matching tasks: ALOD2vec (10 minutes), AML (time not produced), ATMatcher (113), LogMap (9 minutes), LogMapLt (212 minutes) and Wiktionary (157 minutes). These number are not comparable to those from last year given the fact that this year MELT framework was used instead of SEALS framework. These measurements are only indicative of the time the systems require for finishing the task in a common environment.

Overall results

The table below presents the aggregated results for the matching tasks. MultiFarm aggregated results per matcher for different ontologies. Time is measured in minutes (for completing the 55x24 matching tasks), ** runtime is not reported.

AML, LogMap, LogMapLt and Wiktionary have participated last year. Unfortunately, we lost VeeAlign participation this year but we compensate it with two new tools (ALOD2Vec and ATMatcher). Comparing the results from last year, in terms F-measure (cases of type i), AML maintains its overall performance (.47 in 2020, .45 in 2019, .46 in 2018, .46 in 2017, .45 in 2016 and .47 in 2015). On the other hand, LogMap has slightly increased its F-measure (.37 in 2020, .37 in 2019, .37 in 2018, .36 in 2017, and .37 in 2016). The performance in terms of f-measure of Wiktionary slightly increases f-measure to 0.35 (.32 in 2020, .31 in 2019).

Language specific results (type i)

Table below presents the results per pair of language, involving matching different ontologies (test cases of type i).

Figure below presents the aggregated results for the matching tasks. They have been computed using the MELT framework. We haven't applied any threshold on the results. They are measured in terms of macro precision and recall. We do not report the results of non-specific systems here, as we could observe in the last campaigns that they can have intermediate results in tests of type ii) and poor performance in tests i).

AMLoutperforms all other systems in terms of F-measure (0.35), followed by LogMap (0.32). We have the same figure than last year.

In terms of runtime, the results are not comparable to those from last year as the systems have been run in a different environment in terms of memory and number of processors.

Comparing the results from last year, in terms F-measure, AML maintains its overall performance (.47 in 2020, .45 in 2019, .46 in 2018, .46 in 2017, .45 in 2016 and .47 in 2015). LogMap slightly improves F-measure (.44 this year, 37 in 2020, .37 in 2019, .37 in 2018, .36 in 2017, and .37 in 2016). This is also the case of Wiktionary (besides the fact that is losses a little in precision).

Conclusions

Unfortunately, this track does not attract to many participants. In terms of performance, the F-measure for blind tests remains relatively stable across campaigns. As observed in several campaigns, still, all systems privilege precision in detriment to recall (recall below 0.40) and the results are below the ones obtained for the Conference original dataset.

References

[1] Christian Meilicke, Raul Garcia-Castro, Fred Freitas, Willem Robert van Hage, Elena Montiel-Ponsoda, Ryan Ribeiro de Azevedo, Heiner Stuckenschmidt, Ondrej Svab-Zamazal, Vojtech Svatek, Andrei Tamilin, Cassia Trojahn, Shenghui Wang. MultiFarm: A Benchmark for Multilingual Ontology Matching. Accepted for publication at the Journal of Web Semantics.

An authors version of the paper can be found at the MultiFarm homepage, where the data set is described in details.

Contact

This track is organized by Beyza Yaman (Trinity College Dublin, ADAPT Centre) and Cassia Trojahn dos Santos (University of Toulouse). If you have any problems working with the ontologies, any questions or suggestions, feel free to write an email to beyza [.] yaman [at] adaptcentre [.] ie and cassia [.] trojahn [at] irit [.] fr.