MultiFarm Results for OAEI 2018

In this page, we report the results of the OAEI 2018 campaign for the MultiFarm track. The details on this data set can be found at the MultiFarm web page.

If you notice any kind of error (wrong numbers, incorrect information on a matching system, etc.) do not hesitate to contact us (for the mail see below in the last paragraph on this page).

Experimental setting

We have conducted an evaluation based on the blind data set. This data set includes the matching tasks involving the edas and ekaw ontologies (resulting in 55 x 24 tasks). Participants were able to test their systems on the open subset of tasks, available via the SEALS repository. The open subset counts on 45 x 25 tasks and it does not include Italian translations.

We distinguish two types of matching tasks :

those tasks where two different ontologies (edas-ekaw, for instance) have been translated into two different languages;
those tasks where the same ontology (edas-edas) has been translated into two different languages.

As we could observe in previous evaluations, for the tasks of type (ii), good results are not directly related to the use of specific techniques for dealing with cross-lingual ontologies, but on the ability to exploit the fact that both ontologies have an identical structure.

Participants

This year, 6 systems (out of 18) have registered to participate in the MultiFarm track (i.e., those that have been assigned to the task in the registration phase): AML, DOME, EVOCROS, KEPLER, LogMap and XMap. This number slightly decreases with respect to the last campaign (8 in 2017, 7 in 2016, 5 in 2015, 3 in 2014, 7 in 2013, and 7 in 2012). The reader can refer to the OAEI papers for a detailed description of the strategies adopted by each system. In fact, most of them still adopts a translation step before the matching itself.

For this track, we observe that

KEPLER generated some parsing errors for some pairs;
EVOCROS took around 30 minutes to complete a single task (we have hence tested only 50 matching tasks) and generated empty alignments;
DOME was not able to generated any alignment;;
XMap has problems to deals with most pairs involving ar, ru and cn languages.

Evaluation results

Execution setting and runtime

The systems have been executed on a Ubuntu 16.04 machine configured with 16GB of RAM running under a i7-4790K CPU 4.00GHz x 8 processors. All measurements are based on a single run. As below, we can observe large differences in the time required for a system to complete the 55 x 24 matching tasks : AML (26 minutes), KEPLER (900 minutes), LogMap (39 minutes) and XMap (22 minutes). Note as well that the concurrent access to the SEALS repositories during the evaluation period may have an impact in the time required for completing the tasks.

Overall results

The table below presents the aggregated results for the matching tasks. They have been computed using the Alignment API 4.9 and can slightly differ from those computed with the SEALS client. We haven't applied any threshold on the results. They are measured in terms of classical precision and recall. We do not report the results of non-specific systems here, as we could observe in the last campaigns that they can have intermediate results in tests of type ii) and poor performance in tests i). For the evaluated systems, AML outperforms all other systems in terms of F-measure for task i), keeping its top place in this task. AML is followed by LogMap and KEPLER. With respect to the task ii), AML has relatively low performance and KEPLER has provided the higher F-measure, followed by LogMap.

MultiFarm aggregated results per matcher, for each type of matching task -- different ontologies (i) and same ontologies (ii). Time is measured in minutes. #pairs indicates the number of pairs of languages the tool is able to generated (non empty) alignments. Size indicates the average of the number of generated correspondences for the tests where an (non empty) alignment has been generated. Two kinds of results are reported : those do not distinguishing empty and erroneous (or not generated) alignments and those -- indicated between parenthesis -- considering only non empty generated alignments for a pair of languages.
			Different ontologies (i)				Same ontologies (ii)
System	Time	#pairs	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AML	26	55	6.87	.72(.72)	.46(.46)	.35(.35)	23.24	.96(.95)	.27(.27)	.16(.16)
KEPLER	900	53	9.74	.40(.42)	.27(.28)	.21(.22)	58.28	.85(.88)	.49(.51)	.36(.37)
LogMap	39	55	6.99	.72(.72)	.37(.37)	.25(.25)	46.80	.95(.96)	.41(.42)	.28(.28)
XMAP	22	22	94.72	.02(.05)	.03(.07)	.07(.07)	345	.13(.18)	.14(.20)	.19(.19)

Although the numbers of participants registered to participate in MultiFarm remains stable this year with respect to the last campaigns (6 in 2018, 7 in 2016, 5 in 2015, 3 in 2014, 7 in 2013 and 2012 and 3 in 2011), this year only 4 of them were able to provide non-empty alignments. All these systems have also been participated last year (AML, KEPLER, LogMap, and XMap). Unfortunately, we lose some ones on the way (CroLOM, SANOM and WikiV3). Comparing the results from last year, in terms F-measure (cases of type i), AML maintains its overall performance (.46 in 2018, .46 in 2017, .45 in 2016 and .47 in 2015). The same could be observed for LogMap (.37 in 2018, .36 in 2017, and .37 in 2016) and XMAP (and .06 in 2017).

Language specific results (type i)

Table below presents the results per pair of language, involving matching different ontologies (test cases of type i). The top F-measures for these systems include the pairs es-it, it-pt and nl-pt (AML), cz-pt and de-pt (KEPLER), en-nl (LogMap), and cz-en (XMap). Looking for the pairs of languages that appears in the 5 top F-measures, we observe the pairs nl-pt and es-pt. We could also note that the worst results have been obtained for some pairs involving specific translations (ar-cn for AML, and cn for KEPLER and LogMap). For these two systems, some pairs involving it have also poor performance.

MultiFarm results per pair of languages (55 pairs), for the test cases of type (i)

	AML			KEPLER			LogMap			XMap
ar-cn	0,62	0,26	0,16	0,46	0,14	0,08	0,62	0,19	0,11	NaN	NaN	0,00
ar-cz	0,71	0,40	0,28	0,51	0,23	0,15	0,72	0,40	0,28	NaN	NaN	0,00
ar-de	0,67	0,37	0,26	0,51	0,25	0,16	0,73	0,37	0,25	NaN	NaN	0,00
ar-en	0,74	0,39	0,26	0,55	0,26	0,17	0,73	0,41	0,28	NaN	NaN	0,00
ar-es	0,68	0,44	0,33	0,46	0,22	0,15	0,69	0,36	0,25	NaN	NaN	0,00
ar-fr	0,62	0,37	0,26	0,46	0,20	0,13	0,64	0,29	0,19	NaN	NaN	0,00
ar-it	0,73	0,46	0,34	0,43	0,21	0,14	0,69	0,22	0,13	NaN	NaN	0,00
ar-nl	0,69	0,39	0,27	0,49	0,23	0,15	0,74	0,41	0,28	NaN	NaN	0,00
ar-pt	0,71	0,48	0,37	0,53	0,28	0,19	0,72	0,38	0,25	NaN	NaN	0,00
ar-ru	0,65	0,29	0,19	0,60	0,23	0,14	0,77	0,41	0,28	NaN	NaN	0,00
cn-cz	0,63	0,32	0,22	0,31	0,17	0,12	0,72	0,27	0,17	NaN	NaN	0,00
cn-de	0,64	0,35	0,24	0,38	0,23	0,16	0,71	0,23	0,13	NaN	NaN	0,00
cn-en	0,67	0,32	0,21	0,42	0,27	0,20	0,85	0,22	0,13	NaN	NaN	0,00
cn-es	0,69	0,41	0,29	0,34	0,19	0,13	0,66	0,25	0,15	NaN	NaN	0,00
cn-fr	0,68	0,40	0,28	0,35	0,20	0,14	0,69	0,23	0,14	NaN	NaN	0,00
cn-it	0,72	0,39	0,27	0,41	0,24	0,17	0,79	0,12	0,06	NaN	NaN	0,00
cn-nl	0,67	0,34	0,23	0,39	0,19	0,13	0,70	0,21	0,12	NaN	NaN	0,00
cn-pt	0,65	0,41	0,30	0,36	0,21	0,15	0,77	0,25	0,15	NaN	NaN	0,00
cn-ru	0,65	0,39	0,28	0,36	0,21	0,15	0,73	0,31	0,19	NaN	NaN	0,00
cz-de	0,68	0,47	0,36	0,44	0,34	0,28	0,70	0,39	0,27	0,05	0,08	0,24
cz-en	0,81	0,48	0,34	0,41	0,31	0,25	0,79	0,50	0,37	0,18	0,14	0,11
cz-es	0,77	0,57	0,45	0,39	0,31	0,25	0,68	0,39	0,27	0,03	0,05	0,13
cz-fr	0,78	0,54	0,41	0,35	0,27	0,21	0,66	0,39	0,28	0,01	0,01	0,02
cz-it	0,77	0,53	0,40	0,40	0,28	0,21	0,77	0,37	0,24	0,06	0,08	0,11
cz-nl	0,78	0,56	0,44	0,43	0,30	0,23	0,72	0,45	0,33	0,05	0,08	0,17
cz-pt	0,72	0,55	0,45	0,50	0,41	0,35	0,72	0,44	0,32	0,06	0,10	0,25
cz-ru	0,75	0,52	0,39	0,41	0,35	0,30	0,75	0,46	0,33	NaN	NaN	0,00
de-en	0,79	0,47	0,33	0,47	0,39	0,33	0,78	0,44	0,31	0,06	0,10	0,24
de-es	0,67	0,48	0,37	0,44	0,33	0,27	0,73	0,39	0,26	0,02	0,02	0,03
de-fr	0,73	0,50	0,38	0,38	0,29	0,23	0,75	0,43	0,30	0,02	0,03	0,11
de-it	0,75	0,52	0,40	0,44	0,33	0,26	0,70	0,34	0,22	0,05	0,06	0,09
de-nl	0,73	0,48	0,36	0,47	0,34	0,27	0,78	0,45	0,32	0,05	0,08	0,14
de-pt	0,70	0,49	0,37	0,50	0,41	0,35	0,70	0,38	0,26	0,04	0,07	0,13
de-ru	0,67	0,41	0,30	0,49	0,28	0,20	0,78	0,44	0,31	NaN	NaN	0,00
en-es	0,77	0,45	0,32	0,39	0,30	0,25	0,72	0,45	0,33	0,07	0,08	0,10
en-fr	0,80	0,45	0,31	0,36	0,27	0,22	0,70	0,43	0,31	0,07	0,11	0,26
en-it	0,79	0,44	0,30	0,45	0,34	0,27	0,71	0,41	0,29	0,14	0,14	0,14
en-nl	0,80	0,48	0,34	0,42	0,32	0,26	0,80	0,54	0,40	0,08	0,12	0,22
en-pt	0,79	0,49	0,36	0,50	0,39	0,33	0,76	0,52	0,39	0,09	0,12	0,16
en-ru	0,74	0,38	0,26	0,42	0,30	0,24	0,90	0,48	0,33	0,01	0,00	0,00
es-fr	0,76	0,55	0,43	0,30	0,27	0,24	0,69	0,40	0,28	0,01	0,02	0,10
es-it	0,75	0,59	0,49	NaN	NaN	0,00	0,63	0,27	0,17	0,05	0,08	0,38
es-nl	0,74	0,58	0,48	0,37	0,33	0,29	0,71	0,40	0,28	0,00	NaN	0,00
es-pt	0,73	0,58	0,49	0,42	0,39	0,36	0,70	0,45	0,33	0,04	0,08	0,41
es-ru	0,72	0,51	0,39	0,45	0,39	0,34	0,76	0,41	0,28	NaN	NaN	0,00
fr-it	0,74	0,50	0,38	NaN	NaN	0,00	0,67	0,35	0,24	0,01	0,02	0,05
fr-nl	0,74	0,55	0,44	0,33	0,27	0,23	0,71	0,42	0,30	0,06	0,10	0,27
fr-pt	0,75	0,55	0,44	0,39	0,33	0,28	0,67	0,39	0,28	0,01	0,02	0,14
fr-ru	0,73	0,49	0,37	0,39	0,30	0,25	0,74	0,36	0,24	NaN	NaN	0,00
it-nl	0,72	0,53	0,42	0,39	0,30	0,24	0,75	0,36	0,24	NaN	NaN	0,00
it-pt	0,74	0,59	0,50	0,36	0,32	0,28	0,64	0,33	0,23	NaN	NaN	0,00
it-ru	0,68	0,46	0,35	0,36	0,27	0,21	0,82	0,28	0,17	NaN	NaN	0,00
nl-pt	0,76	0,59	0,48	0,44	0,36	0,31	0,70	0,45	0,33	0,06	0,07	0,11
nl-ru	0,74	0,51	0,39	0,40	0,29	0,23	0,79	0,46	0,33	NaN	NaN	0,00
pt-ru	0,72	0,49	0,37	0,51	0,37	0,30	0,75	0,47	0,34	NaN	NaN	0,00

NaN: division per zero, likely due to empty alignment.

Conclusions

From 18 participants, only 4 were evaluated in MultiFarm. In terms of performance, the F-measure for blind tests remains relatively stable across campaigns. AML and LogMap keep their positions with respect to the previous campaigns, followed by KEPLER. As observed in several campaigns, still, all systems privilege precision in detriment to recall and the results are below the ones obtained for the Conference original dataset. As last years, still cross-lingual approaches are mainly based on translation strategies and the combination of other resources (like cross-lingual links in Wikipedia, BabelNet, etc.) and strategies (machine learning, indirect alignment composition) remains under-exploited.

References

[1] Christian Meilicke, Raul Garcia-Castro, Fred Freitas, Willem Robert van Hage, Elena Montiel-Ponsoda, Ryan Ribeiro de Azevedo, Heiner Stuckenschmidt, Ondrej Svab-Zamazal, Vojtech Svatek, Andrei Tamilin, Cassia Trojahn, Shenghui Wang. MultiFarm: A Benchmark for Multilingual Ontology Matching. Accepted for publication at the Journal of Web Semantics.

An authors version of the paper can be found at the MultiFarm homepage, where the data set is described in details.

Contact

This track is organized by Cassia Trojahn dos Santos, with the help of Elodie Thieblin. If you have any problems working with the ontologies, any questions or suggestions, feel free to write an email to cassia [.] trojahn [at] irit [.] fr or elodie [.] thieblin [at] irit [.] fr