Ontology Alignment Evaluation Initiative - OAEI-2012 Campaign

Large biomedical ontology results

We have run the evaluation in a high performance server with 16 CPUs and allocating 15 Gb RAM. In total, 15 out of 23 participating systems/configurations have been able to cope with at least one of the tasks of the track matching problem. Optima and MEDLEY failed to complete the smallest task with a time out of 24 hours, while OMR, OntoK, ASE and WeSeE, threw an Exception during the matching process. CODI was evaluated in a different setting using only 7Gb and threw an exception related to insufficient memory when processing the smallest matching task. TOAST was not evaluated since it was only configured for the Anatomy track and it required a complex installation. LogMapLt, a string matcher that exploits the creation of an inverted file to efficiently compute correspondences, has been used as baseline.

Together with Precision, Recall, F-measure and runtimes we have also evaluated the coherence of alignments. We have reported (1) number of unsatisfiabilities when reasoning (using HermiT) with the input ontologies together with the computed mappings, (2) the ratio/degree of unsatisfiable classes with respect to the size of the merged ontology (based on the Unsatisfiability Measure proposed in [1]), and (3) an approximation of the root unsatisfiability. The root unsatisfiability aims at providing a more precise amount of errors, since many of the unsatisfiabilities may be derived (i.e., a subclass of an unsatisfiable class will also be reported as unsatisfiable). The provided approximation is based on LogMap's (incomplete) repair facility and shows the number of classes that this facility needed to repair in order to solve (most of) the unsatisfiabilities [2].

Precision, recall and F-measure have been computed with respect to the available UMLS based alignments. Systems have been ordered in terms of the average F-measure.

Note that GOMMA has also been evaluated with a configuration that exploits specialised background knowledge (GOMMA-bk). The background knowledge of GOMMA-bk involves the application of mapping composition techniques and the reuse of mappings from FMA-UMLS and NCI-UMLS. LogMap, MaasMatch and YAM++ also use different kinds of background knowledge. LogMap uses normalisations and spelling variants from the UMLS Lexicon. YAM++ and MaasMatch use the general purpose background knowledge provided by WordNet.

LogMap has also been evaluated with two configurations. LogMap's default algorithm computes an estimation of the overlapping between the input ontologies before the matching process, while LogMap-noe has this feature deactivated.

The error-free "Large BioMed 2012 silver standard" reference alignment computed by "harmonising" the output of the participating matching systems will be available soon. We will also perform a debugging of all mapping outputs using Alcomo [3] and LogMap's repair facility [2].

Results OAEI 2012 FMA-NCI matching problem

FMA-NCI small fragments

This year we obtained very high level participation and 11 systems/configurations obtained, on average, an F-measure over 0.80 for the matching problem involving the small fragments of FMA and NCI. GOMMA-bk obtained the best results in terms of both recall and F-measure while ServOMap provided the most precise alignments. LogMap and LogMap-noe provided the same results since the input ontologies are already small fragments of FMA and NCI and thus, the overlapping estimation performed by LogMap did not have any impact. In general, as expected, precision increases when comparing against the original UMLS mapping set, while recall decreases.

Our baseline provided very good results in terms of F-measure and outperformed 8 of the participating systems. MaasMatch and Hertuda provided competitive results in terms of recall, but the low precision damaged the final F-measure. MapSSS and AUTOMSv2 provided a set of mappings with high precision, however, the F-measure was damaged due to the low recall of their mappings.

The runtimes were very positive in general and 8 systems completed the task in less than 2 minutes. MapSSS required less than 10 minutes, while Hertuda and HotMatch required around 1 hour. Finally, MaasMatch, AUTOMSv2 and Wmatch needed 8, 17 and 18 hours to complete the task, respectively.

Regarding mapping coherence, only LogMap (with its two variants) generates an almost clean output. In the table, we can appreciate that even the most precise mappings (ServOMap or YAM++) lead to a huge amount of unsatisfiable classes when reasoning together with the input ontologies; and thus, it proves the importance of using techniques to assess the coherence of the generated alignments. Unfortunately, LogMap and CODI are the unique systems (participating in the OAEI 2012) that have shown to use such techniques.

FMA-NCI big fragments

AUTOMSv2, HotMatch, Hertuda, Wmatch and MaasMatch failed to complete the task involving the big fragments of FMA and NCI after more than 24 hours of execution. Runtimes were in line with the small matching task, apart from the ones for MapSSS and AROMA which suffered an important increase.

YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. F-measures have decreased considerably with respect to the small matching task. This is mostly due to the fact that this matching task involves more possible candidate mappings than the previous one. Nevertheless, seven systems outperformed our baseline and provided high quality mapping sets in terms of both precision and recall. Only, MapSSS and AROMA provided worse results in terms of both precision and recall than LogMapLt.

Regarding mapping coherence, as in the previous task, only LogMap (with its two variants) generates an almost clean output where the mappings together with the input ontologies only lead to 5 unsatisfiable classes.

FMA-NCI whole ontologies

AROMA and MapSSS failed to complete the matching task involving the whole FMA and NCI ontologies in less than 24 hours.

As in the previous task, the remaining 7 matching systems generated high quality mapping sets. YAM++ provided the best results in terms of F-measure, whereas GOMMA-bk and ServOMap got the best recall and precision, respectively. LogMap with its two configurations provided an almost clean output and only 9 classes where unsatisfiable after reasoning with the input ontologies and the computed mappings.

Runtimes were also very positive. YAM++ was slightly slower than the other systems, which gave the outputs in less than 5 minutes, and required around 20 minutes to complete the task.

Results OAEI 2012 FMA-SNOMED matching problem

As it is depicted in the following tables, the FMA-SNOMED matching problem was harder than the FMA-NCI problem both in size and in complexity. Thus, matching systems required more time to complete the task and provided, in general, worse results in terms of F-measure. Furthermore, MaasMatch, Wmatch and AUTOMSv2, which were able to complete the small FMA-NCI task, failed to complete the small FMA-SNOMED task in less than 24 hours.

FMA-SNOMED small fragments

Six systems provided an on average an F-measure greater than 0.75. However, the other 6 systems that completed the task (including our baseline) failed to provide a recall higher than 0.4. GOMMA-bk provided the best results in terms of both recall and F-measure, while the baseline LogMapLt provided the best precision closely followed by ServOMapL. GOMMA-bk is a bit ahead with respect the other systems since managed to provide a mapping set with very high recall. The use of background knowledge was key in this matching task.

As in the FMA-NCI matching problem, precision tend to increase when comparing against the original UMLS mapping set, while recall decreases.

The runtimes were also very positive in general and 8 systems completed the task in less than 6 minutes. MapSSS required almost 1 hour, while Hertuda, HotMatch and AROMA needed 5, 9 and 14 hours to complete the task, respectively.

LogMap, unlike LogMap-noe, failed to detect and repair two unsatisfiable classes since they were outside the computed ontology fragments (overlapping). The rest of the systems, even when providing highly precise mappings like ServOMapL, generated mapping sets with a high incoherence degree.

FMA-SNOMED big fragments

MapSSS, HotMatch and Hertuda failed to complete the task involving the big fragments of FMA and SNOMED after more than 24 hours of execution.

ServOMapL provided the best results in terms of F-measure and precision, whereas GOMMA-bk got the best recall. As in the FMA-NCI matching task involving big fragments, the F-measures suffered, in general, a decrease with respect to the small matching task. The most important variations were suffered by GOMMA-bk and GOMMA where their average precision decreased from 0.893 and 0.875 to 0.571 and 0.389, respectively. This is an interesting fact, since the background knowledge used by GOMMA-bk could not avoid the decrease in precision while keeping a high recall. Furthermore, runtimes were from 4 to 10 times higher for all the systems, with the exception of AROMA's runtime that increased from 14 to 17 hours.

LogMap (with its two variants) generated a clean output where the mappings together with the input ontologies did not lead to any unsatisfiable class.

FMA-SNOMED whole ontologies

AROMA failed to complete the matching task involving the whole FMA and SNOMED ontologies in less than 24 hours.

The results in terms of both precision and recall did not suffer important changes and, as in the previous task, ServOMapL provided the best results in terms of F-measure and precision while GOMMA-bk got the best recall.

Runtimes for ServOMap, ServOMapL LogMapLt and LogMap (with its two variations) were in line with the previous matching task; the computation times for GOMMA, GOMMA-bk and YAM++, however, suffered and important increase. GOMMA (with its two variations) required more than 30 minutes, while YAM++ required more than 6 hours.

LogMap and LogMap-noe mappings, as in previous tasks, had a very low incoherence degree.

Results OAEI 2012 SNOMED-NCI matching problem

The matching outputs in the SNOMED-NCI matching problem have only been compared against the original UMLS mapping and the refined subset computed by LogMap's repair facility. We could not compute a refined UMLS alignment set with Alcomo debugging system since, at the time of creating the datasets, it could not cope with the integration of SNOMED and NCI via mappings. The new version of Alcomo, however, has shown to be able to provide such refined set.

The satisfiability results, since currently no OWL 2 reasoner has shown to cope with the integration of SNOMED and NCI via mappings [url], have been estimated using the Dowling-Gallier algorithm [url] for propositional Horn satisfiability (implemented in LogMap's repair facility).

SNOMED-NCI small fragments

The SNOMED-NCI matching problem moves to a next level of difficulty with respect to the FMA-SNOMED matching problem and, in general, runtimes and results are slightly worse. Furthermore, Hertuda and HotMatch, which were able to complete the small FMA-NCI and the small FMA-SNOMED tasks, failed to complete the small SNOMED-NCI task in less than 24 hours.

Six systems provided an F-measure higher than our baseline LogMapLt and their F-measures were very close to each other. On the other hand, GOMMA, MapSSS and AROMA failed to top LogMapLt results. LogMap-noe provided the best results in terms of recall and F-measure while ServOMap generated the most precise mappings.

As in the FMA-NCI and FMA-SNOMED matching problems, precision tend to increase when comparing against the original UMLS mapping set, while recall decreases.

The runtimes were also positive in general and 7 systems completed the task in less than 4 minutes. YAM++ required more than 30 minutes, while AROMA and MapSSS needed 4 and 8 hours to complete the task, respectively.

LogMap (with its two variants) generated a set of output mappings that did not lead to any unsatisfiable class when reasoning (using Dowling-Gallier algorithm) together with the input ontologies. The rest of the systems generated mapping sets that lead to a degree of incoherence greater than 50%.

SNOMED-NCI big fragments

MapSSS and AROMA failed to complete the task involving the big fragments of FMA and SNOMED after more than 24 hours of execution.

There were not big differences, in general, in terms of F-measure with respect to the small SNOMED-NCI task. Only LogMap decreased their recall and lost its second position and GOMMA-bk generated less precise mappings and was relegated to the sixth position. As in previous task, LogMap-noe provided the best results in terms of recall and F-measure while ServOMap generated the most precise mappings.

Runtimes were between 2 and 3 orders of magnitude bigger than in the small task, but in the most of the cases the task was finished in less than 10 minutes.

Regarding mapping coherence, LogMap-noe provided a clean output while LogMap, since it computes an estimation of the overlapping (fragments) between the input ontologies, failed to detect and repair 3 unsatisfiable classes, which were outside the computed ontology fragments.

SNOMED-NCI whole ontologies

The precision and recall slightly decreased in all systems and none of them could reach an F-measure of 0.7. YAM++ produced the best mapping set in terms of F-measure, while ServOMap and GOMMA-bk generated the mappings with best precision and recall, respectively. LogMap-noe lost its first position since it provided less comprehensive mappings.

ServOMap, ServOMapL and LogMapwere the fastest tools and required 11, 12 and 16 minutes respectively. GOMMA (with its two variations) required more than 30 minutes, while YAM++ required more than 8 hours.

As in previous task, LogMap-noe provided a clean output while LogMap failed to detect and repair a few unsatisfiable classes due to the computation of the overlapping between the input ontologies.

Result summaryk (Top 8 systems)

The following table summarises the results for the 8 systems that completed all 9 tasks in the Large BioMed Track. The table shows the average precision, recall, F-measure and incoherence degree; and the total time to complete the tasks.

The systems have been ordered according to the average F-measure. YAM++ obtained the best average F-measure, GOMMA-Bk the best recall and ServOMap computed the most precise mappings. The first 6 systems obtained very close results in terms of F-measure and there were only a gap of 0.024 between the first (YAM++) and the sixth (ServOMap).

Regarding mapping incoherence, LogMap and LogMap-noe were the unique systems providing mapping sets leading to a small number of unsatisfiable classes.

Finally, LogMapLt, since it implements basic and efficient string similarity techniques, was the fastest system. The rest of the tools, apart from YAM++, were also very fast and only needed between 38 and 97 minutes to complete the tasks. YAM++ was the counterexample and required almost 19 hours to complete the nine tasks.

Harmonization of the mapping outputs

Mapping repair evaluation

References

[1] Christian Meilicke and Heiner Stuckenschmidt. Incoherence as a basis for measuring the quality of ontology mappings. In Proc. of 3rd International Workshop on Ontology Matching (OM), 2008. [url]

[2] Ernesto Jimenez-Ruiz and Bernardo Cuenca Grau. LogMap: Logic-based and scalable ontology matching. In Proc. of 10th International Semantic Web Conference (ISWC), 2011. [url]

[3] Christian Meilicke. Alignment Incoherence in Ontology Matching. University of Mannheim, Chair of Artificial Intelligence (2011) [url]

Contact

This track is organised by Ernesto Jimenez Ruiz, Bernardo Cuenca Grau and Ian Horrocks, and supported by the SEALS and LogMap projects. If you have any question/suggestion related to the results of this track, feel free to write an email to ernesto [at] cs [.] ox [.] ac [.] uk or ernesto [.] jimenez [.] ruiz [at] gmail [.] com