We have collected all generated alignments and make them available in a zip-file via the following link. These alignments are the raw results that the following report is based on.
We conducted experiments by executing each system in its standard setting and we compare precision, recall, F-measure and recall+. The measure recall+ indicates the amount of detected non-trivial correspondences. The matched entities in a non-trivial correspondence do not have the same normalized label.
We ran the matchers on a machine with 16GB RAM installed. As last year, we used the MELT platform to execute our evaluations for all systems except ALIN and AMD that we used the SEALS client.
As in earlier years, we slightly changed the way how precision and recall are computed, i.e., the results generated by the MELT and SEALS clients vary in some cases by 0.5% compared to the results presented below. In particular, we removed trivial correspondences in the oboInOwlnamespace like
http://...oboInOwl#Synonym = http://...oboInOwl#Synonym
as well as correspondences expressing relations different from equivalence.
Using the Pellet reasoner we also checked whether the generated alignment is coherent, i.e., there are no unsatisfiable concepts when the ontologies are merged with the alignment.
In the following, we analyze all participating systems that could generate an alignment. The listing comprises of 10 entries. LogMap participated with different versions, namely LogMap, LogMapBio, and a lightweight version LogMapLite that uses only some core components as previous years. There are three systems which are Matcha, ALIOn and SEBMatcher participating in the anatomy track this year for the first time. LSMatch and AMD participate in the anatomy track this year for the second time (last time in 2021). The rest of the systems have participated in OAEI for more than two years. The previous time ALIN , LSMatch, LogMap (all versions) and ATMatcher participated in the anatomy track was last year. LogMap is a constant participant since 2011 and ALIN joined in 2016. For more details, we refer the reader to the papers presenting the systems. Thus, this year we have 8 different systems (not counting different versions) which generated an alignment.
This year 4 out of 10 systems were able to achieve the alignment task in less than 100 seconds (they require between 3 and 37 seconds to match the ontologies). These are LogMap, LogMapLite, LSMatch and Matcha. In 2021 and 2020, there were 6 out of 15 and 4 out of 11 systems respectively which generated an alignment in this time frame. Similarly to the last 10 years, LogMapLite has the shortest run time. The table shows that there is no correlation between the required time for running and the quality of the generated alignment in specific metric. This result has also been observed in previous OAEI campaigns.
The table also shows the results for F-measure, recall+ and the size of the alignments. Regarding F-measure, the top 3 ranked systems are Matcha, SEBMatcher and LogMapBio. Among these, Matcha achieved the highest F-measure (0.941). ATMatcher and different versions of LogMap show similar results to those from 2021. This year, ALIN has an increase on F-measure from 0.835 in 2021 to 0.852 in 2022. Also, AMD has an increase from 0.835 in 2021 to 0.88 in 2022 on F-measure. Regarding recall+, ATMatcher, LogMap and LogMapLite show similar results as earlier. LogMapBio had a decrease from 0.74 in 2020 to 0.773 in 2021 but an increase to 0.787 in 2022. ALIN and AMD had notable increases on recall+ from 0.438 in 2021 to 0.501 in 2022 and from 0.316 in 2021 to 0.522 in 2022 respectively. SEBMAtcher as a new system in 2022 shows a high value for recall+ (0.674). Regarding the number of correspondences, LogMapLite, LogMap and ATMatcher computed a similar number of correspondences as last year. Compared with last year's results, LogMapBio, ALIN, LSMatch and AMD generated 10, 40, 69 and 132 more correspondences respectively.
This year 8 out of 10 systems achieved a F-measure higher than the baseline which is based on (normalized) string equivalence (StringEquiv in the table). Among these 8 systems, Matcha and SEBMatcher are new participants.This year two systems produced coherent alignments which are LogMap and LogMapBio.
The number of participating systems varies between the years. In 2022, there are five and one participants less than the number of participants in 2021 and 2020 respectively. As noted earlier there are newly-joined systems as well as long-term participants.
This year, Matcha sets the top result for the anatomy track with respect to the F-measure, followed by SEBMatcher and LogMapBio.
This track is organized by Mina Abd Nikooie Pour, Huanyu Li, Ying Li, and Patrick Lambrix. If you have any problems working with the ontologies, any questions related to tool wrapping, or any suggestions related to the anatomy track, feel free to write an email to oaei-anatomy [at] ida [.] liu [.] se.