Results for OAEI 2022 - Common Knowledge Graphs Track

The evaluation have been executed on a Linux virtual machine with 128 GB of RAM and 16 vCPUs (2.4 GHz) processors. Precision, Recall and F-measure have been computed with respect to the reference alignment for this task. The evaluation was performed using MELT evaluation toolkit, which was used to evaluate SEALS matchers as well as those submitted as web packages. This year we added a new benchmark dataset in addition to the original Nell-DBpedia dataset. The new benchmark aligns classes from Yago and Wikidata. The gold standard for Nell-DBpedia is only a partial gold standard [1]. Therefore, to avoid over-penalising systems that may discover reasonable matches that are not coded in the gold standard, we ignore any predicted matches if neither of the classes in that pair is present as a true positive pair with another class in the gold standard. The two datasets annotate numerous instances, which some matchers unfortunately could not handle. Therefore, this year, we evaluate all matchers on a small version of both benchmarks. The latter have the same exact schema information, but with fewer annotated instances.

• LogMap
• ATMatcher
• Matcha
• KGMatcher+
• LogMapLite
• LogMapKG
• LsMatch
• AMD

The tables below show the aggregated results on the two datasets for systems that produced non-empty alignment files. The size column indicates the total number of class alignments discovered by each system. While the majority of the systems discovered alignments at both schema and instance levels, we have only evaluated class alignments, as the two gold standard does not include any instance-level ground truth. Further, Not all systems were able to handle the original dataset versions (i.e., those with all annotated instances). In terms of the Nell-DBpedia test case, LogMap, ATMatcher, KGMatcher+, and AMD were able to generate results when applied to the full-size dataset. While on the Yago-Wikidata dataset, which is large-scale compared to the first dataset, only ATMatcher, KGMatcher+, and AMD were able to generate alignments with the original dataset. Other systems either fail to complete the task within the allocated 24 hours time limit such as LogMapLite, Matcha, and LsMatch, or produce an empty alignment file such as Matcha, LogMap (only on the YagoWikidata dataset). LogMapKG on the other hand tend to only align instances when it is applied to full-size datasets. Similar to 2021 evaluation results, AMD does generate schema alignments but in the wrong format, therefore, they can not be evaluated. On the Nell-DBpedia dataset, all systems were able to outperform the basic string matcher, in terms of F1 score, except for LogMapLite. On the YagoWikidata dataset, two systems were not able to outperform the baseline, which are LogMapLite and LsMatch. In terms of runtime, the tables below present the run time as HH:MM:SS where we can observe that all matching were able to finish the task in less than an hour except for KGMatcher+. Finally, the dataset size column identifies whether the system was able to perform on the original dataset or only on the smaller version.

This track is organized by:

• Omaima Fallatah (The University of Sheffield - Information School)

For any questions or suggestions about the track please email: oafallatah1 at sheffield dot ac dot uk