Results for OAEI 2023 - Common Knowledge Graphs Track

The evaluation have been executed on a virtual machine with 16 GB of RAM and 16 vCPUs (2.4 GHz) processors. Precision, Recall and F-measure have been computed with respect to the reference alignment for this task. The evaluation was performed using MELT evaluation toolkit, which was used to evaluate SEALS matchers as well as those submitted as web packages. This year we added a new benchmark dataset in addition to the original Nell-DBpedia dataset. The new benchmark aligns classes from Yago and Wikidata. The gold standard for Nell-DBpedia is only a partial gold standard [1]. Therefore, to avoid over-penalising systems that may discover reasonable matches that are not coded in the gold standard, we ignore any predicted matches if neither of the classes in that pair is present as a true positive pair with another class in the gold standard. The two datasets annotate numerous instances, which some matchers unfortunately could not handle. Therefore, this year, we evaluate all matchers on a small version of both benchmarks. The latter have the same exact schema information, but with fewer annotated instances.

• LogMap
• OLaLa
• Matcha
• LogMapLite
• LogMapKG
• LsMatch
• AMD

The tables provided below showcase aggregated results from two datasets for systems that generated non-empty alignment files. The "size" column indicates the total number of class alignments discovered by each system. Although most systems found alignments at both schema and instance levels, our evaluation focused solely on class alignments due to the absence of instance-level ground truth in the two gold standards. Regarding the Nell-DBpedia test case, systems like LogMap, OLaLa, Matcha, and AMD successfully produced results on the full-size dataset. However, some systems struggled with the original dataset versions containing all annotated instances. Specifically, LogMapLite and LsMatch failed to complete the task within the allocated 24-hour time limit. LogMap, while handling the smaller version, produced empty alignment files when applied to the YagoWikidata dataset. LogMapKG primarily aligned instances with full-size datasets. Additionally, AMD generated schema alignments, but in an incorrect format, rendering them unassessable. In terms of performance, all systems, except for LogMapLite, outperformed the basic string matcher on the Nell-DBpedia dataset. However, on the YagoWikidata dataset, LogMapLite and LsMatch were unable to surpass the baseline. This year saw the return of different matchers and the introduction of a new one, OLaLa. While most matchers demonstrated similar performance to previous evaluations, Matcha notably improved its results on both datasets. Matcha also showcased the ability to function with the original datasets, a capability it lacked in the 2022 evaluation. Notably, OLaLa outperformed all other matchers in the Nell-DBpedia task, whereas Matcha excelled on the larger dataset, Yago-Wikidata. Furthermore, all matching processes were completed in less than an hour, as indicated in the runtime column. Lastly, the dataset size column specifies whether a system operated on the original dataset or solely on the smaller version.

This track is organized by:

• Omaima Fallatah (Umm AlQura University - College of Computer Science - Saudi Arabia)

For any questions or suggestions about the track please email: oafallatah at uqu dot edu dot sa