Ontology Alignment Evaluation Initiative - OAEI-2020 Campaign OAEI OAEI

Results for OAEI 2020 - Knowledge Graph Track

Matching systems

As a pre-test, we executed all systems submitted as SEALS packages to OAEI (even if they are not registered for the track) on a very small matching example (dataset) with a similar structure and shape like the real knowledge graphs (in fact, they are a small subset of them). It showed that not all matching systems are able to complete this small task due to exceptions or other failures. The following matching systems produced an exception: We wrote an email to the respective system developers to give them the possibility to modify the system configuration to complete this task. However, none of them were ultimately capable of producing results on the test task of the Knowledge Graph track. Thus, we executed the following systems: The source code for the baseline matchers is available. The baselineLabel matcher matches all resources which share the same rdfs:label. (in case multiple resources share the same label, all of them are matched). BaselineAltLabel is additionally using skos:altLabel. Again, in cases where multiple resources share a common label, all those resources are matched in a cross product manner.

Experimental setting

The evaluation is executed on a virtual machine(VM) with 32GB of RAM and 16 vCPUs (2.4 GHz). The operating system is debian 9 with openjdk version "1.8.0_265".

We used the "-o" option in SEALS (version 7.0.5) to provide the two knowledge graphs which should be matched. The two given URLs are file URLs and not HTTP URLs because downloading the knowledge graphs from another server would be a huge time overhead (downloading 500 MB multiple times for different tasks). Thus the call of the SEALS client looks like the following:

java -Xmx25g -Xms15g -jar ../seals-omt-client.jar ${MATCHER_DIR} -z -o file:///data/ont1.xml file:///data/ont2.xml -f out.xml

This also means that the reported time also includes the environment preparation of SEALS (copying the configuration folder of each matcher at the start of the matching task) as well as copying the resulting alignment to the output file. We could not use the "-x" option of SEALS because we had to modify the evaluation routine based on two reasons. The first one is to differentiate between class, property and instance mappings. The second reason is to deal with the partial gold standard of this track.

The alignments were evaluated based on Precision, Recall and F-Measure for classes, properties and instances (each in isolation). Our partial gold standard consist of 1:1 mappings extracted from links contained in wiki pages (cross wiki links). The schema was matched by ontology experts. We assume that in each knowledge graph, only one representation of one concept exists. This means if we have the mapping in our gold standard we can count the mapping as a false positive (the assumption here is that in the seconds knowledge graph no similar concept to B exists). The value of false negatives is only increased if we have a 1:1 mapping and it is not found by a matcher. The source code for generating the evaluation results is also available.

We imposed a maximum execution time of 24h per task, however, that time limit was never exceeded.

Generated dashboard / CSV file

We also generated an online dashboard with the help of the MELT framework. Have a look at the knowledge graph results here (it may take some seconds to load due to 200 000 correspondences).
Moreover, we also generated a CSV file which allows to analyze each matcher on a correspondence level. This should help matcher developers to increase the matcher performance.

Alignment results

The generated alignment files are also available.

Results overview

class property instance overall
SystemTime#testcases SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec.
ALOD2Vec0:13:245 20.01.00 (1.00)0.80 (0.80)0.67 (0.67) 76.80.94 (0.94)0.95 (0.95)0.97 (0.97) 4893.80.91 (0.91)0.87 (0.87)0.83 (0.83) 4990.60.91 (0.91)0.87 (0.87)0.83 (0.83)
AML0:50:555 23.60.98 (0.98)0.89 (0.89)0.81 (0.81) 48.40.92 (0.92)0.70 (0.70)0.57 (0.57) 6802.80.90 (0.90)0.85 (0.85)0.80 (0.80) 6874.80.90 (0.90)0.85 (0.85)0.80 (0.80)
ATBox0:16:225 25.60.97 (0.97)0.87 (0.87)0.79 (0.79) 78.80.97 (0.97)0.96 (0.96)0.95 (0.95) 4858.80.89 (0.89)0.84 (0.84)0.80 (0.80) 4963.20.89 (0.89)0.85 (0.85)0.81 (0.81)
baselineAltLabel0:10:575 16.41.00 (1.00)0.74 (0.74)0.59 (0.59) 47.80.99 (0.99)0.79 (0.79)0.66 (0.66) 4674.80.89 (0.89)0.84 (0.84)0.80 (0.80) 4739.00.89 (0.89)0.84 (0.84)0.80 (0.80)
baselineLabel0:10:445 16.41.00 (1.00)0.74 (0.74)0.59 (0.59) 47.80.99 (0.99)0.79 (0.79)0.66 (0.66) 3641.80.95 (0.95)0.81 (0.81)0.71 (0.71) 3706.00.95 (0.95)0.81 (0.81)0.71 (0.71)
DESKMatcher0:13:545 91.40.76 (0.76)0.71 (0.71)0.66 (0.66) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 3820.60.94 (0.94)0.82 (0.82)0.74 (0.74) 3912.00.93 (0.93)0.81 (0.81)0.72 (0.72)
LogMap2:55:145 24.00.95 (0.95)0.84 (0.84)0.76 (0.76) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 29190.40.40 (0.40)0.54 (0.54)0.86 (0.86) 29214.40.40 (0.40)0.54 (0.54)0.84 (0.84)
LogMapBio4:35:295 24.00.95 (0.95)0.84 (0.84)0.76 (0.76) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 24.00.95 (0.95)0.01 (0.01)0.00 (0.00)
LogMapIM2:49:345 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 29190.40.40 (0.40)0.54 (0.54)0.86 (0.86) 29190.40.40 (0.40)0.54 (0.54)0.84 (0.84)
LogMapKG2:47:515 24.00.95 (0.95)0.84 (0.84)0.76 (0.76) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 29190.40.40 (0.40)0.54 (0.54)0.86 (0.86) 29214.40.40 (0.40)0.54 (0.54)0.84 (0.84)
LogMapLt0:07:194 23.00.80 (1.00)0.56 (0.70)0.43 (0.54) 0.00.00 (0.00)0.00 (0.00)0.00 (0.00) 6653.80.73 (0.91)0.67 (0.84)0.62 (0.78) 6676.80.73 (0.92)0.66 (0.83)0.61 (0.76)
Wiktionary0:30:125 22.41.00 (1.00)0.80 (0.80)0.67 (0.67) 80.00.94 (0.94)0.95 (0.95)0.97 (0.97) 4893.80.91 (0.91)0.87 (0.87)0.83 (0.83) 4996.20.91 (0.91)0.87 (0.87)0.83 (0.83)
Aggregated results per matcher, divided into class, property, instance, and overall alignments. Time is displayed as HH:MM:SS. Column #testcases indicates the number of testcases where the tool is able to generate (non empty) alignments. Column size indicates the averaged number of system correspondences. Two kinds of results are reported: (1) those not distinguishing empty and erroneous (or not generated) alignments, and (2) those considering only non empty alignments (value between parenthesis).
All matchers except LogMapLt can produce an alignment for all test cases (LogMapLt generates a multi GB alignment file for test case marvelcinematicuniverse-marvel). The longest runtime was observed for the LogMap family with more than 2 hours for all 5 test cases.

Test case specific results

Overall results

This result table shows the overall performance (without dividing into class, property or instance) of the matchers.
marvelcinematicuniverse-marvel memoryalpha-memorybeta memoryalpha-stexpanded starwars-swg starwars-swtor
SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec.
ALOD2Vec 30930.860.760.68 134570.920.910.90 34020.920.920.93 21850.920.830.75 28160.930.920.92
AML 46870.850.680.56 184390.910.890.87 37950.930.930.92 35150.900.810.74 39380.930.910.90
ATBox 35180.670.590.52 130020.960.930.91 32810.960.940.92 22440.930.840.76 27710.950.930.91
baselineAltLabel 25740.860.760.68 135140.880.890.89 32300.880.900.92 17120.920.740.63 26650.920.910.90
baselineLabel 18790.900.690.56 105520.950.850.77 25820.980.900.83 12450.960.680.53 22720.950.890.84
DESKMatcher 19380.890.690.56 109460.940.850.77 26770.970.890.82 16380.930.750.63 23610.940.880.83
LogMap 410800.240.350.71 545370.430.580.86 155600.470.630.92 153350.580.670.80 195600.270.420.92
LogMapBio 101.000.000.00 380.750.000.00 311.000.010.00 121.000.010.00 291.000.020.01
LogMapIM 410700.240.350.71 544990.430.580.86 155290.470.620.91 153230.580.670.80 195310.270.420.91
LogMapKG 410800.240.350.71 545370.430.580.86 155600.470.630.92 153350.580.670.80 195600.270.420.92
LogMapLt 00.000.000.00 166880.900.830.77 35770.940.880.82 28070.910.740.62 36350.910.870.83
Wiktionary 30960.860.760.68 134660.920.910.90 34050.920.920.93 21920.920.830.75 28220.930.920.92

Class results

All matchers except LogMapIM were able to generate class correspondences.
marvelcinematicuniverse-marvel memoryalpha-memorybeta memoryalpha-stexpanded starwars-swg starwars-swtor
SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec.
ALOD2Vec 71.001.001.00 211.000.440.29 291.000.760.62 141.000.750.60 291.000.930.87
AML 81.001.001.00 360.910.800.71 321.000.820.69 121.000.890.80 301.000.930.87
ATBox 111.001.001.00 390.830.770.71 341.000.870.77 131.000.750.60 311.000.930.87
baselineAltLabel 81.001.001.00 191.000.440.29 191.000.630.46 91.000.570.40 271.000.890.80
baselineLabel 81.001.001.00 191.000.440.29 191.000.630.46 91.000.570.40 271.000.890.80
DESKMatcher 401.001.001.00 1150.330.310.29 1200.620.620.62 1161.000.750.60 660.860.830.80
LogMap 101.001.001.00 380.750.690.64 311.000.700.54 121.000.890.80 291.000.890.80
LogMapBio 101.001.001.00 380.750.690.64 311.000.700.54 121.000.890.80 291.000.890.80
LogMapIM 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapKG 101.001.001.00 380.750.690.64 311.000.700.54 121.000.890.80 291.000.890.80
LogMapLt 00.000.000.00 231.000.440.29 271.000.700.54 121.000.750.60 301.000.850.73
Wiktionary 91.001.001.00 271.000.440.29 331.000.760.62 141.000.750.60 291.000.930.87

Property results

While last year, most matchers struggled on creating mappings typed as rdf:Property (instead of owl:ObjectProperty or owl:DatatypeProperty, many matchers were able to produce property mappings also in that case.
marvelcinematicuniverse-marvel memoryalpha-memorybeta memoryalpha-stexpanded starwars-swg starwars-swtor
SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec.
ALOD2Vec 211.001.001.00 1110.840.880.92 880.870.910.95 531.001.001.00 1110.980.980.98
AML 91.000.530.36 840.760.730.70 640.890.850.80 121.000.400.25 730.980.820.71
ATBox 240.910.910.91 1030.980.950.92 850.950.950.95 611.001.001.00 1211.000.990.98
baselineAltLabel 71.000.530.36 411.000.510.34 460.970.800.68 421.001.001.00 1031.000.940.89
baselineLabel 71.000.530.36 411.000.510.34 460.970.800.68 421.001.001.00 1031.000.940.89
DESKMatcher 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMap 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapBio 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapIM 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapKG 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapLt 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
Wiktionary 221.001.001.00 1120.840.880.92 890.870.910.95 601.001.001.00 1170.980.980.98

Instance results

Only LogMapBio did not return any instance matches.
marvelcinematicuniverse-marvel memoryalpha-memorybeta memoryalpha-stexpanded starwars-swg starwars-swtor
SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec. SizePrec.F-m.Rec.
ALOD2Vec 30650.860.760.68 133250.920.910.90 32850.920.930.93 21180.920.820.75 26760.920.920.91
AML 46700.850.680.56 183190.910.890.87 36990.930.930.93 34910.900.810.75 38350.930.920.90
ATBox 34830.660.580.52 128600.960.930.91 31620.960.940.92 21700.930.830.76 26190.940.920.91
baselineAltLabel 25590.860.760.68 134540.880.890.89 31650.880.900.93 16610.920.740.62 25350.920.910.90
baselineLabel 18640.900.690.56 104920.950.850.77 25170.980.910.84 11940.950.670.52 21420.950.890.84
DESKMatcher 18980.890.690.56 108310.940.850.78 25570.980.910.84 15220.930.760.64 22950.940.900.86
LogMap 410700.240.350.71 544990.430.580.87 155290.470.630.94 153230.580.680.82 195310.270.420.96
LogMapBio 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00 00.000.000.00
LogMapIM 410700.240.350.71 544990.430.580.87 155290.470.630.94 153230.580.680.82 195310.270.420.96
LogMapKG 410700.240.350.71 544990.430.580.87 155290.470.630.94 153230.580.680.82 195310.270.420.96
LogMapLt 00.000.000.00 166650.900.830.77 35500.940.890.84 27950.910.750.63 36050.910.890.87
Wiktionary 30650.860.760.68 133270.920.910.90 32830.920.920.93 21180.920.820.75 26760.920.920.91

Runtime

Most matchers have a very low runtime which shows that they scale to huge KGs. Except the LogMap family, all matchers solve each task in less than 30 minutes.
marvelcinematicuniverse-marvel memoryalpha-memorybeta memoryalpha-stexpanded starwars-swg starwars-swtor
ALOD2Vec 0:03:05 0:02:10 0:01:20 0:03:22 0:03:27
AML 0:12:34 0:22:17 0:04:43 0:05:30 0:05:51
ATBox 0:03:54 0:03:26 0:01:49 0:03:28 0:03:45
baselineAltLabel 0:02:35 0:01:49 0:01:09 0:02:35 0:02:49
baselineLabel 0:02:30 0:01:43 0:01:17 0:02:38 0:02:36
DESKMatcher 0:03:08 0:02:28 0:01:41 0:03:20 0:03:17
LogMap 1:40:56 0:18:09 0:12:14 0:30:31 0:13:24
LogMapBio 0:37:37 1:24:45 0:59:48 0:26:18 1:07:01
LogMapIM 1:33:33 0:19:48 0:11:47 0:30:51 0:13:35
LogMapKG 1:33:40 0:17:58 0:12:00 0:31:08 0:13:05
LogMapLt 0:00:00 0:01:27 0:00:54 0:02:27 0:02:31
Wiktionary 0:06:46 0:05:50 0:03:57 0:06:49 0:06:50

Organizers

References

[1] Sven Hertling, Heiko Paulheim: The knowledge graph track at OAEI : Gold standards, baselines, and the golden hammer bias. ESWC 2020. [pdf]

[2] Sven Hertling, Heiko Paulheim: DBkWik: A Consolidated Knowledge Graph from Thousands of Wikis. International Conference on Big Knowledge 2018. [pdf]

[3] Alexandra Hofmann, Samresh Perchani, Jan Portisch, Sven Hertling, and Heiko Paulheim. DBkWik: Towards Knowledge Graph Creation from Thousands of Wikis. International Semantic Web Conference (Posters & Demos) 2017. [pdf]