Results for OAEI 2019 - Knowledge Graph Track

Matching systems

As a pre-test, we executed all systems submitted as SEALS packages to OAEI (even if they are not registered for the track) on a very small matching example (dataset) with a similar structure and shape like the real knowledge graphs (in fact, they are a small subset of them). It showed that not all matching systems are able to complete this small task due to exceptions or other failures. The following matching systems produced an exception:

EVOCROS
ALIN

The following matchers produced an empty alignment:

CANARD (only registered for COMPLEX track)
OntMat1 (error: Could not read configuration file)
Lily (error: cannot create file for resulting alignment)

We wrote an email to the respective system developers to give them the possibility to modify the system configuration to complete this task. However, none of them were ultimately capable of producing results on the test task of the Knowledge Graph track. Thus, we executed the following systems:

AGM
AML
baselineAltLabel (seals package)
baselineLabel (seals package)
DOME
FCAMap-KG
LogMap
LogMapBio
LogMapKG
LogMapLt
POMap++
Wiktionary

The source code for the baseline matchers is available. The baselineLabel matcher matches all resources which share the same rdfs:label. (in case multiple resources share the same label, all of them are matched). BaselineAltLabel is additionally using skos:altLabel. Again, in cases where multiple resources share a common label, all those resources are matched in a cross product manner.

Experimental setting

The evaluation is executed on a virtual machine(VM) with 32GB of RAM and 16 vCPUs (2.4 GHz). The operating system is debian 9 with openjdk version "1.8.0_212".

We used the "-o" option in SEALS (version 7.0.5) to provide the two knowledge graphs which should be matched. The two given URLs are file URLs and not HTTP URLs because downloading the knowledge graphs from another server would be a huge time overhead (downloading 500 MB multiple times for different tasks). Thus the call of the SEALS client looks like the following:


java -Xmx25g -Xms15g -jar ../seals-omt-client.jar ${MATCHER_DIR} -z -o file:///data/ont1.xml file:///data/ont2.xml -f out.xml

This also means that the reported time also includes the environment preparation of SEALS (copying the configuration folder of each matcher at the start of the matching task) as well as copying the resulting alignment to the output file. We could not use the "-x" option of SEALS because we had to modify the evaluation routine based on two reasons. The first one is to differentiate between class, property and instance mappings. The second reason is to deal with the partial gold standard of this track.

The alignments were evaluated based on Precision, Recall and F-Measure for classes, properties and instances (each in isolation). Our partial gold standard consist of 1:1 mappings extracted from links contained in wiki pages (cross wiki links). The schema was matched by ontology experts. We assume that in each knowledge graph, only one representation of one concept exists. This means if we have the mapping in our gold standard we can count the mapping as a false positive (the assumption here is that in the seconds knowledge graph no similar concept to B exists). The value of false negatives is only increased if we have a 1:1 mapping and it is not found by a matcher. The source code for generating the evaluation results is also available.

We imposed a maximum execution time of 4h per task, however, that time limit was never exceeded.

Generated dashboard

We also generated an online dashboard with the help of the MELT framework. Have a look at the knowledge graph results here (it may take some seconds to load due to 200 000 correspondences).
It would be nice if you could fill out a questionnaire about this tool.

Results overview

Aggregated results per matcher, divided into class, property, instance, and overall alignments. Time is displayed as HH:MM:SS. Column *#testcases* indicates the number of testcases where the tool is able to generate (non empty) alignments. Column *size* indicates the averaged number of system correspondences. Two kinds of results are reported: (1) those not distinguishing empty and erroneous (or not generated) alignments, and (2) those considering only non empty alignments (value between parenthesis).
			class				property				instance				overall
System	Time	#testcases	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AGM	10:47:38	5	14.6	0.23 (0.23)	0.09 (0.09)	0.06 (0.06)	49.4	0.66 (0.66)	0.32 (0.32)	0.21 (0.21)	5169.0	0.48 (0.48)	0.25 (0.25)	0.17 (0.17)	5233.2	0.48 (0.48)	0.25 (0.25)	0.17 (0.17)
AML	0:45:46	4	27.5	0.78 (0.98)	0.69 (0.86)	0.61 (0.77)	58.2	0.72 (0.91)	0.59 (0.73)	0.49 (0.62)	7529.8	0.72 (0.90)	0.71 (0.88)	0.69 (0.86)	7615.5	0.72 (0.90)	0.70 (0.88)	0.69 (0.86)
baselineAltLabel	0:11:48	5	16.4	1.00 (1.00)	0.74 (0.74)	0.59 (0.59)	47.8	0.99 (0.99)	0.79 (0.79)	0.66 (0.66)	4674.2	0.89 (0.89)	0.84 (0.84)	0.80 (0.80)	4739.0	0.89 (0.89)	0.84 (0.84)	0.80 (0.80)
baselineLabel	0:12:30	5	16.4	1.00 (1.00)	0.74 (0.74)	0.59 (0.59)	47.8	0.99 (0.99)	0.79 (0.79)	0.66 (0.66)	3641.2	0.95 (0.95)	0.81 (0.81)	0.71 (0.71)	3706.0	0.95 (0.95)	0.81 (0.81)	0.71 (0.71)
DOME	1:05:26	4	22.5	0.74 (0.92)	0.62 (0.77)	0.53 (0.66)	75.5	0.79 (0.99)	0.77 (0.96)	0.75 (0.93)	4895.2	0.74 (0.92)	0.70 (0.88)	0.67 (0.84)	4994.8	0.74 (0.92)	0.70 (0.88)	0.67 (0.84)
FCAMap-KG	1:14:49	5	18.6	1.00 (1.00)	0.82 (0.82)	0.70 (0.70)	69.0	1.00 (1.00)	0.98 (0.98)	0.96 (0.96)	4530.6	0.90 (0.90)	0.84 (0.84)	0.79 (0.79)	4792.6	0.91 (0.91)	0.85 (0.85)	0.79 (0.79)
LogMap	0:15:43	5	26.0	0.95 (0.95)	0.84 (0.84)	0.76 (0.76)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	26.0	0.95 (0.95)	0.01 (0.01)	0.00 (0.00)
LogMapBio	2:31:01	5	26.0	0.95 (0.95)	0.84 (0.84)	0.76 (0.76)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	26.0	0.95 (0.95)	0.01 (0.01)	0.00 (0.00)
LogMapKG	2:26:14	5	26.0	0.95 (0.95)	0.84 (0.84)	0.76 (0.76)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	29190.4	0.40 (0.40)	0.54 (0.54)	0.86 (0.86)	29216.4	0.40 (0.40)	0.54 (0.54)	0.84 (0.84)
LogMapLt	0:07:28	4	23.0	0.80 (1.00)	0.56 (0.70)	0.43 (0.54)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	6653.8	0.73 (0.91)	0.67 (0.84)	0.62 (0.78)	6676.8	0.73 (0.91)	0.66 (0.83)	0.61 (0.76)
POMAP++	0:14:39	5	2.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	19.4	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)
Wiktionary	0:20:14	5	21.4	1.00 (1.00)	0.80 (0.80)	0.67 (0.67)	75.8	0.97 (0.97)	0.98 (0.98)	0.98 (0.98)	3483.6	0.91 (0.91)	0.79 (0.79)	0.70 (0.70)	3581.8	0.91 (0.91)	0.80 (0.80)	0.71 (0.71)

The following matchers could output results on all 5 test cases: AGM, baselineAltLabel, baselineLabel, FCAMap-KG, LogMap, LogMapBio, LogMapKG, POMAP++, and Wiktionary. The longest runtime was observed for AGM with more than 10 hours for all 5 test cases. The systems which did not finish a task did so because of exceeding the amount of main memory they were granted.

Test case specific results

Overall results

This result table shows the overall performance (without dividing into class, property or instance) of the matchers.

	marvelcinematicuniverse-marvel				memoryalpha-memorybeta				memoryalpha-stexpanded				starwars-swg				starwars-swtor
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AGM	9085	0.14	0.11	0.09	11449	0.47	0.32	0.24	3684	0.44	0.30	0.23	1101	0.70	0.30	0.19	847	0.65	0.20	0.12
AML	0	0.00	0.00	0.00	18842	0.90	0.88	0.87	3996	0.90	0.92	0.93	3578	0.90	0.81	0.74	4046	0.92	0.91	0.90
baselineAltLabel	2574	0.86	0.76	0.68	13514	0.88	0.89	0.89	3230	0.88	0.90	0.92	1712	0.92	0.74	0.63	2665	0.92	0.91	0.90
baselineLabel	1879	0.90	0.69	0.56	10552	0.95	0.85	0.77	2582	0.98	0.90	0.83	1245	0.96	0.68	0.53	2272	0.95	0.89	0.84
DOME	0	0.00	0.00	0.00	12475	0.91	0.88	0.84	2727	0.93	0.91	0.89	2024	0.92	0.82	0.74	2753	0.93	0.91	0.89
FCAMap-KG	2682	0.84	0.73	0.65	13171	0.92	0.88	0.85	3174	0.94	0.91	0.89	2140	0.90	0.80	0.71	2796	0.93	0.90	0.87
LogMap	12	1.00	0.00	0.00	40	0.75	0.00	0.00	33	1.00	0.01	0.00	14	1.00	0.01	0.00	31	1.00	0.02	0.01
LogMapBio	12	1.00	0.00	0.00	40	0.75	0.00	0.00	33	1.00	0.01	0.00	14	1.00	0.01	0.00	31	1.00	0.02	0.01
LogMapKG	41082	0.24	0.35	0.71	54539	0.43	0.58	0.87	15562	0.48	0.63	0.92	15337	0.58	0.67	0.80	19562	0.27	0.42	0.92
LogMapLt	0	0.00	0.00	0.00	16688	0.90	0.83	0.77	3577	0.94	0.87	0.82	2807	0.91	0.74	0.62	3635	0.91	0.87	0.83
POMAP++	9	0.00	0.00	0.00	20	0.00	0.00	0.00	25	0.00	0.00	0.00	14	0.00	0.00	0.00	29	0.00	0.00	0.00
Wiktionary	1865	0.86	0.68	0.56	9838	0.92	0.79	0.69	2167	0.94	0.87	0.80	1592	0.92	0.75	0.63	2447	0.93	0.88	0.84

Class results

All matchers were able to generate class correspondences.

	marvelcinematicuniverse-marvel				memoryalpha-memorybeta				memoryalpha-stexpanded				starwars-swg				starwars-swtor
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AGM	21	0.00	0.00	0.00	18	0.67	0.24	0.14	15	0.50	0.24	0.15	9	0.00	0.00	0.00	10	0.00	0.00	0.00
AML	0	0.00	0.00	0.00	36	0.91	0.80	0.71	32	1.00	0.82	0.69	12	1.00	0.89	0.80	30	1.00	0.93	0.87
baselineAltLabel	8	1.00	1.00	1.00	19	1.00	0.44	0.29	19	1.00	0.63	0.46	9	1.00	0.57	0.40	27	1.00	0.89	0.80
baselineLabel	8	1.00	1.00	1.00	19	1.00	0.44	0.29	19	1.00	0.63	0.46	9	1.00	0.57	0.40	27	1.00	0.89	0.80
DOME	0	0.00	0.00	0.00	22	1.00	0.67	0.50	25	1.00	0.82	0.69	15	0.75	0.67	0.60	28	0.93	0.90	0.87
FCAMap-KG	8	1.00	1.00	1.00	21	1.00	0.44	0.29	24	1.00	0.76	0.62	12	1.00	0.89	0.80	28	1.00	0.89	0.80
LogMap	12	1.00	1.00	1.00	40	0.75	0.69	0.64	33	1.00	0.70	0.54	14	1.00	0.89	0.80	31	1.00	0.89	0.80
LogMapBio	12	1.00	1.00	1.00	40	0.75	0.69	0.64	33	1.00	0.70	0.54	14	1.00	0.89	0.80	31	1.00	0.89	0.80
LogMapKG	12	1.00	1.00	1.00	40	0.75	0.69	0.64	33	1.00	0.70	0.54	14	1.00	0.89	0.80	31	1.00	0.89	0.80
LogMapLt	0	0.00	0.00	0.00	23	1.00	0.44	0.29	27	1.00	0.70	0.54	12	1.00	0.75	0.60	30	1.00	0.85	0.73
POMAP++	2	0.00	0.00	0.00	2	0.00	0.00	0.00	2	0.00	0.00	0.00	2	0.00	0.00	0.00	2	0.00	0.00	0.00
Wiktionary	9	1.00	1.00	1.00	24	1.00	0.44	0.29	31	1.00	0.76	0.62	14	1.00	0.75	0.60	29	1.00	0.93	0.87

Property results

While last year, most matchers struggled on creating mappings typed as rdf:Property (instead of owl:ObjectProperty or owl:DatatypeProperty, many matchers were able to produce property mappings also in that case. Only POMAP++ and all LogMap variants could not return any property mappings.

	marvelcinematicuniverse-marvel				memoryalpha-memorybeta				memoryalpha-stexpanded				starwars-swg				starwars-swtor
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AGM	66	0.22	0.20	0.18	71	0.54	0.35	0.26	45	0.74	0.47	0.34	31	1.00	0.18	0.10	34	0.82	0.27	0.16
AML	0	0.00	0.00	0.00	84	0.76	0.73	0.70	64	0.89	0.85	0.80	12	1.00	0.40	0.25	73	0.98	0.82	0.71
baselineAltLabel	7	1.00	0.53	0.36	41	1.00	0.51	0.34	46	0.97	0.80	0.68	42	1.00	1.00	1.00	103	1.00	0.94	0.89
baselineLabel	7	1.00	0.53	0.36	41	1.00	0.51	0.34	46	0.97	0.80	0.68	42	1.00	1.00	1.00	103	1.00	0.94	0.89
DOME	0	0.00	0.00	0.00	88	0.98	0.94	0.91	70	0.97	0.95	0.93	42	1.00	1.00	1.00	102	1.00	0.94	0.89
FCAMap-KG	19	1.00	0.95	0.91	93	1.00	0.97	0.94	73	0.98	0.98	0.98	48	1.00	1.00	1.00	112	1.00	0.99	0.98
LogMap	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
LogMapBio	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
LogMapKG	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
LogMapLt	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
POMAP++	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
Wiktionary	22	1.00	1.00	1.00	102	0.93	0.93	0.94	80	0.95	0.96	0.98	58	1.00	1.00	1.00	117	0.98	0.98	0.98

Instance results

Only POMAP++, LogMap, and LogMapBio did not return any instance matches.

	marvelcinematicuniverse-marvel				memoryalpha-memorybeta				memoryalpha-stexpanded				starwars-swg				starwars-swtor
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AGM	8997	0.14	0.11	0.09	11360	0.47	0.32	0.24	3624	0.43	0.30	0.23	1061	0.71	0.30	0.19	803	0.64	0.20	0.12
AML	0	0.00	0.00	0.00	18722	0.90	0.88	0.87	3900	0.90	0.92	0.94	3554	0.90	0.82	0.75	3943	0.92	0.91	0.90
baselineAltLabel	2559	0.86	0.76	0.68	13451	0.88	0.89	0.89	3165	0.87	0.90	0.93	1661	0.92	0.74	0.62	2535	0.92	0.91	0.90
baselineLabel	1864	0.90	0.69	0.56	10489	0.95	0.85	0.77	2517	0.98	0.91	0.84	1194	0.95	0.67	0.52	2142	0.95	0.89	0.84
DOME	0	0.00	0.00	0.00	12360	0.91	0.88	0.84	2632	0.93	0.91	0.89	1966	0.91	0.82	0.74	2623	0.93	0.91	0.89
FCAMap-KG	2603	0.84	0.73	0.65	12474	0.92	0.88	0.85	3008	0.94	0.91	0.89	2004	0.90	0.79	0.70	2564	0.93	0.89	0.86
LogMap	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
LogMapBio	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
LogMapKG	41070	0.24	0.35	0.71	54499	0.43	0.58	0.87	15529	0.47	0.63	0.94	15323	0.58	0.68	0.82	19531	0.27	0.43	0.96
LogMapLt	0	0.00	0.00	0.00	16665	0.90	0.83	0.77	3550	0.94	0.89	0.84	2795	0.91	0.75	0.63	3605	0.91	0.89	0.87
POMAP++	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00
Wiktionary	1832	0.86	0.68	0.56	9709	0.92	0.79	0.69	2056	0.94	0.86	0.80	1520	0.92	0.74	0.63	2301	0.92	0.88	0.84

Runtime

AGM had the highest runtimes followed by LogMapBio, and LogMapKG. LogMapLt were quite fast and returned class as well as instance matches. POMAP++ is also fast but returns only class mappings.

	marvelcinematicuniverse-marvel	memoryalpha-memorybeta	memoryalpha-stexpanded	starwars-swg	starwars-swtor
AGM	2:59:49	1:58:09	2:59:35	1:29:48	1:20:17
AML	0:00:00	0:28:05	0:05:15	0:06:27	0:05:59
baselineAltLabel	0:02:57	0:01:53	0:01:11	0:02:51	0:02:56
baselineLabel	0:03:03	0:02:07	0:01:12	0:02:53	0:03:15
DOME	0:00:00	0:08:45	0:05:45	0:28:29	0:22:27
FCAMap-KG	0:10:56	0:11:08	0:05:20	0:23:46	0:23:39
LogMap	0:03:57	0:02:09	0:01:20	0:04:12	0:04:05
LogMapBio	0:23:31	0:36:58	0:40:36	0:23:56	0:26:00
LogMapKG	1:14:09	0:17:52	0:11:44	0:30:30	0:11:59
LogMapLt	0:00:00	0:01:29	0:00:58	0:02:31	0:02:30
POMAP++	0:04:23	0:01:58	0:01:15	0:03:37	0:03:26
Wiktionary	0:05:43	0:02:41	0:01:44	0:04:55	0:05:11

Organizers

Sven Hertling (University of Mannheim, Germany), main contact for the track, sven at informatik dot uni-mannheim dot de
Heiko Paulheim (University of Mannheim, Germany)

References

[1] Sven Hertling, Heiko Paulheim: DBkWik: A Consolidated Knowledge Graph from Thousands of Wikis. International Conference on Big Knowledge 2018. [pdf]

[2] Alexandra Hofmann, Samresh Perchani, Jan Portisch, Sven Hertling, and Heiko Paulheim. DBkWik: Towards Knowledge Graph Creation from Thousands of Wikis. International Semantic Web Conference (Posters & Demos) 2017. [pdf]