Results for OAEI 2018 - Knowledge Graph Track

Matching systems

We executed all systems submitted as SEALS packages to OAEI (even if they are not registered for the track) first on a very small matching example (dataset) with a similar structure and shape like the real knowledge graphs (in fact they are a small subset of them). It shows that not all matching systems are able to complete this small task due to exceptions or other failures. The following matching systems produce an exception:

ALOD2Vec
FCAMapX
EVOCROS
XMap

We wrote an email to the system implementers to possibly modify the system configuration to complete this task. Unfortunately we were not be able to get the matchers running. The following matchers produce an empty alignment or state that the system is not able to complete this task:

CANARD (only registered for COMPLEX track)
Lily (stated that it can not support this task)
ALIN (only registered for INTERACTIVE track)

Using a bigger KG task we further encounter systems not being able to complete. KEPLER returned with a "NegativeArraySizeException". Thus we executed the following systems:

AML
POMap++
Hontology
DOME
LogMap (in its KG version)
LogMapBio
LogMapLt
String baseline (seals package, source code)

Experimental setting

The evaluation is executed on a virtual machine(VM) with 32GB of RAM and 16 vCPUs (2.4 GHz). The operating system is debian 9 with openjdk version "1.8.0_181".

We executed it not on HOBBIT because not enough matchers registered in HOBBIT for this task. Just AML, DOME, LogMap, Lily, SANOM, RADON and Silk are registered in the HOBBIT evaluation platform. The first four have a SEALS counterpart which can be used for the evaluation. The latter two are only registered for the "Link discovery" track. SANOM only registered for "Anatomy" and "Conference" track of HOBBIT. Thus we have to stick to the SEALS evaluation routine.

We used the "-o" option in SEALS (version 7.0.5) to provide the two knowledge graphs which should be matched. The two given URLs are file URLs and not HTTP URLs because downloading the knowledge graphs from another server would be a huge time overhead (downloading 500 MB multiple times for different tasks). Thus the call of the SEALS client looks like the following:


java -Xmx25g -Xms15g -jar ../seals-omt-client.jar ${MATCHER_DIR} -z -o file:///data/ont1.xml file:///data/ont2.xml -f out.xml

This also means that the reported time (obtained with linux time command) also includes the environment preparation of SEALS (copying the configuration folder of each matcher at the start of the matching task) as well as copying the resulting alignment to the output file. We could not use the "-x" option of SEALS because we had to modify the evaluation routine based on two reasons. The first one is to differentiate between class, property and instance mappings. The second reason is to deal with the partial gold standard of this track.

The alignments were evaluated based on Precision, Recall and F-Measure for classes, properties and instances (each in isolation). Our partial gold standard contained 1:1 mappings as well as mappings in the form of which represent that resourceA has no correspondence in the second knowledge graph. This allows to increase the value of false positives if the matcher nevertheless finds a mapping (e.g. maps resourceA to resourceB). We further assume that in each knowledge graph only one representation of the concept exist. This means if we have the mapping in our gold standard we can count the mapping as a false positive (the assumption here is that in the seconds knowledge graph no similar concept to B exists). The value of false negatives is only increased if we have a 1:1 mapping and it is not found by a matcher. The whole source code for generating the evaluation results is also uploaded.

Results overview

Aggregated results per matcher, divided into class, property, instance an overall mappings. Time is measured in seconds. #tracks indicates the number of tracks where the tool is able to generate (non empty) alignments. Size indicates the average of the number of generated correspondences for the tracks where an (non empty) alignment has been generated. Two kinds of results are reported : those do not distinguishing empty and erroneous (or not generated) alignments and those -- indicated between parenthesis -- considering only non empty generated alignments.
			class				property				instance				overall
System	Time	#tracks	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
AML	88448	5	11.6	0.85 (0.85)	0.64 (0.87)	0.51 (0.88)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	82380.9	0.16 (0.16)	0.23 (0.26)	0.38 (0.63)	102471.1	0.19 (0.19)	0.23 (0.28)	0.31 (0.52)
POMAP++	438	9	15.1	0.79 (0.79)	0.74 (0.74)	0.69 (0.69)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	16.9	0.79 (0.79)	0.14 (0.14)	0.08 (0.08)
Holontology	318	9	16.8	0.80 (0.80)	0.83 (0.83)	0.87 (0.87)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	18.8	0.80 (0.80)	0.17 (0.17)	0.10 (0.10)
DOME	13747	9	16.0	0.73 (0.73)	0.73 (0.73)	0.73 (0.73)	207.3	0.86 (0.86)	0.84 (0.84)	0.81 (0.81)	15688.7	0.61 (0.61)	0.61 (0.61)	0.61 (0.61)	15912.0	0.68 (0.68)	0.68 (0.68)	0.67 (0.67)
LogMap	14083	7	21.7	0.66 (0.66)	0.77 (0.80)	0.91 (1.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	97081.4	0.08 (0.08)	0.14 (0.15)	0.81 (0.93)	97104.8	0.09 (0.09)	0.16 (0.16)	0.64 (0.74)
LogMapBio	2340	9	22.1	0.68 (0.68)	0.81 (0.81)	1.00 (1.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	24.1	0.68 (0.68)	0.19 (0.19)	0.11 (0.11)
LogMapLt	500	6	22.0	0.61 (0.61)	0.72 (0.76)	0.87 (1.00)	0.0	0.00 (0.00)	0.00 (0.00)	0.00 (0.00)	82388.3	0.39 (0.39)	0.52 (0.56)	0.76 (0.96)	88893.1	0.42 (0.42)	0.49 (0.54)	0.60 (0.77)
Baseline	412	9	18.9	0.75 (0.75)	0.79 (0.79)	0.84 (0.84)	213.8	0.86 (0.86)	0.84 (0.84)	0.82 (0.82)	17743.3	0.59 (0.59)	0.69 (0.69)	0.82 (0.82)	17976.0	0.65 (0.65)	0.73 (0.73)	0.82 (0.82)

The following matchers could output results on all 9 tracks: POMAP++, Holontology, DOME, LogMapBio and the baseline. AML was running out of time(12 hours) on some tracks (heykidscomics~dc is currently running and not finished yet). LogMap needed more than the given 32 GB RAM for the bigger knowledge graphs. LogMapLt created alignment files bigger than 1GB (we stopped the matcher because it produced more than 50 GB in some runs).

Track specific results

Overall results

This result table shows the overall performance (without dividing into class, property or instance) of the matchers.

	AML				POMAP++				Holontology				DOME				LogMap				LogMapBio				LogMapLt				Baseline
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
darkscape~oldschoolrunescape	91122	0.17	0.26	0.59	37	0.80	0.20	0.11	42	0.75	0.22	0.13	8997	0.59	0.59	0.58	32535	0.23	0.36	0.77	57	0.55	0.24	0.15	71711	0.45	0.57	0.77	16028	0.52	0.67	0.92
runescape~darkscape	587247	0.46	0.46	0.45	54	0.85	0.20	0.11	62	0.93	0.25	0.14	19022	0.68	0.68	0.68	96084	0.12	0.20	0.88	76	0.83	0.26	0.15	575371	0.48	0.62	0.89	19970	0.77	0.78	0.80
runescape~oldschoolrunescape	0	0.00	0.00	0.00	41	0.67	0.18	0.11	47	0.62	0.22	0.13	19796	0.61	0.57	0.54	198968	0.08	0.15	0.76	66	0.59	0.27	0.17	123638	0.31	0.45	0.80	28846	0.51	0.62	0.78
heykidscomics~dc	0	0.00	0.00	0.00	4	1.00	0.05	0.03	4	1.00	0.10	0.05	54026	0.68	0.72	0.76	0	0.00	0.00	0.00	4	1.00	0.10	0.05	0	0.00	0.00	0.00	54877	0.74	0.78	0.84
marvel~dc	0	0.00	0.00	0.00	4	1.00	0.11	0.06	4	1.00	0.21	0.12	2980	0.70	0.76	0.82	0	0.00	0.00	0.00	4	1.00	0.21	0.12	0	0.00	0.00	0.00	2546	0.74	0.78	0.82
marvel~heykidscomics	0	0.00	0.00	0.00	5	1.00	0.11	0.06	4	1.00	0.11	0.06	17175	0.86	0.87	0.88	147241	0.07	0.13	0.59	4	1.00	0.11	0.06	0	0.00	0.00	0.00	16945	0.89	0.90	0.91
memory-alpha~memory-beta	120854	0.20	0.31	0.62	3	0.00	0.00	0.00	2	0.00	0.00	0.00	13695	0.77	0.73	0.69	333864	0.06	0.11	0.62	2	0.00	0.00	0.00	19031	0.53	0.58	0.66	14483	0.72	0.72	0.72
memory-alpha~stexpanded	39919	0.08	0.14	0.44	2	0.00	0.00	0.00	2	0.00	0.00	0.00	3297	0.76	0.74	0.72	21301	0.02	0.04	0.44	2	0.00	0.00	0.00	4650	0.42	0.43	0.44	3552	0.78	0.78	0.78
memory-beta~stexpanded	83098	0.09	0.16	0.55	2	0.00	0.00	0.00	2	0.00	0.00	0.00	4220	0.65	0.70	0.75	43950	0.08	0.14	0.55	2	0.00	0.00	0.00	5637	0.56	0.53	0.50	4537	0.77	0.81	0.85

Class results

All matchers are able to generate class correspondences. Only the tracks darkscape~oldschoolrunescape, runescape~darkscape and runescape~oldschoolrunescape have enough classes to be meaningfully matched.

	AML				POMAP++				Holontology				DOME				LogMap				LogMapBio				LogMapLt				Baseline
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
darkscape~oldschoolrunescape	41	0.75	0.78	0.82	35	0.80	0.76	0.73	40	0.75	0.78	0.82	37	0.55	0.55	0.55	55	0.55	0.71	1.00	55	0.55	0.71	1.00	55	0.55	0.71	1.00	46	0.62	0.67	0.73
runescape~darkscape	60	0.93	0.93	0.93	52	0.85	0.79	0.73	60	0.93	0.93	0.93	54	0.87	0.87	0.87	74	0.83	0.91	1.00	74	0.83	0.91	1.00	80	0.68	0.81	1.00	62	0.87	0.87	0.87
runescape~oldschoolrunescape	0	0.00	0.00	0.00	39	0.67	0.64	0.62	45	0.62	0.69	0.77	41	0.62	0.62	0.62	64	0.59	0.74	1.00	64	0.59	0.74	1.00	63	0.59	0.74	1.00	50	0.65	0.73	0.85
heykidscomics~dc	0	0.00	0.00	0.00	2	1.00	0.67	0.50	2	1.00	1.00	1.00	3	1.00	1.00	1.00	0	0.00	0.00	0.00	2	1.00	1.00	1.00	0	0.00	0.00	0.00	3	1.00	1.00	1.00
marvel~dc	0	0.00	0.00	0.00	2	1.00	0.67	0.50	2	1.00	1.00	1.00	3	1.00	1.00	1.00	0	0.00	0.00	0.00	2	1.00	1.00	1.00	0	0.00	0.00	0.00	3	1.00	1.00	1.00
marvel~heykidscomics	0	0.00	0.00	0.00	3	1.00	1.00	1.00	2	1.00	1.00	1.00	3	1.00	1.00	1.00	2	1.00	1.00	1.00	2	1.00	1.00	1.00	0	0.00	0.00	0.00	3	1.00	1.00	1.00
memory-alpha~memory-beta	1	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00
memory-alpha~stexpanded	1	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00
memory-beta~stexpanded	1	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	1	0.00	0.00	0.00

Property results

Only DOME and the baseline could generate property mappings whereas most matchers do not return property mappings. The reason behind that is most probably that all properties are typed as rdf:Property and not subdivided into owl:DatatypeProperty and owl:ObjectProperty. But this cannot be done easily in a preprocessing step because the usage of the properties is rather not strict i.e., some properties are used both with literals and resources as their object.. Given a matcher which maps only DatatypeProperty to DatatypeProperty and ObjectProperty to ObjectProperty will not take care of the properties defined in such an ontology. An improvement of these matchers would be to include also the mapping of rdf:Properties in case no more types are defined.

	AML				POMAP++				Holontology				DOME				LogMap				LogMapBio				LogMapLt				Baseline
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
darkscape~oldschoolrunescape	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	339	0.92	0.89	0.86	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	341	1.00	0.96	0.93
runescape~darkscape	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	601	1.00	1.00	1.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	649	1.00	1.00	1.00
runescape~oldschoolrunescape	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	407	0.92	0.92	0.92	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	409	0.92	0.92	0.92
heykidscomics~dc	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	131	0.80	0.80	0.80	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	134	0.80	0.80	0.80
marvel~dc	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	50	0.88	0.88	0.88	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	50	0.88	0.88	0.88
marvel~heykidscomics	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	82	0.80	0.80	0.80	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	83	0.80	0.80	0.80
memory-alpha~memory-beta	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	91	0.50	0.37	0.30	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	91	0.43	0.35	0.30
memory-alpha~stexpanded	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	72	0.89	0.89	0.89	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	72	0.89	0.89	0.89
memory-beta~stexpanded	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	93	0.88	0.88	0.88	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	95	0.88	0.88	0.88

Instance results

AML, DOME, LogMap, LogMapLt and the baseline could return instance mappings.

	AML				POMAP++				Holontology				DOME				LogMap				LogMapBio				LogMapLt				Baseline
	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.	Size	Prec.	F-m.	Rec.
darkscape~oldschoolrunescape	57290	0.14	0.24	0.72	0	0.00	0.00	0.00	0	0.00	0.00	0.00	8621	0.51	0.51	0.50	32478	0.20	0.34	0.96	0	0.00	0.00	0.00	60423	0.43	0.59	0.96	15641	0.45	0.61	0.96
runescape~darkscape	533355	0.38	0.39	0.41	0	0.00	0.00	0.00	0	0.00	0.00	0.00	18367	0.60	0.60	0.60	96008	0.10	0.18	0.97	0	0.00	0.00	0.00	543740	0.45	0.62	0.99	19259	0.72	0.74	0.75
runescape~oldschoolrunescape	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	19348	0.52	0.47	0.43	198902	0.07	0.12	0.88	0	0.00	0.00	0.00	111200	0.27	0.42	0.94	28387	0.43	0.54	0.73
heykidscomics~dc	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	53892	0.62	0.67	0.72	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	54740	0.70	0.76	0.84
marvel~dc	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	2927	0.50	0.59	0.71	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	2493	0.56	0.63	0.71
marvel~heykidscomics	0	0.00	0.00	0.00	0	0.00	0.00	0.00	0	0.00	0.00	0.00	17090	0.87	0.89	0.91	147236	0.07	0.12	0.82	0	0.00	0.00	0.00	0	0.00	0.00	0.00	16859	0.91	0.93	0.95
memory-alpha~memory-beta	62065	0.20	0.33	0.95	0	0.00	0.00	0.00	0	0.00	0.00	0.00	13603	0.85	0.87	0.89	333862	0.06	0.12	0.95	0	0.00	0.00	0.00	16477	0.53	0.69	1.00	14391	0.82	0.88	0.95
memory-alpha~stexpanded	21664	0.08	0.15	0.89	0	0.00	0.00	0.00	0	0.00	0.00	0.00	3224	0.62	0.59	0.56	21299	0.02	0.04	0.89	0	0.00	0.00	0.00	4294	0.42	0.57	0.89	3479	0.67	0.67	0.67
memory-beta~stexpanded	67054	0.09	0.17	0.92	0	0.00	0.00	0.00	0	0.00	0.00	0.00	4126	0.53	0.59	0.67	43948	0.08	0.15	0.92	0	0.00	0.00	0.00	5361	0.56	0.67	0.83	4441	0.71	0.77	0.83

Runtime

AML had the highest runtimes followed by DOME and LogMap. POMAP++ and Holontology were quite fast but only return class mappings.

	AML	POMAP++	Holontology	DOME	LogMap	LogMapBio	LogMapLt	Baseline
	Time	Time	Time	Time	Time	Time	Time	Time
darkscape~oldschoolrunescape	7695	25	23	348	139	530	53	33
runescape~darkscape	28309	51	38	1173	364	597	288	50
runescape~oldschoolrunescape	0	51	35	1112	740	552	87	53
heykidscomics~dc	0	106	67	4901	0	167	0	97
marvel~dc	0	36	26	76	0	88	0	25
marvel~heykidscomics	0	90	61	1199	10522	167	0	76
memory-alpha~memory-beta	33148	31	26	1997	1207	77	33	33
memory-alpha~stexpanded	10270	24	21	1534	518	62	20	22
memory-beta~stexpanded	9026	24	21	1407	593	100	19	23

Organizers

Sven Hertling (University of Mannheim, Germany), main contact for the track, sven at informatik dot uni-mannheim dot de
Heiko Paulheim (University of Mannheim, Germany)

References

[1] Sven Hertling, Heiko Paulheim: DBkWik: A Consolidated Knowledge Graph from Thousands of Wikis. International Conference on Big Knowledge 2018. [pdf]

[2] Alexandra Hofmann, Samresh Perchani, Jan Portisch, Sven Hertling, and Heiko Paulheim. DBkWik: Towards Knowledge Graph Creation from Thousands of Wikis. International Semantic Web Conference (Posters & Demos) 2017. [pdf]