A lot of Ontology Matching systems have been developed over the last years. After several years of experience, the results can only be slightly improved in terms of the alignment quality (precision/recall resp. F-Measure). Based on this insight, it is clear that fully automatic ontology matching systems slowly reach an upper bound of the results they can achieve. By incorporating user interaction, we expect to improve the alignments even further and to push this upper boundary. Semi-automatic ontology matching approaches are quite promising since humans can really help the systems, for example by detecting incorrect correspondences.
Whenever the user is directly involved, all required efforts of the human have taken into account and it has to be in an appropriate proportion to the result. Thus, beside the quality of the alignment, other measures like the number of interactions are interesting and meaningful to decide which matching system is best suitable for a certain matching task. By now, all OAEI tracks focus on fully automatic matching and semi-automatic matching is not evaluated although such systems already exist, e.g. LogMap2 (Jiménez-Ruiz et al., 2011) As long as the evaluation of such systems is not driven forward, it is hardly possible to systematically compare the quality of interactive matching approaches. With this new track, we like to change this unfavorable situation by explicitly offering a systematic, automated evaluation of matching systems with user interaction.
For the first edition of the interactive track, we use the well-established OAEI Conference data set. This data set covers 16 ontologies describing the domain of conference organization. Over the last years, the quality of the generated alignments has been constantly increased but only to small amount (by a few percent). In 2012, the best system according to F-Measure (YAM++) achieves a value of 70% (Aguirre et al., 2012). This shows that there is significant room for improvement, which could be filled by interactive means. Moreover, the Conference set has a suitable size such that most of the systems can participate and do not run into problems concerning the run time or memory consumption.
The interactive client works exactly the same way as the usual client. It only includes one additional Class "Oracle" in the "eu.sealsproject.omt.client.interactive" package. This class provides the method "check" which takes two strings and a relation as input (uri1, uri2, relation) which are the URIs of two concepts and the relation can be one of these three: EQUIVALENCE, SUBSUMES, SUBSUMED_BY. The method returns true, if the correspondence between these entities holds and false otherwise.
An example call: Oracle.check("http://cmt#Paper", "http://ekaw#Paper", Oracle.Relation.EQUIVALENCE)
You do not need to include the Oracle-class in your system (as library) but then your system might indicate that the class cannot be found. However, when starting the client itself, the class should be loaded without problems. Whenever it is not an interactive track, which is encoded in the name of the SEALS repository name, it does not allow any interactions and the method cannot be called. To check whether the track is interactive, you can call the method Oracle.isInteractive(). This method returns a boolean value, TRUE if it is an interactive track and false otherwise.
One goal of this track is to show in general that the exploitation user interaction is able to further improve the results of ontology matching systems in terms of F-measure. Furthermore, we like to see which semi-automatic methods exist, which ones perform best, and which ones need the smallest amount of interactions, i.e., make best use of the scarce resource of users' time. Beside the amount of user interactions, the type of the interaction and the involvement time is interesting. Do matching systems involve the user interaction before or during the process? Do they ask the user only to verify single correspondences or complete alignments? Altogether, we aim to promote the development of semi-automatic ontology matching systems and methods to overcome the limitations which are caused by fully automatic techniques. Furthermore, the track will encourage a discussion of different interactive matching techniques as well as a set of relevant interaction primitives.
Since the reference alignment of the Conference set is already available, we can use this to check the quality of the created alignments. Moreover, we can use this reference alignment to simulate a user. Whenever the system requires a user interaction, e.g. to verify a correspondence, the reference alignment acts as oracle, see (Paulheim et al., 2013). Thus, it is not necessary to take “real” users into account, but it is nevertheless possible to evaluate the systems according to their interactive component. Of course, to realize this evaluation, the systems need to provide methods to automatically react to the interaction requests. However, the interface is kept quite minimal, demanding only the implementation of a few additional methods. It is ipart of the SEALS plugin, so the matching systems only need to be slightly adapted. A further description can be found in the section about the client.
Aguirre, J. L., Eckert, K., Euzenat, J., & et al. (2012). Results of the Ontology Alignment Evaluation Initiative 2012. Proceedings of the 7th Ontology Matching Workshop, (S. 73-115).
Jiménez-Ruiz, E., Grau, B. C., & Yujiao, Z. (2011). LogMap 2.0: towards logic-based, scalable and interactive ontology matching. Proceedings of the 4th International Workshop on Semantic Web Applications and Tools for the Life Sciences, (S. 45-46).
Paulheim, H., Hertling, S., & Ritze, D. (2013). Towards Evaluating Interactive Ontology Matching Tools. Proceedings of the 10th Extended Semantic Web Conference, (S. 31-45).