Ontology Alignment Evaluation Initiative - SEALS evaluation for OAEI 2018 Campaign

SEALS evaluation for OAEI 2018

Instructions for organisers

Creating a new track for SEALS is very simple. One only needs to create a metada file where the source and target ontologies are defined. Additionally, if available, a reference alignment can be attached to each pair of sorice-target ontologies. The metadata file can also describe several matching tasks (i.e. several pairs of source-target ontologies). See example of the HP-MP phenotype task and the metadata file of the largebio track.

Instructions for participants

Participants have to follow this procedure:

  1. Register your matching tool. Participants must register their intention to participate in the OAEI 2018 by filling this online form.
  2. Wrap the current version of your matching tool. This step is described in detail in a tutorial (see Section below). It requires to create a small java class that acts as a bridge. Further, you have to put the libraries used by your tool in a certain folder structure.
  3. Test your tool with the data sets available for each track. Ids of data sets for testing your tool prior to the evaluation process are given in each track web page. Participants can test their tools with the SEALS client on those data-sets. It is convenient that participants (locally) test their wrapped tool way before the deadline. We can also give you support and perform a dry run in our machines.
  4. Prepare, wrap and upload the final version of your matching tool. Submit a zipped SEALS package (e.g., LogMap.zip) using this form (requires a google account and a valid email).

Once these steps have been conducted, we run all systems on the SEALS infrastructure and generate the results.

Tutorial on Tool Wrapping

We have prepared a comprehensive PDF-tutorial for wrapping an ontology matching tool and testing your tool. You can download it here:

In the tutorial there are mentioned several additional materials that are available here.

LogMap system can also be used as reference LogMap-seals-oaei-2017.zip

We have also prepared a tutorial to communicate with the Oracle in the interactive track:

We detected some problems in the past when using libraries in conflict with the libraries used by the SEALS client. We encourage when possible to use the same versions as the SEALS client. Please check the maven dependencies of the SEALS client here. Note that you must not include the SEALS client jar file within your wrapped system.

We also encourage developers to use the Alignment API. For developers using it, the following ant package is available for packaging and validating the wrapped tools:

Within the Tutorial we show how you can use your wrapped tool to run locally a full evaluation. Thus, you can compute precision and recall for all of those testsuites listed in the track web pages at any time in your development process.

Contacts

Do not hesitate to contact Daniel Faria (daniel [.] faria [.] 81 [at] gmail [.] com) and Ernesto Jimenez-Ruiz (ernesto [.] jimenez [.] ruiz [at] gmail [.] com) for any question you may have.

Acknowledgement

While developing and improving the tutorial, we have been in contact with several matching tool developer to have some 'reference' matcher for testing our tutorial and the client that comes with it. Thanks go out to Hua Wei Khong Watson (Eff2match), Peigang Xu (Falcon-OA), Faycal Hamdi (Taxomap), Peng Wang (Lily), Zhichun Wang (RiMOM), and Cosmin Stroe (AgreementMaker).