Ontology Alignment Evaluation Initiative - SEALS evaluation for OAEI 2018 Campaign

SEALS evaluation for OAEI 2018

Instructions for organisers

Creating a new track for SEALS is very simple. One only needs to create a metada file where the source and target ontologies are defined. Additionally, if available, a reference alignment can be attached to each pair of sorice-target ontologies. The metadata file can also describe several matching tasks (i.e. several pairs of source-target ontologies). See example of the HP-MP phenotype task and the metadata file of the largebio track.

Instructions for participants

Participants have to follow this procedure:

1. Register your matching tool. Participants must register their intention to participate in the OAEI 2018 by filling this online form.
2. Wrap the current version of your matching tool. This step is described in detail in a tutorial (see Section below). It requires to create a small java class that acts as a bridge. Further, you have to put the libraries used by your tool in a certain folder structure.
3. Test your tool with the data sets available for each track. Ids of data sets for testing your tool prior to the evaluation process are given in each track web page. Participants can test their tools with the SEALS client on those data-sets. It is convenient that participants (locally) test their wrapped tool way before the deadline. We can also give you support and perform a dry run in our machines.
4. Prepare, wrap and upload the final version of your matching tool. Submit a zipped SEALS package (e.g., LogMap.zip) using this form (requires a google account and a valid email).

Once these steps have been conducted, we run all systems on the SEALS infrastructure and generate the results.

• Download Tutorial (May 2017) and MatcherBridge.java presented in the appendix.

In the tutorial there are mentioned several additional materials that are available here.

• Download demomatcher-package.zip for LINUX (example of a tool correctly packaged)
  or download demomatcher-package.zip for WINDOWS
   
• Download seals-omt-validator.jar (validates the structure and content of the zipped tool package)
• Download latest seals-omt-client.jar (interfaces and client to execute the final tool package)

LogMap system can also be used as reference LogMap-seals-oaei-2017.zip

We have also prepared a tutorial to communicate with the Oracle in the interactive track:

• The Oracle tutorial (May 2017) include details about how matching systems can use the Oracle class for Interactive Matching.

We detected some problems in the past when using libraries in conflict with the libraries used by the SEALS client. We encourage when possible to use the same versions as the SEALS client. Please check the maven dependencies of the SEALS client here. Note that you must not include the SEALS client jar file within your wrapped system.

We also encourage developers to use the Alignment API. For developers using it, the following ant package is available for packaging and validating the wrapped tools:

• Download package-template.zip Use 'ant pack' to generate the tool package.

Within the Tutorial we show how you can use your wrapped tool to run locally a full evaluation. Thus, you can compute precision and recall for all of those testsuites listed in the track web pages at any time in your development process.

Contacts

Do not hesitate to contact Daniel Faria (daniel [.] faria [.] 81 [at] gmail [.] com) and Ernesto Jimenez-Ruiz (ernesto [.] jimenez [.] ruiz [at] gmail [.] com) for any question you may have.

Acknowledgement

While developing and improving the tutorial, we have been in contact with several matching tool developer to have some 'reference' matcher for testing our tutorial and the client that comes with it. Thanks go out to Hua Wei Khong Watson (Eff2match), Peigang Xu (Falcon-OA), Faycal Hamdi (Taxomap), Peng Wang (Lily), Zhichun Wang (RiMOM), and Cosmin Stroe (AgreementMaker).