Ontology Alignment Evaluation Initiative - HOBBIT evaluation for OAEI 2020 Campaign

HOBBIT evaluation for OAEI 2020

Instructions for organisers

OAEI track organisers should follow the HOBBIT benchmark specification.

These benchmark definitions can be used as reference:

Instructions for participants

System developers willing to participate in the HOBBIT tracks are required to follow different instructions with respect to the SEALS wrapping.

Check the following examples to submit a HOBBIT compliant system:

In addition, Sven Hertling (sven [at] informatik [dot] uni-mannheim [dot] de) has implemented a framework to create SEALS and HOBBIT compliant matching systems. Check details here and/or get in touch with him. It also contains a small client which allows to evaluate the matcher locally (i.e., it downloads the benchmarks and computes the KPIs).

The standalone software library called the HOBBIT Java SDK may also enable an easier way to implement a benchmark or a system. It will also make possible to run local tests before submitting to the platform.

Participants have to follow this procedure:

  1. Register your matching tool. Participants must register their intention to participate in the OAEI 2020 by filling this online form.
  2. Wrap the current version of your matching tool. Follow the instruction in the examples above and push the docker images to the HOBBIT platform (a personal access token is required).
  3. Test your tool with the data sets available. Test your system (locally when possible) and remotely using HOBBIT's public instance. More details in the HOBBIT wiki pages.