Welcome to evalutils’s documentation!

The automated evaluation system on Grand Challenge will run an instance of your evaluation docker container image on each new submission. The users submission will be extracted on /input/, and your container is expected to calculate all of the metrics for this submission, and write them to /output/metrics.json. If the metrics cannot be calculated, the container should write to stderr and the last line of this will be passed to the user so that they can debug their submission.

evalutils helps you do this by providing a package that helps you create a project structure, load and validate the submissions, score the submission, write the json file, and package this in a docker container image so that you can then upload it to Grand Challenge.

Indices and tables