Reader Studies

A reader study enables you to have a set of readers answer a set of questions about a set of images.

Editors

You can add multiple editors to your reader study. An editor is someone who can edit the reader study settings, add other editors, add and remove readers, add images and edit questions.

Readers

A user who can read this study, creating an answer for each question and image in the study.

Cases

The set of images that will be used in the study.

Hanging List

How the each image will be presented to the user as a set of hanging protocols. For instance, you might want to present two images side by side and have a reader answer a question about both, or overlay one image on another.

Creating a Reader Study

A ReaderStudy can use any available Workstation. A WorkstationConfig can also be used for the study to customise the default appearance of the workstation.

Cases

Cases can be added to a reader study by adding Image instances. Multiple image formats are supported:

  • .mha

  • .mhd with the accompanying .zraw or .raw file

  • .tif/.tiff

  • .jpg/.jpeg

  • .png

  • 3D/4D DICOM support is also available, though this is experimental and not guaranteed to work on all .dcm images.

Defining the Hanging List

When you upload a set of images you have the option to automatically generate the default hanging list. The default hanging list presents each reader with 1 image per protocol.

You are able to customise the hanging list in the study edit page. Here, you are able to assign multiple images and overlays to each protocol.

Available image ports are: * main * secondary * tertiary * quaternary * quinary * senary * septenary * octonary * nonary * denary

Overlays can be applied to the image ports by using the image-port name with the suffix ‘-overlay’ (e.g. main-overlay).

Questions

A Question can be optional and the following answer_type options are available:

  • Heading (not answerable)

  • Bool

  • Single line text

  • Multiline text

The following annotation answer types are also available:

  • Distance measurement

  • Multiple distance measurements

  • 2D bounding box

To use an annotation answer type you must also select the image port where the annotation will be made.

Adding Ground Truth

To monitor the performance of the readers you are able add ground truth to a reader study by uploading a csv file.

If ground truth has been added to a ReaderStudy, any Answer given by a reader is evaluated by applying the scoring_function chosen for the Question.

The scores can then be compared on the leaderboard. Statistics are also available based on these scores: the average and total scores for each question as well as for each case are displayed in the statistics view.

class grandchallenge.reader_studies.models.Answer(*args, **kwargs)[source]

An Answer can be provided to a Question that is a part of a ReaderStudy.

Parameters

  • id (UUIDField) – Id

  • modified (DateTimeField) – Modified

  • created (DateTimeField) – Created

  • creator (ForeignKey to User) – Creator

  • question (ForeignKey to Question) – Question

  • display_set (ForeignKey to DisplaySet) – Display set

  • answer (JSONField) – Answer

  • answer_image (ForeignKey to Image) – Answer image

  • is_ground_truth (BooleanField) – Is ground truth

  • score (FloatField) – Score

  • explanation (TextField) – Explanation

  • last_edit_duration (DurationField) – Last edit duration

  • total_edit_duration (DurationField) – Total edit duration

exception DoesNotExist
exception MultipleObjectsReturned
property api_url

API url for this Answer.

calculate_score(ground_truth)[source]

Calculate the score for this Answer based on ground_truth.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

save_without_historical_record(*args, **kwargs)

Save model without saving a historical record

Make sure you know what you’re doing before you use this method.

static validate(*, creator, question, answer, display_set, is_ground_truth=False, instance=None)[source]

Validates all fields provided for answer.

class grandchallenge.reader_studies.models.AnswerType(value)[source]

An enumeration.

class grandchallenge.reader_studies.models.CategoricalOption(id, question, title, default)[source]
Parameters

  • id (AutoField) – Id

  • question (ForeignKey to Question) – Question

  • title (CharField) – Title

  • default (BooleanField) – Default

exception DoesNotExist
exception MultipleObjectsReturned
class grandchallenge.reader_studies.models.DisplaySet(id, created, modified, reader_study, order)[source]
Parameters

  • id (UUIDField) – Id

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • reader_study (ForeignKey to ReaderStudy) – Reader study

  • order (PositiveSmallIntegerField) – Order

  • values (ManyToManyField) – Values

exception DoesNotExist
exception MultipleObjectsReturned
property api_url

API url for this DisplaySet.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

workstation_url

The URL to answer this display set in a workstation

class grandchallenge.reader_studies.models.Question(id, created, modified, overlay_segments, look_up_table, reader_study, question_text, help_text, answer_type, image_port, required, direction, scoring_function, order, interface)[source]
Parameters

  • id (UUIDField) – Id

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • overlay_segments (JSONField) – Overlay segments. The schema that defines how categories of values in the overlay images are differentiated.

  • look_up_table (ForeignKey to LookUpTable) – Look up table. The look-up table that is applied when an overlay image is first shown

  • reader_study (ForeignKey to ReaderStudy) – Reader study

  • question_text (TextField) – Question text

  • help_text (TextField) – Help text

  • answer_type (CharField) – Answer type

  • image_port (CharField) – Image port

  • required (BooleanField) – Required

  • direction (CharField) – Direction

  • scoring_function (CharField) – Scoring function

  • order (PositiveSmallIntegerField) – Order

  • interface (ForeignKey to ComponentInterface) – Interface

class AnswerType(value)

An enumeration.

class Direction(value)[source]

An enumeration.

exception DoesNotExist
class ImagePort(value)[source]

An enumeration.

exception MultipleObjectsReturned
class ScoringFunction(value)[source]

An enumeration.

property api_url

API url for this Question.

calculate_score(answer, ground_truth)[source]

Calculates the score for answer by applying scoring_function to answer and ground_truth.

clean()[source]

Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.

is_answer_valid(*, answer)[source]

Validates answer against ANSWER_TYPE_SCHEMA.

property is_fully_editable

True if no Answer has been given for this Question.

property read_only_fields

question_text, answer_type, image_port, required if this Question is fully editable, an empty list otherwise.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

class grandchallenge.reader_studies.models.ReaderStudy(*args, **kwargs)[source]

Reader Study model.

A reader study is a tool that allows users to have a set of readers answer a set of questions on a set of images (cases).

Parameters

  • title (CharField) – Title

  • description (TextField) – Description

  • slug (AutoSlugField) – Slug

  • id (UUIDField) – Id

  • created (DateTimeField) – Created

  • modified (DateTimeField) – Modified

  • view_content (JSONField) – View content

  • editors_group (OneToOneField to Group) – Editors group

  • readers_group (OneToOneField to Group) – Readers group

  • workstation (ForeignKey to Workstation) – Workstation

  • workstation_config (ForeignKey to WorkstationConfig) – Workstation config

  • hanging_protocol (ForeignKey to HangingProtocol) – Hanging protocol

  • public (BooleanField) – Public. Should this reader study be visible to all users on the overview page? This does not grant all users permission to read this study. Users will still need to be added to the study’s readers group in order to do that.

  • access_request_handling (CharField) – Access request handling. How would you like to handle access requests?

  • logo (JPEGField) – Logo

  • social_image (JPEGField) – Social image. An image for this reader study which is displayed when you post the link on social media. Should have a resolution of 640x320 px (1280x640 px for best display).

  • help_text_markdown (TextField) – Help text markdown

  • shuffle_hanging_list (BooleanField) – Shuffle hanging list

  • is_educational (BooleanField) – Is educational. If checked, readers get the option to verify their answers against the uploaded ground truth. This also means that the uploaded ground truth will be readily available to the readers.

  • case_text (JSONField) – Case text

  • allow_answer_modification (BooleanField) – Allow answer modification. If true, readers are allowed to modify their answers for a case by navigating back to previous cases. ‘Allow case navigation’ must be checked as well to enable this setting.

  • allow_case_navigation (BooleanField) – Allow case navigation. If true, readers are allowed to navigate back and forth between cases in this reader study.

  • allow_show_all_annotations (BooleanField) – Allow show all annotations. If true, readers are allowed to show/hide all annotations for a case.

  • roll_over_answers_for_n_cases (PositiveSmallIntegerField) – Roll over answers for n cases. The number of cases for which answers should roll over. It can be used for repeated readings with slightly different hangings. For instance, if set to 1. Case 2 will start with the answers from case 1; whereas case 3 starts anew but its answers will rollover to case 4.

  • publications (ManyToManyField) – Publications. The publications associated with this reader study

  • modalities (ManyToManyField) – Modalities. The imaging modalities contained in this reader study

  • structures (ManyToManyField) – Structures. The structures contained in this reader study

  • organizations (ManyToManyField) – Organizations. The organizations associated with this reader study

  • actor_actions (GenericRelation) – Actor actions

  • target_actions (GenericRelation) – Target actions

  • action_object_actions (GenericRelation) – Action object actions

exception DoesNotExist
exception MultipleObjectsReturned
add_editor(user)[source]

Adds user as an editor for this ReaderStudy.

add_ground_truth(*, data, user)[source]

Add ground truth answers provided by data for this ReaderStudy.

add_reader(user)[source]

Adds user as a reader for this ReaderStudy.

answerable_question_count

The number of answerable questions for this ReaderStudy.

answerable_questions

All questions for this ReaderStudy except those with answer type heading.

clean()[source]

Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.

get_progress_for_user(user)[source]

Returns the percentage of completed hangings and questions for user.

property help_text

The cleaned help text from the markdown sources

property image_groups

Names of the images as they are grouped in the hanging list.

is_editor(user)[source]

Checks if user is an editor for this ReaderStudy.

is_reader(user)[source]

Checks if user is a reader for this ReaderStudy.

leaderboard

The leaderboard for this ReaderStudy.

remove_editor(user)[source]

Removes user as an editor for this ReaderStudy.

remove_reader(user)[source]

Removes user as a reader for this ReaderStudy.

save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

score_for_user(user)[source]

Returns the average and total score for answers given by user.

scores_by_user

The average and total scores for this ReaderStudy grouped by user.

statistics

Statistics per question and case based on the total / average score.

study_image_names

Names for all images added to this ReaderStudy.

class grandchallenge.reader_studies.models.ReaderStudyPermissionRequest(*args, **kwargs)[source]

When a user wants to read a reader study, editors have the option of reviewing each user before accepting or rejecting them. This class records the needed info for that.

Parameters

  • id (AutoField) – Id

  • user (ForeignKey to User) – User. which user requested to participate?

  • created (DateTimeField) – Created

  • changed (DateTimeField) – Changed

  • status (CharField) – Status

  • reader_study (ForeignKey to ReaderStudy) – Reader study. To which reader study has the user requested access?

  • rejection_text (TextField) – Rejection text. The text that will be sent to the user with the reason for their rejection.

  • actor_actions (GenericRelation) – Actor actions

  • target_actions (GenericRelation) – Target actions

  • action_object_actions (GenericRelation) – Action object actions

exception DoesNotExist
exception MultipleObjectsReturned
save(*args, **kwargs)[source]

Save the current instance. Override this in a subclass if you want to control the saving process.

The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.

grandchallenge.reader_studies.models.delete_reader_study_groups_hook(*_, instance, using, **__)[source]

Deletes the related groups.

We use a signal rather than overriding delete() to catch usages of bulk_delete.