Reader Studies
A reader study enables you to have a set of readers answer a set of questions about a set of images.
- Editors
You can add multiple editors to your reader study. An editor is someone who can edit the reader study settings, add other editors, add and remove readers, add images and edit questions.
- Readers
A user who can read this study, creating an answer for each question and image in the study.
- Cases
The set of images that will be used in the study.
- Hanging List
How the each image will be presented to the user as a set of hanging protocols. For instance, you might want to present two images side by side and have a reader answer a question about both, or overlay one image on another.
Creating a Reader Study
A ReaderStudy
can use any available Workstation
.
A WorkstationConfig
can also be used for the study to customise the default
appearance of the workstation.
Cases
Cases can be added to a reader study by adding Image
instances.
Multiple image formats are supported:
.mha
.mhd
with the accompanying.zraw
or.raw
file.tif
/.tiff
.jpg
/.jpeg
.png
3D/4D DICOM support is also available, though this is experimental and not guaranteed to work on all
.dcm
images.
Defining the Hanging List
When you upload a set of images you have the option to automatically generate the default hanging list. The default hanging list presents each reader with 1 image per protocol.
You are able to customise the hanging list in the study edit page. Here, you are able to assign multiple images and overlays to each protocol.
Available image ports are:
* main
* secondary
* tertiary
* quaternary
* quinary
* senary
* septenary
* octonary
* nonary
* denary
Overlays can be applied to the image ports by using the image-port name with
the suffix ‘-overlay’ (e.g. main-overlay
).
Questions
A Question
can be optional and the following answer_type
options are
available:
Heading (not answerable)
Bool
Single line text
Multiline text
The following annotation answer types are also available:
Distance measurement
Multiple distance measurements
2D bounding box
To use an annotation answer type you must also select the image port where the annotation will be made.
Adding Ground Truth
To monitor the performance of the readers you are able add ground truth to a reader study by uploading a csv file.
If ground truth has been added to a ReaderStudy
, any Answer
given by a
reader is evaluated by applying the scoring_function
chosen for the Question
.
The scores can then be compared on the leaderboard
. Statistics are also available
based on these scores: the average and total scores for each question as well
as for each case are displayed in the statistics
view.
- class grandchallenge.reader_studies.models.Answer(*args, **kwargs)[source]
An
Answer
can be provided to aQuestion
that is a part of aReaderStudy
.- Parameters
id (UUIDField) – Id
modified (DateTimeField) – Modified
created (DateTimeField) – Created
creator (ForeignKey to
User
) – Creatorquestion (ForeignKey to
Question
) – Questiondisplay_set (ForeignKey to
DisplaySet
) – Display setanswer (JSONField) – Answer
answer_image (ForeignKey to
Image
) – Answer imageis_ground_truth (BooleanField) – Is ground truth
score (FloatField) – Score
explanation (TextField) – Explanation
last_edit_duration (DurationField) – Last edit duration
total_edit_duration (DurationField) – Total edit duration
- exception DoesNotExist
- exception MultipleObjectsReturned
- property api_url
API url for this
Answer
.
- save(*args, **kwargs)[source]
Save the current instance. Override this in a subclass if you want to control the saving process.
The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.
- save_without_historical_record(*args, **kwargs)
Save model without saving a historical record
Make sure you know what you’re doing before you use this method.
- class grandchallenge.reader_studies.models.CategoricalOption(id, question, title, default)[source]
- Parameters
id (AutoField) – Id
question (ForeignKey to
Question
) – Questiontitle (CharField) – Title
default (BooleanField) – Default
- exception DoesNotExist
- exception MultipleObjectsReturned
- class grandchallenge.reader_studies.models.DisplaySet(id, created, modified, reader_study, order)[source]
- Parameters
id (UUIDField) – Id
created (DateTimeField) – Created
modified (DateTimeField) – Modified
reader_study (ForeignKey to
ReaderStudy
) – Reader studyorder (PositiveSmallIntegerField) – Order
values (ManyToManyField) – Values
- exception DoesNotExist
- exception MultipleObjectsReturned
- property api_url
API url for this
DisplaySet
.
- save(*args, **kwargs)[source]
Save the current instance. Override this in a subclass if you want to control the saving process.
The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.
- workstation_url
The URL to answer this display set in a workstation
- class grandchallenge.reader_studies.models.Question(id, created, modified, overlay_segments, look_up_table, reader_study, question_text, help_text, answer_type, image_port, required, direction, scoring_function, order, interface)[source]
- Parameters
id (UUIDField) – Id
created (DateTimeField) – Created
modified (DateTimeField) – Modified
overlay_segments (JSONField) – Overlay segments. The schema that defines how categories of values in the overlay images are differentiated.
look_up_table (ForeignKey to
LookUpTable
) – Look up table. The look-up table that is applied when an overlay image is first shownreader_study (ForeignKey to
ReaderStudy
) – Reader studyquestion_text (TextField) – Question text
help_text (TextField) – Help text
answer_type (CharField) – Answer type
image_port (CharField) – Image port
required (BooleanField) – Required
direction (CharField) – Direction
scoring_function (CharField) – Scoring function
order (PositiveSmallIntegerField) – Order
interface (ForeignKey to
ComponentInterface
) – Interface
- class AnswerType(value)
An enumeration.
- exception DoesNotExist
- exception MultipleObjectsReturned
- property api_url
API url for this
Question
.
- calculate_score(answer, ground_truth)[source]
Calculates the score for
answer
by applyingscoring_function
toanswer
andground_truth
.
- clean()[source]
Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.
- property is_fully_editable
True
if noAnswer
has been given for thisQuestion
.
- property read_only_fields
question_text
,answer_type
,image_port
,required
if thisQuestion
is fully editable, an empty list otherwise.
- save(*args, **kwargs)[source]
Save the current instance. Override this in a subclass if you want to control the saving process.
The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.
- class grandchallenge.reader_studies.models.ReaderStudy(*args, **kwargs)[source]
Reader Study model.
A reader study is a tool that allows users to have a set of readers answer a set of questions on a set of images (cases).
- Parameters
title (CharField) – Title
description (TextField) – Description
slug (AutoSlugField) – Slug
id (UUIDField) – Id
created (DateTimeField) – Created
modified (DateTimeField) – Modified
view_content (JSONField) – View content
editors_group (OneToOneField to
Group
) – Editors groupreaders_group (OneToOneField to
Group
) – Readers groupworkstation (ForeignKey to
Workstation
) – Workstationworkstation_config (ForeignKey to
WorkstationConfig
) – Workstation confighanging_protocol (ForeignKey to
HangingProtocol
) – Hanging protocolpublic (BooleanField) – Public. Should this reader study be visible to all users on the overview page? This does not grant all users permission to read this study. Users will still need to be added to the study’s readers group in order to do that.
access_request_handling (CharField) – Access request handling. How would you like to handle access requests?
logo (JPEGField) – Logo
social_image (JPEGField) – Social image. An image for this reader study which is displayed when you post the link on social media. Should have a resolution of 640x320 px (1280x640 px for best display).
help_text_markdown (TextField) – Help text markdown
shuffle_hanging_list (BooleanField) – Shuffle hanging list
is_educational (BooleanField) – Is educational. If checked, readers get the option to verify their answers against the uploaded ground truth. This also means that the uploaded ground truth will be readily available to the readers.
case_text (JSONField) – Case text
allow_answer_modification (BooleanField) – Allow answer modification. If true, readers are allowed to modify their answers for a case by navigating back to previous cases. ‘Allow case navigation’ must be checked as well to enable this setting.
allow_case_navigation (BooleanField) – Allow case navigation. If true, readers are allowed to navigate back and forth between cases in this reader study.
allow_show_all_annotations (BooleanField) – Allow show all annotations. If true, readers are allowed to show/hide all annotations for a case.
roll_over_answers_for_n_cases (PositiveSmallIntegerField) – Roll over answers for n cases. The number of cases for which answers should roll over. It can be used for repeated readings with slightly different hangings. For instance, if set to 1. Case 2 will start with the answers from case 1; whereas case 3 starts anew but its answers will rollover to case 4.
publications (ManyToManyField) – Publications. The publications associated with this reader study
modalities (ManyToManyField) – Modalities. The imaging modalities contained in this reader study
structures (ManyToManyField) – Structures. The structures contained in this reader study
organizations (ManyToManyField) – Organizations. The organizations associated with this reader study
actor_actions (GenericRelation) – Actor actions
target_actions (GenericRelation) – Target actions
action_object_actions (GenericRelation) – Action object actions
- exception DoesNotExist
- exception MultipleObjectsReturned
- add_ground_truth(*, data, user)[source]
Add ground truth answers provided by
data
for thisReaderStudy
.
- answerable_question_count
The number of answerable questions for this
ReaderStudy
.
- answerable_questions
All questions for this
ReaderStudy
except those with answer type heading.
- clean()[source]
Hook for doing any extra model-wide validation after clean() has been called on every field by self.clean_fields. Any ValidationError raised by this method will not be associated with a particular field; it will have a special-case association with the field defined by NON_FIELD_ERRORS.
- get_progress_for_user(user)[source]
Returns the percentage of completed hangings and questions for
user
.
- property help_text
The cleaned help text from the markdown sources
- property image_groups
Names of the images as they are grouped in the hanging list.
- leaderboard
The leaderboard for this
ReaderStudy
.
- save(*args, **kwargs)[source]
Save the current instance. Override this in a subclass if you want to control the saving process.
The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.
- scores_by_user
The average and total scores for this
ReaderStudy
grouped by user.
- statistics
Statistics per question and case based on the total / average score.
- study_image_names
Names for all images added to this
ReaderStudy
.
- class grandchallenge.reader_studies.models.ReaderStudyPermissionRequest(*args, **kwargs)[source]
When a user wants to read a reader study, editors have the option of reviewing each user before accepting or rejecting them. This class records the needed info for that.
- Parameters
id (AutoField) – Id
user (ForeignKey to
User
) – User. which user requested to participate?created (DateTimeField) – Created
changed (DateTimeField) – Changed
status (CharField) – Status
reader_study (ForeignKey to
ReaderStudy
) – Reader study. To which reader study has the user requested access?rejection_text (TextField) – Rejection text. The text that will be sent to the user with the reason for their rejection.
actor_actions (GenericRelation) – Actor actions
target_actions (GenericRelation) – Target actions
action_object_actions (GenericRelation) – Action object actions
- exception DoesNotExist
- exception MultipleObjectsReturned
- save(*args, **kwargs)[source]
Save the current instance. Override this in a subclass if you want to control the saving process.
The ‘force_insert’ and ‘force_update’ parameters can be used to insist that the “save” must be an SQL insert or update (or equivalent for non-SQL backends), respectively. Normally, they should not be set.