Modules¶
Evalutils¶
- class evalutils.evalutils.Algorithm(*, index_key='input_image', file_loaders=None, file_filters=None, input_path=PosixPath('/input'), output_path=PosixPath('/output/images'), file_sorter_key=None, validators=None, output_file=PosixPath('/output/results.json'))[source]¶
- __init__(*, index_key='input_image', file_loaders=None, file_filters=None, input_path=PosixPath('/input'), output_path=PosixPath('/output/images'), file_sorter_key=None, validators=None, output_file=PosixPath('/output/results.json'))[source]¶
The base class for all algorithms. Sets the environment and controls the flow of the processing once process is called.
- Parameters
index_key (
str
) – Fileloader key which must be used for the index. Default: input_imagefile_loaders (
Optional
[Dict
[str
,FileLoader
]]) – The loaders that will be used to get all files. Default: evalutils.io.SimpleITKLoader for input_imagefile_filters (
Optional
[Dict
[str
,Optional
[Pattern
[str
]]]]) – Regular expressions for filtering certain FileLoaders. Default: no filtering.input_path (
Path
) – The path in the container where the ground truth will be loaded from. Default: /inputoutput_path (
Path
) – The path in the container where the output images will be written. Default: /output/imagesfile_sorter_key (
Optional
[Callable
]) – A function that determines how files in the input_path are sorted. Default: None (alphanumerical)validators (
Optional
[Dict
[str
,Tuple
[DataFrameValidator
,...
]]]) – A dictionary containing the validators that will be used on the loaded data per file_loader key. Default: evalutils.validators.UniqueImagesValidator for input_imageoutput_file (
PathLike
) – The path to the location where the results will be written. Default: /output/results.json
- class evalutils.evalutils.BaseEvaluation(*, ground_truth_path=PosixPath('/opt/evaluation/ground-truth'), predictions_path=PosixPath('/input'), file_sorter_key=<function first_int_in_filename_key>, file_loader, validators, join_key=None, aggregates=None, output_file=PosixPath('/output/metrics.json'))[source]¶
- __init__(*, ground_truth_path=PosixPath('/opt/evaluation/ground-truth'), predictions_path=PosixPath('/input'), file_sorter_key=<function first_int_in_filename_key>, file_loader, validators, join_key=None, aggregates=None, output_file=PosixPath('/output/metrics.json'))[source]¶
The base class for all evaluations. Sets the environment and controls the flow of the evaluation once evaluate is called.
- Parameters
ground_truth_path (
Path
) – The path in the container where the ground truth will be loaded frompredictions_path (
Path
) – The path in the container where the submission will be loaded fromfile_sorter_key (
Callable
) – A function that determines how files are sorted and matched togetherfile_loader (
FileLoader
) – The loader that will be used to get all filesvalidators (
Tuple
[DataFrameValidator
,...
]) – A tuple containing all the validators that will be used on the loaded datajoin_key (
Optional
[str
]) – The column that will be used to join the predictions and ground truth tablesaggregates (
Optional
[Set
[str
]]) – The set of aggregates that will be calculated by pandas.DataFrame.describeoutput_file (
PathLike
) – The path to the location where the results will be written
- class evalutils.evalutils.ClassificationAlgorithm(*, index_key='input_image', file_loaders=None, file_filters=None, input_path=PosixPath('/input'), output_path=PosixPath('/output/images'), file_sorter_key=None, validators=None, output_file=PosixPath('/output/results.json'))[source]¶
- class evalutils.evalutils.ClassificationEvaluation(*, ground_truth_path=PosixPath('/opt/evaluation/ground-truth'), predictions_path=PosixPath('/input'), file_sorter_key=<function first_int_in_filename_key>, file_loader, validators, join_key=None, aggregates=None, output_file=PosixPath('/output/metrics.json'))[source]¶
ClassificationEvaluations have the same number of predictions as the number of ground truth cases. These can be things like, what is the stage of this case, or segment some things in this case.
- class evalutils.evalutils.DetectionAlgorithm(*, index_key='input_image', file_loaders=None, file_filters=None, input_path=PosixPath('/input'), output_path=PosixPath('/output/images'), file_sorter_key=None, validators=None, output_file=PosixPath('/output/results.json'))[source]¶
- class evalutils.evalutils.DetectionEvaluation(*args, detection_radius, detection_threshold, **kwargs)[source]¶
DetectionEvaluations have a different number of predictions from the number of ground truth annotations. An example would be detecting lung nodules in a CT volume, or malignant cells in a pathology slide.
- __init__(*args, detection_radius, detection_threshold, **kwargs)[source]¶
The base class for all evaluations. Sets the environment and controls the flow of the evaluation once evaluate is called.
- Parameters
ground_truth_path – The path in the container where the ground truth will be loaded from
predictions_path – The path in the container where the submission will be loaded from
file_sorter_key – A function that determines how files are sorted and matched together
file_loader – The loader that will be used to get all files
validators – A tuple containing all the validators that will be used on the loaded data
join_key – The column that will be used to join the predictions and ground truth tables
aggregates – The set of aggregates that will be calculated by pandas.DataFrame.describe
output_file – The path to the location where the results will be written
- class evalutils.evalutils.Evaluation(*args, **kwargs)[source]¶
Legacy class, you should use ClassificationEvaluation instead.
- __init__(*args, **kwargs)[source]¶
The base class for all evaluations. Sets the environment and controls the flow of the evaluation once evaluate is called.
- Parameters
ground_truth_path – The path in the container where the ground truth will be loaded from
predictions_path – The path in the container where the submission will be loaded from
file_sorter_key – A function that determines how files are sorted and matched together
file_loader – The loader that will be used to get all files
validators – A tuple containing all the validators that will be used on the loaded data
join_key – The column that will be used to join the predictions and ground truth tables
aggregates – The set of aggregates that will be calculated by pandas.DataFrame.describe
output_file – The path to the location where the results will be written
IO¶
- class evalutils.io.CSVLoader[source]¶
- load(*, fname)[source]¶
Tries to load the file given by the path fname.
Notes
For this to work with the validators you must:
If you load an image it must save the hash in the hash column
If you reference a Path it must be saved in the path column
- Parameters
fname (
Path
) – The file that the loader will try to load- Return type
A list containing all of the cases in this file
- Raises
FileLoaderError – If a file cannot be loaded as the specified type
- class evalutils.io.FileLoader[source]¶
- abstract load(*, fname)[source]¶
Tries to load the file given by the path fname.
Notes
For this to work with the validators you must:
If you load an image it must save the hash in the hash column
If you reference a Path it must be saved in the path column
- Parameters
fname (
Path
) – The file that the loader will try to load- Return type
A list containing all of the cases in this file
- Raises
FileLoaderError – If a file cannot be loaded as the specified type
- class evalutils.io.ImageIOLoader[source]¶
- class evalutils.io.ImageLoader[source]¶
A specialised file loader for images. As images are large they will not all be loaded into memory, so score_case needs to load them again later via load_image.
- static hash_image(image)[source]¶
Generates a hash of the image
- Parameters
image – The image to hash
- Return type
The hash of the image
- load(*, fname)[source]¶
Tries to load the file given by the path fname.
Notes
For this to work with the validators you must:
If you load an image it must save the hash in the hash column
If you reference a Path it must be saved in the path column
- Parameters
fname (
Path
) – The file that the loader will try to load- Return type
A list containing all of the cases in this file
- Raises
FileLoaderError – If a file cannot be loaded as the specified type
Validators¶
- class evalutils.validators.ExpectedColumnNamesValidator(*, expected, extra_cols_check=True)[source]¶
- __init__(*, expected, extra_cols_check=True)[source]¶
Validates that the DataFrame has the expected columns
- Parameters
expected (
Tuple
[str
,...
]) – The expected columns in the DataFrameextra_cols_check (
bool
) – Perform the check for extra columns, default is true but you may want to disable this if you’re sure that extra columns can be ignored.
- Raises
ValueError – If no columns are defined
- class evalutils.validators.NumberOfCasesValidator(*, num_cases)[source]¶
- class evalutils.validators.UniqueImagesValidator[source]¶
Validates that each image in the set is unique
Scorers¶
- class evalutils.scorers.DetectionScore(true_positives, false_negatives, false_positives)¶
- property false_negatives¶
Alias for field number 1
- property false_positives¶
Alias for field number 2
- property true_positives¶
Alias for field number 0
- evalutils.scorers.find_hits_for_targets(*, targets, predictions, radius)[source]¶
Generates a list of the predicted points that are within a radius r of the targets. The indicies are returned in sorted order, from closest to farthest point.
- Parameters
targets (
List
[Tuple
[float
,...
]]) – A list of target pointspredictions (
List
[Tuple
[float
,...
]]) – A list of predicted pointsradius (
float
) – The maximum distance that two points can be apart for them to be considered a hit
- Return type
List
[Tuple
[int
,...
]]- Returns
A list which has the same length as the targets list. Each element within
this list contains another list that contains the indicies of the
predictions that are considered hits.
- evalutils.scorers.score_detection(*, ground_truth, predictions, radius=1.0)[source]¶
Generates the number of true positives, false positives and false negatives for the ground truth points given the predicted points.
If multiple predicted points hit one ground truth point then this is considered as 1 true positive, and 0 false negatives.
If one predicted point is a hit for N ground truth points then this is considered as 1 true positive, and N-1 false negatives.
- Parameters
ground_truth (
List
[Tuple
[float
,...
]]) – A list of the ground truth pointspredictions (
List
[Tuple
[float
,...
]]) – A list of the predicted pointsradius (
float
) – The maximum distance that two points can be separated by in order to be considered a hit
- Return type
- Returns
A tuple containing the number of true positives, false positives and
false negatives.
Annotations¶
- class evalutils.annotations.BoundingBox(*, x1, x2, y1, y2)[source]¶
- __init__(*, x1, x2, y1, y2)[source]¶
A bounding box is a face defined by 4 edges on a 2D plane. It must have a non-zero width and height.
- Parameters
x1 (
float
) – Left edge of the bounding boxx2 (
float
) – Right edge of the bounding boxy1 (
float
) – Bottom edge of the bounding boxy2 (
float
) – Top edge of the bounding box
- Raises
ValueError – If the bounding box has zero width or height
- property area: float¶
Return the area of the bounding box in natural units
- Return type
float
- intersection(*, other)[source]¶
Calculates the intersection area between this bounding box and another, axis aligned, bounding box.
- Parameters
other (
BoundingBox
) – The other bounding box- Returns
The intersection area in natural units if the two bounding boxes overlap, zero otherwise.
- Return type
float
- jaccard_index(*, other)[source]¶
Calculates the intersection over union between this bounding box and a second, axis aligned, bounding box.
- Parameters
other (
BoundingBox
) – The other bounding box- Returns
The intersection over union in natural units
- Return type
float
- union(*, other)[source]¶
Calculates the union between this bounding box and another, axis aligned, bounding box.
- Parameters
other (
BoundingBox
) – The other bounding box- Returns
The union area in natural units
- Return type
float
Statistics¶
- class evalutils.stats.HausdorffMeasures(distance, modified_distance, percentile_distance)¶
- property distance¶
Alias for field number 0
- property modified_distance¶
Alias for field number 1
- property percentile_distance¶
Alias for field number 2
- evalutils.stats.absolute_volume_difference(s1, s2, voxelspacing=None)[source]¶
Calculate absolute volume difference from s2 to s1
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else. s1 is taken to be the reference.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.voxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.
- Return type
float
- Returns
The absolute volume difference between the object(s) in
input1
and the object(s) in
input2
. This is a percentage value in therange \([0, +inf]\) for which a \(0\) denotes an ideal score.
Notes
This is a real metric
- evalutils.stats.accuracies_from_confusion_matrix(cm)[source]¶
Computes accuracy scores from a confusion matrix
- Parameters
cm (
ndarray
) – N x N Input confusion matrix- Return type
1d ndarray containing accuracy scores for all N classes
- evalutils.stats.calculate_confusion_matrix(y_true, y_pred, labels)[source]¶
Efficient confusion matrix calculation, based on sklearn interface
- Parameters
y_true (
ndarray
) – Target multi-object segmentation masky_pred (
ndarray
) – Predicted multi-object segmentation masklabels (
List
[int
]) – Inclusive list of N labels to compute the confusion matrix for.
- Return type
N x N confusion matrix for Y_pred w.r.t. Y_true
Notes
By definition a confusion matrix \(C\) is such that \(C_{i, j}\) is equal to the number of observations known to be in group \(i\) but predicted to be in group \(j\).
- evalutils.stats.dice_from_confusion_matrix(cm)[source]¶
Computes Dice scores from a confusion matrix
- Parameters
cm (
ndarray
) – N x N Input confusion matrix- Return type
1d ndarray containing Dice scores for all N classes
- evalutils.stats.dice_to_jaccard(dice)[source]¶
Conversion computation from Dice to Jaccard
- Parameters
dice (
ndarray
) – 1 or N Dice values within [0 .. 1]- Return type
1 or N Jaccard values within [0 .. 1]
- evalutils.stats.distance_transform_edt_float32(input, sampling=None, return_distances=True, return_indices=False, distances=None, indices=None)[source]¶
Memory efficient version of scipy.ndimage.distance_transform_edt
The same as scipy.ndimage.distance_transform_edt but using float32 and better memory cleaning internally.
In addition to the distance transform, the feature transform can be calculated. In this case the index of the closest background element is returned along the first axis of the result.
- Parameters
input – Input data to transform. Can be any type but will be converted into binary: 1 wherever input equates to True, 0 elsewhere.
sampling – Spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.
return_distances – Whether to return distance matrix. At least one of return_distances/return_indices must be True. Default is True.
return_indices – Whether to return indices matrix. Default is False.
distances – Used for output of distance array, must be of type float64.
indices – Used for output of indices, must be of type int32.
- Returns
Either distance matrix, index matrix, or a list of the two, depending on
return_x
flags anddistance
andindices
input parameters.- Return type
distance_transform_edt
Notes
The euclidean distance transform gives values of the euclidean distance:
n y_i = sqrt(sum (x[i]-b[i])**2) i
where b[i] is the background point (value 0) with the smallest Euclidean distance to input points x[i], and n is the number of dimensions.
Copyright (C) 2003-2005 Peter J. Verveer
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
The name of the author may not be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE AUTHOR ‘’AS IS’’ AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
- evalutils.stats.hausdorff_distance(s1, s2, voxelspacing=None, connectivity=1, edt_method=<function distance_transform_edt_float32>)[source]¶
Computes the (symmetric) Hausdorff Distance (HD) between the binary objects in two images. It is defined as the maximum surface distance between the objects.
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.voxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.connectivity (
int
) – The neighbourhood/connectivity considered when determining the surface of the binary objects. This value is passed to scipy.ndimage.generate_binary_structure and should usually be \(> 1\).edt_method (
Callable
) – Method used for computing the euclidean distance transform. By default it uses a variant on the scipy.ndimage.distance_transform_edt method that uses float32 data to reduce memory costs at the cost of some additional compute time.
- Return type
float
- Returns
The symmetric Hausdorff Distance between the object(s) in
`s1`
andthe object(s) in
`s2`
. The distance unit is the same as for thespacing of elements along each dimension, which is usually given in mm.
Notes
This is a real metric. Implementation inspired by medpy.metric.binary http://pythonhosted.org/MedPy/_modules/medpy/metric/binary.html
- evalutils.stats.hausdorff_distance_measures(s1, s2, voxelspacing=None, connectivity=1, percentile=0.95, edt_method=<function distance_transform_edt_float32>)[source]¶
Returns multiple Hausdorff measures - (hd, modified_hd, percentile_hd) Since measures share common calculations, together the measures can be calculated more efficiently
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.voxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.connectivity (
int
) – The neighbourhood/connectivity considered when determining the surface of the binary objects. This value is passed to scipy.ndimage.generate_binary_structure and should usually be \(> 1\).percentile (
float
) – The percentile at which to calculate the Hausdorff Distanceedt_method (
Callable
) – Method used for computing the euclidean distance transform. By default it uses a variant on the scipy.ndimage.distance_transform_edt method that uses float32 data to reduce memory costs at the cost of some additional compute time.
- Return type
- Returns
The hausdorff distance, modified hausdorff distance and percentile
hausdorff distance
Notes
This returns real metrics.
- evalutils.stats.jaccard_from_confusion_matrix(cm)[source]¶
Computes Jaccard scores from a confusion matrix a.k.a. intersection over union (IoU)
- Parameters
cm (
ndarray
) – N x N Input confusion matrix- Return type
1d ndarray containing Jaccard scores for all N classes
- evalutils.stats.jaccard_to_dice(jacc)[source]¶
Conversion computation from Jaccard to Dice
- Parameters
jacc (
ndarray
) – 1 or N Jaccard values within [0 .. 1]- Return type
1 or N Dice values within [0 .. 1]
- evalutils.stats.mean_contour_distance(s1, s2, voxelspacing=None, edt_method=<function distance_transform_edt_float32>)[source]¶
Computes the (symmetric) Mean Contour Distance between the binary objects in two images. It is defined as the maximum average surface distance between the objects.
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.voxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.edt_method (
Callable
) – Method used for computing the euclidean distance transform. By default it uses a variant on the scipy.ndimage.distance_transform_edt method that uses float32 data to reduce memory costs at the cost of some additional compute time.
- Return type
float
- Returns
The symmetric Mean Contour Distance between the object(s) in
`s1`
and the object(s) in
`s2`
. The distance unit is the same as for thespacing of elements along each dimension, which is usually given in mm.
Notes
This is a real metric that mimics the ITK MeanContourDistanceFilter.
- evalutils.stats.modified_hausdorff_distance(s1, s2, voxelspacing=None, connectivity=1, edt_method=<function distance_transform_edt_float32>)[source]¶
Computes the (symmetric) Modified Hausdorff Distance (MHD) between the binary objects in two images. It is defined as the maximum average surface distance between the objects.
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.voxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.connectivity (
int
) – The neighbourhood/connectivity considered when determining the surface of the binary objects. This value is passed to scipy.ndimage.generate_binary_structure and should usually be \(> 1\).edt_method (
Callable
) – Method used for computing the euclidean distance transform. By default it uses a variant on the scipy.ndimage.distance_transform_edt method that uses float32 data to reduce memory costs at the cost of some additional compute time.
- Return type
float
- Returns
The symmetric Modified Hausdorff Distance between the object(s) in
`s1`
and the object(s) in`s2`
. The distance unit is the sameas for the spacing of elements along each dimension, which is usually
given in mm.
Notes
This is a real metric.
- evalutils.stats.percentile_hausdorff_distance(s1, s2, percentile=0.95, voxelspacing=None, connectivity=1, edt_method=<function distance_transform_edt_float32>)[source]¶
Nth Percentile Hausdorff Distance.
Computes a percentile for the (symmetric) Hausdorff Distance between the binary objects in two images. It is defined as the maximum surface distance between the objects at the nth percentile.
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.percentile (
Union
[int
,float
]) – The percentile to perform the comparison on the two sorted distance setsvoxelspacing (
Union
[Tuple
[Union
[float
,int
],...
],List
[Union
[float
,int
]],float
,int
,None
]) – The voxelspacing in a distance unit i.e. spacing of elements along each dimension. If a sequence, must be of length equal to the input rank; if a single number, this is used for all axes. If not specified, a grid spacing of unity is implied.connectivity (
int
) – The neighbourhood/connectivity considered when determining the surface of the binary objects. This value is passed to scipy.ndimage.generate_binary_structure and should usually be \(> 1\).edt_method (
Callable
) – Method used for computing the euclidean distance transform. By default it uses a variant on the scipy.ndimage.distance_transform_edt method that uses float32 data to reduce memory costs at the cost of some additional compute time.
- Return type
float
- Returns
The maximum Percentile Hausdorff Distance between the object(s) in
`s1`
and the object(s) in`s2`
at the`percentile`
percentile.
The distance unit is the same as for the spacing of elements along each
dimension, which is usually given in mm.
See also
hd()
Notes
This is a real metric.
- evalutils.stats.relative_absolute_volume_difference(s1, s2)[source]¶
Calculate relative absolute volume difference from s2 to s1
- Parameters
s1 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else. s1 is taken to be the reference.s2 (
ndarray
) – Input data containing objects. Can be any type but will be converted into binary: background where 0, object everywhere else.
- Return type
float
- Returns
The relative absolute volume difference between the object(s) in
input1
and the object(s) ininput2
. This is a percentage valuein the range \([0, +inf]\) for which a \(0\) denotes an ideal
score.
Notes
This is not a real metric! it is asymmetric.
ROC¶
- class evalutils.roc.BootstrappedCIPointError(mean_fprs, mean_tprs, low_tpr_vals, high_tpr_vals, low_fpr_vals, high_fpr_vals)[source]¶
- property high_fpr_vals¶
Alias for field number 5
- property high_tpr_vals¶
Alias for field number 3
- property low_fpr_vals¶
Alias for field number 4
- property low_tpr_vals¶
Alias for field number 2
- property mean_fprs¶
Alias for field number 0
- property mean_tprs¶
Alias for field number 1
- class evalutils.roc.BootstrappedROCCICurves(fpr_vals, mean_tpr_vals, low_tpr_vals, high_tpr_vals, low_az_val, high_az_val)[source]¶
- property fpr_vals¶
Alias for field number 0
- property high_az_val¶
Alias for field number 5
- property high_tpr_vals¶
Alias for field number 3
- property low_az_val¶
Alias for field number 4
- property low_tpr_vals¶
Alias for field number 2
- property mean_tpr_vals¶
Alias for field number 1
- evalutils.roc.average_roc_curves(roc_curves, bins=200)[source]¶
Averages ROC curves using vertical averaging (fixed FP rates), which gives a 1D measure of variability.
- Parameters
curves – List of BootstrappedROCCICurves to be averaged
(optional) (bins) – Number of false-positives to iterate over. (Default: 200)
- Returns
ROC class containing the average over all ROCs.
- Return type
- evalutils.roc.get_bootstrapped_ci_point_error(y_score, y_true, num_bootstraps=100, ci_to_use=0.95, exclude_first_last=True)[source]¶
Produces Confidence-Interval errors for individual points from ROC Useful when only few ROC points exist so they will be plotted individually e.g. when range of score values in y_score is very small (e.g. manual observer scores)
Note that this method only works by analysing the cloud of boostrapped points generatedfor a particular threshold value. A fixed number of threshold values is essential. Therefore the scores in y_score must be from a fixed discrete set of values, eg. [1,2,3,4,5]
Bootstrapping is done by selecting len(y_score) samples randomly (with replacement) from y_score and y_true. This is done num_boostraps times.
- Parameters
y_score (
ndarray
) – The scores produced by the system being evaluated. A discrete set of possible scores must be used.y_true (
ndarray
) – The true labels (1 or 0) which are the reference standard being usednum_bootstraps (integer) – How many times to make a random sample with replacement
ci_to_use (
float
) – Which confidence interval is required.exclude_first_last (
bool
) – The first and last ROC point (0,0 and 1,1) are usually irrelevant in these scenarios where only a few ROC points will be individually plotted. Set this to true to ignore these first and last points.
- Return type
- Returns
mean_fprs – The array of mean fpr values (1 per possible ROC point)
mean_tprs – The array of mean tpr values (1 per possible ROC point)
low_tpr_vals – The tpr vals (one per ROC point) representing lowest val in CI
high_tpr_vals – The tpr vals (one per ROC point) representing the highest val in CI
low_fpr_vals – The fpr vals (one per ROC point) representing lowest val in CI_to_use
high_fpr_vals – The fpr vals (one per ROC point) representing the highest val in CI
- evalutils.roc.get_bootstrapped_roc_ci_curves(y_pred, y_true, num_bootstraps=100, ci_to_use=0.95)[source]¶
Produces Confidence-Interval Curves to go alongside a regular ROC curve This is done by using boostrapping. Bootstrapping is done by selecting len(y_pred) samples randomly (with replacement) from y_pred and y_true. This is done num_boostraps times.
- Parameters
y_pred (
ndarray
) – The predictions (scores) produced by the system being evaluatedy_true (
ndarray
) – The true labels (1 or 0) which are the reference standard being usednum_bootstraps (
int
) – How many times to make a random sample with replacementci_to_use (
float
) – Which confidence interval is required.
- Return type
- Returns
fpr_vals – An equally spaced set of fpr vals between 0 and 1
mean_tpr_vals – The mean tpr vals (one per fpr_val) obtained by boostrapping
low_tpr_vals – The tpr vals (one per fpr_val) representing lower curve for CI
high_tpr_vals – The tpr vals (one per fpr_val) representing the upper curve for CI
low_Az_val – The lower Az (AUC) val for the given CI_to_use
high_Az_val – The higher Az (AUC) val for the given CI_to_use