threedb.evaluators.base_evaluator

Provides an abstract base class for implementing evaluators.

See threedb.evaluators.classification.SimpleClassificationEvaluator for further details.

class threedb.evaluators.base_evaluator.BaseEvaluator

Bases: abc.ABC

Abstract Base Class for Evaluators

An evaluator is in charge of taking the output of an inference model, analyzing it, and returning the relevant summary outputs (e.g., correctness for classification, IoU for detection, MSE for segmentation, etc.)

output_type: str = ''
KEYS: List[str] = ['is_correct', 'loss']
abstract get_segmentation_label(model_uid: str) → int

Given a model_uid, return a scalar label corresponding to the 3D model. This label is only used for the purpopses of generating a segmentation map. If the only goal is to separate the object from its background, this function can return anything greater than zero.

Parameters

model_uid (str) – Unique identifier for the model.

Returns

An integer which will be the color of the model in the segmentation map.

Return type

int

abstract get_target(model_uid: str, render_output: Dict[str, torch.Tensor]) → Any

Given a model_uid, returns the corresponding label for that 3D model, which will be used to evaluate accuracy, loss, and other statistics.

Parameters

  • model_uid (str) – The unique id of the model, as returned by threedb.rendering.base_renderer.BaseRenderer.get_model() (for blender, this will be the object ID)

  • render_output (Dict[str, Tensor]) – The dictionary containing the results of the renderer (as returned by threedb.rendering.base_renderer.BaseRenderer.render())

Returns

The class label for that object, most likely looked up through the evaluator metadata.

Return type

int

abstract summary_stats(pred: Any, label: Any, input_shape: List[int]) → Dict[str, Union[bool, int, float, str, torch.Tensor]]

Given a tensor corresponding to a model prediction on a single rendered image, and a tensor containing the label for that image, return a dictionary of summary statistics. The keys of the dictionary should match the KEYS static property of the concrete class.

Parameters

  • pred (Any) – The output of the model being inspected with 3DB.

  • label (Any) – The label corresponding to the rendered image.

Returns

A dictionary mapping each key declared in type(self).KEYS to a serializable value (int, str, float, or bool).

Return type

Dict[str, Output]

abstract declare_outputs() → Dict[str, Tuple[List[int], str]]

This function declares what the output of render() will be, based on the renderer settings. Returns a dictionary mapping keys to (dtype, size) tuples—the output of render() is string-to-tensor dictionary whose tensors will be checked against the return value of this function for both size and type.

A basic implementation which suffices for most applications is provided in the abstract class threedb.rendering.base_renderer.BaseRenderer.