auswahl.benchmarking.util.metrics.StabilityScore¶
- class auswahl.benchmarking.util.metrics.StabilityScore(metric_name: str)[source]¶
Base class for all stability scores useable by the benchmarking system
- Parameters
- metric_name: str, default=None
Unique Name of the metric. If no name is provided, the name of the class inheriting from this function is used
- add_stabilities(pod: DataHandler)[source]¶
Conducts the evaluation of the stability metric across all datasets and methods in the
DataHandlerobject, which is extended with the results of the stability evaluation.- Parameters
- pod: DataHandler
instance of
DataHandlercontaining the results of the benchmarking procedure
- abstract evaluate_stability(meta_data: dict, selections: array, features: FeatureDescriptor) float[source]¶
Conducts the stability evaluation of a set of executions of a selector algorithm on one dataset with a specific feature configuration under different data splits and seeds
- Parameters
- meta_data: dict
information about the data set, which might be relevant for stability calculations. See
get_meta()for the contained data- selections: np.ndarray
The selected features of the different executions of the selector algorithm as integer indices of features. Shape (#executions, #features to select)
- features: FeatureDescriptor
FeatureDescriptor describing the configuration of features to be selected
- Returns
- stability: float
