auswahl.benchmarking.util.metrics.PairwiseStabilityScore¶
- class auswahl.benchmarking.util.metrics.PairwiseStabilityScore(metric_name: str)[source]¶
The class provides the infrastructure for the introduction of new symmetric and pairwise defined stability metrics.
- add_stabilities(pod: DataHandler)¶
Conducts the evaluation of the stability metric across all datasets and methods in the
DataHandlerobject, which is extended with the results of the stability evaluation.- Parameters
- pod: DataHandler
instance of
DataHandlercontaining the results of the benchmarking procedure
- evaluate_stability(meta_data: dict, selections: array, features: FeatureDescriptor)[source]¶
Conducts the stability evaluation of a set of executions of a selector algorithm on one dataset with a specific feature configuration under different data splits and seeds
- Parameters
- meta_data: dict
information about the data set, which might be relevant for stability calculations. See
get_meta()for the contained data- selections: np.ndarray
The selected features of the different executions of the selector algorithm as integer indices of features. Shape (#executions, #features to select)
- features: FeatureDescriptor
FeatureDescriptor describing the configuration of features to be selected
- Returns
- stability: float
- abstract pairwise_sim_func(meta_data: dict, set_1: ndarray, set_2: ndarray) float[source]¶
Function calculating the stability score for a single pair of selections of features.
- Parameters
- meta_data: dict
Dict containing meta information about the dataset for which the stability metric is evaluated. See the documentation of
get_meta()for the available data.- set_1: np.nadarray
array of integer indices of selected features of shape (n_features_to_select,)
- set_2: np.nadarray
array of integer indices of selected features of shape (n_features_to_select,)
- Returns
- stability score for the given pair of selections: float
