auswahl.benchmarking.util.metrics.ZucknickScore¶
- class auswahl.benchmarking.util.metrics.ZucknickScore(correlation_threshold: float = 0.8, metric_name: str = 'zucknick_score')[source]¶
Wraps the calculation of the stability score according to Zucknick et al. [1]. The stability score features a correlation-adjusting mechanism assessing stability not only with respect to set theoretical stabilities, but also according to the correlation between the features selected in different runs. A detailed overview is provided in the userguide.
- Parameters
- correlation_threshold: float, default=0.8
Parameter of the calculation of stability according to Zucknick et al. [1] . The parameter determines the minimum required correlation between two features to be considered similar.
- metric_name: str, default=”zucknick_score”
References
- 1(1,2)
Zucknick, M., Richardson, S., Stronach, E.A.: Comparing the characteristics of gene expression profiles derived by univariate and multivariate classification methods. Stat. Appl. Genet. Molecular Biol. 7(1), 7 (2008)
- add_stabilities(pod: DataHandler)¶
Conducts the evaluation of the stability metric across all datasets and methods in the
DataHandlerobject, which is extended with the results of the stability evaluation.- Parameters
- pod: DataHandler
instance of
DataHandlercontaining the results of the benchmarking procedure
- evaluate_stability(meta_data: dict, selections: array, features: FeatureDescriptor)¶
Conducts the stability evaluation of a set of executions of a selector algorithm on one dataset with a specific feature configuration under different data splits and seeds
- Parameters
- meta_data: dict
information about the data set, which might be relevant for stability calculations. See
get_meta()for the contained data- selections: np.ndarray
The selected features of the different executions of the selector algorithm as integer indices of features. Shape (#executions, #features to select)
- features: FeatureDescriptor
FeatureDescriptor describing the configuration of features to be selected
- Returns
- stability: float
- pairwise_sim_func(meta_data: dict, set_1: ndarray, set_2: ndarray) float[source]¶
Function calculating the stability score for a single pair of selections of features.
- Parameters
- meta_data: dict
Dict containing meta information about the dataset for which the stability metric is evaluated. See the documentation of
get_meta()for the available data.- set_1: np.nadarray
array of integer indices of selected features of shape (n_features_to_select,)
- set_2: np.nadarray
array of integer indices of selected features of shape (n_features_to_select,)
- Returns
- stability score for the given pair of selections: float
