lf2i.calibration package

Submodules

lf2i.calibration.critical_values module

lf2i.calibration.critical_values.multi_quantile_mean_pinball_loss(y_true: ndarray, y_pred: ndarray, alpha: float | Sequence[float]) → float

lf2i.calibration.critical_values.train_qr_algorithm(test_statistics: ndarray | Tensor, parameters: ndarray | Tensor, algorithm: str | Any, alpha: float | Sequence[float], param_dim: int, algorithm_kwargs: Dict[str, Any] | Dict[str, Dict[str, Any]] = {}, verbose: bool = True, n_jobs: int = -2) → Any

Dispatcher to train different quantile regressors and estimate critical values.

Parameters:

• test_statistics (Union[np.ndarray, torch.Tensor]) – The i-th element is the test statistics evaluated on the i-th element of poi (i.e., :math:` heta_i`) and on \(x \sim F_{ heta_i}\).

• parameters (Union[np.ndarray, torch.Tensor]) – Parameters of interest in the calibration set.

• algorithm (Union[str, Any]) – Either ‘cat-gb’ for gradient boosted trees, ‘nn’ for a feed-forward neural network, or a custom algorithm (Any). The latter must implement the fit(X=…, y=…) method.

• alpha (Union[float, Sequence[float]]) – The alpha quantile of the test statistic to be estimated. E.g., for 90% confidence intervals, it should be 0.9 if the acceptance region of the test statistic is on the left of the critical value. Similarly, it should be 0.1 if the acceptance region of the test statistic is on the right of the critical value. Must be in the range (0, 1). NOTE: There is currently no explicit way of avoiding quantile crossings when estimating multiple quantiles simultaneously.

• param_dim (int) – Dimensionality of the parameter.

• algorithm_kwargs (Union[Dict[str, Any], Dict[str, Dict[str, Any]]], optional) – Keyword arguments for the desired algorithm, by default {}. If algorithm == ‘nn’, then ‘hidden_layer_shapes’, ‘epochs’ and ‘batch_size’ must be present. If algorithm == ‘cat-gb’, pass {‘cv’: hp_dist} to do a randomized search over the hyperparameters in hp_dist (a Dict) via 5-fold cross validation. Include ‘n_iter’ as a key to decide how many hyperparameter setting to sample for randomized search. Defaults to 10.

• verbose (bool, optional) – Whether to print information on the hyper-parameter search for quantile regression, by default True.

• n_jobs (int, optional) – Number of workers to use when doing random search with 5-fold CV. By default -2, which uses all cores minus one. If -1, use all cores. n_jobs == -1 uses all cores. If n_jobs < -1, then n_jobs = os.cpu_count()+1+n_jobs.

Returns:

Any – Fitted quantile regressor.

Raises:

ValueError – Only one of ‘cat-gb’, ‘nn’ or an instantiated custom quantile regressor (Any) is currently accepted as algorithm.

lf2i.calibration.p_values module

lf2i.calibration.p_values.estimate_rejection_proba(test_statistics: ndarray | Tensor, parameters: ndarray | Tensor, algorithm: str | Any, algorithm_kwargs: Dict[str, Any] = {}, augment_kwargs: Dict[str, Any] | None = None, acceptance_region: str | None = None, verbose: bool = True) → Any

Fit a calibration model to estimate rejection probabilities (p-values).

Handles augmentation internally: CDF estimators receive raw (T, θ) pairs and model p(T | θ) directly; probabilistic classifiers receive augmented (τ, θ) pairs with 1[T ≤ τ] labels produced by augment_calibration_set().

Parameters:

• test_statistics (array-like of shape (N,)) – Test statistics T_i evaluated at the i-th calibration parameter θ_i and corresponding sample x_i ~ F_{θ_i}.

• parameters (array-like of shape (N,) or (N, d)) – Parameters of interest θ_i for each calibration sample. Any width d ≥ 1 is accepted; 1-D input is promoted to (N, 1).

• algorithm (str or AbstractCDFEstimator or AbstractProbabilisticClassifier) –

  • 'nn': constructs a default ParametricCDFEstimator. Pass algorithm_kwargs to override any constructor argument.

  • CDF estimator (fit(X) with no y argument): receives (T, θ) stacked as X and learns p(T | θ) directly. augment_kwargs are ignored for this branch.

  • Probabilistic classifier (fit(X, y)): receives augmented (τ, θ) inputs and 1[T ≤ τ] labels produced internally by augment_calibration_set().

• algorithm_kwargs (dict, optional) – Keyword arguments forwarded to the constructor when algorithm='nn'. Ignored for pre-instantiated estimators.

• augment_kwargs (dict, optional) –

  Forwarded to augment_calibration_set() for probabilistic classifiers. Recognised keys (with defaults):

  • num_augment (int, default 1): cutoffs resampled per observation.

  • conditional_resampling (bool, default True): resample from p(τ | θ) rather than the marginal.

  • min_points_per_bin (int, default 50): minimum bin occupancy for conditional resampling.

  Silently ignored (with a verbose note) when algorithm is a CDF estimator.

• acceptance_region (str, optional) – Deprecated and ignored. Directionality is the caller’s responsibility. Passing a value emits a DeprecationWarning.

• verbose (bool, default True) – Print a one-line summary of the preprocessing performed.

Returns:

Any – The fitted algorithm object.

lf2i.calibration.p_values.augment_calibration_set(test_statistics: ndarray | Tensor, poi: ndarray | Tensor, num_augment: int, conditional_resampling: bool = True, min_points_per_bin: int = 50) → Tuple[ndarray, ndarray]

Augment the calibration set by resampling cutoffs from the empirical distribution of the test statistics. This allows to estimate p-values that are amortized with respect to all levels \(lpha\).

The rejection indicator is always defined as \(\mathbb{1}[T \le \tau]\), so the trained classifier estimates the CDF \(F(\tau \mid \theta) = P(T \le \tau \mid \theta)\). This is monotone increasing and produces predict_proba output with columns [1-CDF, CDF]. Directional p-value selection (based on the test statistic’s acceptance region) is the responsibility of the caller (e.g. lf2i.inference.lf2i.LF2I).

Parameters:

• test_statistics (Union[np.ndarray, torch.Tensor]) – The i-th element is the test statistics evaluated on the i-th element of poi (i.e., :math:` heta_i`) and on \(x \sim F_{ heta_i}\).

• poi (Union[np.ndarray, torch.Tensor]) – Parameters of interest in the calibration set.

• num_augment (int) – Number of cutoffs to resample for each value in test_statistics. The augmented calibration set will be of size num_augment :math:` imes B^prime`, where \(B^\prime\) is the size of the original calibration set.

• conditional_resampling (bool, optional) – Whether to re-sample cutoffs for augmentation from \(p( au \mid heta)\) or from the marginal \(p( au)\). Default is True. Conditional sampling should yield better estimates of p-values since it is designed to better represent the tails of each conditional distribution, but it could be impractical with a high-dimensional parameter.

• min_points_per_bin (int, optional) – Minimum number of points required per bin for constructing the POI bins. The POI space will be divided into bins such that each bin contains at least this number of points. Default is 50.

Returns:

Tuple[np.ndarray, np.ndarray] – Augmented inputs (cutoffs and POIs) and outputs (CDF indicators) to estimate amortized p-values.

lf2i.calibration.p_values.conditional_sampling(poi: ndarray, test_statistics: ndarray, num_augment: int, min_points_per_bin: int = 50) → ndarray

Perform conditional sampling of test statistics based on the parameters of interest (POI). This method divides the POI space into multidimensional bins, associates each bin with the corresponding test statistics, and resamples conditionally from the empirical distribution of test statistics within each bin.

Parameters:

• poi (np.ndarray) – A 2D array where each row represents a parameter of interest (POI) and each column corresponds to a dimension in the parameter space. Shape: (num_samples, num_dimensions).

• test_statistics (np.ndarray) – A 1D array of test statistics evaluated for each parameter of interest. Shape: (num_samples,).

• num_augment (int) – Number of samples to draw from the conditional distribution of test statistics for each POI bin.

• min_points_per_bin (int, optional) – Minimum number of points required per bin for constructing the POI bins. The POI space will be divided into bins such that each bin contains at least this number of points. Default is 50.

Returns:

np.ndarray – A 2D array of resampled test statistics. Shape: (num_samples, num_augment), where num_samples corresponds to the number of rows in poi.

Raises:

AssertionError – If bin assignments fail or there is no data available for a specific bin.