lf2i.diagnostics package

Submodules

lf2i.diagnostics.coverage_probability module

lf2i.diagnostics.coverage_probability.estimate_coverage_proba(indicators: ndarray, parameters: ndarray, estimator: str, estimator_kwargs: Dict, param_dim: int, new_parameters: ndarray | None = None, n_sigma: int = 2) → Tuple[Any, ndarray, ndarray | None, ndarray | None]

Estimate conditional coverage probabilities by regressing indicators, which signal if the corresponding value in parameters was included or not in the parameter region, against the parameters themselves.

Note that indicators can be computed from any parameter region (posterior credible sets, confidence sets, prediction sets, etc …).

Parameters:

• indicators (np.ndarray) – Array of zeros and ones to mark which parameters were included or not in the corresponding parameter regions.

• parameters (np.ndarray) – Array of p-dimensional parameters.

• estimator (str) – Name of the probabilistic classifier to use to estimate coverage probabilities.

• estimator_kwargs (Dict) – Settings for estimator. If estimator=’cat-gb’, passing {‘cv’: hp_dist} triggers a randomized hyperparameter search over hp_dist (a Dict) using 5-fold cross validation; include n_iter to control how many sampled hyperparameter settings are evaluated (defaults to 10). If cv is not provided, remaining keys in estimator_kwargs are passed directly to CatBoostClassifier.

• param_dim (int) – Dimensionality of the parameter.

• new_parameters (Optional[np.ndarray], optional) – Array of parameters over which to estimate/evaluate coverage probabilities. If not provided, both training and evaluation of the probabilistic classifier are done over parameters.

• n_sigma (int, optional) – Uncertainties around the estimated mean coverage proabilities are computed as \(\mu \pm se \cdot n\_sigma\).

Returns:

Tuple[Any, np.ndarray, Optional[np.ndarray], Optional[np.ndarray]] – Fitted estimator, evaluated parameters, and estimated coverage probabilities – mean, upper-n_sigma bound, lower-n_sigma bound.

Raises:

ValueError – Estimator must be one of [cat-gb].

lf2i.diagnostics.coverage_probability.compute_indicators_lf2i(calibration_method: str, test_statistics: ndarray, parameters: ndarray, critical_values: ndarray | None, p_values: ndarray | None, alpha: float | None, acceptance_region: str | None, param_dim: int) → ndarray

Construct an array of indicators which mark whether each value in parameters is included or not in the corresponding LF2I confidence region.

This assumes that parameters is an array containing the “true” values, as simulated for the diagnostics branch. Instead of actually checking if the parameter is geometrically included in the confidence region or not, this allows to simply deem a value as included if the corresponding test does not reject it.

Parameters:

• calibration_method (str) – Either critical-values or p-values.

• test_statistics (np.ndarray) – Array of test statistics. Each value must be computed for the test with corresponding (null) value of parameters, given a sample generate from it. Only used if calibration_method = ‘critical-values.

• parameters (np.ndarray) – True (simulated) parameter values. If a parameter is in the acceptance region of the corresponding test, then it is included in the confidence set. Only used if calibration_method = ‘critical-values.

• critical_values (np.ndarray, optional) – Array of critical values, each computed for the test with corresponding (null) value of parameters, against which to compare the test statistics. Only used if calibration_method = ‘critical-values.

• p_values (np.ndarray, optional) – Array of p-values, each computed for the test with corresponding (null) value of parameters, against which to compare the provided level \(lpha\). Only used if calibration_method = ‘p-values.

• alpha (float, optional) – If calibration_method = ‘p-values, used to decide whether the test rejects or not, otherwise ignored.

• acceptance_region (str, optional) – Whether the acceptance region for the corresponding test is defined to be on the right or on the left of the critical value. Must be either left or right. Only used if calibration_method = ‘critical-values.

• param_dim (int) – Dimensionality of the parameter.

Returns:

np.ndarray – Array of zeros and ones that indicate whether the corresponding value in parameters is included or not in the confidence region.

Raises:

ValueError – acceptance_region must be either left or right.

lf2i.diagnostics.coverage_probability.isin_with_tol(a, B, atol=1e-06)

lf2i.diagnostics.coverage_probability.compute_indicators_posterior(posterior: AbstractNeuralPosterior | AbstractKDE | Sequence[AbstractNeuralPosterior | AbstractKDE], parameters: Tensor, samples: Tensor, parameter_grid: Tensor, credible_level: float, param_dim: int, batch_size: int, num_level_sets: int = 10000, tol: float = 0.01, return_credible_regions: bool = False, return_size: bool = False, verbose: bool = True, n_jobs: int = -2, **posterior_kwargs) → ndarray | Tuple[ndarray, Sequence[Tensor]] | Tuple[ndarray, ndarray]

Construct an array of indicators which mark whether each value in parameters is included or not in the corresponding posterior credible region.

Parameters:

• posterior (Union[AbstractNeuralPosterior, AbstractKDE, Sequence[Union[AbstractNeuralPosterior, AbstractKDE]]],) – Estimated posterior distribution. If Sequence of posteriors, we assume i-th posterior is estimated given i-th element of samples. Must have log_prob() method.

• parameters (torch.Tensor,) – True (simulated) parameter values, for which inclusion in the corresponding credible region is checked.

• samples (torch.Tensor,) – Array of d-dimensional samples, each generated from the corresponding value in parameters.

• parameter_grid (torch.Tensor,) – Parameter space over which posterior is defined. This is used to construct the credible region.

• credible_level (float) – Desired credible level for the HPD regions. Must be in (0, 1).

• param_dim (int) – Dimensionality of the parameter.

• batch_size (int) – Number of samples drawn from the same parameter value, for each batch in samples. Each element of samples is of size (batch_size, data_dim).

• num_level_sets (int, optional) – Number of level sets to consider to construct the high-posterior-density region, by default 10_000. A high number of level sets ensures the actual credible level is as close as possible to the specified one.

• tol (float, optional) – Actual credible levels within tol of the specified credible_level will be considered acceptable, by default 0.01. NOTE: this is used as a stopping criterion, but if the closest actual credible level is not within tol of credible_level, a warning is raised but the HPD region is still used.

• return_credible_regions (bool, optional) – Whether to return the credible regions computed along the way or not.

• return_size (bool, optional) – Whether to return the size (%points retained from the parameter grid) of each credible region.

• verbose (bool, optional) – Whether to print progress bars or not, by default True.

• n_jobs (int, optional) – Number of workers to use when computing indicators over a sequence of inputs. By default -2, which uses all cores minus one.

• **posterior_kwargs – Any keyword argument needed when calling the log_prob method of the posterior.

Returns:

Union[np.ndarray, Tuple[np.ndarray, Sequence[torch.Tensor]]] – Array of zeros and ones that indicates whether the corresponding value in parameters is included or not in the credible region. If return_credible_regions, then return a tuple whose second element is a sequence of credible regions (one for each parameter/sample).

lf2i.diagnostics.coverage_probability.compute_indicators_prediction(test_statistic: Waldo, parameters: ndarray, samples: ndarray, confidence_level: float, param_dim: int) → ndarray

Construct an array of indicators which mark whether each value in parameters is included or not in the corresponding prediction set. The (central) prediction set is computed using a gaussian approximation.

Parameters:

• test_statistic (Waldo) – An instance of the Waldo test statistic object, where Waldo.estimator and Waldo.cond_variance_estimator have been trained and have a predict(X=…) method to estimate the conditional mean and conditional variance given samples.

• parameters (np.ndarray) – True (simulated) parameter values, for which inclusion in the corresponding prediction set is checked.

• confidence_level (float) – Confidence level of the credible regions to be constructed. Must be in (0, 1).

• param_dim (int) – Dimensionality of the parameter.

Returns:

np.ndarray – Array of zeros and ones that indicate whether the corresponding value in parameters is included or not in the prediction set.

Raises:

NotImplementedError – Only implemented for one-dimensional parameters.

lf2i.diagnostics.coverage_probability.fit_r_estimator(estimator: str, indicators: ndarray, parameters: ndarray, param_dim: int) → Any

Estimate coverage probabilities across the whole parameter space using a pre-defined estimator available in R. (DEPRECATED)

Parameters:

• estimator (str) – Name of the estimator to use.

• indicators (np.ndarray) – Array of zeros and ones that indicate whether the corresponding value in parameters is included or not in the parameter region.

• parameters (np.ndarray) – True (simulated) parameter values.

• param_dim (int) – Dimensionality of the parameter.

Returns:

Any – A model object returned by the corresponding R code.

Raises:

NotImplementedError – Estimator must be one of [gam, TBD]

lf2i.diagnostics.coverage_probability.predict_r_estimator(fitted_estimator: Any, parameters: ndarray, param_dim: int, n_sigma: int) → Tuple[ndarray, ndarray, ndarray]

Evaluate the trained R estimator and estimate the coverage probabilities given parameters. (DEPRECATED)

Parameters:

• fitted_estimator (Any) – Trained estimator, as returned by fit_r_estimator.

• param_dim (int) – Dimensionality of the parameter.

• n_sigma (int) – Uncertainties around the estimated mean coverage proabilities are computed as mean +- se * n_sigma, by default 2.

Returns:

Tuple[np.ndarray, np.ndarray, np.ndarray] – Estimated conditional coverage probabilities – mean, upper-n_sigma bound. lower-n_sigma bound

lf2i.diagnostics.monte_carlo_methods module

lf2i.diagnostics.monte_carlo_methods.monte_carlo_confidence_region(test_statistic: TestStatistic, simulator: Simulator, test_param: Tensor, param_grid: Tensor, x: Tensor, confidence_level: float, monte_carlo_size: int = 2000, critical_values: Tensor = None)

lf2i.diagnostics.monte_carlo_methods.monte_carlo_critical_values(test_statistic: TestStatistic, simulator: Simulator, param_grid: Tensor, confidence_level: float | list, monte_carlo_size: int)

lf2i.diagnostics.monte_carlo_methods.monte_carlo_coverage(test_statistic: TestStatistic, calibration_model, simulator: Simulator, evaluation_grid: ndarray, confidence_level: float | Sequence[float], calibration_method: str, monte_carlo_size: int = 500) → Tuple[ndarray, ndarray] | Tuple[ndarray, Dict[float, ndarray]]

MC-exact coverage at each point of evaluation_grid.

Parameters:

confidence_level (Union[float, Sequence[float]]) – One or more nominal confidence levels in (0, 1). Simulation and test-statistic evaluation are performed only once regardless of how many levels are requested.

Returns:

• Tuple[np.ndarray, np.ndarray] – (evaluation_grid, coverage_per_grid_point) if confidence_level is a scalar float, where coverage_per_grid_point has shape (n_grid,).

• Tuple[np.ndarray, Dict[float, np.ndarray]] – (evaluation_grid, {cl: coverage_per_grid_point, ...}) if confidence_level is a sequence, one entry per requested level.

lf2i.diagnostics.monte_carlo_methods.monte_carlo_coverage_posterior(posterior_estimator, simulator: Simulator, evaluation_grid: ndarray, credible_level: float | Sequence[float], parameter_grid: Tensor, monte_carlo_size: int = 500, num_level_sets: int = 10000, n_jobs: int = -2, **posterior_kwargs) → Tuple[ndarray, ndarray] | Tuple[ndarray, Dict[float, ndarray]]

MC-exact coverage of HPD credible regions at each point of evaluation_grid.

For each parameter value in evaluation_grid, draws monte_carlo_size samples from the simulator, computes the HPD credible region of the posterior at the given level, and checks whether the true parameter is included.

Parameters:

• posterior_estimator – Trained posterior estimator with a log_prob method (e.g. from sbi).

• simulator (Simulator) – lf2i Simulator used to draw samples.

• evaluation_grid (np.ndarray, shape (n_grid, param_dim)) – Parameter values at which to evaluate coverage.

• credible_level (Union[float, Sequence[float]]) – Nominal credible level(s), each in (0, 1).

• parameter_grid (torch.Tensor, shape (n_param_grid, param_dim)) – Dense grid over the parameter space used to approximate the HPD region.

• monte_carlo_size (int, optional) – MC draws per grid point. Default 500.

• num_level_sets (int, optional) – Number of level sets for the HPD binary search. Default 10_000.

• n_jobs (int, optional) – Joblib parallelism for HPD computation. Default -2.

• **posterior_kwargs – Extra keyword arguments forwarded to the posterior’s log_prob method.

Returns:

• Tuple[np.ndarray, np.ndarray] – (evaluation_grid, coverage_per_grid_point) when credible_level is a scalar.

• Tuple[np.ndarray, Dict[float, np.ndarray]] – (evaluation_grid, {cl: coverage_per_grid_point, ...}) when a sequence.

lf2i.diagnostics.monte_carlo_methods.monte_carlo_pvalue_diagnostics(test_statistic: TestStatistic, calibration_model, simulator: Simulator, evaluation_grid: Tensor | ndarray, monte_carlo_size: int = 500, pinball_levels: Sequence[float] = None)

For each theta in evaluation_grid, draw MC samples, evaluate the test statistic, and compare the calibration model’s predicted CDF to the empirical CDF.

Returns a dict with per-theta arrays:

• mse: mean((p_hat_(i) - u_i)^2) over sorted samples

• crps: integral (F_hat - F_emp)^2 dt via trapezoid rule at sample points

• pinball_<alpha>: pinball loss L_alpha(u_i, p_hat_i) for each alpha in pinball_levels

where u_i = (i - 0.5) / M is the midpoint empirical CDF estimate.