lf2i.inference package

class lf2i.inference.LF2I(test_statistic: str | TestStatistic, **test_statistic_kwargs: Any)[source]

Bases: object

High-level entry point to do inference with LF2I (https://arxiv.org/abs/2107.03920). This allows to quickly construct confidence regions for parameters of interest in an SBI setting leveraging an arbitrary estimator

Alternatively, one can define a custom TestStatistic appropriate for the problem at hand.

NOTE: although this entry point contains all the main LF2I functionalities, using the single implemented components (test statistics, critical values, neyman inversion) provides a bit more flexibility and allows to control every single hyper-parameter.

Parameters:

  • test_statistic (Union[str, TestStatistic]) – Either acore, bff, waldo or an instance of a custom lf2i.test_statistics._base.TestStatistic

  • test_statistic_kwargs (Any) – Arguments specific to the chosen test statistic if one of acore, bff, waldo. See the dedicated documentation for each of them in lf2i/test_statistics/

inference(x: ndarray | Tensor, evaluation_grid: ndarray | Tensor, confidence_level: float | Sequence[float], calibration_method: str, calibration_model: str | Any = 'cat-gb', calibration_model_kwargs: Dict = {}, T: Tuple[ndarray | Tensor] | None = None, T_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b: int | None = None, b_prime: int | None = None, num_augment: int = 5, retrain_calibration: bool = False, recalibrate_p_values: bool = False, verbose: bool = True) → List[ndarray] | Dict[str, List[ndarray]][source]

Estimate test statistic and critical values, and construct confidence sets for all observations in x.

Parameters:

  • x (Union[np.ndarray, torch.Tensor]) – Observed sample(s).

  • evaluation_grid (Union[np.ndarray, torch.Tensor]) – Grid of points over the parameter space over which to invert hypothesis tests. Each confidence set will be a subset of this grid.

  • confidence_level (Union[float, Sequence[float]]) – Desired confidence level(s), must be in \((0, 1)\).

  • calibration_method (str) – Either critical-values (via quantile regression) or p-values (via monotonic probabilistic classification).

  • calibration_model (Union[str, Any], optional) – If str, identifier for the calibration model, by default ‘cat-gb’.

  • calibration_model_kwargs (Dict, optional) – Settings for the chosen calibration model, by default {}.

  • T (Tuple[Union[np.ndarray, torch.Tensor]], optional) – Simulated dataset to train the test statistic estimator.

  • T_prime (Tuple[Union[np.ndarray, torch.Tensor]], optional) – Simulated dataset to train the calibration model.

  • simulator (Simulator, optional) – If T and T_prime are not given, must pass an instance of lf2i.simulator.Simulator.

  • b (int, optional) – Number of simulations for the test statistic. Used only if simulator is provided.

  • b_prime (int, optional) – Number of simulations for calibration. Used only if simulator is provided.

  • num_augment (int) – If calibration_method = ‘p-values’, number of cutoffs to resample per value.

  • retrain_calibration (bool, optional) – Whether to retrain the calibration model, by default False.

  • recalibrate_p_values (bool, optional) – Whether to hold out part of the calibration set to recalibrate p-value thresholds, by default False.

  • verbose (bool, optional) – Whether to print checkpoints and progress bars, by default True.

Returns:

List[np.ndarray] or List[List[np.ndarray]] for multiple confidence levels.

coverage(region_type: str, confidence_level: float, calibration_method: str | None = None, coverage_estimator: str = 'cat-gb', coverage_estimator_kwargs: Dict = {}, T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b_double_prime: int | None = None, new_parameters: ndarray | None = None, indicators: ndarray | None = None, parameters: ndarray | None = None, posterior_estimator: Any | None = None, evaluation_grid: ndarray | Tensor = None, num_level_sets: int | None = 10000, n_jobs: int | None = -2, verbose: bool = True, exact: bool | None = None, monte_carlo_size: int = 500, parameter_grid: Tensor = None, **posterior_kwargs)[source]

Estimate or compute exactly the coverage probability of a confidence/credible region method.

Dispatches to _estimated_coverage() (ML-based estimator) or _exact_coverage() (Monte Carlo), depending on exact and whether a simulator is available. Only parameters specific to this dispatch are documented below; for all other parameters, see diagnostics().

Parameters:

  • exact (bool, optional) – If True, use Monte Carlo exact coverage (requires simulator). If False, use the ML estimator (requires T_double_prime or simulator). If None (default), use MC when a simulator is provided, otherwise estimated.

  • monte_carlo_size (int, optional) – MC draws per grid point for _exact_coverage. Default 500.

  • parameter_grid (torch.Tensor, optional) – Dense parameter grid for region_type='posterior' exact coverage.

diagnostics(region_type: str, confidence_level: float, calibration_method: str | None = None, coverage_estimator: str = 'cat-gb', coverage_estimator_kwargs: Dict = {}, T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b_double_prime: int | None = None, new_parameters: ndarray | None = None, indicators: ndarray | None = None, parameters: ndarray | None = None, posterior_estimator: Any | None = None, evaluation_grid: ndarray | Tensor = None, num_level_sets: int | None = 10000, n_jobs: int | None = -2, verbose: bool = True, **posterior_kwargs) → Tuple[Any, ndarray, ndarray, ndarray, ndarray][source]

ML-based coverage diagnostics. Deprecated alias for _estimated_coverage().

Deprecated since version ``diagnostics``: will be removed in a future release. Use coverage() instead.

mc_diagnostics(simulator, evaluation_grid: ndarray, confidence_level: float | Sequence[float], calibration_method: str = None, monte_carlo_size: int = 500, region_type: str = 'lf2i', posterior_estimator=None, parameter_grid: Tensor = None, num_level_sets: int = 10000, n_jobs: int = -2, **posterior_kwargs) → Tuple[ndarray, ndarray] | Tuple[ndarray, Dict[float, ndarray]][source]

MC-exact coverage diagnostics. Deprecated alias for _exact_coverage().

Deprecated since version ``mc_diagnostics``: will be removed in a future release. Use coverage() instead.

power(evaluation_grid: ndarray | Tensor, confidence_level: float | Sequence[float], T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, calibration_method: str = 'critical-values', batch_size: int = 1000, monte_carlo_size: int = 100, exact: bool | None = None, verbose: bool = True)[source]

Estimate or compute exactly the power (expected confidence set size) of the LF2I procedure.

Dispatches to _estimated_power() (regression-based) or _exact_power() (Monte Carlo), depending on exact and whether a simulator is available.

Parameters:

  • evaluation_grid (Union[np.ndarray, torch.Tensor]) – Grid of parameter values at which to evaluate power.

  • confidence_level (Union[float, Sequence[float]]) – Nominal confidence level(s), each in (0, 1).

  • T_double_prime (Tuple, optional) – Pre-simulated dataset (parameters, samples) for estimated power.

  • simulator (Simulator, optional) – Required for exact (MC) power.

  • calibration_method (str, optional) – 'critical-values' or 'p-values'. Default 'critical-values'.

  • batch_size (int, optional) – Batch size for estimated power. Default 1000.

  • monte_carlo_size (int, optional) – MC draws per grid point for exact power. Default 100.

  • exact (bool, optional) – If None (default), use MC when simulator is available, else estimated.

  • verbose (bool, optional) – Whether to print progress. Default True.

Returns:

  • For estimated power (np.ndarray of confidence set sizes (length = n_samples in T_double_prime).)

  • For exact power ((evaluation_grid, mean_size_per_grid_point).)

Submodules

lf2i.inference.lf2i module

class lf2i.inference.lf2i.LF2I(test_statistic: str | TestStatistic, **test_statistic_kwargs: Any)[source]

Bases: object

High-level entry point to do inference with LF2I (https://arxiv.org/abs/2107.03920). This allows to quickly construct confidence regions for parameters of interest in an SBI setting leveraging an arbitrary estimator

Alternatively, one can define a custom TestStatistic appropriate for the problem at hand.

NOTE: although this entry point contains all the main LF2I functionalities, using the single implemented components (test statistics, critical values, neyman inversion) provides a bit more flexibility and allows to control every single hyper-parameter.

Parameters:

  • test_statistic (Union[str, TestStatistic]) – Either acore, bff, waldo or an instance of a custom lf2i.test_statistics._base.TestStatistic

  • test_statistic_kwargs (Any) – Arguments specific to the chosen test statistic if one of acore, bff, waldo. See the dedicated documentation for each of them in lf2i/test_statistics/

inference(x: ndarray | Tensor, evaluation_grid: ndarray | Tensor, confidence_level: float | Sequence[float], calibration_method: str, calibration_model: str | Any = 'cat-gb', calibration_model_kwargs: Dict = {}, T: Tuple[ndarray | Tensor] | None = None, T_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b: int | None = None, b_prime: int | None = None, num_augment: int = 5, retrain_calibration: bool = False, recalibrate_p_values: bool = False, verbose: bool = True) → List[ndarray] | Dict[str, List[ndarray]][source]

Estimate test statistic and critical values, and construct confidence sets for all observations in x.

Parameters:

  • x (Union[np.ndarray, torch.Tensor]) – Observed sample(s).

  • evaluation_grid (Union[np.ndarray, torch.Tensor]) – Grid of points over the parameter space over which to invert hypothesis tests. Each confidence set will be a subset of this grid.

  • confidence_level (Union[float, Sequence[float]]) – Desired confidence level(s), must be in \((0, 1)\).

  • calibration_method (str) – Either critical-values (via quantile regression) or p-values (via monotonic probabilistic classification).

  • calibration_model (Union[str, Any], optional) – If str, identifier for the calibration model, by default ‘cat-gb’.

  • calibration_model_kwargs (Dict, optional) – Settings for the chosen calibration model, by default {}.

  • T (Tuple[Union[np.ndarray, torch.Tensor]], optional) – Simulated dataset to train the test statistic estimator.

  • T_prime (Tuple[Union[np.ndarray, torch.Tensor]], optional) – Simulated dataset to train the calibration model.

  • simulator (Simulator, optional) – If T and T_prime are not given, must pass an instance of lf2i.simulator.Simulator.

  • b (int, optional) – Number of simulations for the test statistic. Used only if simulator is provided.

  • b_prime (int, optional) – Number of simulations for calibration. Used only if simulator is provided.

  • num_augment (int) – If calibration_method = ‘p-values’, number of cutoffs to resample per value.

  • retrain_calibration (bool, optional) – Whether to retrain the calibration model, by default False.

  • recalibrate_p_values (bool, optional) – Whether to hold out part of the calibration set to recalibrate p-value thresholds, by default False.

  • verbose (bool, optional) – Whether to print checkpoints and progress bars, by default True.

Returns:

List[np.ndarray] or List[List[np.ndarray]] for multiple confidence levels.

coverage(region_type: str, confidence_level: float, calibration_method: str | None = None, coverage_estimator: str = 'cat-gb', coverage_estimator_kwargs: Dict = {}, T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b_double_prime: int | None = None, new_parameters: ndarray | None = None, indicators: ndarray | None = None, parameters: ndarray | None = None, posterior_estimator: Any | None = None, evaluation_grid: ndarray | Tensor = None, num_level_sets: int | None = 10000, n_jobs: int | None = -2, verbose: bool = True, exact: bool | None = None, monte_carlo_size: int = 500, parameter_grid: Tensor = None, **posterior_kwargs)[source]

Estimate or compute exactly the coverage probability of a confidence/credible region method.

Dispatches to _estimated_coverage() (ML-based estimator) or _exact_coverage() (Monte Carlo), depending on exact and whether a simulator is available. Only parameters specific to this dispatch are documented below; for all other parameters, see diagnostics().

Parameters:

  • exact (bool, optional) – If True, use Monte Carlo exact coverage (requires simulator). If False, use the ML estimator (requires T_double_prime or simulator). If None (default), use MC when a simulator is provided, otherwise estimated.

  • monte_carlo_size (int, optional) – MC draws per grid point for _exact_coverage. Default 500.

  • parameter_grid (torch.Tensor, optional) – Dense parameter grid for region_type='posterior' exact coverage.

diagnostics(region_type: str, confidence_level: float, calibration_method: str | None = None, coverage_estimator: str = 'cat-gb', coverage_estimator_kwargs: Dict = {}, T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, b_double_prime: int | None = None, new_parameters: ndarray | None = None, indicators: ndarray | None = None, parameters: ndarray | None = None, posterior_estimator: Any | None = None, evaluation_grid: ndarray | Tensor = None, num_level_sets: int | None = 10000, n_jobs: int | None = -2, verbose: bool = True, **posterior_kwargs) → Tuple[Any, ndarray, ndarray, ndarray, ndarray][source]

ML-based coverage diagnostics. Deprecated alias for _estimated_coverage().

Deprecated since version ``diagnostics``: will be removed in a future release. Use coverage() instead.

mc_diagnostics(simulator, evaluation_grid: ndarray, confidence_level: float | Sequence[float], calibration_method: str = None, monte_carlo_size: int = 500, region_type: str = 'lf2i', posterior_estimator=None, parameter_grid: Tensor = None, num_level_sets: int = 10000, n_jobs: int = -2, **posterior_kwargs) → Tuple[ndarray, ndarray] | Tuple[ndarray, Dict[float, ndarray]][source]

MC-exact coverage diagnostics. Deprecated alias for _exact_coverage().

Deprecated since version ``mc_diagnostics``: will be removed in a future release. Use coverage() instead.

power(evaluation_grid: ndarray | Tensor, confidence_level: float | Sequence[float], T_double_prime: Tuple[ndarray | Tensor] | None = None, simulator: Simulator | None = None, calibration_method: str = 'critical-values', batch_size: int = 1000, monte_carlo_size: int = 100, exact: bool | None = None, verbose: bool = True)[source]

Estimate or compute exactly the power (expected confidence set size) of the LF2I procedure.

Dispatches to _estimated_power() (regression-based) or _exact_power() (Monte Carlo), depending on exact and whether a simulator is available.

Parameters:

  • evaluation_grid (Union[np.ndarray, torch.Tensor]) – Grid of parameter values at which to evaluate power.

  • confidence_level (Union[float, Sequence[float]]) – Nominal confidence level(s), each in (0, 1).

  • T_double_prime (Tuple, optional) – Pre-simulated dataset (parameters, samples) for estimated power.

  • simulator (Simulator, optional) – Required for exact (MC) power.

  • calibration_method (str, optional) – 'critical-values' or 'p-values'. Default 'critical-values'.

  • batch_size (int, optional) – Batch size for estimated power. Default 1000.

  • monte_carlo_size (int, optional) – MC draws per grid point for exact power. Default 100.

  • exact (bool, optional) – If None (default), use MC when simulator is available, else estimated.

  • verbose (bool, optional) – Whether to print progress. Default True.

Returns:

  • For estimated power (np.ndarray of confidence set sizes (length = n_samples in T_double_prime).)

  • For exact power ((evaluation_grid, mean_size_per_grid_point).)