Package 'pmsims'

Description

Results from four completed sample-size searches, one for each of the package's entry points. A search repeatedly simulates datasets, fits a prediction model to each and evaluates its performance across a range of candidate sample sizes, so a realistic run takes minutes to hours. These objects were computed once, with the calls shown under Source, so that the examples and the vignette can demonstrate the output without recomputing it.

Usage

binary_example

continuous_example

survival_example

custom_example

Format

A list of class "pmsims". The most useful components are:

min_n

The estimated minimum sample size.

perf_n

Expected performance at min_n.

target_performance

The performance target the search aimed for.

metric

The performance metric used.

model

The model fitted to each simulated dataset.

mlpwr_ds

The sampled designs and simulated performance values behind the fitted curve, used by plot().

summaries

Aggregated performance summaries across replications.

simulation_time

Elapsed time of the original run, in seconds.

Objects produced by the wrapper functions additionally record the data-generating configuration (for example outcome_prevalence, correlation and complexity); see simulate_binary() for the full set.

An object of class pmsims of length 36.

An object of class pmsims of length 35.

An object of class pmsims of length 37.

An object of class pmsims of length 24.

Details

Each object is a "pmsims" object, as returned by simulate_binary(), simulate_continuous(), simulate_survival() and simulate_custom(), and can be used with print() and plot() in the usual way.

Source

Generated by data-raw/precomputed-examples.R.

binary_example:

set.seed(123)
simulate_binary(
  signal_parameters = 20,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0.3),
  outcome_prevalence = 0.30,
  maximum_achievable_cstatistic = 0.80,
  model = "glm",
  metric = "calibration_slope",
  target_performance = 0.85,
  n_reps_total = 1000,
  mean_or_assurance = "assurance"
)

continuous_example:

set.seed(123)
simulate_continuous(
  signal_parameters = 15,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0.3),
  maximum_achievable_rsquared = 0.50,
  model = "lm",
  metric = "calibration_slope",
  target_performance = 0.95,
  n_reps_total = 1000,
  mean_or_assurance = "assurance"
)

survival_example:

set.seed(123)
simulate_survival(
  signal_parameters = 15,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0.3),
  maximum_achievable_cindex = 0.70,
  baseline_hazard = 0.01,
  censoring_rate = 0.30,
  model = "coxph",
  metric = "calibration_slope",
  target_performance = 0.90,
  n_reps_total = 1000,
  mean_or_assurance = "assurance"
)

custom_example: a linear model with three independent predictors and a population $R^2$ of 0.5, targeting a calibration slope of 0.9 with 80% assurance. The search uses bounds of 25 to 1,000 participants, 1,000 replications and 30,000 test observations. See simulate_custom() for the generating functions and executable call (seed 123).

See Also

simulate_binary(), simulate_continuous(), simulate_survival(), simulate_custom()

Examples

binary_example
binary_example$min_n

Description

Compute the minimum sample size required to develop a prediction model with a binary outcome. The function wraps a simulation-based engine that combines a bisection search with Gaussian-process curve fitting. From user inputs (outcome prevalence, maximum achievable performance, target performance, etc.) it constructs a data-generating function, a model-fitting function, and a metric function, then searches for the smallest $n$ that meets the chosen performance criterion.

Usage

simulate_binary(
  signal_parameters,
  noise_parameters = 0,
  complexity = 1,
  data_control = NULL,
  outcome_prevalence,
  maximum_achievable_cstatistic,
  model = c("glm", "lasso", "ridge", "rf", "xgboost"),
  metric = "calibration_slope",
  target_performance,
  n_reps_total = 1000,
  mean_or_assurance = "assurance",
  ...
)

Arguments

Value

An object of class "pmsims" containing the estimated minimum sample size and simulation diagnostics (inputs, fitted GP curve, intermediate evaluations, and summary metrics).

Criteria

Two formulations are supported.

• Mean-based: find the smallest $n$ such that the expected model performance exceeds the target $M^*$, i.e.

  $\min_n \; \mathbb{E}_{D_n}\{ M \mid D_n \} \ge M^*.$

• Assurance-based: find the smallest $n$ such that the probability the performance exceeds $M^*$ is at least $\delta$ (e.g. 0.80), i.e.

  $\min_n \; \mathbb{P}_{D_n}\!\left( M \mid D_n \ge M^* \right) \ge \delta.$

Here, $M$ is the chosen performance metric and the probability/expectation is over repeated samples of training data of size $n$. The assurance criterion explicitly accounts for variability across training sets; models with higher variance typically require larger $n$ to satisfy it.

Data control

complexity selects the signal structure of the data-generating mechanism: 1 purely linear, 2 linear + quadratic, 3 linear + quadratic + interaction, 4 the Friedman function. data_control is an optional list fine-tuning the predictors:

nonlinear_strength

Numeric in ⁠[0, 1)⁠. Fraction of the signal variance carried by the nonlinear, linearly-inaccessible component. Applies to complexity 2 and 3 only; ignored (with a warning) for 1 and 4. If omitted, the generator's per-complexity default is used.

correlation

Numeric in $[-1, 1]$. Pairwise correlation among the candidate predictors. Default 0.3.

predictor_distribution

One of "normal", "uniform", "binary", "exponential", "lognormal", "t", "laplace". "binary" selects 0/1 predictors and requires binary_predictor_prevalence; any other value selects continuous predictors from that family. Default "normal".

binary_predictor_prevalence

Numeric in ⁠(0, 1)⁠. Prevalence of the binary predictors; required when predictor_distribution = "binary", ignored (with a warning) otherwise. Note: binary predictors are incompatible with complexity 2/3 because squaring a 0/1 variable returns itself.

See Also

simulate_continuous(), simulate_survival(), simulate_custom()

Examples

set.seed(123)
est <- simulate_binary(
  signal_parameters = 3,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0),
  outcome_prevalence = 0.50,
  maximum_achievable_cstatistic = 0.80,
  model = "glm",
  metric = "calibration_slope",
  target_performance = 0.9,
  mean_or_assurance = "assurance",
  min_sample_size = 50,
  max_sample_size = 1000,
  n_reps_total = 1000,
  test_n = 30000,
  progress = FALSE
)
est
est$min_n
plot(est)

Description

Compute the minimum sample size required to develop a prediction model with a continuous outcome. This wraps the same simulation engine as simulate_binary(), combining bisection search with Gaussian-process learning-curve modelling. From user inputs (maximum achievable performance, target performance, etc.) it constructs a data-generating function, model-fitting function, and metric function, then searches for the smallest $n$ meeting the chosen criterion.

Usage

simulate_continuous(
  signal_parameters,
  noise_parameters = 0,
  complexity = 1,
  data_control = NULL,
  maximum_achievable_rsquared,
  model = c("lm", "lasso", "ridge", "rf", "xgboost"),
  metric = "calibration_slope",
  target_performance,
  n_reps_total = 1000,
  mean_or_assurance = "assurance",
  ...
)

Arguments

Value

An object of class "pmsims" containing the estimated minimum sample size and simulation diagnostics (inputs, fitted GP curve, intermediate evaluations, and summary metrics).

Criteria

Two formulations are supported.

• Mean-based: find the smallest $n$ such that the expected model performance exceeds the target $M^*$, i.e.

  $\min_n \; \mathbb{E}_{D_n}\{ M \mid D_n \} \ge M^*.$

• Assurance-based: find the smallest $n$ such that the probability the performance exceeds $M^*$ is at least $\delta$ (e.g. 0.80), i.e.

  $\min_n \; \mathbb{P}_{D_n}\!\left( M \mid D_n \ge M^* \right) \ge \delta.$

Here, $M$ is the chosen performance metric and the probability/expectation is over repeated samples of training data of size $n$. The assurance criterion explicitly accounts for variability across training sets; models with higher variance typically require larger $n$ to satisfy it.

Data control

complexity selects the signal structure of the data-generating mechanism: 1 purely linear, 2 linear + quadratic, 3 linear + quadratic + interaction, 4 the Friedman function. data_control is an optional list fine-tuning the predictors:

nonlinear_strength

Numeric in ⁠[0, 1)⁠. Fraction of the signal variance carried by the nonlinear, linearly-inaccessible component. Applies to complexity 2 and 3 only; ignored (with a warning) for 1 and 4. If omitted, the generator's per-complexity default is used.

correlation

Numeric in $[-1, 1]$. Pairwise correlation among the candidate predictors. Default 0.3.

predictor_distribution

One of "normal", "uniform", "binary", "exponential", "lognormal", "t", "laplace". "binary" selects 0/1 predictors and requires binary_predictor_prevalence; any other value selects continuous predictors from that family. Default "normal".

binary_predictor_prevalence

Numeric in ⁠(0, 1)⁠. Prevalence of the binary predictors; required when predictor_distribution = "binary", ignored (with a warning) otherwise. Note: binary predictors are incompatible with complexity 2/3 because squaring a 0/1 variable returns itself.

See Also

simulate_binary(), simulate_survival(), simulate_custom()

Examples

set.seed(123)
est <- simulate_continuous(
  signal_parameters = 3,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0),
  maximum_achievable_rsquared = 0.50,
  model = "lm",
  metric = "calibration_slope",
  target_performance = 0.9,
  mean_or_assurance = "assurance",
  min_sample_size = 50,
  max_sample_size = 1000,
  n_reps_total = 1000,
  test_n = 30000,
  progress = FALSE
)
est
est$min_n
plot(est)

Description

Compute the minimum sample size required to achieve a target level of predictive performance using user-defined simulation components. simulate_custom() is the low-level interface in pmsims: users supply a data-generating function, a model-fitting function, and a metric function, and the chosen search engine estimates the smallest $n$ meeting the selected performance criterion.

Usage

simulate_custom(
  data_function,
  model_function,
  metric_function,
  target_performance,
  c_statistic = NULL,
  mean_or_assurance = "assurance",
  test_n = 30000,
  min_sample_size = NULL,
  max_sample_size = NULL,
  n_reps_total = 1000,
  n_reps_per = 20,
  method = "mlpwr",
  progress = TRUE,
  verbose = FALSE,
  ...
)

Arguments

Value

An object of class "pmsims" containing the estimated minimum sample size.

See Also

simulate_binary(), simulate_continuous(), simulate_survival()

Examples

# Three independent predictors with a population R-squared of 0.5.
data_fun <- function(n) {
  x1 <- rnorm(n)
  x2 <- rnorm(n)
  x3 <- rnorm(n)
  y <- (x1 + x2 + x3) / sqrt(3) + rnorm(n)
  data.frame(y = y, x1 = x1, x2 = x2, x3 = x3)
}

model_fun <- function(dat) {
  stats::lm(y ~ ., data = dat)
}

# Calibration slope evaluated on independent test data.
metric_fun <- function(test_data, fit, model) {
  preds <- stats::predict(fit, newdata = test_data)
  unname(stats::coef(stats::lm(test_data$y ~ preds))[2])
}
attr(metric_fun, "metric") <- "calibration_slope"


set.seed(123)
est <- simulate_custom(
  data_function = data_fun,
  model_function = model_fun,
  metric_function = metric_fun,
  target_performance = 0.9,
  mean_or_assurance = "assurance",
  min_sample_size = 25,
  max_sample_size = 1000,
  n_reps_total = 1000,
  test_n = 30000,
  progress = FALSE
)
est
est$min_n
plot(est)

Description

Compute the minimum sample size required to develop a prediction model with a time-to-event (survival) outcome. As with the other wrappers, this uses a simulation-based learning-curve approach with Gaussian-process surrogate modelling to locate the smallest $n$ meeting the chosen performance criterion.

Usage

simulate_survival(
  signal_parameters,
  noise_parameters = 0,
  complexity = 1,
  data_control = NULL,
  maximum_achievable_cindex,
  baseline_hazard = 1,
  censoring_rate,
  model = c("coxph", "lasso", "ridge", "rf", "xgboost"),
  metric = "calibration_slope",
  target_performance,
  n_reps_total = 1000,
  mean_or_assurance = "assurance",
  ...
)

Arguments

Value

An object of class "pmsims" containing the estimated minimum sample size and simulation diagnostics (inputs, fitted GP curve, intermediate evaluations, and summary metrics).

Criteria

Two formulations are supported.

• Mean-based: find the smallest $n$ such that the expected model performance exceeds the target $M^*$, i.e.

  $\min_n \; \mathbb{E}_{D_n}\{ M \mid D_n \} \ge M^*.$

• Assurance-based: find the smallest $n$ such that the probability the performance exceeds $M^*$ is at least $\delta$ (e.g. 0.80), i.e.

  $\min_n \; \mathbb{P}_{D_n}\!\left( M \mid D_n \ge M^* \right) \ge \delta.$

Here, $M$ is the chosen performance metric and the probability/expectation is over repeated samples of training data of size $n$. The assurance criterion explicitly accounts for variability across training sets; models with higher variance typically require larger $n$ to satisfy it.

Data control

complexity selects the signal structure of the data-generating mechanism: 1 purely linear, 2 linear + quadratic, 3 linear + quadratic + interaction, 4 the Friedman function. data_control is an optional list fine-tuning the predictors:

nonlinear_strength

Numeric in ⁠[0, 1)⁠. Fraction of the signal variance carried by the nonlinear, linearly-inaccessible component. Applies to complexity 2 and 3 only; ignored (with a warning) for 1 and 4. If omitted, the generator's per-complexity default is used.

correlation

Numeric in $[-1, 1]$. Pairwise correlation among the candidate predictors. Default 0.3.

predictor_distribution

One of "normal", "uniform", "binary", "exponential", "lognormal", "t", "laplace". "binary" selects 0/1 predictors and requires binary_predictor_prevalence; any other value selects continuous predictors from that family. Default "normal".

binary_predictor_prevalence

Numeric in ⁠(0, 1)⁠. Prevalence of the binary predictors; required when predictor_distribution = "binary", ignored (with a warning) otherwise. Note: binary predictors are incompatible with complexity 2/3 because squaring a 0/1 variable returns itself.

See Also

simulate_binary(), simulate_continuous(), simulate_custom()

Examples

set.seed(123)
est <- simulate_survival(
  signal_parameters = 1,
  noise_parameters = 0,
  complexity = 1,
  data_control = list(correlation = 0),
  maximum_achievable_cindex = 0.70,
  baseline_hazard = 0.01,
  censoring_rate = 0.30,
  model = "coxph",
  metric = "calibration_slope",
  target_performance = 0.9,
  mean_or_assurance = "assurance",
  min_sample_size = 25,
  max_sample_size = 500,
  n_reps_total = 1000,
  test_n = 30000,
  progress = FALSE
)
est
est$min_n
plot(est)