Model diagnostics

On this page you can perform model diagnostics with Visual Predictive Check plot (VPC Plot), Covariate sensitivity plot and Table of model odds ratios.

VPC plot is used to evaluate the fit and predictive performance of a logistic regression model relating drug exposure to the probability of response. It visualizes the model-predicted curve alongside empirical summaries of observed responses.

The Covariate sensitivity plot is used to explore how covariates (both continuous and categorical) impact the odds of response across the range of drug exposure. Main purposes of this plot are to assess the sensitivity of predicted odds to changes in key covariates across the exposure range and to visualize whether covariate effects are constant, increasing, or decreasing with exposure.

Table of odds ratios presents the results of a logistic regression analysis, showing the estimated effects of exposure and covariates on the outcome, including regression coefficients, p-values, and odds ratios with confidence intervals for both unit-based and user-defined changes.

1 Order of operation

1.1. Model selection

In the Model diagnostics tab, the models generated in the Covariate search section are used. Make sure that you save file with models on this tab.

Click Select a file with model .RData, navigate to the working directory, and select the file FinalModels.RData.

After selecting the model file, click Confirm model. Then, from the drop-down list Select response metric, you can select a model by response. Each response corresponds to one model.

Next, click Go to diagnostics.

1.2. Table of continuous covariates

Before running diagnostics one can fill in the tables with additional information about covariates. Using this tables one can add user-friendly names to plot labels and rescale the model parameters. To edit a cell, double-click it with the left mouse button.

Note that in the model transformed covariates can be used. Two types of transformed covariates are available:

  • Log-transformed

  • Median-centered

A median-centered covariate is a continuous covariate that has been transformed by subtracting its median value from each individual value. This results in a covariate whose median is zero, while the distribution and range of values remain the same (only shifted).

The Table of continuous covariates contains the following columns:

  • COV (“Covariate”) – automatically filled in from the model file. The first row corresponds to the exposure metric. The following rows contain the names of model continuous covariates.

  • BTR (“Back Transformed”) – contain the name of the corresponding untransformed covariate from the dataset, if the covariate listed in the ”COV” column is transformed. Only filled in for transformed covariates. Example: If the COV column contains “LOGCAVG” ("Log-transformed C average"), which is obtained by log-transforming the values in the “CAVG” column, then "BTR" should be set to “CAVG”. If the COV column contains “MEDBWT” ("Median-centered Body Weight"), which is the “BWT” covariate transformed by subtracting its median value from each observation, then "BTR" should be set to “BWT”.

  • TRTYPE (“Transformation Type”) – two options are available: “LOG” – for log-transformed covariates and “MED” – for median-centered covariates. Only filled in for transformed covariates.

  • STEP – fill in to change the scale of odds-ratio. Odds-ratio will be calculated per STEP units of continuous covariate. By default odds-ratio is calculated per one unit of continuous covariate.

  • NICENAME – add a user-friendly name for covariate that will appear in plot labels and table

1.3. Table of categorical covariates

The Table of categorical covariates contains the following columns:

  • COV (“Covariate”) – contains the names of model categorical covariates. Automatically filled in from the model file.

  • VAL (“Value”) - contains numeric codes of categories from the dataset. Filled in automatically.

  • NICENAME - add a user-friendly name for the category that will appear in plot labels and table.

  • REFFL ("Reference Flag") - value 1 indicates the reference category, while 0 corresponds to the other categories. Filled in automatically.

1.4. Running diagnostics and saving results

Visualization parameters can be modified from the side panel.

Click Run diagnostics button to start the analysis.

Click Save all to save plots and table. The results will be saved to the “Model diagnostics” folder in the working directory.

2. VPC plot

Plot Description

  • X-axis: Exposure
  • Y-axis: Predicted probability of response (in percent)
  • Line: Median predicted probability curve from the model
  • Shaded Area: Confidence Interval
  • Points: Median observed response probability within each quantile of exposure

Visualization Options

On the sidebar panel you can change the following parameters:

Number of replicas - number of simulated datasets generated using the model to estimate prediction intervals and assess the model's predictive performance.

Number of tiles - number of quartiles of exposure.

log x - add log-transformation of x-scale.

3. Covariate Sensitivity Plot

Plot Description

  • X-axis: Odds ratios — representing the effect of each covariate on the probability of response
  • Y-axis: continuous and categorical covariates.
  • Points: Estimated odds ratios for each covariate at different exposure levels
  • Error Bars: Confidence intervals for the odds ratios

Visualization Options

On the sidebar panel you can change the following parameters:

Select CI of parameters - select confidence interval for odds ratio values (e.g. value 0.95 means 95% confidence interval)

The following fields refer to Predictor distribution:

Central tendency – used for transformed continuous covariates. Specify median for covariates centered on the median, and mean for those centered on the mean.

Sensitivity analysis for continuous covariates is performed using the extreme quantiles of the covariate (e.g., 0.05 and 0.95 by default). On the plot, two points are shown for each continuous covariate, corresponding to the left quantile and right quantile.

Left quantile - left quantile value of continuous predictor.

Right quantile - right quantile value of continuous predictor.

log y - log-transformation of y-scale

add reference group

4. Table of odds ratios

Table represents numerical values of odds ratios and contains the following columns:

Term - names of the model terms (predictors). This includes the intercept, continuous covariates (e.g., age, weight), and categorical variables (e.g. sex, race) with their reference categories.

Estimate (CI) - estimated regression coefficient and its confidence interval (CI) from the logistic regression model. This value represents the change in the log-odds of the response per unit increase in the predictor.

p-value - statistical significance of the predictor. A small p-value (typically < 0.05) suggests that the predictor has a statistically significant effect on the response.

Odds ratio (CI) (per unit of measurement) - odds ratio and its CI for a one-unit increase in the predictor (e.g., 1 year for age, 1 kg for weight). For categorical variables, it represents the odds ratio relative to the reference category.

Odds ratio (CI) (per user-defined change) - odds ratio and its CI based on a user-specified change in the predictor value. For example, this might be a 10-year change in age or a defined change in drug concentration. This column allows users to interpret effect sizes more meaningfully in the context of practical changes.