Cellwise-robust iterative regression imputation for mixed data

Source: R/imputeCellwise.R

Extends IRMI (Templ, Kowarik, and Filzmoser, 2011) with cellwise contamination handling. Each conditional regression uses a cell-weighted IRWLS engine where per-cell weights in the design matrix downweight contaminated cells without discarding entire observations.

imputeCellIRMI(
  data,
  method = "tukey",
  alpha = NULL,
  maxit = 100,
  maxit_irwls = 50,
  eps = 0.005,
  eps_irwls = 1e-06,
  uncert = "pmm",
  weight_update = "multivariate",
  init_weights = "ddc",
  hard_threshold = 0.5,
  trace = FALSE
)

Arguments

data

a data.frame with missing values (mixed continuous and categorical variables are supported).

method

weight function: "tukey" (default, Tukey bisquare) or "huber" (Huber).

alpha

tuning constant. NULL (default) uses 1.345 for Huber and 4.685 for Tukey, giving 95% efficiency at the normal model.

maxit

maximum number of outer IRMI iterations (default: 100).

maxit_irwls

maximum number of inner IRWLS iterations per regression (default: 50).

eps

convergence tolerance for the outer loop (default: 5e-3). Convergence is declared when the relative change in imputed values falls below this threshold.

eps_irwls

convergence tolerance for the inner IRWLS (default: 1e-6).

uncert

imputation uncertainty method: "pmm" (predictive mean matching, default), "normalerror" (add normal noise), or "resid" (bootstrap residual).

weight_update

strategy for updating cell weights between outer iterations: "multivariate" (default) uses an MCD-based multivariate update for weight coherence across variables, or "univariate" updates each variable independently from its residuals.

init_weights

method for initialising cell weights, one of "ddc" (default; DetectDeviatingCells, requires the cellWise package and falls back to univariate weights when it is unavailable), "univariate" (per-column median/MAD standardisation), or "mcd" (minimum covariance determinant on the continuous block). The default is "ddc" because "mcd" downweights high-leverage points that carry the regression signal, which can make imputation worse than unconditional median imputation.

hard_threshold

numeric in \([0, 1]\). After convergence, cells with weight below this value are flagged as contaminated (default: 0.5).

trace

logical; if TRUE, print progress information.

Value

A list with components:

data_imputed

the imputed data.frame.

cellweights

\(n \times p\) matrix of final cell weights (1 = clean, 0 = fully downweighted). Categorical columns always have weight 1.

converged

logical indicating whether the outer loop converged.

iterations

number of outer iterations used.

Details

The algorithm works iteratively: in each outer iteration, every variable with missing values is used as response in a conditional regression on all remaining variables. For continuous responses, the custom cellIRWLS() engine fits a weighted regression where each cell in the design matrix receives its own weight reflecting potential cellwise contamination. For categorical responses, a weighted multinomial model is used. After each regression, cell weights for the response variable are updated from the residuals.

The algorithm proceeds as follows:

  1. Missing values are initialised using initialise.

  2. Initial cell weights are computed with cellWeights() on all continuous variables in the initialised data.

  3. Outer loop (up to maxit iterations):

    • For each variable \(j\) with missing values:

      • Form predictor matrix \(X\) (all other variables) and response \(y\) (variable \(j\)).

      • If \(j\) is continuous: fit cellIRWLS(X, y, w_cell, w_response) and impute missing values in \(j\) using the fitted model plus uncertainty.

      • If \(j\) is categorical: fit nnet::multinom() with row weights derived from the cell weight matrix and impute by sampling from predicted probabilities.

      • Update cell weights for \(j\) from residuals via cellWeightsFromResiduals().

    • Check convergence: relative change in imputed values falls below eps.

References

Templ, M., Kowarik, A. and Filzmoser, P. (2011). Iterative stepwise regression imputation using standard and robust methods. Computational Statistics & Data Analysis, 55(10), 2793–2806.

Author

Matthias Templ

Examples

# \donttest{
data(sleep, package = "VIM")
result <- imputeCellIRMI(sleep)
head(result$data_imputed)
#>   BodyWgt BrainWgt NonD Dream Sleep Span Gest Pred Exp Danger
#> 1   1.000      5.0  8.3   2.1  10.3 38.6  100    3   3      3
#> 2   1.000      6.6  6.3   2.0   8.3  4.5   42    3   1      3
#> 3   3.385     44.5  8.4   2.0  10.6 14.0   60    1   1      1
#> 4   0.920      5.7  8.3   2.1  10.7  3.5   25    5   2      3
#> 5   1.400      8.1  2.1   1.8   3.9 34.0   42    3   1      4
#> 6   0.425     11.4  9.1   0.7   9.8 27.0  180    4   4      4
image(result$cellweights, main = "Cell weights")


# With Huber weights (less aggressive downweighting)
result2 <- imputeCellIRMI(sleep, method = "huber", trace = TRUE)
#>   hard thresholding removed 79 outlying cells (set to NA)
#> --------------------------------------
#> cellIRMI: start of iteration 1
#>   imputing variable: 1 ( BodyWgt ) - continuous
#>   imputing variable: 2 ( BrainWgt ) - continuous
#>   imputing variable: 3 ( NonD ) - continuous
#>   imputing variable: 4 ( Dream ) - continuous
#>   imputing variable: 5 ( Sleep ) - continuous
#>   imputing variable: 6 ( Span ) - continuous
#>   imputing variable: 7 ( Gest ) - continuous
#>   imputing variable: 9 ( Exp ) - continuous
#>   multivariate weight update: flagged cells = 0
#>   convergence criterion: 0.738809
#> --------------------------------------
#> cellIRMI: start of iteration 2
#>   imputing variable: 1 ( BodyWgt ) - continuous
#>   imputing variable: 2 ( BrainWgt ) - continuous
#>   imputing variable: 3 ( NonD ) - continuous
#>   imputing variable: 4 ( Dream ) - continuous
#>   imputing variable: 5 ( Sleep ) - continuous
#>   imputing variable: 6 ( Span ) - continuous
#>   imputing variable: 7 ( Gest ) - continuous
#>   imputing variable: 9 ( Exp ) - continuous
#>   multivariate weight update: flagged cells = 1
#>   convergence criterion: 0.021655
#> --------------------------------------
#> cellIRMI: start of iteration 3
#>   imputing variable: 1 ( BodyWgt ) - continuous
#>   imputing variable: 2 ( BrainWgt ) - continuous
#>   imputing variable: 3 ( NonD ) - continuous
#>   imputing variable: 4 ( Dream ) - continuous
#>   imputing variable: 5 ( Sleep ) - continuous
#>   imputing variable: 6 ( Span ) - continuous
#>   imputing variable: 7 ( Gest ) - continuous
#>   imputing variable: 9 ( Exp ) - continuous
#>   multivariate weight update: flagged cells = 1
#>   convergence criterion: 0.005233
#> --------------------------------------
#> cellIRMI: start of iteration 4
#>   imputing variable: 1 ( BodyWgt ) - continuous
#>   imputing variable: 2 ( BrainWgt ) - continuous
#>   imputing variable: 3 ( NonD ) - continuous
#>   imputing variable: 4 ( Dream ) - continuous
#>   imputing variable: 5 ( Sleep ) - continuous
#>   imputing variable: 6 ( Span ) - continuous
#>   imputing variable: 7 ( Gest ) - continuous
#>   imputing variable: 9 ( Exp ) - continuous
#>   multivariate weight update: flagged cells = 1
#>   convergence criterion: 0.00244
#> cellIRMI converged.

# Mixed data example
data(testdata)
result3 <- imputeCellIRMI(testdata$wna)
# }