Robust and sparse LW-PLS

Why

LW-PLS already gives tiny weights to training samples that are far from the query, so outliers far away have little influence. Outliers among the neighbors of the query are a different matter: a wrong reference value (vertical outlier) or an unusual spectrum (leverage point) close to the query enters its local model with a large weight.

With robust = TRUE, every local model is fitted by partial robust M-regression (PRM; Serneels et al., 2005). Each training sample gets the weight

\[ \omega_i \, w_i^r \, w_i^t, \]

the product of its similarity weight \(\omega_i\), a weight \(w_i^r\) that decreases with its residual in the local model, and a weight \(w_i^t\) that decreases with its distance to the center of the local scores. The local model is refitted with the new weights until it is stable. The weights come from the Fair function (default) or the Hampel function (weight_function = "hampel"), which gives a zero weight to the most outlying samples. All the medians that PRM uses are weighted by the similarity weights, so that the robust model stays local.

Example: wrong reference values

We replace 20% of the water contents of the training set by wrong values, shifted by 15 to 30 percentage points, as could happen with sample mix-ups or transcription errors. The test set is left untouched.

contaminate <- function(data, fraction, seed = 1) {
  set.seed(seed)
  bad <- sample(nrow(data), round(fraction * nrow(data)))
  shift <- sample(c(-1, 1), length(bad), replace = TRUE) * runif(length(bad), 15, 30)
  data$water[bad] <- data$water[bad] + shift
  data
}

meats_train_20 <- contaminate(meats_train, 0.2)

The outliers are also in the assessment sets of the resamples, where their errors would dominate the usual RMSE and favor the model that follows them. We tune with the trimmed RMSE instead, which leaves out the 20% largest errors (trimmed_rmse(), provided by lwpls):

set.seed(2)
folds_20 <- vfold_cv(meats_train_20, v = 5)
grid <- expand.grid(num_comp = c(4, 6, 8, 10, 12), localization = c(0.5, 1, 2, 4))

tune_lwpls <- function(robust) {
  wflow <- workflow(
    water ~ .,
    lwpls(num_comp = tune(), localization = tune()) |>
      set_mode("regression") |>
      set_engine("lwpls", robust = robust)
  )
  res <- tune_grid(wflow, folds_20, grid = grid, metrics = metric_set(trimmed_rmse))
  finalize_workflow(wflow, select_best(res, metric = "trimmed_rmse"))
}

lwpls_wflow <- tune_lwpls(robust = FALSE)
robust_wflow <- tune_lwpls(robust = TRUE)

Robust models are refitted for each number of components, so tuning them takes longer than tuning ordinary LW-PLS models.

We then train both tuned workflows on training sets with 0% to 30% of wrong reference values and measure their errors on the clean test set:

fractions <- c(0, 0.1, 0.2, 0.3)
test_rmse <- function(wflow, fraction) {
  fitted <- fit(wflow, contaminate(meats_train, fraction))
  rmse_vec(meats_test$water, predict(fitted, meats_test)$.pred)
}

results <- data.frame(
  fraction = rep(fractions, 2),
  model = rep(c("LW-PLS", "Robust LW-PLS"), each = length(fractions)),
  rmse = c(
    vapply(fractions, test_rmse, numeric(1), wflow = lwpls_wflow),
    vapply(fractions, test_rmse, numeric(1), wflow = robust_wflow)
  )
)
results
#>   fraction         model     rmse
#> 1      0.0        LW-PLS 2.583581
#> 2      0.1        LW-PLS 3.101320
#> 3      0.2        LW-PLS 3.478736
#> 4      0.3        LW-PLS 3.869655
#> 5      0.0 Robust LW-PLS 2.418944
#> 6      0.1 Robust LW-PLS 2.354993
#> 7      0.2 Robust LW-PLS 2.501034
#> 8      0.3 Robust LW-PLS 2.865060

ggplot(results, aes(100 * fraction, rmse, color = model)) +
  geom_line(linewidth = 0.8) +
  geom_point(size = 2) +
  scale_color_manual(values = c("LW-PLS" = "#969696", "Robust LW-PLS" = "#08519C")) +
  labs(
    x = "Wrong reference values in the training set (%)",
    y = "Test RMSE (water, %)",
    color = NULL
  ) +
  theme_bw()

The error of ordinary LW-PLS grows with the number of wrong reference values, while the robust local models are hardly affected. Both models were tuned on the contaminated training set, which is why the robust model is also better without contamination here. On clean data, robust models are usually a little less efficient than ordinary ones: use them when outliers are expected, and compare both by resampling with a robust metric.