LOO-adjusted R-squared of draws

brms's loo_R2() on draws from frm_sample(): one minus the ratio of the variance of the leave-one-out residuals to the variance of the response, 1 - Var(y - E_loo(ypred)) / Var(y), with uncertainty from the Bayesian bootstrap of Vehtari and Lampinen (2002). The leave-one-out prediction of each row is its posterior_epred() column under the row's Pareto-smoothed importance weights, as in brms; each bootstrap draw reweights the observations by a flat Dirichlet. Values are clipped to [-1, 1], as brms clips them.

Usage

# S3 method for class 'frmtmb_draws'
loo_R2(
  object,
  resp = NULL,
  summary = TRUE,
  robust = FALSE,
  probs = c(0.025, 0.975),
  seed = NULL,
  args_epred = list(),
  args_loglik = list(),
  ...
)

Arguments

object: A frmtmb_draws from frm_sample().
resp: For a multivariate model, which responses, in brms's spelling (ya for a column y_a), as in brms and frmtmb::bayes_R2(). NULL, the default, is every response, one R2<resp> column each.
summary: If TRUE (the default), summarize the bootstrap draws into estimate, error and quantiles; if FALSE, return the ndraws x responses matrix of them.
robust: If TRUE, the summary is the median and MAD.
probs: Quantiles for the summary.
seed: A seed for the Bayesian bootstrap. The random number state is restored afterwards, as in brms.
args_epred, args_loglik: Named lists of arguments for posterior_epred() alone and for frmtmb::log_lik() alone.
...: ndraws or draw_ids, which choose the draws for both, and further arguments of posterior_epred().

Value

A summary matrix with one row per response, or with summary = FALSE the matrix of bootstrap draws.

Details

The weights come from one frmtmb::log_lik() and one loo::psis() per response, on the same draws as the prediction. With add_criterion(x, "loo_R2") the bootstrap draws are stored, and a later loo_R2() without further arguments reads them, as brms's does.

An ordinal or categorical family is refused, as frmtmb::bayes_R2() refuses it: brms refuses the categorical families and treats ordinal probabilities as a continuous prediction, with a warning. newdata is refused for the reason loo_epred() gives: a row the model was not fitted on has no observation to leave out.

References

Vehtari, A. and Lampinen, J. (2002). Bayesian model assessment and comparison using cross-validation predictive densities. Neural Computation, 14(10), 2439-2468.

Examples

# \donttest{
if (requireNamespace("tmbstan", quietly = TRUE) &&
    requireNamespace("rstan", quietly = TRUE) &&
    requireNamespace("loo", quietly = TRUE) &&
    !frmtmb.sample:::tmbstan_build_broken()) {
  set.seed(9)
  dd <- data.frame(x = rnorm(60))
  dd$y <- rnorm(60, 1 + 0.5 * dd$x, 1)
  ds <- frm_sample(bf(y ~ x), family = gaussian(), data = dd,
                   chains = 2, iter = 500, refresh = 0)
  loo_R2(ds, seed = 1)
  # in-sample R2 is larger: it scores rows the draws were fitted to
  bayes_R2(ds)
}
#> frm_sample(): default priors (brms 2.23 defaults; prior = "flat" puts brms's (flat) in every slot)
#>   Intercept          student_t(3, 0.8, 2.5)
#>   sigma              student_t(3, 0, 2.5)  [natural scale]
#>   b                  (flat), as brms leaves slopes
#>     Estimate  Est.Error      Q2.5     Q97.5
#> R2 0.3184854 0.07978614 0.1556851 0.4585772
# }