Skip to contents

Approximate leave-one-out cross-validation

Source: R/loo.R
sample-loo.Rd

loo() runs Pareto-smoothed importance-sampling LOO and waic() the widely applicable information criterion, both on the frmtmb::log_lik() matrix, by handing it to loo::loo.matrix() and loo::waic.matrix() unchanged. The returned objects are the loo package's own, so print() and loo::pareto_k_table() work on them directly. loo_compare() computes the criterion for each draws object it is given and ranks them; handed criteria instead of draws, it is loo::loo_compare() itself. psis() returns the smoothed importance weights alone.

Usage

# S3 method for class 'frmtmb_draws'
loo(x, ..., compare = TRUE, resp = NULL, ndraws = NULL, model_names = NULL)

# S3 method for class 'frmtmb_draws'
waic(x, ..., compare = TRUE, resp = NULL, ndraws = NULL, model_names = NULL)

# S3 method for class 'frmtmb_draws'
loo_compare(x, ..., criterion = c("loo", "waic"), model_names = NULL)

psis(log_ratios, ...)

# S3 method for class 'frmtmb_draws'
psis(
  log_ratios,
  newdata = NULL,
  resp = NULL,
  model_name = NULL,
  ndraws = NULL,
  ...
)

# S3 method for class 'frmtmb_draws'
LOO(x, ...)

# S3 method for class 'frmtmb_draws'
WAIC(x, ...)

Arguments

x

A frmtmb_draws from frm_sample(), or (for loo_compare()) already-computed criteria.

...

Further models: for loo() and waic() the draws objects given without a name, which brms compares in one call, and brms's named options, refused by name where they would change the answer (newdata and the other arguments log_lik() refuses, moment_match, reloo) and passed on where they are the loo package's (r_eff, save_psis, cores). For loo_compare(), further models.

compare

With several models, whether to add their comparison, loo::loo_compare() of the criteria, as brms does.

ndraws, resp

Passed to frmtmb::log_lik().

model_names

Row names for the comparison; the default deparses the arguments, as loo does.

criterion

Which criterion loo_compare() computes for each draws object.

log_ratios

For psis(), the draws object whose negative pointwise log-likelihood supplies the importance ratios.

newdata

For psis(), accepted in brms's own second position and refused, because frmtmb::log_lik() does not take it.

model_name

For psis(), brms's label for the model. Accepted and unused: a psis object has nothing to label.

Value

A loo, waic, compare.loo or psis object from the loo package; for several models, loo() and waic() give brms's list of them, loos, with their comparison diffs and brms's ic_diffs__, of class frmtmb_loolist.

Details

LOO() and WAIC() are brms's deprecated capitalized spellings and are defined only to name their replacements.

A time series, and what is left out

On a model with brms's default cov = FALSE ar(), ma() or arma() term, a column of the frmtmb::log_lik() matrix is the density of one row GIVEN THE OBSERVED EARLIER ROWS of its group, the one-step-ahead density. That is what brms's log_lik() returns, so this is brms's elpd. It also fixes what the estimate means: leaving out column t leaves out one conditional density, not the series and not the row's influence on its neighbors, because every retained column still reads y_t through its own lagged residuals. Read it as "how well the one-step density fits each row", and refit without a group for a held-out-series answer. Under the cov = TRUE covariance form a column is the row's density given the rest of its group, before and after it, as brms's log_lik() gives it (sample-log_lik, "Residual covariance terms").

Priors, and what these numbers mean

These are posterior quantities, and they inherit the standing of the draws they are computed from. Both of frm_sample()'s routes apply brms's default priors, so these numbers are regularized the way brms regularizes them unless the call opted out with prior = "flat". Under that opt-out the elpd is likelihood-shaped and unregularized: expect Pareto k warnings for models with many group-level parameters, because a flat prior leaves those to be identified by the data alone, and an influential observation then moves them a long way. The maximum-likelihood answer to the same question is AIC() on the fits, or frmtmb::frm_bootstrap().

Relative efficiency

r_eff defaults to loo::relative_eff() on the chain structure of the draws, which is what brms does. A subset of the draws (ndraws, draw_ids) no longer has that structure, so its relative efficiency is taken on one chain, as brms takes it. The same rule gives the weights of psis(), the loo_*() functions and pp_check()'s loo_* types.

See also

frmtmb::log_lik(), frmtmb::bayes_R2()

Examples

# \donttest{
if (requireNamespace("tmbstan", quietly = TRUE) &&
    requireNamespace("rstan", quietly = TRUE) &&
    requireNamespace("loo", quietly = TRUE) &&
    !frmtmb.sample:::tmbstan_build_broken()) {
  set.seed(9)
  dd <- data.frame(x = rnorm(60), g = factor(rep(1:6, 10)))
  dd$y <- rnorm(60, 1 + 0.5 * dd$x + rnorm(6, 0, 0.5)[dd$g], 1)

  # sample with priors: an elpd is a posterior quantity
  d1 <- frm_sample(bf(y ~ x + (1 | g)), family = gaussian(),
                   data = dd, chains = 1, iter = 500, refresh = 0)
  d2 <- frm_sample(bf(y ~ 1 + (1 | g)), family = gaussian(),
                   data = dd, chains = 1, iter = 500, refresh = 0)
  loo_compare(d1, d2)
  # brms's spelling of the same comparison
  loo(d1, d2)
  # the same thing, one step at a time
  loo_compare(loo(d1), loo(d2))
}
#> frm_sample(): default priors (brms 2.23 defaults; prior = "flat" puts brms's (flat) in every slot)
#>   Intercept          student_t(3, 0.8, 2.5)
#>   sigma              student_t(3, 0, 2.5)  [natural scale]
#>   sd                 student_t(3, 0, 2.5)  [natural sd scale]
#>   b                  (flat), as brms leaves slopes
#> Warning: The largest R-hat is 1.08, indicating chains have not mixed.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#r-hat
#> Warning: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#bulk-ess
#> Warning: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#tail-ess
#> frm_sample(): default priors (brms 2.23 defaults; prior = "flat" puts brms's (flat) in every slot)
#>   Intercept          student_t(3, 0.8, 2.5)
#>   sigma              student_t(3, 0, 2.5)  [natural scale]
#>   sd                 student_t(3, 0, 2.5)  [natural sd scale]
#>   b                  (flat), as brms leaves slopes
#> Warning: Bulk Effective Samples Size (ESS) is too low, indicating posterior means and medians may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#bulk-ess
#> Warning: Tail Effective Samples Size (ESS) is too low, indicating posterior variances and tail quantiles may be unreliable.
#> Running the chains for more iterations may help. See
#> https://mc-stan.org/misc/warnings.html#tail-ess
#>   model elpd_diff se_diff p_worse diag_diff diag_elpd
#>  model1       0.0     0.0      NA                    
#>  model2      -4.9     2.8    0.96   N < 100          
#> 
#> Diagnostic flags present.
#> See ?`loo-glossary` (sections `diag_diff` and `diag_elpd`)
#> or https://mc-stan.org/loo/reference/loo-glossary.html.
# }