Does performance_cv() use Nakigawa-R2 on lmer() mixed models when using k-fold cv? #576
Replies: 1 comment
-
performance_cv() uses the same formula for all models. For lmer() models, that would generally be a conditionals R2 estimate because it is based on simple level-1 residuals (so any random effects are included). If you supply a data frame to use for prediction and these are all from new groups, it would be a marginal R2 because no information is available in the model to estimate the random effect values for the new groups. Note that simple k-fold or holdout cross-validation may not be appropriate for nested data depending on the target of your inference (new observations of the same groups vs new groups). If your target is new groups, then you would need to do the resampling of groups for group-level-k-fold cross-validation manually. |
Beta Was this translation helpful? Give feedback.
-
I have been trying to understand the code within
performance_cv()
and as it appears to me, it uses a very standard R-squared formula for the CV.R2 <- 1 - MSE / mean((test_resp - mean(test_resp, na.rm = TRUE))^2, na.rm = TRUE)
When using
lmer()
, normallyperformance()
would default to using Nakigawa R2's.Is this state being inherited?
If it is, is the resultant R2 based on the Conditional or the Marginal R2?
I have a question here up on Cross Validated that may explain the issue more, unfortunately no-one has addressed it, and I need to know. I also feel it is important to that
performance_cv()
also uses Nakigawa methods on mixed models.Beta Was this translation helpful? Give feedback.
All reactions