Model assesment, selection and inference after selection
Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.
FAQ
Talks
- Use of reference models in variable selection at Laplace’s demon seminar series
- Model assessment and model selection lectures of Bayesian data analysis course
- Model assessment and model selection at StanCon 2019 Cambridge
- Model assessment, comparison and selection at Master class in Bayesian statistics, CIRM, Marseille
- Model assessment and model selection aka Basics of cross-validation tutorial at StanCon 2018 Helsinki
- Regularized horseshoe talk at StanCon 2018 Asilomar
- Model selection tutorial at StanCon 2018 Asilomar
Links to notebooks
- When cross-validation is not needed
- When cross-validation is useful
- We don’t trust the model - roaches
- Complex model with posterior dependencies - collinear
- When cross-validation is not enough
- LOO-R^2
- Cross-validation for hierarchical models
- Projection predictive model selection (projpred)
- loo 2.0
- Projection predictive model selection (projpred) examples
See also
References
- Afrabandpey, H., Peltola, T., Piironen, J., Vehtari, A., and Kaski, S. (2019). Making Bayesian predictive models interpretable: A decision theoretic approach. arXiv preprint arXiv:1910.09358
- Bürkner, P.-C., Gabry, J., Vehtari, A. (2018). Leave-one-out
cross-validation for non-factorizable normal
models. arXiv:1810.10559
- Bürkner, P.-C., Gabry, J., Vehtari, A. (2020). Approximate leave-future-out
cross-validation for time series models. Journal of Statistical Computation and Simulation, doi:10.1080/00949655.2020.1783262. Online. Preprint arXiv:1902.06281
- Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding
predictive information criteria for Bayesian models. Statistics and
Computing, 24(6):997–1016.
Preprint
- Gelman, A., Goodrich, B., Gabry, J., and Vehtari, A. (2018). R-squared for Bayesian regression models. The American Statistician, doi:10.1080/00031305.2018.1549100. Online.
- Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2019). Bayesian leave-one-out
cross-validation for large data. Thirty-sixth International Conference on Machine Learning,
PMLR 97:4244–4253. Online.
-
- Magnusson, M., Andersen, M.R., Jonasson, J., Vehtari, A. (2020). Leave-one-out cross-validation for Bayesian model comparison in large data. Proceedings of the 23rd International Conference on Artificial Intelligence and Statistics (AISTATS), PMLR 108:341-351. Online. preprint arXiv:2001.00980.
- Paananen, T., Piironen, J., Bürkner, P.-C., and Vehtari, A. (2020). Implicitly adaptive importance sampling.
arXiv:1906.08850
- Pavone, F., Piironen, J., Bürkner, P.-C., and Vehtari, A- (2020). Using reference models in variable selection. arXiv preprint arXiv:2004.13118
- Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian
predictive methods for model selection, Statistics and Computing
27(3), 711–735. Online
- Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for
the global shrinkage parameter in the horseshoe prior. Proceedings
of the 20th International Conference on Artificial Intelligence and
Statistics, PMLR 54:905-913.
Online
- Piironen, J., and Vehtari, A. (2017). Sparsity information and
regularization in the horseshoe and other shrinkage priors. In
Electronic Journal of Statistics, 11(2):5018-5051.
Online
- Piironen, J., and Vehtari, A. (2018). Iterative supervised principal
components. Proceedings of the 21th International Conference on
Artificial Intelligence and Statistics, accepted for
publication.
arXiv preprint arXiv:1710.06229
- Piironen, J., Paasiniemi, M., and Vehtari, A. (2020). Projective
Inference in High-dimensional Problems: Prediction and Feature
Selection. Electronic Journal of Statistics, 14(1):2155-2197. Online. Preprint arXiv:1810.02406
- Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model
evaluation using leave-one-out cross-validation and WAIC. Statistics
and Computing. 27(5):1413–1432. arXiv
preprint.
- Vehtari, A., Simpson, D., Gelman, A., Yao, Y., and Gabry, J. (2019). Pareto smoothed
importance sampling. arXiv
preprint.
- Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016).
Bayesian leave-one-out cross-validation approximations for Gaussian
latent variable models. JMLR, 17(103):1–38.
Online
- Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive
methods for model assessment, selection and comparison, Statistics
Surveys 6, 142–228. Online
- Williams, D. R., Piironen, J., Vehtari, A., and Rast,
P. (2018). Bayesian estimation of Gaussian graphical models with
projection predictive selection. arXiv:1801.05725
- Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using
stacking to average Bayesian predictive distributions. In Bayesian
Analysis, doi:10.1214/17-BA1091,
Online