Model assesment, selection and inference after selection

Example notebooks in R using rstanarm, rstan, bayesplot, loo, projpred.

Talks

• Model assessment, comparison and selection at Master class in Bayesian statistics, CIRM, Marseille
  • Slides
  • Video
• Model assessmen and model selection aka Basics of cross-validation tutorial at StanCon 2018 Helsinki
  • Slides
  • Video
  • Code for figures
• Regularized horseshoe talk at StanCon 2018 Asilomar
  • Slides
  • Video
• Model selection tutorial at StanCon 2018 Asilomar
  • Slides
  • Video

Outline of the StanCon 2018 Asilomar tutorial and links to notebooks

• Basics of predictive performance estimation
• When cross-validation is not needed
  • Simple model we trust - betablockers
• When cross-validation is useful
  • We don’t trust the model - roaches
  • Complex model with posterior dependencies - collinear
• On accuracy of cross-validation
• Cross-validation and hierarchical models
• When cross-validation is not enough
  • large number of models - diabetes
• loo 2.0
• Projection predictive model selection
  • collinear
  • diabetes
  • projpred

Additional case studies

• loo 2.0
  • Bayesian Stacking and Pseudo-BMA weights using the loo package
  • Leave-one-out cross-validation for non-factorizable models
  • Approximate leave-future-out cross-validation for time series models
• Projection predictive model selection (projpred) examples
  • collinearity - mesquite
  • random data vs original data - candy
  • winequality-red
  • bodyfat
  • See also projpred quick start vignette
• LOO-R^2
• Cross-validation for hierarchical models

See also

• When LOO and other cross-validation approaches are valid
• Parsimonious principle vs integration over all uncertainties
• Comments on Limitations of Bayesian Leave-One-Out Cross-Validation for Model Selection
• I am the supercargo
• Nice horseshoe example in Bayes Sparse Regression case study by Michael Betancourt

References

• Bürkner, P.-C., Gabry, J., Vehtari, A. (2018). Leave-one-out cross-validation for non-factorizable normal models. arXiv:1810.10559
• Gelman, A., Hwang, J., and Vehtari, A. (2014). Understanding predictive information criteria for Bayesian models. Statistics and Computing, 24(6):997–1016. Preprint
• Piironen, J., Paasiniemi, M., and Vehtari, A. (2018). Projective Inference in High-dimensional Problems: Prediction and Feature Selection. arXiv:1810.02406
• Piironen, J. and Vehtari, A. (2016), Comparison of Bayesian predictive methods for model selection, Statistics and Computing 27(3), 711–735. Online
• Piironen, J., and Vehtari, A. (2017). On the hyperprior choice for the global shrinkage parameter in the horseshoe prior. Proceedings of the 20th International Conference on Artificial Intelligence and Statistics, PMLR 54:905-913. Online
• Piironen, J., and Vehtari, A. (2017). Sparsity information and regularization in the horseshoe and other shrinkage priors. In Electronic Journal of Statistics, 11(2):5018-5051. Online
• Piironen, J., and Vehtari, A. (2018). Iterative supervised principal components. Proceedings of the 21th International Conference on Artificial Intelligence and Statistics, accepted for publication. arXiv preprint arXiv:1710.06229
• Vehtari, A., Gelman, A., Gabry, J. (2017). Practical Bayesian model evaluation using leave-one-out cross-validation and WAIC. Statistics and Computing. 27(5):1413–1432. arXiv preprint.
• Vehtari, A., Gelman, A., Gabry, J. (2017). Pareto smoothed importance sampling. arXiv preprint.
• Vehtari, A., Mononen, T., Tolvanen, V., and Winther, O. (2016). Bayesian leave-one-out cross-validation approximations for Gaussian latent variable models. JMLR, 17(103):1–38. Online
• Vehtari, A. and Ojanen, J.: 2012, A survey of Bayesian predictive methods for model assessment, selection and comparison, Statistics Surveys 6, 142–228. Online
• Williams, D. R., Piironen, J., Vehtari, A., and Rast, P. (2018). Bayesian estimation of Gaussian graphical models with projection predictive selection. arXiv:1801.05725
• Yao, Y., Vehtari, A., Simpson, D., and Gelman, A. (2017). Using stacking to average Bayesian predictive distributions. In Bayesian Analysis, doi:10.1214/17-BA1091, Online