This assignment is related to Lecture 3 and BDA3 Chapters 2 and 3. Use Frank Harrell’s recommendations on how to state results in Bayesian two group comparisons (and note that there is no point null hypothesis testing in this assignment).
The maximum amount of points from this assignment is 9.
We have prepared two quarto templates specific to this assignment to help you get started:
A recommended template (html, qmd, pdf) which uses some additional packages, which however requires a bit more set-up work to run and
a simple template (html, qmd, pdf) which doesn’t use those additional packages and is therefore easier to get to run.
The grading will be done in peergrade. All grading questions and evaluations for this assignment are contained within this document in the collapsible Rubric blocks.
Further information
The recommended tool in this course is R (with the IDE RStudio).
If you want to install R and RStudio locally, download R and RStudio.
There are tons of tutorials, videos and introductions to R and RStudio online. You can find some initial hints from RStudio Education pages.
When working with R, we recommend writing the report using quarto and the provided template. The template includes the formatting instructions and how to include code and figures.
Instead of quarto, you can use other software to make the PDF report, but the the same instructions for formatting should be used.
Report all results in a single, anonymous *.pdf -file and submit it in peergrade.io.
The course has its own R package aaltobda with data and functionality to simplify coding. The package is pre-installed in JupyterHub. To install the package on your own system, run the following code (upgrade="never" skips question about updating other packages):
Many of the exercises can be checked automatically using the R package markmyassignment (pre-installed in JupyterHub). Information on how to install and use the package can be found in the markmyassignment documentation. There is no need to include markmyassignment results in the report.
Recommended additional self study exercises for each chapter in BDA3 are listed in the course web page. These will help to gain deeper understanding of the topic.
Deadlines for all assignments can be found on the course web page and in Peergrade. You can set email alerts for the deadlines in Peergrade settings.
You are allowed to discuss assignments with your friends, but it is not allowed to copy solutions directly from other students or from internet.
You can copy, e.g., plotting code from the course demos, but really try to solve the actual assignment problems with your own code and explanations.
Do not share your answers publicly.
Do not copy answers from the internet or from previous years. We compare the answers to the answers from previous years and to the answers from other students this year.
Do not submit empty PDFs, almost empty PDFs, copy of the questions, nonsense generated by yourself or AI, as these are just harming the other students as they can’t do peergrading for the empty or nonsense submissions. Violations of this rule will be reported and investigated in the same way was plagiarism.
If you have any suggestions or improvements to the course material, please post in the course chat feedback channel, create an issue, or submit a pull request to the public repository!
Rubric S1: General information - 7.5/100 points
Q1:
Can you open the PDF and it’s not blank nor nonsense? If the pdf is blank, nonsense, or something like only a copy of the questions, 1) report it as problematic in Peergrade-interface to get another report to review, and 2) send a message to TAs.
Q2:
Is the report anonymous?
2 Inference for normal mean and deviation (3 points)
A factory has a production line for manufacturing car windshields. A sample of windshields has been taken for testing hardness. The observed hardness values \(\mathbf{y}_1\) can be found in the dataset windshieldy1 in the aaltobda package.
We may assume that the observations follow a normal distribution with an unknown standard deviation \(\sigma\). We wish to obtain information about the unknown average hardness \(\mu\). For simplicity we assume standard uninformative prior discussed in the book, that is, \(p(\mu, \sigma) \propto \sigma^{-1}\). It is not necessary to derive the posterior distribution in the report, as it has already been done in the book (see section 3.2).
Subtask 2.a)
Formulate
the likelihood,
the prior, and
the resulting posterior.
Subtask 2.b)
What can you say about the unknown \(\mu\)?
Compute and report the point estimate \(E(\mu|y)\),
compute and report a posterior 95%-interval,
plot the density, and
write interpretation of the result in text.
Tip
Posterior intervals are also called credible intervals and are different from confidence intervals.
Subtask 2.c)
What can you say about the hardness of the next windshield coming from the production line before actually measuring the hardness?
Compute and report the point estimate \(E(\tilde{y}|y)\),
compute and report a posterior predictive 95%-interval,
plot the density, and
write interpretation of the result in text.
Tip
Predictive intervals are different from posterior intervals.
With a conjugate prior a closed form posterior is Student’s \(t\) form (see equations in the book).
Rubric S2: Inference for normal mean and deviation (3 points)
Q3:
Is the source code included?
No
Yes
Q4:
Are the likelihood, prior and the posterior for computing the average hardness value reported? It is ok to refer to the book instead of deriving the distributions.
No
Yes, but some are missing
Yes
Q5:
In part a), were the point estimates and posterior interval provided? (The posterior mean should be close to … and 95% posterior interval should be around …)
No
Yes, but seem incorrect or only one estimate was reported
Yes, and the reported values seem plausible
Q6:
In part b), was the density plotted?
No
Yes, but seem incorrect
Yes, and the plot seems plausible
Q7:
For b)-part, was a formula or a simulation method presented for computing the posterior predictive distribution? It is ok to refer to the book.
No
Yes, but seems incorrect
Yes
Q8:
For c)-part, were the point estimate and predictive interval provided? (95% predictive interval should be around … and the mean the same as in a)-part).
No
Yes, but seems incorrect
Yes, and the reported values seem plausible
Q9:
For c)-part, was the density plotted?
No
Yes, but seems incorrect
Yes, and the plot seem plausible
3 Inference for the difference between proportions (3 points)
An experiment was performed to estimate the effect of beta-blockers on mortality of cardiac patients. A group of patients was randomly assigned to treatment and control groups: out of 674 patients receiving the control, 39 died, and out of 680 receiving the treatment, 22 died. Assume that the outcomes are independent and binomially distributed, with probabilities of death of \(p_0\) and \(p_1\) under the control and treatment, respectively. Set up a noninformative or weakly informative prior distribution on \((p_0,p_1)\).
Subtask 3.a)
Formulate
the likelihood,
the prior, and
the resulting posterior.
Subtask 3.b)
Summarize the posterior distribution for the odds ratio, \[
\mathrm{OR} = (p_1/(1-p_1))/(p_0/(1-p_0)).
\]
Compute and report the point estimate \(E(\mathrm{OR}|y_0,y_1)\),
With a conjugate prior, a closed-form posterior is the Beta form for each group separately (see equations in the book). You can use rbeta() to sample from the posterior distributions of \(p_0\) and \(p_1\), and use this sample and odds ratio equation to get a sample from the distribution of the odds ratio.
Subtask 3.c)
Use at least two different priors, and discuss the sensitivity of your inference to your choice of prior density with a couple of sentences.
Rubric S3: Inference for the difference between proportions (3 points)
Q10:
Is the source code included?
No
Yes
Q11:
Are the likelihood, prior and the posterior for the death probabilities reported? It is ok to refer to the book instead of deriving the distributions.
No
Yes, but some are missing
Yes
Q12:
In part a), was the simulation algorithm for computing the posterior of the odds ratio presented or implemented?
No
Yes, but seems incorrect
Yes
Q13:
In part a), was the odds ratio summarized with a point estimate and a posterior interval? (The mean should be close to … and 95% posterior interval approximately …)
No
Yes, but results seem incorrect
Yes, and the results seem plausible
Q14:
In part b), was some discussion about testing alternative priors provided? (For example, one could have repeated the computations in a)-part with a couple of alternative priors and reported these results or some related general conclusions briefly)
Not at all
Some analysis was provided but it was lacking or did not make sense
Some alternative priors were tested and some sensible discussion provided
4 Inference for the difference between normal means (3 points)
Consider a case where the same factory has two production lines for manufacturing car windshields. Independent samples from the two production lines were tested for hardness. The hardness measurements for the two samples \(\mathbf{y}_1\) and \(\mathbf{y}_2\) be found in the datasets windshieldy1 and windshieldy2 in the aaltobda package.
We assume that the samples have unknown standard deviations \(\sigma_1\) and \(\sigma_2\). Use uninformative or weakly informative priors and answer the following questions:
Subtask 4.a)
Formulate
the likelihood,
the prior, and
the resulting posterior.
Subtask 4.b)
What can you say about \(\mu_d = \mu_1 - \mu_2\)?
Compute and report the point estimate \(E(\mu_d|y_1, y_2)\),
With a conjugate prior, a closed-form posterior is Student’s \(t\) form for each group separately (see equations in the book). You can use the rtnew() function to sample from the posterior distributions of \(\mu_1\) and \(\mu_2\), and use this sample to get a sample from the distribution of the difference \(\mu_d = \mu_1 - \mu_2\).
Subtask 4.c)
Given this specific model, what is the probability that the means are exactly the same (\(\mu_1 = \mu_2\))? Explain your reasoning.
Rubric S4: Inference for the difference between normal means (3 points)
Q15:
Is source code included?
No
Yes
Q16:
Are the likelihood, prior and the posterior for the windshield hardness values reported? (It is also ok to refer to the book or related formulas from exercise 1)
No
Yes, but some are missing
Yes
Q17:
In part a), was the simulation algorithm for computing the difference in the means presented or implemented?
No
Yes, but seems to be incorrect
Yes
Q18:
In part a), was the posterior for the difference between the means summarized with point and interval estimates? (The mean should be close to … and 95% posterior interval … or something close to it)
No answer
Yes, but results seem incorrect or only one estimate was given
Yes, and results seem reasonable
Q19:
Were some analysis or discussion provided for assessing whether the means could be the same? …
No analysis or explanation is given
Yes, but the analysis or explanation seems incorrect
Yes, and the analysis or explanation seems plausible
5 Overall quality of the report
Rubric S5: Overall quality of the report - 7.5/100 points
Q20:
Does the report include comment on whether AI was used, and if AI was used, explanation on how it was used?
No
Yes
Q21:
Does the report follow the formatting instructions?
Not at all
Little
Mostly
Yes
Q22:
In case the report doesn’t fully follow the general and formatting instructions, specify the instructions that have not been followed. If applicable, specify the page of the report, where this difference is visible. This will help the other student to improve their reports so that they are easier to read and review. If applicable, specify the page of the report, where this difference in formatting is visible.
Q23:
Please also provide feedback on the presentation (e.g. text, layout, flow of the responses, figures, figure captions). Part of the course is practicing making data analysis reports. By providing feedback on the report presentation, other students can learn what they can improve or what they already did well. You should be able to provide constructive or positive feedback for all non-empty and non-nonsense reports. If you think the report is perfect, and you can’t come up with any suggestions how to improve, you can provide feedback on what you liked and why you think some part of the report is better than yours.
---title: "Assignment 3"author: "Aki Vehtari et al."format: html: toc: true code-tools: true code-line-numbers: true number-sections: true mainfont: Georgia, serif page-layout: articleeditor: sourcefilters: - includes/assignments.lua ---# General informationThis assignment is related to Lecture 3 and BDA3 Chapters 2 and 3. Use [Frank Harrell'srecommendations](https://hbiostat.org/blog/post/bayes-freq-stmts/) onhow to state results in Bayesian two group comparisons (and note thatthere is no *point null hypothesis testing* in this assignment).**The maximum amount of points from this assignment is 9.**We have prepared **two quarto templates specific to this assignment** to help you get started:- A recommended template ([html](template3.html), [qmd](https://avehtari.github.io/BDA_course_Aalto/assignments/template3.qmd), [pdf](template3.pdf)) which uses some additional packages, which however requires a bit more set-up work to run and- a simple template ([html](simple_template3.html), [qmd](https://avehtari.github.io/BDA_course_Aalto/assignments/simple_template3.qmd), [pdf](simple_template3.pdf)) which doesn't use those additional packages and is therefore easier to get to run.::: {.hint} **Reading instructions:** - [**The reading instructions for BDA3 Chapter 1**](../BDA3_notes.html#ch2).- [**The reading instructions for BDA3 Chapter 2**](../BDA3_notes.html#ch3).{{< include includes/_grading_instructions.md >}}:::{{< include includes/_general_info.md >}}# Inference for normal mean and deviation (3 points) A factory has a production line for manufacturing car windshields. Asample of windshields has been taken for testing hardness. The observed hardness values $\mathbf{y}_1$ can be found in the dataset `windshieldy1` in the `aaltobda` package.We may assume that the observations follow a normal distribution with anunknown standard deviation $\sigma$. We wish to obtain information aboutthe unknown average hardness $\mu$. For simplicity we assume standarduninformative prior discussed in the book, that is,$p(\mu, \sigma) \propto \sigma^{-1}$. It is not necessary to derive theposterior distribution in the report, as it has already been done in thebook (see section 3.2).::: {.subtask letter=a}Formulate 1. the likelihood,2. the prior, and 3. the resulting posterior.:::::: {.subtask letter=b}What can you say about the unknown $\mu$?* Compute and report the point estimate $E(\mu|y)$, * compute and report a posterior 95%-interval, * plot the density, and* write interpretation of the result in text.:::::: {.hint}Posterior intervals are also called credibleintervals and are different from confidence intervals. :::::: {.subtask letter=c}What can you say about the hardness of the next windshield comingfrom the production line before actually measuring the hardness?* Compute and report the point estimate $E(\tilde{y}|y)$, * compute and report a posterior **predictive** 95%-interval,* plot the density, and* write interpretation of the result in text.:::::: {.hint}Predictive intervals are different from posteriorintervals.With a conjugate prior a closed form posterior is Student's $t$form (see equations in the book). :::::: {.rubric}* Is the source code included? * No * Yes* Are the likelihood, prior and the posterior for computing the average hardness value reported? It is okto refer to the book instead of deriving the distributions. * No * Yes, but some are missing * Yes* In part a), were the point estimates and posterior interval provided? (The posterior mean should beclose to **...** and 95% posterior interval should be around **...**) * No * Yes, but seem incorrect or only one estimate was reported * Yes, and the reported values seem plausible* In part b), was the density plotted? * No * Yes, but seem incorrect * Yes, and the plot seems plausible* For b)-part, was a formula or a simulation method presented for computing the posterior predictivedistribution? It is ok to refer to the book. * No * Yes, but seems incorrect * Yes* For c)-part, were the point estimate and predictive interval provided? (95% predictive interval shouldbe around **...** and the mean the same as in a)-part). * No * Yes, but seems incorrect * Yes, and the reported values seem plausible* For c)-part, was the density plotted? * No * Yes, but seems incorrect * Yes, and the plot seem plausible:::# Inference for the difference between proportions (3 points) An experiment was performed to estimate the effect of beta-blockers onmortality of cardiac patients. A group of patients was randomly assignedto treatment and control groups: out of 674 patients receiving thecontrol, 39 died, and out of 680 receiving the treatment, 22 died.Assume that the outcomes are independent and binomially distributed,with probabilities of death of $p_0$ and $p_1$ under the control andtreatment, respectively. Set up a noninformative or weakly informativeprior distribution on $(p_0,p_1)$.::: {.subtask letter=a}Formulate 1. the likelihood,2. the prior, and 3. the resulting posterior.:::::: {.subtask letter=b}Summarize the posterior distribution for the [odds ratio](https://en.wikipedia.org/wiki/Odds_ratio),$$\mathrm{OR} = (p_1/(1-p_1))/(p_0/(1-p_0)).$$* Compute and report the point estimate $E(\mathrm{OR}|y_0,y_1)$, * compute and report a posterior 95%-interval, * plot the histogram, and* write interpretation of the result in text.Use [Frank Harrell's recommendations](https://hbiostat.org/blog/post/bayes-freq-stmts/)how to state results in Bayesian two group comparison.:::::: {.hint}With a conjugate prior, a closed-form posterior is the Betaform for each group separately (see equations in the book). You can use`rbeta()` to sample from the posterior distributions of $p_0$ and $p_1$,and use this sample and odds ratio equation to get a sample from thedistribution of the odds ratio.:::::: {.subtask letter=c}Use at least two different priors, and discuss the sensitivity ofyour inference to your choice of prior density with a couple ofsentences.:::::: {.rubric}* Is the source code included? * No * Yes* Are the likelihood, prior and the posterior for the death probabilities reported? It is ok to refer to thebook instead of deriving the distributions. * No * Yes, but some are missing * Yes* In part a), was the simulation algorithm for computing the posterior of the odds ratio presented orimplemented? * No * Yes, but seems incorrect * Yes* In part a), was the odds ratio summarized with a point estimate and a posterior interval? (The meanshould be close to **...** and 95% posterior interval approximately **...**) * No * Yes, but results seem incorrect * Yes, and the results seem plausible* In part b), was some discussion about testing alternative priors provided? (For example, one couldhave repeated the computations in a)-part with a couple of alternative priors and reported theseresults or some related general conclusions briefly) * Not at all * Some analysis was provided but it was lacking or did not make sense * Some alternative priors were tested and some sensible discussion provided:::# Inference for the difference between normal means (3 points) Consider a case where the same factory has two production lines formanufacturing car windshields. Independent samples from the twoproduction lines were tested for hardness. The hardness measurements forthe two samples $\mathbf{y}_1$ and $\mathbf{y}_2$ be found in the datasets `windshieldy1` and `windshieldy2` in the `aaltobda` package.We assume that the samples have unknown standard deviations $\sigma_1$and $\sigma_2$. Use uninformative or weakly informative priors and answer the followingquestions:::: {.subtask letter=a}Formulate 1. the likelihood,2. the prior, and 3. the resulting posterior.:::::: {.subtask letter=b}What can you say about $\mu_d = \mu_1 - \mu_2$?* Compute and report the point estimate $E(\mu_d|y_1, y_2)$, * compute and report a posterior 95%-interval, * plot the histogram, and* write interpretation of the result in text.Use [Frank Harrell's recommendations](https://hbiostat.org/blog/post/bayes-freq-stmts/)how to state results in Bayesian two group comparison.:::::: {.hint}With a conjugate prior, a closed-form posterior is Student's$t$ form for each group separately (see equations in the book). You canuse the `rtnew()` function to sample from the posterior distributions of$\mu_1$ and $\mu_2$, and use this sample to get a sample from thedistribution of the difference $\mu_d = \mu_1 - \mu_2$.:::::: {.subtask letter=c}Given this specific model, what is the probability that the meansare exactly the same ($\mu_1 = \mu_2$)? Explain your reasoning.:::::: {.rubric}* Is source code included? * No * Yes* Are the likelihood, prior and the posterior for the windshield hardness values reported? (It is also ok torefer to the book or related formulas from exercise 1) * No * Yes, but some are missing * Yes* In part a), was the simulation algorithm for computing the difference in the means presented orimplemented? * No * Yes, but seems to be incorrect * Yes* In part a), was the posterior for the difference between the means summarized with point and intervalestimates? (The mean should be close to **...** and 95% posterior interval **...** or somethingclose to it) * No answer * Yes, but results seem incorrect or only one estimate was given * Yes, and results seem reasonable* Were some analysis or discussion provided for assessing whether the means could be the same? **...** * No analysis or explanation is given * Yes, but the analysis or explanation seems incorrect * Yes, and the analysis or explanation seems plausible:::{{< include includes/_overall_quality.md >}}