# Assignment 4

Author

Aki Vehtari et al.

# 1 General information

This assignment is related to Lecture 4 and Chapters 3 and 10.

The maximum amount of points from this assignment is 6.

We have prepared a quarto template specific for this assignment (html, qmd, pdf) to help you get started.

Tip

The grading will be done in peergrade. All grading questions and evaluations for this assignment are contained within this document in the collapsible Rubric blocks.

Reporting accuracy

For posterior statistics of interest, only report digits that are not completely random based on the Monte Carlo standard error (MCSE).

Example: If you estimate $$E(\mu) \approx 1.234$$ with MCSE($$E(\mu)$$) = 0.01, then the true expectation is likely to be between $$1.204$$ and $$1.264$$, it makes sense to report $$E(\mu) \approx 1.2$$.

See Lecture video 4.1, the chapter notes, and a case study for more information.

• The recommended tool in this course is R (with the IDE RStudio).
• Instead of installing R and RStudio on you own computer, see how to use R and RStudio remotely.
• If you want to install R and RStudio locally, download R and RStudio.
• There are tons of tutorials, videos and introductions to R and RStudio online. You can find some initial hints from RStudio Education pages.
• When working with R, we recommend writing the report using quarto and the provided template. The template includes the formatting instructions and how to include code and figures.
• Instead of quarto, you can use other software to make the PDF report, but the the same instructions for formatting should be used.
• Report all results in a single, anonymous *.pdf -file and submit it in peergrade.io.
• The course has its own R package aaltobda with data and functionality to simplify coding. The package is pre-installed in JupyterHub. To install the package on your own system, run the following code (upgrade="never" skips question about updating other packages):
install.packages("aaltobda", repos = c("https://avehtari.r-universe.dev", getOption("repos")))
• Many of the exercises can be checked automatically using the R package markmyassignment (pre-installed in JupyterHub). Information on how to install and use the package can be found in the markmyassignment documentation. There is no need to include markmyassignment results in the report.
• Recommended additional self study exercises for each chapter in BDA3 are listed in the course web page. These will help to gain deeper understanding of the topic.
• Common questions and answers regarding installation and technical problems can be found in Frequently Asked Questions (FAQ).
• You are allowed to discuss assignments with your friends, but it is not allowed to copy solutions directly from other students or from internet.
• You can copy, e.g., plotting code from the course demos, but really try to solve the actual assignment problems with your own code and explanations.
• Do not copy answers from the internet or from previous years. We compare the answers to the answers from previous years and to the answers from other students this year.
• Use of AI is allowed on the course, but the most of the work needs to by the student, and you need to report whether you used AI and in which way you used them (See points 5 and 6 in Aalto guidelines for use of AI in teaching).
• All suspected plagiarism will be reported and investigated. See more about the Aalto University Code of Academic Integrity and Handling Violations Thereof.
• Do not submit empty PDFs, almost empty PDFs, copy of the questions, nonsense generated by yourself or AI, as these are just harming the other students as they can’t do peergrading for the empty or nonsense submissions. Violations of this rule will be reported and investigated in the same way was plagiarism.
• If you have any suggestions or improvements to the course material, please post in the course chat feedback channel, create an issue, or submit a pull request to the public repository!
Rubric S1: General information - 7.5/100 points
• Q1: Can you open the PDF and it’s not blank nor nonsense? If the pdf is blank, nonsense, or something like only a copy of the questions, 1) report it as problematic in Peergrade-interface to get another report to review, and 2) send a message to TAs.
• Q2: Is the report anonymous?

# 2 Bioassay model

In this exercise, you will use a dose-response relation model that is used in BDA3 Section 3.7 and in the chapter reading instructions. The used likelihood is the same, but instead of uniform priors, we will use a bivariate normal distribution as the joint prior distribution of the parameters $$\alpha$$ and $$\beta$$.

In the prior distribution for $$(\alpha,\beta)$$, the marginal distributions are $$\alpha \sim N(0,2^2)$$ and $$\beta \sim N(10,10^2)$$, and the correlation between them is $$\mathrm{corr}(\alpha, \beta)=0.6$$.

Report the mean (vector of two values) and covariance (two by two matrix) of the bivariate normal distribution.

Tip

The mean and covariance of the bivariate normal distribution are a length–$$2$$ vector and a $$2 \times 2$$ matrix. The elements of the covariance matrix can be computed using the relation of correlation and covariance.

You are given 4000 independent draws from the posterior distribution of the model in the dataset bioassay_posterior in the aaltobda package.

Report

• the mean as well as
• 5 $$\%$$ and 95 $$\%$$ quantiles separately

for both

• $$\alpha$$ and
• $$\beta$$.

Report also the Monte Carlo standard errors (MCSEs) for the mean and quantile estimates and explain in text what does Monte Carlo standard error mean and how you decided the number of digits to show.

Tip

The answer is graded as correct only if the number of digits reported is correct. The number of significant digits can be different for the mean and quantile estimates. In some other cases, the number of digits reported can be less than MCSE allows for practical reasons as discussed in the lecture.

Hint:

Quantiles can be computed with the quantile function. With $$S$$ draws, the MCSE for $$\text{E}[\theta]$$ is $$\sqrt{\text{Var} [\theta]/S}$$. MCSE for the quantile estimates can be computed with the mcse_quantile function from the aaltobda package.

Rubric S2: Bioassay model
• Q3: Are the mean and covariance of the prior in a) reported? The correct answers are :
• Not reported
• Yes, but they are not correct
• Yes, and they are correct
• Q4: Are the means and their MCSEs of alpha and beta in b) reported? Note that the number of digits reported for the means must follow the rule given in the assignment. The correct answers are alpha: mean and beta: mean .
• Not reported
• Yes, but one or both means are incorrect
• Yes, and the means are correct
• Q5: Are the quantiles and their MCSEs of alpha and beta in b) reported? Note that the number of digits reported for the quantiles must follow the rule given in the assignment. The correct answers are alpha: 5% quantile , 95% quantile and beta: 5% quantile , 95% quantile .
• Not reported
• Yes, but one or more quantiles are incorrect
• Yes, and the quantiles are correct

# 3 Importance sampling

Now we discard our posterior draws and switch to importance sampling.

Implement a function for computing the log importance ratios (log importance weights) when the importance sampling target distribution is the posterior distribution, and the proposal distribution is the prior distribution from a). Explain in words why it’s better to compute log ratios instead of ratios.

Tip

Non-log importance ratios are given by equation (10.3) in the course book. The fact that our proposal distribution is the same as the prior distribution makes this task easier. The logarithm of the likelihood can be computed with the bioassaylp function from the aaltobda package. The data required for the likelihood can be loaded with data("bioassay").

Implement a function for computing normalized importance ratios from the unnormalized log ratios in c). In other words, exponentiate the log ratios and scale them such that they sum to one. Explain in words what is the effect of exponentiating and scaling so that sum is one.

Sample 4000 draws of $$\alpha$$ and $$\beta$$ from the prior distribution from a). Compute and plot a histogram of the 4000 normalized importance ratios. Use the functions you implemented in c) and d).

Tip

Use the function rmvnorm from the aaltobda package for sampling.

Using the importance ratios, compute the importance sampling effective sample size $$S_{\text{eff}}$$ and report it.

Tip

Equation (10.4) in the course book.

BDA3 1st (2013) and 2nd (2014) printing have an error for $$\tilde{w}(\theta^s)$$ used in the effective sample size equation (10.4). The normalized weights equation should not have the multiplier S (the normalized weights should sum to one). The later printings, the online version, and the slides have the correct equation.

Explain in your own words what the importance sampling effective sample size represents. Also explain how the effective sample size is seen in the histogram of the weights that you plotted in e).

Implement a function for computing the posterior mean using importance sampling, and compute the mean using your 4000 draws. Explain in your own words the computation for importance sampling. Report the means for $$\alpha$$ and $$\beta$$, and also the Monte Carlo standard errors (MCSEs) for the mean estimates. Report the number of digits for the means based on the MCSEs.

Tip

The values below are only a test case, you need to use 4000 draws for $$\alpha$$ and $$\beta$$ in the final report.

Use the same equation for the MCSE of $$\text{E}[\theta]$$ as earlier ($$\sqrt{\text{Var} [\theta]/S}$$), but now replace $$S$$ with $$S_{\text{eff}}$$. To compute $$\text{Var} [\theta]$$ with importance sampling, use the identity $$\text{Var}[\theta] = \text{E}[\theta^2] - \text{E}[\theta]^2$$.

Rubric S3: Importance sampling
• Q6: Is the source code for the function in c) reported?
• No
• Yes
• Q7: Is the source code for the function in d) reported?
• No
• Yes
• Q8: Does the histogram in e) look something like this figure? If it is evident that the normalized importance ratios are computed correctly, but the prior was incorrect, you can still grade “Reported and looks similar”.
• Not reported
• Reported, but looks different
• Reported and looks similar
• Q9: Is the effective sample size in f) reported? The correct range for the effective sample size is between . However, if it is evident that the effective sample size is computed correctly, but the prior was incorrect, you can still grade “Yes, and it is correct”.
• No
• Yes, but it is not correct
• Yes, and it is correct
• Q10: The correct explanation for g) is roughly the following:
• Q11: What is the connection between S_eff and the histogram of weights: How is the answer?
• Totally wrong/has not tried
• Something is a bit wrong
• Explanation is sensible
• Q12: Is the source code for the function in h) reported?
• No
• Yes
• Q13: Are the means and their MCSEs of alpha and beta in h) reported? Note that the number of digits reported for the means must follow the rule given in the assignment. The correct answers should be close to these: alpha: mean and beta: mean
• Not reported
• Yes, but they are incorrect
• Yes, and they are correct

# 4 Overall quality of the report

Rubric S4: Overall quality of the report - 7.5/100 points
• Q14: Does the report include comment on whether AI was used, and if AI was used, explanation on how it was used?
• No
• Yes
• Q15: Does the report follow the formatting instructions?
• Not at all
• Little
• Mostly
• Yes
• Q16: In case the report doesn’t fully follow the general and formatting instructions, specify the instructions that have not been followed. If applicable, specify the page of the report, where this difference is visible. This will help the other student to improve their reports so that they are easier to read and review. If applicable, specify the page of the report, where this difference in formatting is visible.
• Q17: Please also provide feedback on the presentation (e.g. text, layout, flow of the responses, figures, figure captions). Part of the course is practicing making data analysis reports. By providing feedback on the report presentation, other students can learn what they can improve or what they already did well. You should be able to provide constructive or positive feedback for all non-empty and non-nonsense reports. If you think the report is perfect, and you can’t come up with any suggestions how to improve, you can provide feedback on what you liked and why you think some part of the report is better than yours.