In this exercise, we will see some basics in hypothesis testing and more specifically on t-tests. As an example, we will work with the captopril dataset that we already explored in the exercise on data exploration.
The goal is to answer these two research questions;
Is the average systolic baseline blood pressure of the patients is higher than the treshold for hypertension of 140 mmHg?
Is the average SBP before captopril treatment different from the average SBP after captopril treatment?
First, load the required R libraries:
Import the data
(captopril <- read.table("https://raw.githubusercontent.com/statOmics/PSLS21/data/captopril.txt",
header = TRUE,
sep = ","))
Data exploration
Before we start delving into the data in order to solve our research hypothesis, we have to explore our data. Our dataset looks like this;
We have 15 patients, for which the systolic blood pressures and diastolyic blood pressures were measured, before and after treatment with the captopril drug.
Note, that the dataset is not in the tidy format. We could tidy the data using the following code.
captopril %>%
gather(type,bp,-id) %>%
filter(type%in%c("SBPa","SBPb")) %>%
mutate(id = as.factor(id))
# we are only interested in the systolic blood pressure
We can visualize the dataframe using boxplots;
captopril %>%
gather(type,bp,-id) %>%
filter(type%in%c("SBPa","SBPb")) %>%
ggplot(aes(x=type, y=bp, fill=type)) +
scale_fill_brewer(palette="RdGy") +
theme_bw() +
geom_boxplot(outlier.shape=NA) +
geom_jitter(width = 0.2) +
ggtitle("Boxplot of blood pressure measures before and after treatment") +
ylab("blood pressure (mmHg)") +
stat_summary(fun=mean, geom="point", shape=5, size=3, color="black", fill="black")
Clearly, it seems that on average the measurements after treatment are lower than those before treatment.
Note that the boxplot is useful in light of the first question where we have to assess if the average systolic blood pressure of patients is above 140mm Hg, the threshold for hypertension.
However, to assess the question if there is an affect of administering captopril it is essential to account for the fact that the blood pressure before and after administering captopril was measured on the same patients.
captopril %>%
gather(type,bp,-id) %>%
filter(type%in%c("SBPa","SBPb")) %>%
ggplot(aes(x=type, y=bp, group=id)) +
geom_line() +
geom_point() +
ylab("Systolic blood pressure (mmHg)")
The key question is to know is if there is significant effect of the treatment? To answer this question, we will need to perform a hypothesis test.
Let’s start of with question 1.
Question 1
Is the average baseline systolic blood pressure (SBP) higher than the threshold for hypertension of 140 mmHg?
To test the effect of the captopril on subjects with hypertension (patients), we need to select patients with hypertension, hence they should have elevated SBP levels, that are on average higher than 140 mmHg. We can assess this hypothesis using a one sample t-test.
Assess the assumptions
Before we can perform a t-test, we must check that the required assumptions are met!
- The observations are independent
- The data (SBPb) must be normally distributed
The first assumption requires us to think about how the data were collected. Are there any underlying correlation structures (that we know of) in the data? For instance, if all the 15 subjects are members of the same family, we expect that the data will give us a good representation of the underlying population of interest, i.e., all past, present and future patients with elevated SBP levels.
Here, we have no reason to believe that this assumption was violated; we may assume 15 unrelated patients with elevated SBP levels were selected at random from the population.
We can assess the second assumption with a quantile-quantile plot.
captopril %>%
ggplot(aes(sample=SBPa)) +
geom_qq() +
geom_qq_line()
captopril %>%
ggplot(aes(sample=SBPb)) +
geom_qq() +
geom_qq_line()
# or, equivalently
captoprilTidy <- captopril %>% gather(type,bp,-id)
captoprilTidy %>%
filter(type%in%c("SBPa","SBPb")) %>%
ggplot(aes(sample=bp)) +
geom_qq() +
geom_qq_line() +
facet_wrap(~type)
We can see that all of the data lies nicely around the quantile-quantile line (black line). As such, we may conclude that our data is normally distributed.
Hypothesis test
Here, we will test if mean systolic blood pressure at baseline is significantly higher than 140 mmHg. More specifically, we will test the null hypothesis;
\(H_0:\) the mean SBPb is equal to 140 mmHg
versus the alternative hypothesis;
\(H_1:\) the mean SBPb is greater than 140 mmHg
output1 <- t.test(captopril$SBPb,
mu=140,
alternative = "greater",
conf.level = 0.95)
output1
##
## One Sample t-test
##
## data: captopril$SBPb
## t = 6.9556, df = 14, p-value = 3.352e-06
## alternative hypothesis: true mean is greater than 140
## 95 percent confidence interval:
## 167.581 Inf
## sample estimates:
## mean of x
## 176.9333
Conclusion
When writing a conclusion on your research hypothesis, it is very important to be precise, concise, and complete.
An example of such a conclusion for our research question is given below:
We can conclude that the mean systolic baseline blood pressure of patients in the captopril study is extremely significantly higher than the threshold for hypertension of 140 mmHg (p << 0.001). The mean SBPb equals 176.93 mmHg with a 95% confidence interval of [167.58, 167.58, +\(\infty\)]).
As we have seen in the theory class, the 95% confidence interval can be interpreted as;
With 95% confidence we can conclude that the true average of baseline systolic blood pressure of diseased patients in the population is above 167.58.
Note, that we only report a one sided confidence interval because we only test against the alternative hypothesis that the blood pressure is on average larger than the threshold for hypertension of 140 mmHg.
Question 2
Is the average SBP before captopril treatment different from the average SBP after captopril treatment?
As the data are paired, there will be a strong correlation between the BP values before and after treatment of each individual patient. We can show this with a scatterplot.
captopril %>%
ggplot(aes(x=SBPb,y=SBPa)) +
geom_point() +
ggtitle("correlation between SBPb and SBPa") +
ylab("SBPa (mmHg)") +
xlab("SBPb (mmHg)")
We clearly see that if a patient’s SBPb value is high, its SBPa value will be comparatively high as well.
We can immediately calculate the impact of the treatment for every patient by calculating the difference between the blood pressure after and before the treatment.
captopril <-
captopril %>% mutate(deltaSBP = SBPa-SBPb)
captopril%>%
ggplot(aes(x="Systolic blood pressure",y=deltaSBP)) +
geom_boxplot(outlier.shape=NA) +
geom_point(position="jitter")+
ylab("Difference (mm mercury)") +
xlab("")
Check the assumptions
The paired t-test has 2 assumptions:
The blood pressure differences are independent of each other.
The blood pressure differences are normally distributed.
The first assumption is met given the experimental design.
Secondly, we assess if the data are normally distributed.
captopril %>%
ggplot(aes(sample=deltaSBP)) +
geom_qq() +
geom_qq_line()
We can see that all of the data lies nicely around the quantile-quantile line. As such, we may assume that our data are normally distributed. As our assumptions are met we may continue with performing the unpaired t-test.
Hypothesis test
As such, we will perform a paired
t-test.
- The null hypothesis of the test is that the blood pressure before and after the treatment with captopril is on average equal.
- Which will be tested against the alternative that the blood pressure before and after the treatment with captopril is on average different.
output2 <- t.test(captopril$SBPb, captopril$SBPa, paired = TRUE)
output2
##
## Paired t-test
##
## data: captopril$SBPb and captopril$SBPa
## t = 8.1228, df = 14, p-value = 1.146e-06
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
## 13.93409 23.93258
## sample estimates:
## mean of the differences
## 18.93333
Conclusion
There is on average an extremely significant blood pressure drop upon administering captopril to patients with hypertension (p << 0.001). The systolic blood pressure decreases on average with 18.9 mmHg upon the treatment with captopril (95% CI [13.9, 23.9]).
Alternative solution: One-sample t-test on the difference
Performing a paired t-test is analogous to performing a one-sample t-test on the difference between both groups.
This can be easily seen from the output of the paired two-sample t-test. The alternative hypothesis \(HA\) there states that the “true difference in means is not equal to 0”. So internally, R will actually perform a one-sample t-test on the difference, and check whether or not the true mean difference is equal to 0. We can also set this up manually.
t.test(captopril$deltaSBP, mu=0)
##
## One Sample t-test
##
## data: captopril$deltaSBP
## t = -8.1228, df = 14, p-value = 1.146e-06
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
## -23.93258 -13.93409
## sample estimates:
## mean of x
## -18.93333
Indeed, the output is equivalent to that of the paired t-test.
LS0tCnRpdGxlOiAiRXhlcmNpc2UgNS4xOiBIeXBvdGhlc2lzIHRlc3Rpbmcgb24gdGhlIGNhcHRvcHJpbCBkYXRhc2V0IC0gc29sdXRpb24iICAgCmF1dGhvcjogIkxpZXZlbiBDbGVtZW50IGFuZCBKZXJvZW4gR2lsaXMiCmRhdGU6ICJzdGF0T21pY3MsIEdoZW50IFVuaXZlcnNpdHkgKGh0dHBzOi8vc3RhdG9taWNzLmdpdGh1Yi5pbykiICAKb3V0cHV0OgogICAgaHRtbF9kb2N1bWVudDoKICAgICAgY29kZV9kb3dubG9hZDogdHJ1ZSAgICAKICAgICAgdGhlbWU6IGNvc21vCiAgICAgIHRvYzogdHJ1ZQogICAgICB0b2NfZmxvYXQ6IHRydWUKICAgICAgaGlnaGxpZ2h0OiB0YW5nbwogICAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUKLS0tCgpJbiB0aGlzIGV4ZXJjaXNlLCB3ZSB3aWxsIHNlZSBzb21lIGJhc2ljcyBpbiBoeXBvdGhlc2lzIHRlc3RpbmcgYW5kIG1vcmUgCnNwZWNpZmljYWxseSBvbiB0LXRlc3RzLiBBcyBhbiBleGFtcGxlLCB3ZSB3aWxsIHdvcmsgd2l0aCB0aGUKY2FwdG9wcmlsIGRhdGFzZXQgdGhhdCB3ZSBhbHJlYWR5IGV4cGxvcmVkIGluIHRoZSBleGVyY2lzZSBvbiBkYXRhIGV4cGxvcmF0aW9uLgoKVGhlIGdvYWwgaXMgdG8gYW5zd2VyIHRoZXNlIHR3byByZXNlYXJjaCBxdWVzdGlvbnM7CgoxLiBJcyB0aGUgYXZlcmFnZSBzeXN0b2xpYyBiYXNlbGluZSBibG9vZCBwcmVzc3VyZSBvZiB0aGUgcGF0aWVudHMgaXMgaGlnaGVyIHRoYW4gdGhlIHRyZXNob2xkIGZvciBoeXBlcnRlbnNpb24gb2YgMTQwIG1tSGc/IAoKMi4gSXMgdGhlIGF2ZXJhZ2UgU0JQIGJlZm9yZSBjYXB0b3ByaWwgdHJlYXRtZW50IApkaWZmZXJlbnQgZnJvbSB0aGUgYXZlcmFnZSBTQlAgYWZ0ZXIKY2FwdG9wcmlsIHRyZWF0bWVudD8KCkZpcnN0LCBsb2FkIHRoZSByZXF1aXJlZCBSIGxpYnJhcmllczoKCmBgYHtyLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIEltcG9ydCB0aGUgZGF0YQoKYGBge3J9CihjYXB0b3ByaWwgPC0gcmVhZC50YWJsZSgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3N0YXRPbWljcy9QU0xTMjEvZGF0YS9jYXB0b3ByaWwudHh0IiwgCiAgICAgICAgICAgICAgICAgICAgICAgICBoZWFkZXIgPSBUUlVFLCAKICAgICAgICAgICAgICAgICAgICAgICAgIHNlcCA9ICIsIikpCmBgYAoKIyBEYXRhIGV4cGxvcmF0aW9uCgpCZWZvcmUgd2Ugc3RhcnQgZGVsdmluZyBpbnRvIHRoZSBkYXRhIGluIG9yZGVyIHRvIHNvbHZlIApvdXIgcmVzZWFyY2ggaHlwb3RoZXNpcywgd2UgaGF2ZSB0byBleHBsb3JlIG91ciBkYXRhLiBPdXIgZGF0YXNldCBsb29rcyBsaWtlIHRoaXM7CgpgYGB7cn0KaGVhZChjYXB0b3ByaWwpCmBgYAoKV2UgaGF2ZSAxNSBwYXRpZW50cywgZm9yIHdoaWNoIHRoZSBzeXN0b2xpYyAKYmxvb2QgcHJlc3N1cmVzIGFuZCBkaWFzdG9seWljIGJsb29kIHByZXNzdXJlcyB3ZXJlIG1lYXN1cmVkLCBiZWZvcmUgYW5kIGFmdGVyCnRyZWF0bWVudCB3aXRoIHRoZSBjYXB0b3ByaWwgZHJ1Zy4KCk5vdGUsIHRoYXQgdGhlIGRhdGFzZXQgaXMgbm90IGluIHRoZSB0aWR5IGZvcm1hdC4gV2UgY291bGQgdGlkeSB0aGUgZGF0YSB1c2luZwp0aGUgZm9sbG93aW5nIGNvZGUuCgpgYGB7cn0KY2FwdG9wcmlsICU+JSAKICAgIGdhdGhlcih0eXBlLGJwLC1pZCkgJT4lCiAgICBmaWx0ZXIodHlwZSVpbiVjKCJTQlBhIiwiU0JQYiIpKSAlPiUKICAgIG11dGF0ZShpZCA9IGFzLmZhY3RvcihpZCkpCiAgICAjIHdlIGFyZSBvbmx5IGludGVyZXN0ZWQgaW4gdGhlIHN5c3RvbGljIGJsb29kIHByZXNzdXJlCmBgYAoKV2UgY2FuIHZpc3VhbGl6ZSB0aGUgZGF0YWZyYW1lIHVzaW5nIGJveHBsb3RzOwoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2F0aGVyKHR5cGUsYnAsLWlkKSAlPiUKICBmaWx0ZXIodHlwZSVpbiVjKCJTQlBhIiwiU0JQYiIpKSAlPiUKICBnZ3Bsb3QoYWVzKHg9dHlwZSwgeT1icCwgZmlsbD10eXBlKSkgKyAKICAgICAgc2NhbGVfZmlsbF9icmV3ZXIocGFsZXR0ZT0iUmRHeSIpICsKICAgICAgdGhlbWVfYncoKSArCiAgICAgIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArIAogICAgICBnZW9tX2ppdHRlcih3aWR0aCA9IDAuMikgKwogICAgICBnZ3RpdGxlKCJCb3hwbG90IG9mIGJsb29kIHByZXNzdXJlIG1lYXN1cmVzIGJlZm9yZSBhbmQgYWZ0ZXIgdHJlYXRtZW50IikgKwogICAgICB5bGFiKCJibG9vZCBwcmVzc3VyZSAobW1IZykiKSArCiAgICAgIHN0YXRfc3VtbWFyeShmdW49bWVhbiwgZ2VvbT0icG9pbnQiLCBzaGFwZT01LCBzaXplPTMsIGNvbG9yPSJibGFjayIsIGZpbGw9ImJsYWNrIikKYGBgCgpDbGVhcmx5LCBpdCBzZWVtcyB0aGF0IG9uIGF2ZXJhZ2UgdGhlIG1lYXN1cmVtZW50cwphZnRlciB0cmVhdG1lbnQgYXJlIGxvd2VyIHRoYW4gdGhvc2UgYmVmb3JlIHRyZWF0bWVudC4KCk5vdGUgdGhhdCB0aGUgYm94cGxvdCBpcyB1c2VmdWwgaW4gbGlnaHQgb2YgdGhlIGZpcnN0IHF1ZXN0aW9uIHdoZXJlIHdlIGhhdmUgdG8gYXNzZXNzIGlmIHRoZSBhdmVyYWdlIHN5c3RvbGljIGJsb29kIHByZXNzdXJlIG9mIHBhdGllbnRzIGlzIGFib3ZlIDE0MG1tIEhnLCB0aGUgdGhyZXNob2xkIGZvciBoeXBlcnRlbnNpb24uIAoKSG93ZXZlciwgdG8gYXNzZXNzIHRoZSBxdWVzdGlvbiBpZiB0aGVyZSBpcyBhbiBhZmZlY3Qgb2YgYWRtaW5pc3RlcmluZyBjYXB0b3ByaWwgaXQgaXMgZXNzZW50aWFsIHRvIGFjY291bnQgZm9yIHRoZSBmYWN0IHRoYXQgdGhlIGJsb29kIHByZXNzdXJlIGJlZm9yZSBhbmQgYWZ0ZXIgYWRtaW5pc3RlcmluZyBjYXB0b3ByaWwgd2FzIG1lYXN1cmVkIG9uIHRoZSBzYW1lIHBhdGllbnRzLiAKCmBgYHtyfQpjYXB0b3ByaWwgJT4lIAogIGdhdGhlcih0eXBlLGJwLC1pZCkgJT4lCiAgZmlsdGVyKHR5cGUlaW4lYygiU0JQYSIsIlNCUGIiKSkgJT4lCiAgZ2dwbG90KGFlcyh4PXR5cGUsIHk9YnAsIGdyb3VwPWlkKSkgKwogIGdlb21fbGluZSgpICsKICBnZW9tX3BvaW50KCkgKyAKICB5bGFiKCJTeXN0b2xpYyBibG9vZCBwcmVzc3VyZSAobW1IZykiKQpgYGAKClRoZSBrZXkgcXVlc3Rpb24gaXMgdG8ga25vdyBpcyBpZiB0aGVyZSBpcyAqKnNpZ25pZmljYW50KiogZWZmZWN0IG9mIHRoZSB0cmVhdG1lbnQ/IFRvIGFuc3dlciB0aGlzCnF1ZXN0aW9uLCB3ZSB3aWxsIG5lZWQgdG8gcGVyZm9ybSBhIGh5cG90aGVzaXMgdGVzdC4KCkxldCdzIHN0YXJ0IG9mIHdpdGggcXVlc3Rpb24gMS4KCiMgUXVlc3Rpb24gMQoKSXMgdGhlIGF2ZXJhZ2UgYmFzZWxpbmUgc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgKFNCUCkgaGlnaGVyIHRoYW4gdGhlIHRocmVzaG9sZCBmb3IgaHlwZXJ0ZW5zaW9uIG9mIDE0MCBtbUhnPwoKVG8gdGVzdCB0aGUgZWZmZWN0IG9mIHRoZSBjYXB0b3ByaWwgb24gc3ViamVjdHMgd2l0aCBoeXBlcnRlbnNpb24KKHBhdGllbnRzKSwgd2UgbmVlZCB0byBzZWxlY3QgcGF0aWVudHMgd2l0aCBoeXBlcnRlbnNpb24sIGhlbmNlIHRoZXkgc2hvdWxkIGhhdmUgCmVsZXZhdGVkIFNCUCBsZXZlbHMsIHRoYXQgYXJlIG9uIGF2ZXJhZ2UgaGlnaGVyIHRoYW4gMTQwIG1tSGcuIApXZSBjYW4gYXNzZXNzIHRoaXMgaHlwb3RoZXNpcyB1c2luZyBhIG9uZSBzYW1wbGUgdC10ZXN0LgoKIyMgQXNzZXNzIHRoZSBhc3N1bXB0aW9ucwoKQmVmb3JlIHdlIGNhbiBwZXJmb3JtIGEgdC10ZXN0LCB3ZSBtdXN0IGNoZWNrIHRoYXQgdGhlIHJlcXVpcmVkCmFzc3VtcHRpb25zIGFyZSBtZXQhCgoxLiBUaGUgb2JzZXJ2YXRpb25zIGFyZSBpbmRlcGVuZGVudAoyLiBUaGUgZGF0YSAoU0JQYikgbXVzdCBiZSBub3JtYWxseSBkaXN0cmlidXRlZAoKVGhlIGZpcnN0IGFzc3VtcHRpb24gcmVxdWlyZXMgdXMgdG8gdGhpbmsgYWJvdXQgaG93IHRoZSBkYXRhIHdlcmUgY29sbGVjdGVkLiAKQXJlIHRoZXJlIGFueSB1bmRlcmx5aW5nIGNvcnJlbGF0aW9uIHN0cnVjdHVyZXMgKHRoYXQgd2Uga25vdyBvZikKaW4gdGhlIGRhdGE/IEZvciBpbnN0YW5jZSwgaWYgYWxsIHRoZSAxNSBzdWJqZWN0cyBhcmUgbWVtYmVycyBvZgp0aGUgc2FtZSBmYW1pbHksIHdlIGV4cGVjdCB0aGF0IHRoZSBkYXRhIHdpbGwgZ2l2ZSB1cyBhIGdvb2QgCnJlcHJlc2VudGF0aW9uIG9mIHRoZSB1bmRlcmx5aW5nIHBvcHVsYXRpb24gb2YgaW50ZXJlc3QsIGkuZS4sIAphbGwgcGFzdCwgcHJlc2VudCBhbmQgZnV0dXJlIHBhdGllbnRzIHdpdGggZWxldmF0ZWQgU0JQIGxldmVscy4KCkhlcmUsIHdlIGhhdmUgbm8gcmVhc29uIHRvIGJlbGlldmUgdGhhdCB0aGlzIAphc3N1bXB0aW9uIHdhcyB2aW9sYXRlZDsgd2UgbWF5IGFzc3VtZSAxNSB1bnJlbGF0ZWQgcGF0aWVudHMgd2l0aCBlbGV2YXRlZCBTQlAgbGV2ZWxzIHdlcmUgc2VsZWN0ZWQgYXQgcmFuZG9tIGZyb20gdGhlIHBvcHVsYXRpb24uCgpXZSBjYW4gYXNzZXNzIHRoZSBzZWNvbmQgYXNzdW1wdGlvbiB3aXRoIGEgcXVhbnRpbGUtcXVhbnRpbGUgcGxvdC4KCmBgYHtyfQpjYXB0b3ByaWwgJT4lCiAgZ2dwbG90KGFlcyhzYW1wbGU9U0JQYSkpICsKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpCgpjYXB0b3ByaWwgJT4lCiAgZ2dwbG90KGFlcyhzYW1wbGU9U0JQYikpICsKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpCgojIG9yLCBlcXVpdmFsZW50bHkKY2FwdG9wcmlsVGlkeSA8LSBjYXB0b3ByaWwgJT4lIGdhdGhlcih0eXBlLGJwLC1pZCkgCmNhcHRvcHJpbFRpZHkgJT4lCiAgZmlsdGVyKHR5cGUlaW4lYygiU0JQYSIsIlNCUGIiKSkgJT4lCiAgZ2dwbG90KGFlcyhzYW1wbGU9YnApKSArCiAgZ2VvbV9xcSgpICsKICBnZW9tX3FxX2xpbmUoKSArCiAgZmFjZXRfd3JhcCh+dHlwZSkKYGBgCgpXZSBjYW4gc2VlIHRoYXQgYWxsIG9mIHRoZSBkYXRhIGxpZXMgbmljZWx5IGFyb3VuZCB0aGUgcXVhbnRpbGUtcXVhbnRpbGUKbGluZSAoYmxhY2sgbGluZSkuIEFzIHN1Y2gsIHdlIG1heSBjb25jbHVkZSB0aGF0IG91ciBkYXRhIGlzIG5vcm1hbGx5IApkaXN0cmlidXRlZC4KCiMjIEh5cG90aGVzaXMgdGVzdAoKSGVyZSwgd2Ugd2lsbCB0ZXN0IGlmIG1lYW4gc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgYXQgYmFzZWxpbmUgaXMgc2lnbmlmaWNhbnRseSBoaWdoZXIgdGhhbiAxNDAgbW1IZy4gTW9yZSAKc3BlY2lmaWNhbGx5LCB3ZSB3aWxsIHRlc3QgdGhlIG51bGwgaHlwb3RoZXNpczsKCiRIXzA6JCB0aGUgbWVhbiBTQlBiIGlzIGVxdWFsIHRvIDE0MCBtbUhnCgp2ZXJzdXMgdGhlIGFsdGVybmF0aXZlIGh5cG90aGVzaXM7CgokSF8xOiQgdGhlIG1lYW4gU0JQYiBpcyBncmVhdGVyIHRoYW4gMTQwIG1tSGcKCmBgYHtyfQpvdXRwdXQxIDwtIHQudGVzdChjYXB0b3ByaWwkU0JQYiwgCiAgICAgICAgICAgICAgICAgIG11PTE0MCwKICAgICAgICAgICAgICAgICAgYWx0ZXJuYXRpdmUgPSAiZ3JlYXRlciIsCiAgICAgICAgICAgICAgICAgIGNvbmYubGV2ZWwgPSAwLjk1KQpvdXRwdXQxCmBgYAoKIyMgQ29uY2x1c2lvbgoKV2hlbiB3cml0aW5nIGEgY29uY2x1c2lvbiBvbiB5b3VyIHJlc2VhcmNoIGh5cG90aGVzaXMsCml0IGlzIHZlcnkgaW1wb3J0YW50IHRvIGJlIHByZWNpc2UsIGNvbmNpc2UsIGFuZCBjb21wbGV0ZS4KCkFuIGV4YW1wbGUgb2Ygc3VjaCBhIGNvbmNsdXNpb24gZm9yIG91ciByZXNlYXJjaCBxdWVzdGlvbgppcyBnaXZlbiBiZWxvdzoKCldlIGNhbiBjb25jbHVkZSB0aGF0IHRoZSBtZWFuIHN5c3RvbGljIGJhc2VsaW5lIGJsb29kIHByZXNzdXJlIG9mIHBhdGllbnRzIGluIHRoZSBjYXB0b3ByaWwgc3R1ZHkgaXMgZXh0cmVtZWx5IHNpZ25pZmljYW50bHkgaGlnaGVyICB0aGFuIHRoZSB0aHJlc2hvbGQgZm9yIGh5cGVydGVuc2lvbiBvZiAxNDAgbW1IZyAocCA8PCAwLjAwMSkuIApUaGUgbWVhbiBTQlBiIGVxdWFscyBgciByb3VuZCh1bm5hbWUob3V0cHV0MSRlc3RpbWF0ZSksMilgIG1tSGcgd2l0aCAKYSA5NSUgY29uZmlkZW5jZSBpbnRlcnZhbCBvZiBbYHIgcm91bmQob3V0cHV0MSRjb25mLmludFtjKDEsMSldLDIpYCwgKyRcaW5mdHkkXSkuCgpBcyB3ZSBoYXZlIHNlZW4gaW4gdGhlIHRoZW9yeSBjbGFzcywgdGhlIDk1JSBjb25maWRlbmNlIAppbnRlcnZhbCBjYW4gYmUgaW50ZXJwcmV0ZWQgYXM7CgpXaXRoIDk1JSBjb25maWRlbmNlIHdlIGNhbiBjb25jbHVkZSB0aGF0IHRoZSB0cnVlIGF2ZXJhZ2Ugb2YgYmFzZWxpbmUgc3lzdG9saWMgYmxvb2QgcHJlc3N1cmUgb2YgZGlzZWFzZWQgcGF0aWVudHMgaW4gdGhlIHBvcHVsYXRpb24gaXMgYWJvdmUgYHIgcm91bmQob3V0cHV0MSRjb25mLmludFsxXSwyKWAuCgpOb3RlLCB0aGF0IHdlIG9ubHkgcmVwb3J0IGEgb25lIHNpZGVkIGNvbmZpZGVuY2UgaW50ZXJ2YWwgYmVjYXVzZSB3ZSBvbmx5IHRlc3QgYWdhaW5zdCB0aGUgYWx0ZXJuYXRpdmUgaHlwb3RoZXNpcyB0aGF0IHRoZSBibG9vZCBwcmVzc3VyZSBpcyBvbiBhdmVyYWdlIGxhcmdlciB0aGFuIHRoZSB0aHJlc2hvbGQgZm9yIGh5cGVydGVuc2lvbiBvZiAxNDAgbW1IZy4gCgojIFF1ZXN0aW9uIDIKCklzIHRoZSBhdmVyYWdlIFNCUCBiZWZvcmUgY2FwdG9wcmlsIHRyZWF0bWVudCBkaWZmZXJlbnQgZnJvbSB0aGUgYXZlcmFnZSAKU0JQIGFmdGVyIGNhcHRvcHJpbCB0cmVhdG1lbnQ/CgpBcyB0aGUgZGF0YSBhcmUgcGFpcmVkLCB0aGVyZSB3aWxsIGJlIGEgc3Ryb25nIGNvcnJlbGF0aW9uIGJldHdlZW4gdGhlIEJQIHZhbHVlcyAKYmVmb3JlIGFuZCBhZnRlciB0cmVhdG1lbnQgb2YgZWFjaCBpbmRpdmlkdWFsIHBhdGllbnQuIFdlIGNhbiBzaG93IHRoaXMKd2l0aCBhIHNjYXR0ZXJwbG90LgoKYGBge3J9CmNhcHRvcHJpbCAlPiUgCiAgZ2dwbG90KGFlcyh4PVNCUGIseT1TQlBhKSkgKwogICAgZ2VvbV9wb2ludCgpICsKICAgIGdndGl0bGUoImNvcnJlbGF0aW9uIGJldHdlZW4gU0JQYiBhbmQgU0JQYSIpICsKICAgIHlsYWIoIlNCUGEgKG1tSGcpIikgKwogICAgeGxhYigiU0JQYiAobW1IZykiKQpgYGAKCldlIGNsZWFybHkgc2VlIHRoYXQgaWYgYSBwYXRpZW50J3MgU0JQYiB2YWx1ZSBpcyBoaWdoLCBpdHMKU0JQYSB2YWx1ZSB3aWxsIGJlIGNvbXBhcmF0aXZlbHkgaGlnaCBhcyB3ZWxsLgoKV2UgY2FuIGltbWVkaWF0ZWx5IGNhbGN1bGF0ZSB0aGUgaW1wYWN0IG9mIHRoZSB0cmVhdG1lbnQgZm9yIGV2ZXJ5IHBhdGllbnQgYnkgY2FsY3VsYXRpbmcgdGhlIGRpZmZlcmVuY2UgYmV0d2VlbiB0aGUgYmxvb2QgcHJlc3N1cmUgYWZ0ZXIgYW5kIGJlZm9yZSB0aGUgdHJlYXRtZW50LiAKCmBgYHtyfQpjYXB0b3ByaWwgPC0gCiAgY2FwdG9wcmlsICU+JSBtdXRhdGUoZGVsdGFTQlAgPSBTQlBhLVNCUGIpCgpjYXB0b3ByaWwlPiUKICBnZ3Bsb3QoYWVzKHg9IlN5c3RvbGljIGJsb29kIHByZXNzdXJlIix5PWRlbHRhU0JQKSkgKwogIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlPU5BKSArIAogIGdlb21fcG9pbnQocG9zaXRpb249ImppdHRlciIpKwogIHlsYWIoIkRpZmZlcmVuY2UgKG1tIG1lcmN1cnkpIikgKwogIHhsYWIoIiIpIApgYGAKCiMjIENoZWNrIHRoZSBhc3N1bXB0aW9ucwoKVGhlIHBhaXJlZCB0LXRlc3QgaGFzIDIgYXNzdW1wdGlvbnM6CgoxLiBUaGUgYmxvb2QgcHJlc3N1cmUgZGlmZmVyZW5jZXMgYXJlIGluZGVwZW5kZW50IG9mIGVhY2ggb3RoZXIuCgoyLiBUaGUgYmxvb2QgcHJlc3N1cmUgZGlmZmVyZW5jZXMgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLiAKClRoZSBmaXJzdCBhc3N1bXB0aW9uIGlzIG1ldCBnaXZlbiB0aGUgZXhwZXJpbWVudGFsIGRlc2lnbi4KClNlY29uZGx5LCB3ZSBhc3Nlc3MgaWYgdGhlIGRhdGEgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLgoKYGBge3J9CmNhcHRvcHJpbCAlPiUKICBnZ3Bsb3QoYWVzKHNhbXBsZT1kZWx0YVNCUCkpICsKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpCmBgYAoKV2UgY2FuIHNlZSB0aGF0IGFsbCBvZiB0aGUgZGF0YSBsaWVzIG5pY2VseSBhcm91bmQgdGhlIHF1YW50aWxlLXF1YW50aWxlCmxpbmUuIEFzIHN1Y2gsIHdlIG1heSBhc3N1bWUgdGhhdCBvdXIgZGF0YSBhcmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQuCkFzIG91ciBhc3N1bXB0aW9ucyBhcmUgbWV0IHdlIG1heSBjb250aW51ZSB3aXRoCnBlcmZvcm1pbmcgdGhlIHVucGFpcmVkIHQtdGVzdC4KCiMjIEh5cG90aGVzaXMgdGVzdAoKQXMgc3VjaCwgd2Ugd2lsbCBwZXJmb3JtIGEgYHBhaXJlZGAgdC10ZXN0LgoKLSBUaGUgbnVsbCBoeXBvdGhlc2lzIG9mIHRoZSB0ZXN0IGlzIHRoYXQgdGhlIGJsb29kIHByZXNzdXJlIGJlZm9yZSBhbmQgYWZ0ZXIgCnRoZSB0cmVhdG1lbnQgd2l0aCBjYXB0b3ByaWwgaXMgb24gYXZlcmFnZSBlcXVhbC4gCi0gV2hpY2ggd2lsbCBiZSB0ZXN0ZWQgYWdhaW5zdCB0aGUgYWx0ZXJuYXRpdmUgdGhhdCB0aGUgYmxvb2QgcHJlc3N1cmUgYmVmb3JlCmFuZCBhZnRlciB0aGUgdHJlYXRtZW50IHdpdGggY2FwdG9wcmlsIGlzIG9uIGF2ZXJhZ2UgZGlmZmVyZW50LiAKCmBgYHtyfQpvdXRwdXQyIDwtIHQudGVzdChjYXB0b3ByaWwkU0JQYiwgY2FwdG9wcmlsJFNCUGEsIHBhaXJlZCA9IFRSVUUpCm91dHB1dDIKYGBgCgoKIyMgQ29uY2x1c2lvbgoKVGhlcmUgaXMgb24gYXZlcmFnZSBhbiBleHRyZW1lbHkgc2lnbmlmaWNhbnQgYmxvb2QgcHJlc3N1cmUgZHJvcCB1cG9uIGFkbWluaXN0ZXJpbmcgY2FwdG9wcmlsIHRvIHBhdGllbnRzIHdpdGggaHlwZXJ0ZW5zaW9uIChwIDw8IDAuMDAxKS4gVGhlIHN5c3RvbGljIGJsb29kIHByZXNzdXJlIGRlY3JlYXNlcyBvbiBhdmVyYWdlIHdpdGggYHIgcm91bmQodW5uYW1lKG91dHB1dDIkZXN0aW1hdGUpLDEpYCBtbUhnIHVwb24gdGhlIHRyZWF0bWVudCB3aXRoIGNhcHRvcHJpbCAoOTUlIENJIFtgciByb3VuZChvdXRwdXQyJGNvbmYuaW50W2MoMSwyKV0sMSlgXSkuCgojIEFsdGVybmF0aXZlIHNvbHV0aW9uOiBPbmUtc2FtcGxlIHQtdGVzdCBvbiB0aGUgZGlmZmVyZW5jZQoKUGVyZm9ybWluZyBhICBwYWlyZWQgdC10ZXN0IGlzCmFuYWxvZ291cyB0byBwZXJmb3JtaW5nIGEgb25lLXNhbXBsZSB0LXRlc3Qgb24gdGhlIGRpZmZlcmVuY2UKYmV0d2VlbiBib3RoIGdyb3Vwcy4KClRoaXMgY2FuIGJlIGVhc2lseSBzZWVuIGZyb20gdGhlIG91dHB1dCBvZiB0aGUgcGFpcmVkIHR3by1zYW1wbGUKdC10ZXN0LiBUaGUgYWx0ZXJuYXRpdmUgaHlwb3RoZXNpcyAkSEEkIHRoZXJlIHN0YXRlcyB0aGF0IAp0aGUgInRydWUgZGlmZmVyZW5jZSBpbiBtZWFucyBpcyBub3QgZXF1YWwgdG8gMCIuIFNvIGludGVybmFsbHksClIgd2lsbCBhY3R1YWxseSBwZXJmb3JtIGEgb25lLXNhbXBsZSB0LXRlc3Qgb24gdGhlIGRpZmZlcmVuY2UsIGFuZApjaGVjayB3aGV0aGVyIG9yIG5vdCB0aGUgdHJ1ZSBtZWFuIGRpZmZlcmVuY2UgaXMgZXF1YWwgdG8gMC4KV2UgY2FuIGFsc28gc2V0IHRoaXMgdXAgbWFudWFsbHkuCgpgYGB7cn0KdC50ZXN0KGNhcHRvcHJpbCRkZWx0YVNCUCwgbXU9MCkKYGBgCgpJbmRlZWQsIHRoZSBvdXRwdXQgaXMgZXF1aXZhbGVudCB0byB0aGF0IG9mIHRoZSAgcGFpcmVkIHQtdGVzdC4KCgoKCgoKCg==