1 Aims

In this exercise, you will revisit the basics of statistical hypothesis testing.

You will acquire the skills

  1. to assess the assumptions of a one-sample and a paired t-test in data exploration.
  2. to conduct a paired t-test in R and to interpret the results.
  3. to conduct a one-sample t-test in R and to interpret the results.

2 The diabetes dataset

The diabetes dataset holds information on a small experiment with 8 patients that are subjected to a glucose tolerance test.

Patients had to fast for eight hours before the test. When the patients entered the hospital their baseline glucose level was measured (mmol/l).

Patients then had to drink 250 ml of a syrupy glucose solution containing 100 grams of sugar. Two hours later, their blood glucose level was measured again.

The data consist of three variables:

  • before: glucose concentration upon 8 hours of fasting (mmol/l)
  • after: glucose concentration 2 hours after drinking glucose solution (mmol/l).
  • patient: identifier for the patient

2.1 Research questions

The goal is to answer two research questions;

  1. Is the average glucose level after the sugar intake different from the average glucose level before sugar intake?

  2. Is the average glucose level of the patients two hours after the sugar intake higher than 7.8 mmol/L?

2.2 Import the data

First, load the required R libraries:

library(tidyverse)
diabetes <- read_delim("https://raw.githubusercontent.com/statOmics/PSLSData/main/diabetes.txt", delim = " ")
## Rows: 8 Columns: 3
## ── Column specification ──────────────────────────────────────────────
## Delimiter: " "
## chr (1): patient
## dbl (2): before, after
## 
## ℹ Use `spec()` to retrieve the full column specification for this data.
## ℹ Specify the column types or set `show_col_types = FALSE` to quiet this message.

3 Data exploration

We will start with a data exploration. Have a first look at the raw data. How is the data structured? Is this data tidy?

glimpse(diabetes)
## Rows: 8
## Columns: 3
## $ before  <dbl> 4.67, 4.97, 5.11, 5.17, 5.33, 6.22, 6.50, 7.00
## $ after   <dbl> 5.44, 5.78, 8.49, 6.71, 10.67, 5.67, 10.11, 9.89
## $ patient <chr> "pat1", "pat2", "pat3", "pat4", "pat5", "pat6", "pat…
head(diabetes)

We have 8 patients, for which glucose concentrations were measured, before and after sugar intake.

Note, that the dataset is not in the tidy format. The glucose concentration variable is spread around 2 columns: before and after, while the “time” variable is encoded in the column names instead of in a dedicated column. Data in this form is also called wide data. Instead, we want to transform the data to a long format.

To tidy the data, we can use the gather() function to pivot the data. In this case, we want to “gather” the time (encoded in the column names before and after) and concentration variables (which is encoded in the actual values). The patient column should stay the same. We can specify this with the following syntax.

diabetes_tidy <- diabetes %>%
  gather(key = "time", value = "concentration", -patient)

## Take a look at the new data
glimpse(diabetes_tidy)
## Rows: 16
## Columns: 3
## $ patient       <chr> "pat1", "pat2", "pat3", "pat4", "pat5", "pat6"…
## $ time          <chr> "before", "before", "before", "before", "befor…
## $ concentration <dbl> 4.67, 4.97, 5.11, 5.17, 5.33, 6.22, 6.50, 7.00…
diabetes_tidy

To improve the visualizations below, we can also sligthly recode the time variable, so that the “before” level is the first one. We can do this by converting time to a factor.

diabetes_tidy <- diabetes_tidy %>%
  mutate(
    time = as.factor(time),
    time = relevel(time, ref = "before")
  )

We could visualize the data using boxplots. Note, however, that we lose the paired characteristic of the data with this visualization. Nevertheless, it is still a useful visualization to assess any differences in mean and variance between the two time points.

boxplot <- diabetes_tidy %>%
  ggplot(aes(x = time, y = concentration, fill = time)) +
  theme_bw() +
  geom_boxplot(outlier.shape = NA) +
  geom_jitter(width = 0.2) +
  stat_summary(
    fun = mean, geom = "point",
    shape = 5, size = 3,
    color = "black"
  ) +
  ylab("Concentration (mmol/l)")

boxplot

We observe that the variability of the glucose levels are higher after than before the sugar intake. The difference plot also shows that glucose levels in the blood are higher for most of the patients two hours after the sugar intake. Only for one patient the glucose concentration two hours after treatment is lower.

To visualize the paired nature of the data, we can use a line plot, connecting the before and after points for each patient.

diabetes_tidy %>%
  ggplot(aes(time, concentration)) +
  geom_point() +
  geom_line(aes(group = patient))

4 Question 1

Is the glucose level after 8 hours of fasting on average different from the glucose level two hours after intake of 100g of glucose?

As the data are paired, we can expect that the measurements before and after the glucose intake are correlated. We can illustrate this with a scatterplot.

Note that in this case, the non-tidy data is actually a more suitable format to visualize.

diabetes %>%
  ggplot(aes(x = before, y = after)) +
  geom_point() +
  ylab("before (mmol/l)") +
  xlab("after (mmol/l)")

cor(diabetes$before, diabetes$after, method = "spearman")
## [1] 0.5952381

4.1 Check the assumptions

The paired t-test has 2 assumptions:

  1. The differences in glucose levels are independent of each other.

  2. The differences in glucose levels are normally distributed.

The first assumption is met given the experimental design. Upon differencing the measurements before and after the glucose intake, we obtained one difference for each patient. As the patients were sampled at random from the population we can expect them to be independent.

Secondly, we assess if the differences are normally distributed.

We can calculate the differences for each patient using a group_by() + summarize() combination on the tidy data.

diabetes_diff <- diabetes_tidy %>%
  group_by(patient) %>%
  summarize(difference = diff(concentration))
diabetes_diff
diabetes_diff %>%
  ggplot(aes(sample = difference)) +
  geom_qq() +
  geom_qq_line()

Based on the QQ-plot, we may assume that our data are normally distributed. As our assumptions are met we may continue with performing the paired t-test.

4.2 Hypothesis test

We will perform a paired t-test.

  • The null hypothesis of the test is that the glucose levels before and two hours after glucose intake are on average equal.
  • Which will be tested against the alternative hypothesis that the glucose levels before and two hours after the glucose intake are on average different.
paired_t <- t.test(diabetes$after,diabetes$before, paired = TRUE)
paired_t
## 
##  Paired t-test
## 
## data:  diabetes$after and diabetes$before
## t = 3.2834, df = 7, p-value = 0.01342
## alternative hypothesis: true difference in means is not equal to 0
## 95 percent confidence interval:
##  0.6222645 3.8252355
## sample estimates:
## mean of the differences 
##                 2.22375

4.3 Conclusion

There is on average an significant blood increase in the blood pressure upon administering 100g of sugar to patients (p = 0.01). The glucose levels two hours after administering 100g of glucose are on average 2.2 mmol/l higher than upon 8 hours of fastening (95% CI [0.6, 3.8]mmol/l).

5 Alternative solution: One-sample t-test on the difference

Since the data is paired, we can also simply calculate the differences in glucose level before and after sugar intake for each patient. We can then perform a one-sample t-test on these differences, testing whether they are significantly different from zero. This is equivalent to the paired t-test we performed above.

We can verify this with the analysis below

t.test(difference ~ 1, data = diabetes_diff, mu = 0)
## 
##  One Sample t-test
## 
## data:  difference
## t = 3.2834, df = 7, p-value = 0.01342
## alternative hypothesis: true mean is not equal to 0
## 95 percent confidence interval:
##  0.6222645 3.8252355
## sample estimates:
## mean of x 
##   2.22375

Indeed, the output is equivalent to that of the paired t-test.


6 Question 2

Is the average glucose level two hours after sugar intake higher than the threshold of 7.8 mmol/l for pre-diabetes?

We can test this hypothesis using a one-sample t-test. Indeed we are interested to compare the average glucose level to a known threshold for pre-diabetes.

6.1 Assess the assumptions

Before we can perform a one sample t-test, we must check that the required assumptions are met!

  1. The observations are independent
  2. The glucose levels two hours after the treatment are normally distributed

The first assumption requires us to think about how the data were collected. Are there dependences in the data? The 8 glucose levels upon sugar intake are collected on 8 random patients so we can assume that the glucose levels 2 hours after sugar intake are independent.

We can assess the second assumption with a quantile-quantile plot. First we need to filter the data, so we only retain the values for the time == "after" level

diabetes_after <- diabetes_tidy %>%
  filter(time == "after")
diabetes_after %>%
  ggplot(aes(sample = concentration)) +
  geom_qq() +
  geom_qq_line()

The data seem to be nicely scattered around the quantile-quantile line (black line). We also do not observe large deviations in the plot.

So we will assume that our data is normally distributed. Note, that there are not many observations to assess normality.

6.2 Hypothesis test

Here, we will test if mean glucose level 2 hours after sugar intake is significantly higher than the threshold for pre-diabetes of 7.8 mmol/l. More specifically, we will test the null hypothesis;

\(H_0:\) the mean glucose level two hours after 100g of sugar intake is equal to 7.8 mmol/l

versus the alternative hypothesis;

\(H_1:\) the mean glucose level two hours after 100g of sugar intake is greater than 7.8 mmol/l

t_test_after <- t.test(concentration ~ 1,
  data = diabetes_after,
  mu = 7.8,
  alternative = "greater",
  conf.level = 0.95
)
t_test_after
## 
##  One Sample t-test
## 
## data:  concentration
## t = 0.057943, df = 7, p-value = 0.4777
## alternative hypothesis: true mean is greater than 7.8
## 95 percent confidence interval:
##  6.373627      Inf
## sample estimates:
## mean of x 
##     7.845

6.3 Conclusion

When writing a conclusion on your research hypothesis, it is very important to be precise, concise, and complete.

An example of such a conclusion for our research question is given below:

We can conclude that the mean glucose level of patients two hours after 100g of glucose intake is not significantly higher than the threshold for pre-diabetes of 7.8 mmol/l (p = 0.48). The mean glucose level two hours after the intake of 100g of sugar equals 7.8 mmol/l (95% CI [6.4, ]).

As we have seen in the theory class, the 95% confidence interval can be interpreted as;

With 95% confidence we can conclude that the true average of glucose level of the patients in the population is above 6.37.

Note, that we only report a one sided confidence interval because we only test against the alternative hypothesis that the glucose level two hours after sugar intake is on average larger than the threshold for pre-diabetes of 7.8 mmol/l.

LS0tCnRpdGxlOiAiRXhlcmNpc2UgNS4xOiBIeXBvdGhlc2lzIHRlc3Rpbmcgb24gdGhlIGRpYWJldGVzIGV4YW1wbGUgLSBzb2x1dGlvbiIKYXV0aG9yOiAiTGlldmVuIENsZW1lbnQsIEplcm9lbiBHaWxpcyBhbmQgTWlsYW4gTWFsZmFpdCIKZGF0ZTogInN0YXRPbWljcywgR2hlbnQgVW5pdmVyc2l0eSAoaHR0cHM6Ly9zdGF0b21pY3MuZ2l0aHViLmlvKSIKLS0tCgojIEFpbXMKCkluIHRoaXMgZXhlcmNpc2UsIHlvdSB3aWxsIHJldmlzaXQgdGhlIGJhc2ljcyBvZiBzdGF0aXN0aWNhbCBoeXBvdGhlc2lzIHRlc3RpbmcuCgpZb3Ugd2lsbCBhY3F1aXJlIHRoZSBza2lsbHMKCjEuIHRvIGFzc2VzcyB0aGUgYXNzdW1wdGlvbnMgb2YgYSBvbmUtc2FtcGxlIGFuZCBhIHBhaXJlZCB0LXRlc3QgaW4gIGRhdGEgZXhwbG9yYXRpb24uCjIuIHRvIGNvbmR1Y3QgYSBwYWlyZWQgdC10ZXN0IGluIFIgYW5kIHRvIGludGVycHJldCB0aGUgcmVzdWx0cy4KMy4gdG8gY29uZHVjdCBhIG9uZS1zYW1wbGUgdC10ZXN0IGluIFIgYW5kIHRvIGludGVycHJldCB0aGUgcmVzdWx0cy4KCiMgVGhlIGRpYWJldGVzIGRhdGFzZXQKClRoZSBgZGlhYmV0ZXNgICBkYXRhc2V0IGhvbGRzIGluZm9ybWF0aW9uIG9uIGEgc21hbGwgZXhwZXJpbWVudCB3aXRoCjggcGF0aWVudHMgdGhhdCBhcmUgc3ViamVjdGVkIHRvIGEgZ2x1Y29zZSB0b2xlcmFuY2UgdGVzdC4KClBhdGllbnRzIGhhZCB0byBmYXN0IGZvciBlaWdodCBob3VycyBiZWZvcmUgdGhlIHRlc3QuCldoZW4gdGhlIHBhdGllbnRzIGVudGVyZWQgdGhlIGhvc3BpdGFsIHRoZWlyIGJhc2VsaW5lIGdsdWNvc2UgbGV2ZWwgd2FzIG1lYXN1cmVkIChtbW9sL2wpLgoKUGF0aWVudHMgdGhlbiAgaGFkIHRvIGRyaW5rIDI1MCBtbCBvZiBhIHN5cnVweSBnbHVjb3NlIHNvbHV0aW9uIGNvbnRhaW5pbmcgMTAwCmdyYW1zIG9mIHN1Z2FyLiAgVHdvIGhvdXJzIGxhdGVyLCB0aGVpciBibG9vZCBnbHVjb3NlIGxldmVsIHdhcyBtZWFzdXJlZCBhZ2Fpbi4KClRoZSBkYXRhIGNvbnNpc3Qgb2YgdGhyZWUgdmFyaWFibGVzOgoKLSBgYmVmb3JlYDogZ2x1Y29zZSBjb25jZW50cmF0aW9uIHVwb24gOCBob3VycyBvZiBmYXN0aW5nIChtbW9sL2wpCi0gYGFmdGVyYDogZ2x1Y29zZSBjb25jZW50cmF0aW9uIDIgaG91cnMgYWZ0ZXIgZHJpbmtpbmcgZ2x1Y29zZSBzb2x1dGlvbiAobW1vbC9sKS4KLSBgcGF0aWVudGA6IGlkZW50aWZpZXIgZm9yIHRoZSBwYXRpZW50CgojIyBSZXNlYXJjaCBxdWVzdGlvbnMKClRoZSBnb2FsIGlzIHRvIGFuc3dlciB0d28gcmVzZWFyY2ggcXVlc3Rpb25zOwoKMS4gSXMgdGhlIGF2ZXJhZ2UgZ2x1Y29zZSBsZXZlbCBhZnRlciB0aGUgc3VnYXIgaW50YWtlIGRpZmZlcmVudCBmcm9tIHRoZSBhdmVyYWdlIGdsdWNvc2UgbGV2ZWwgYmVmb3JlIHN1Z2FyIGludGFrZT8KCjIuIElzIHRoZSBhdmVyYWdlIGdsdWNvc2UgbGV2ZWwgb2YgdGhlIHBhdGllbnRzIHR3byBob3VycyBhZnRlciB0aGUgc3VnYXIgaW50YWtlIGhpZ2hlciB0aGFuIDcuOCBtbW9sL0w/CgojIyBJbXBvcnQgdGhlIGRhdGEKCkZpcnN0LCBsb2FkIHRoZSByZXF1aXJlZCBSIGxpYnJhcmllczoKCmBgYHtyLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgpgYGB7cn0KZGlhYmV0ZXMgPC0gcmVhZF9kZWxpbSgiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3N0YXRPbWljcy9QU0xTRGF0YS9tYWluL2RpYWJldGVzLnR4dCIsIGRlbGltID0gIiAiKQpgYGAKCiMgRGF0YSBleHBsb3JhdGlvbgoKV2Ugd2lsbCBzdGFydCB3aXRoIGEgZGF0YSBleHBsb3JhdGlvbi4KSGF2ZSBhIGZpcnN0IGxvb2sgYXQgdGhlIHJhdyBkYXRhLiBIb3cgaXMgdGhlIGRhdGEgc3RydWN0dXJlZD8KSXMgdGhpcyBkYXRhICp0aWR5Kj8KCmBgYHtyfQpnbGltcHNlKGRpYWJldGVzKQpoZWFkKGRpYWJldGVzKQpgYGAKCldlIGhhdmUgOCBwYXRpZW50cywgZm9yIHdoaWNoIGdsdWNvc2UgY29uY2VudHJhdGlvbnMgd2VyZSBtZWFzdXJlZCwgYmVmb3JlIGFuZCBhZnRlciBzdWdhciBpbnRha2UuCgpOb3RlLCB0aGF0IHRoZSBkYXRhc2V0IGlzIG5vdCBpbiB0aGUgdGlkeSBmb3JtYXQuIFRoZSBnbHVjb3NlIGNvbmNlbnRyYXRpb24KdmFyaWFibGUgaXMgc3ByZWFkIGFyb3VuZCAyIGNvbHVtbnM6IGBiZWZvcmVgIGFuZCBgYWZ0ZXJgLCB3aGlsZSB0aGUgInRpbWUiCnZhcmlhYmxlIGlzIGVuY29kZWQgaW4gdGhlIGNvbHVtbiBuYW1lcyBpbnN0ZWFkIG9mIGluIGEgZGVkaWNhdGVkIGNvbHVtbi4gRGF0YQppbiB0aGlzIGZvcm0gaXMgYWxzbyBjYWxsZWQgKndpZGUqIGRhdGEuIEluc3RlYWQsIHdlIHdhbnQgdG8gdHJhbnNmb3JtIHRoZSBkYXRhCnRvIGEgKmxvbmcqIGZvcm1hdC4KClRvIHRpZHkgdGhlIGRhdGEsIHdlIGNhbiB1c2UgdGhlIGBnYXRoZXIoKWAgZnVuY3Rpb24gdG8KW3Bpdm90XShodHRwczovL3I0ZHMuaGFkLmNvLm56L3RpZHktZGF0YS5odG1sI3Bpdm90aW5nKSB0aGUgZGF0YS4gSW4gdGhpcyBjYXNlLAp3ZSB3YW50IHRvICJnYXRoZXIiIHRoZSBgdGltZWAgKGVuY29kZWQgaW4gdGhlIGNvbHVtbiBuYW1lcyBgYmVmb3JlYCBhbmQKYGFmdGVyYCkgYW5kIGBjb25jZW50cmF0aW9uYCB2YXJpYWJsZXMgKHdoaWNoIGlzIGVuY29kZWQgaW4gdGhlIGFjdHVhbCB2YWx1ZXMpLgpUaGUgYHBhdGllbnRgIGNvbHVtbiBzaG91bGQgc3RheSB0aGUgc2FtZS4gV2UgY2FuIHNwZWNpZnkgdGhpcyB3aXRoIHRoZQpmb2xsb3dpbmcgc3ludGF4LgoKYGBge3J9CmRpYWJldGVzX3RpZHkgPC0gZGlhYmV0ZXMgJT4lCiAgZ2F0aGVyKGtleSA9ICJ0aW1lIiwgdmFsdWUgPSAiY29uY2VudHJhdGlvbiIsIC1wYXRpZW50KQoKIyMgVGFrZSBhIGxvb2sgYXQgdGhlIG5ldyBkYXRhCmdsaW1wc2UoZGlhYmV0ZXNfdGlkeSkKZGlhYmV0ZXNfdGlkeQpgYGAKClRvIGltcHJvdmUgdGhlIHZpc3VhbGl6YXRpb25zIGJlbG93LCB3ZSBjYW4gYWxzbyBzbGlndGhseSByZWNvZGUgdGhlIGB0aW1lYCB2YXJpYWJsZSwgc28gdGhhdCB0aGUgImJlZm9yZSIgbGV2ZWwgaXMgdGhlIGZpcnN0IG9uZS4gV2UgY2FuIGRvIHRoaXMgYnkgY29udmVydGluZyBgdGltZWAgdG8gYSBmYWN0b3IuCgpgYGB7cn0KZGlhYmV0ZXNfdGlkeSA8LSBkaWFiZXRlc190aWR5ICU+JQogIG11dGF0ZSgKICAgIHRpbWUgPSBhcy5mYWN0b3IodGltZSksCiAgICB0aW1lID0gcmVsZXZlbCh0aW1lLCByZWYgPSAiYmVmb3JlIikKICApCmBgYAoKV2UgY291bGQgdmlzdWFsaXplIHRoZSBkYXRhIHVzaW5nIGJveHBsb3RzLiBOb3RlLCBob3dldmVyLCB0aGF0IHdlIGxvc2UgdGhlCnBhaXJlZCBjaGFyYWN0ZXJpc3RpYyBvZiB0aGUgZGF0YSB3aXRoIHRoaXMgdmlzdWFsaXphdGlvbi4gTmV2ZXJ0aGVsZXNzLCBpdCBpcwpzdGlsbCBhIHVzZWZ1bCB2aXN1YWxpemF0aW9uIHRvIGFzc2VzcyBhbnkgZGlmZmVyZW5jZXMgaW4gbWVhbiBhbmQgdmFyaWFuY2UKYmV0d2VlbiB0aGUgdHdvIHRpbWUgcG9pbnRzLgoKYGBge3J9CmJveHBsb3QgPC0gZGlhYmV0ZXNfdGlkeSAlPiUKICBnZ3Bsb3QoYWVzKHggPSB0aW1lLCB5ID0gY29uY2VudHJhdGlvbiwgZmlsbCA9IHRpbWUpKSArCiAgdGhlbWVfYncoKSArCiAgZ2VvbV9ib3hwbG90KG91dGxpZXIuc2hhcGUgPSBOQSkgKwogIGdlb21faml0dGVyKHdpZHRoID0gMC4yKSArCiAgc3RhdF9zdW1tYXJ5KAogICAgZnVuID0gbWVhbiwgZ2VvbSA9ICJwb2ludCIsCiAgICBzaGFwZSA9IDUsIHNpemUgPSAzLAogICAgY29sb3IgPSAiYmxhY2siCiAgKSArCiAgeWxhYigiQ29uY2VudHJhdGlvbiAobW1vbC9sKSIpCgpib3hwbG90CmBgYAoKV2Ugb2JzZXJ2ZSB0aGF0IHRoZSB2YXJpYWJpbGl0eSBvZiB0aGUgZ2x1Y29zZSBsZXZlbHMgYXJlIGhpZ2hlciBhZnRlciB0aGFuCmJlZm9yZSB0aGUgc3VnYXIgaW50YWtlLiAgVGhlIGRpZmZlcmVuY2UgcGxvdCBhbHNvIHNob3dzIHRoYXQgZ2x1Y29zZSBsZXZlbHMgaW4KdGhlIGJsb29kIGFyZSBoaWdoZXIgZm9yIG1vc3Qgb2YgdGhlIHBhdGllbnRzIHR3byBob3VycyBhZnRlciB0aGUgc3VnYXIgaW50YWtlLgpPbmx5IGZvciBvbmUgcGF0aWVudCB0aGUgZ2x1Y29zZSBjb25jZW50cmF0aW9uIHR3byBob3VycyBhZnRlciB0cmVhdG1lbnQgaXMKbG93ZXIuCgpUbyB2aXN1YWxpemUgdGhlIHBhaXJlZCBuYXR1cmUgb2YgdGhlIGRhdGEsIHdlIGNhbiB1c2UgYSAqKmxpbmUgcGxvdCoqLApjb25uZWN0aW5nIHRoZSBgYmVmb3JlYCBhbmQgYGFmdGVyYCBwb2ludHMgZm9yIGVhY2ggcGF0aWVudC4KCmBgYHtyfQpkaWFiZXRlc190aWR5ICU+JQogIGdncGxvdChhZXModGltZSwgY29uY2VudHJhdGlvbikpICsKICBnZW9tX3BvaW50KCkgKwogIGdlb21fbGluZShhZXMoZ3JvdXAgPSBwYXRpZW50KSkKYGBgCgoKIyBRdWVzdGlvbiAxCgpJcyB0aGUgZ2x1Y29zZSBsZXZlbCBhZnRlciA4IGhvdXJzIG9mIGZhc3Rpbmcgb24gYXZlcmFnZSBkaWZmZXJlbnQgZnJvbSB0aGUgZ2x1Y29zZSBsZXZlbCB0d28gaG91cnMgYWZ0ZXIgaW50YWtlIG9mIDEwMGcgb2YgZ2x1Y29zZT8KCkFzIHRoZSBkYXRhIGFyZSBwYWlyZWQsIHdlIGNhbiBleHBlY3QgdGhhdCB0aGUgbWVhc3VyZW1lbnRzIGJlZm9yZSBhbmQgYWZ0ZXIgdGhlIGdsdWNvc2UgaW50YWtlIGFyZSBjb3JyZWxhdGVkLgpXZSBjYW4gaWxsdXN0cmF0ZSB0aGlzIHdpdGggYSBzY2F0dGVycGxvdC4KCk5vdGUgdGhhdCBpbiB0aGlzIGNhc2UsIHRoZSBub24tdGlkeSBkYXRhIGlzIGFjdHVhbGx5IGEgbW9yZSBzdWl0YWJsZSBmb3JtYXQgdG8gdmlzdWFsaXplLgoKYGBge3J9CmRpYWJldGVzICU+JQogIGdncGxvdChhZXMoeCA9IGJlZm9yZSwgeSA9IGFmdGVyKSkgKwogIGdlb21fcG9pbnQoKSArCiAgeWxhYigiYmVmb3JlIChtbW9sL2wpIikgKwogIHhsYWIoImFmdGVyIChtbW9sL2wpIikKYGBgCgpgYGB7cn0KY29yKGRpYWJldGVzJGJlZm9yZSwgZGlhYmV0ZXMkYWZ0ZXIsIG1ldGhvZCA9ICJzcGVhcm1hbiIpCmBgYAoKCiMjIENoZWNrIHRoZSBhc3N1bXB0aW9ucwoKVGhlIHBhaXJlZCB0LXRlc3QgaGFzIDIgYXNzdW1wdGlvbnM6CgoxLiBUaGUgZGlmZmVyZW5jZXMgaW4gZ2x1Y29zZSBsZXZlbHMgYXJlIGluZGVwZW5kZW50IG9mIGVhY2ggb3RoZXIuCgoyLiBUaGUgZGlmZmVyZW5jZXMgaW4gZ2x1Y29zZSBsZXZlbHMgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkLgoKVGhlIGZpcnN0IGFzc3VtcHRpb24gaXMgbWV0IGdpdmVuIHRoZSBleHBlcmltZW50YWwgZGVzaWduLgpVcG9uIGRpZmZlcmVuY2luZyB0aGUgbWVhc3VyZW1lbnRzIGJlZm9yZSBhbmQgYWZ0ZXIgdGhlIGdsdWNvc2UgaW50YWtlLCB3ZSBvYnRhaW5lZCBvbmUgZGlmZmVyZW5jZSBmb3IgZWFjaCBwYXRpZW50LiBBcyB0aGUgcGF0aWVudHMgd2VyZSBzYW1wbGVkIGF0IHJhbmRvbSBmcm9tIHRoZSBwb3B1bGF0aW9uIHdlIGNhbiBleHBlY3QgdGhlbSB0byBiZSBpbmRlcGVuZGVudC4KClNlY29uZGx5LCB3ZSBhc3Nlc3MgaWYgdGhlIGRpZmZlcmVuY2VzIGFyZSBub3JtYWxseSBkaXN0cmlidXRlZC4KCldlIGNhbiBjYWxjdWxhdGUgdGhlIGRpZmZlcmVuY2VzIGZvciBlYWNoIHBhdGllbnQgdXNpbmcgYSBgZ3JvdXBfYnkoKWAgKwpgc3VtbWFyaXplKClgIGNvbWJpbmF0aW9uIG9uIHRoZSB0aWR5IGRhdGEuCgpgYGB7cn0KZGlhYmV0ZXNfZGlmZiA8LSBkaWFiZXRlc190aWR5ICU+JQogIGdyb3VwX2J5KHBhdGllbnQpICU+JQogIHN1bW1hcml6ZShkaWZmZXJlbmNlID0gZGlmZihjb25jZW50cmF0aW9uKSkKZGlhYmV0ZXNfZGlmZgpgYGAKCmBgYHtyfQpkaWFiZXRlc19kaWZmICU+JQogIGdncGxvdChhZXMoc2FtcGxlID0gZGlmZmVyZW5jZSkpICsKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpCmBgYAoKQmFzZWQgb24gdGhlIFFRLXBsb3QsIHdlIG1heSBhc3N1bWUgdGhhdCBvdXIgZGF0YSBhcmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQuCkFzIG91ciBhc3N1bXB0aW9ucyBhcmUgbWV0IHdlIG1heSBjb250aW51ZSB3aXRoIHBlcmZvcm1pbmcgdGhlIHBhaXJlZCB0LXRlc3QuCgojIyBIeXBvdGhlc2lzIHRlc3QKCldlIHdpbGwgcGVyZm9ybSBhICoqcGFpcmVkIHQtdGVzdCoqLgoKLSBUaGUgbnVsbCBoeXBvdGhlc2lzIG9mIHRoZSB0ZXN0IGlzIHRoYXQgdGhlIGdsdWNvc2UgbGV2ZWxzIGJlZm9yZSBhbmQgdHdvIGhvdXJzIGFmdGVyCmdsdWNvc2UgaW50YWtlIGFyZSBvbiBhdmVyYWdlIGVxdWFsLgotIFdoaWNoIHdpbGwgYmUgdGVzdGVkIGFnYWluc3QgdGhlIGFsdGVybmF0aXZlIGh5cG90aGVzaXMgdGhhdCB0aGUgZ2x1Y29zZSBsZXZlbHMgYmVmb3JlIGFuZCB0d28gaG91cnMgYWZ0ZXIgdGhlIGdsdWNvc2UgaW50YWtlIGFyZSBvbiBhdmVyYWdlIGRpZmZlcmVudC4KCmBgYHtyfQpwYWlyZWRfdCA8LSB0LnRlc3QoZGlhYmV0ZXMkYWZ0ZXIsZGlhYmV0ZXMkYmVmb3JlLCBwYWlyZWQgPSBUUlVFKQpwYWlyZWRfdApgYGAKCgojIyBDb25jbHVzaW9uCgpUaGVyZSBpcyBvbiBhdmVyYWdlIGFuIHNpZ25pZmljYW50IGJsb29kIGluY3JlYXNlIGluIHRoZSBibG9vZCBwcmVzc3VyZSB1cG9uCmFkbWluaXN0ZXJpbmcgMTAwZyBvZiBzdWdhciB0byBwYXRpZW50cyAocCA9IGByIHJvdW5kKHBhaXJlZF90JHAudmFsdWUsIDIpYCkuClRoZSBnbHVjb3NlIGxldmVscyB0d28gaG91cnMgYWZ0ZXIgYWRtaW5pc3RlcmluZyAxMDBnIG9mIGdsdWNvc2UgYXJlIG9uIGF2ZXJhZ2UgYHIgcm91bmQocGFpcmVkX3QkZXN0aW1hdGUsMSlgIG1tb2wvbCBoaWdoZXIgdGhhbiB1cG9uIDggaG91cnMgb2YgZmFzdGVuaW5nCig5NSUgQ0kgW2ByIHJvdW5kKHBhaXJlZF90JGNvbmYuaW50LDEpYF1tbW9sL2wpLgoKIyBBbHRlcm5hdGl2ZSBzb2x1dGlvbjogT25lLXNhbXBsZSB0LXRlc3Qgb24gdGhlIGRpZmZlcmVuY2UKClNpbmNlIHRoZSBkYXRhIGlzIHBhaXJlZCwgd2UgY2FuIGFsc28gc2ltcGx5IGNhbGN1bGF0ZSB0aGUgZGlmZmVyZW5jZXMgaW4KZ2x1Y29zZSBsZXZlbCBiZWZvcmUgYW5kIGFmdGVyIHN1Z2FyIGludGFrZSBmb3IgZWFjaCBwYXRpZW50LiBXZSBjYW4gdGhlbgpwZXJmb3JtIGEgb25lLXNhbXBsZSB0LXRlc3Qgb24gdGhlc2UgZGlmZmVyZW5jZXMsIHRlc3Rpbmcgd2hldGhlciB0aGV5IGFyZQpzaWduaWZpY2FudGx5IGRpZmZlcmVudCBmcm9tIHplcm8uIFRoaXMgaXMgZXF1aXZhbGVudCB0byB0aGUgcGFpcmVkIHQtdGVzdCB3ZQpwZXJmb3JtZWQgYWJvdmUuCgpXZSBjYW4gdmVyaWZ5IHRoaXMgd2l0aCB0aGUgYW5hbHlzaXMgYmVsb3cKCmBgYHtyfQp0LnRlc3QoZGlmZmVyZW5jZSB+IDEsIGRhdGEgPSBkaWFiZXRlc19kaWZmLCBtdSA9IDApCmBgYAoKSW5kZWVkLCB0aGUgb3V0cHV0IGlzIGVxdWl2YWxlbnQgdG8gdGhhdCBvZiB0aGUgcGFpcmVkIHQtdGVzdC4KCioqKgoKIyBRdWVzdGlvbiAyCgpJcyB0aGUgYXZlcmFnZSBnbHVjb3NlIGxldmVsIHR3byBob3VycyBhZnRlciBzdWdhciBpbnRha2UKaGlnaGVyIHRoYW4gdGhlIHRocmVzaG9sZCBvZiA3LjggbW1vbC9sIGZvciBwcmUtZGlhYmV0ZXM/CgpXZSBjYW4gdGVzdCB0aGlzIGh5cG90aGVzaXMgdXNpbmcgYSAqKm9uZS1zYW1wbGUgdC10ZXN0KiouCkluZGVlZCB3ZSBhcmUgaW50ZXJlc3RlZCB0byBjb21wYXJlIHRoZSBhdmVyYWdlIGdsdWNvc2UgbGV2ZWwgdG8gYSBrbm93biB0aHJlc2hvbGQgZm9yIHByZS1kaWFiZXRlcy4KCiMjIEFzc2VzcyB0aGUgYXNzdW1wdGlvbnMKCkJlZm9yZSB3ZSBjYW4gcGVyZm9ybSBhIG9uZSBzYW1wbGUgdC10ZXN0LCB3ZSBtdXN0IGNoZWNrIHRoYXQgdGhlIHJlcXVpcmVkCmFzc3VtcHRpb25zIGFyZSBtZXQhCgoxLiBUaGUgb2JzZXJ2YXRpb25zIGFyZSBpbmRlcGVuZGVudAoyLiBUaGUgZ2x1Y29zZSBsZXZlbHMgdHdvIGhvdXJzIGFmdGVyIHRoZSB0cmVhdG1lbnQgYXJlIG5vcm1hbGx5IGRpc3RyaWJ1dGVkCgpUaGUgZmlyc3QgYXNzdW1wdGlvbiByZXF1aXJlcyB1cyB0byB0aGluayBhYm91dCBob3cgdGhlIGRhdGEgd2VyZSBjb2xsZWN0ZWQuCkFyZSB0aGVyZSBkZXBlbmRlbmNlcyBpbiB0aGUgZGF0YT8KVGhlIDggZ2x1Y29zZSBsZXZlbHMgdXBvbiBzdWdhciBpbnRha2UgYXJlIGNvbGxlY3RlZCBvbiA4IHJhbmRvbSBwYXRpZW50cyBzbyB3ZSBjYW4gYXNzdW1lIHRoYXQgdGhlIGdsdWNvc2UgbGV2ZWxzIDIgaG91cnMgYWZ0ZXIgc3VnYXIgaW50YWtlIGFyZSBpbmRlcGVuZGVudC4KCldlIGNhbiBhc3Nlc3MgdGhlIHNlY29uZCBhc3N1bXB0aW9uIHdpdGggYSBxdWFudGlsZS1xdWFudGlsZSBwbG90LiBGaXJzdCB3ZSBuZWVkIHRvIGZpbHRlciB0aGUgZGF0YSwgc28gd2Ugb25seSByZXRhaW4gdGhlIHZhbHVlcyBmb3IgdGhlIGB0aW1lID09ICJhZnRlciJgIGxldmVsCgpgYGB7cn0KZGlhYmV0ZXNfYWZ0ZXIgPC0gZGlhYmV0ZXNfdGlkeSAlPiUKICBmaWx0ZXIodGltZSA9PSAiYWZ0ZXIiKQpgYGAKCmBgYHtyfQpkaWFiZXRlc19hZnRlciAlPiUKICBnZ3Bsb3QoYWVzKHNhbXBsZSA9IGNvbmNlbnRyYXRpb24pKSArCiAgZ2VvbV9xcSgpICsKICBnZW9tX3FxX2xpbmUoKQpgYGAKClRoZSBkYXRhIHNlZW0gdG8gYmUgbmljZWx5IHNjYXR0ZXJlZCBhcm91bmQgdGhlIHF1YW50aWxlLXF1YW50aWxlCmxpbmUgKGJsYWNrIGxpbmUpLgpXZSBhbHNvIGRvIG5vdCBvYnNlcnZlIGxhcmdlIGRldmlhdGlvbnMgaW4gdGhlIHBsb3QuCgpTbyB3ZSB3aWxsIGFzc3VtZSB0aGF0IG91ciBkYXRhIGlzIG5vcm1hbGx5CmRpc3RyaWJ1dGVkLgpOb3RlLCB0aGF0IHRoZXJlIGFyZSBub3QgbWFueSBvYnNlcnZhdGlvbnMgdG8gYXNzZXNzIG5vcm1hbGl0eS4KCiMjIEh5cG90aGVzaXMgdGVzdAoKSGVyZSwgd2Ugd2lsbCB0ZXN0IGlmIG1lYW4gZ2x1Y29zZSBsZXZlbCAyIGhvdXJzIGFmdGVyIHN1Z2FyIGludGFrZSBpcyBzaWduaWZpY2FudGx5CmhpZ2hlciB0aGFuIHRoZSB0aHJlc2hvbGQgZm9yIHByZS1kaWFiZXRlcyBvZiA3LjggbW1vbC9sLiBNb3JlIHNwZWNpZmljYWxseSwgd2Ugd2lsbCB0ZXN0IHRoZSBudWxsIGh5cG90aGVzaXM7CgokSF8wOiQgdGhlIG1lYW4gZ2x1Y29zZSBsZXZlbCB0d28gaG91cnMgYWZ0ZXIgMTAwZyBvZiBzdWdhciBpbnRha2UgaXMgZXF1YWwgdG8gNy44IG1tb2wvbAoKdmVyc3VzIHRoZSBhbHRlcm5hdGl2ZSBoeXBvdGhlc2lzOwoKJEhfMTokIHRoZSBtZWFuIGdsdWNvc2UgbGV2ZWwgdHdvIGhvdXJzIGFmdGVyIDEwMGcgb2Ygc3VnYXIgaW50YWtlIGlzIGdyZWF0ZXIgdGhhbiA3LjggbW1vbC9sCgpgYGB7cn0KdF90ZXN0X2FmdGVyIDwtIHQudGVzdChjb25jZW50cmF0aW9uIH4gMSwKICBkYXRhID0gZGlhYmV0ZXNfYWZ0ZXIsCiAgbXUgPSA3LjgsCiAgYWx0ZXJuYXRpdmUgPSAiZ3JlYXRlciIsCiAgY29uZi5sZXZlbCA9IDAuOTUKKQp0X3Rlc3RfYWZ0ZXIKYGBgCgojIyBDb25jbHVzaW9uCgpXaGVuIHdyaXRpbmcgYSBjb25jbHVzaW9uIG9uIHlvdXIgcmVzZWFyY2ggaHlwb3RoZXNpcywKaXQgaXMgdmVyeSBpbXBvcnRhbnQgdG8gYmUgcHJlY2lzZSwgY29uY2lzZSwgYW5kIGNvbXBsZXRlLgoKQW4gZXhhbXBsZSBvZiBzdWNoIGEgY29uY2x1c2lvbiBmb3Igb3VyIHJlc2VhcmNoIHF1ZXN0aW9uCmlzIGdpdmVuIGJlbG93OgoKV2UgY2FuIGNvbmNsdWRlIHRoYXQgdGhlIG1lYW4gZ2x1Y29zZSBsZXZlbCBvZiBwYXRpZW50cyB0d28gaG91cnMgYWZ0ZXIgMTAwZyBvZiBnbHVjb3NlIGludGFrZSBpcyBub3Qgc2lnbmlmaWNhbnRseSBoaWdoZXIgdGhhbiB0aGUgdGhyZXNob2xkICBmb3IgcHJlLWRpYWJldGVzIG9mIDcuOCBtbW9sL2wgKHAgPSBgciByb3VuZCh0X3Rlc3RfYWZ0ZXIkcC52YWx1ZSwyKWApLgpUaGUgbWVhbiBnbHVjb3NlIGxldmVsIHR3byBob3VycyBhZnRlciB0aGUgaW50YWtlIG9mIDEwMGcgb2Ygc3VnYXIgZXF1YWxzIGByIHJvdW5kKHRfdGVzdF9hZnRlciRlc3RpbWF0ZSwxKWAgbW1vbC9sICg5NSUgQ0kgW2ByIHJvdW5kKHRfdGVzdF9hZnRlciRjb25mLmludCwxKWBdKS4KCkFzIHdlIGhhdmUgc2VlbiBpbiB0aGUgdGhlb3J5IGNsYXNzLCB0aGUgOTUlIGNvbmZpZGVuY2UKaW50ZXJ2YWwgY2FuIGJlIGludGVycHJldGVkIGFzOwoKV2l0aCA5NSUgY29uZmlkZW5jZSB3ZSBjYW4gY29uY2x1ZGUgdGhhdCB0aGUgdHJ1ZSBhdmVyYWdlIG9mIGdsdWNvc2UgbGV2ZWwgb2YgdGhlIHBhdGllbnRzIGluIHRoZSBwb3B1bGF0aW9uIGlzIGFib3ZlIGByIHJvdW5kKHRfdGVzdF9hZnRlciRjb25mLmludFsxXSwyKWAuCgpOb3RlLCB0aGF0IHdlIG9ubHkgcmVwb3J0IGEgb25lIHNpZGVkIGNvbmZpZGVuY2UgaW50ZXJ2YWwgYmVjYXVzZSB3ZSBvbmx5IHRlc3QgYWdhaW5zdCB0aGUgYWx0ZXJuYXRpdmUgaHlwb3RoZXNpcyB0aGF0IHRoZSBnbHVjb3NlIGxldmVsIHR3byBob3VycyBhZnRlciBzdWdhciBpbnRha2UgaXMgb24gYXZlcmFnZSBsYXJnZXIgdGhhbiB0aGUgdGhyZXNob2xkIGZvciBwcmUtZGlhYmV0ZXMgb2YgNy44IG1tb2wvbC4K