1 Cuckoo dataset

The common cuckoo does not build its own nest: it prefers to lay its eggs in another birds’ nest. It is known, since 1892, that the type of cuckoo bird eggs are different between different locations. In a study from 1940, it was shown that cuckoos return to the same nesting area each year, and that they always pick the same bird species to be a “foster parent” for their eggs.

Over the years, this has lead to the development of geographically determined subspecies of cuckoos. These subspecies have evolved in such a way that their eggs look as similar as possible as those of their foster parents.

The cuckoo dataset contains information on 120 Cuckoo eggs, obtained from randomly selected “foster” nests. For these eggs, researchers have measured the length (in mm) and established the type (species) of foster parent. The type column is coded as follows:

  • type=1: Meadow pipit
  • type=2: Tree pipit
  • type=3: Dunnock
  • type=4: European robin
  • type=5: White wagtail
  • type=6: Eurasian wren

2 Goal

The researchers want totest if the type of foster parent has an effect on the average length of the cuckoo eggs.

In theory, they want to study this for all six species. Previously, we looked at a single pairwise comparison between the European robin and the Eurasian wren with a t-test. Here, we will analyse all types simultaneously with ANOVA.

Load the required libraries

library(tidyverse)

2.1 Import the data

Cuckoo <- read_tsv("https://raw.githubusercontent.com/statOmics/PSLS21/data/Cuckoo.txt")
head(Cuckoo)

3 Data tidying

It seems that the tpye column is a double rather than a factor. Let’s fix this:

Cuckoo <- Cuckoo %>%
  mutate(type = as.factor(type)) %>%
  mutate(type = recode(type, 
           "1" = "meadow", 
           "2" = "tree", 
           "3" = "dunnock",  
           "4" = "robin", 
           "5" = "wagtail", 
           "6" = "wren"))

4 Data exploration

How many birds do we have for each type?

Cuckoo %>%
  count(type)

Visualize the data

Cuckoo %>%
  ggplot(aes(x=type,y=length,fill=type)) +
    theme_bw() +
    scale_fill_brewer(palette="RdGy") +
    geom_boxplot() +
    geom_jitter(width = 0.2) +
    ggtitle("Boxplot of the length of eggs per type") +
    ylab("length (mm)") + 
    stat_summary(fun=mean, geom="point", shape=5, size=3, color="black", fill="black")

5 ANOVA

To study if the observed differences in average egg length between the different foster bird types are significant, we may perform an ANOVA.

5.1 Formulate null and alternative hypothesis

The null hypothesis of ANOVA states that: \(H0\): The mean egg length is equal between the different bird types.

The alternative hypothesis of ANOVA states that: \(HA\): The mean egg length for at least one bird type is different from the mean egg length in at least one other bird type.

5.2 Parameter estimation

mod1 <- lm(length~type, Cuckoo)

5.3 Check assumptions

Before we may proceed with the analysis, we must make sure that all assumptions for ANOVA are met. ANOVA has three assumptions:

  1. The observations are independent of each other (in all groups)
  2. The data (length) must be normally distributed (in all groups)
  3. The variability within all groups is similar

The first assumption is met, as we may assume that there are no specific patterns of correlation between the randomly selected nests.

To check the normality assumption, we will use QQ plots.

plot(mod1, which = 2)

Cuckoo %>% 
  ggplot(aes(sample=length)) +
  geom_qq() +
  geom_qq_line() + 
  facet_grid(~type)

There seem to be no clear deviations from normality.

The third assumption of equal variances seems to be met based on the visualization with the boxplots (see above).

As such, we may proceed with the ANOVA analysis.

5.3.1 Simulate to train your skills in assessing assumptions

set.seed(1031)
sigma <- mod1 %>% sigma 
dataHlp <- Cuckoo

simModels <- list()
plotList <- list()
plotListQQ1 <- list() 

par(mfrow=c(3,3))
for (i in 1:9)
{
  nobs <- Cuckoo %>% nrow
  dataHlp$ySim <- mod1$fit + rnorm(nobs, sd = sigma)
  simModels[[i]] <- lm(ySim~type, dataHlp)
  plot(simModels[[i]], which = 2)
  plotList[[i]] <- dataHlp %>%  
    ggplot(aes(type,ySim)) + 
    geom_boxplot(outlier.shape = NA) +
    geom_jitter()
  plotListQQ1[[i]] <- dataHlp %>%  
    filter(type == "meadow")  %>% 
    ggplot(aes(sample = ySim)) + 
    geom_qq() +
    geom_qq_line() + 
    xlab("QQ for meadow")
}

do.call(gridExtra::grid.arrange, plotList)

do.call(gridExtra::grid.arrange, plotListQQ1)

The most deviating QQ-plot of the residuals in the nine simulation runs:

par(mfrow=c(1,1))
plot(simModels[[8]], which = 2)

5.4 ANOVA model

mod1_anova <- anova(mod1)
mod1_anova

The p-value of the ANOVA analysis is extremely significant (p-value < 0.001), so we reject the null hypothesis that the mean egg length is equal between the different bird types. We can say that the mean egg length is significantly different between at least two bird types on the 5% significance level.

Based on this analysis, we do not yet know between which particular bird types there is a significant difference. To study this, we will perform the Tuckey post-hoc analysis.

5.5 Post-hoc analysis

We will perform a post-hoc analysis, to look at the difference in egg length between each pairwise comparison of bird types. Importantly, with this strategy, the p-values will be correctly adjusted for multiple testing.

The null hypothesis for each pairwise test is that there is no difference in the mean egg length between both bird types.

The alternative hypothesis for each pairwise test states that there is indeed a difference in the mean egg length between both bird types.

We will also calculate the confidence interval on the mean differences.

library(multcomp, quietly = TRUE)
mcp <- glht(mod1, linfct = mcp(type = "Tukey"))
summary(mcp)
## 
##   Simultaneous Tests for General Linear Hypotheses
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: lm(formula = length ~ type, data = Cuckoo)
## 
## Linear Hypotheses:
##                        Estimate Std. Error t value Pr(>|t|)   
## tree - meadow == 0      0.59000    0.39263   1.503  0.65686   
## dunnock - meadow == 0   0.62143    0.40301   1.542  0.63147   
## robin - meadow == 0     0.07500    0.38332   0.196  0.99996   
## wagtail - meadow == 0   0.40333    0.39263   1.027  0.90518   
## wren - meadow == 0     -1.37000    0.39263  -3.489  0.00847 **
## dunnock - tree == 0     0.03143    0.48939   0.064  1.00000   
## robin - tree == 0      -0.51500    0.47330  -1.088  0.88202   
## wagtail - tree == 0    -0.18667    0.48087  -0.388  0.99878   
## wren - tree == 0       -1.96000    0.48087  -4.076  0.00114 **
## robin - dunnock == 0   -0.54643    0.48195  -1.134  0.86268   
## wagtail - dunnock == 0 -0.21810    0.48939  -0.446  0.99764   
## wren - dunnock == 0    -1.99143    0.48939  -4.069  0.00115 **
## wagtail - robin == 0    0.32833    0.47330   0.694  0.98170   
## wren - robin == 0      -1.44500    0.47330  -3.053  0.03198 * 
## wren - wagtail == 0    -1.77333    0.48087  -3.688  0.00433 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## (Adjusted p values reported -- single-step method)
confint(mcp)
## 
##   Simultaneous Confidence Intervals
## 
## Multiple Comparisons of Means: Tukey Contrasts
## 
## 
## Fit: lm(formula = length ~ type, data = Cuckoo)
## 
## Quantile = 2.8889
## 95% family-wise confidence level
##  
## 
## Linear Hypotheses:
##                        Estimate lwr      upr     
## tree - meadow == 0      0.59000 -0.54429  1.72429
## dunnock - meadow == 0   0.62143 -0.54285  1.78571
## robin - meadow == 0     0.07500 -1.03239  1.18239
## wagtail - meadow == 0   0.40333 -0.73096  1.53763
## wren - meadow == 0     -1.37000 -2.50429 -0.23571
## dunnock - tree == 0     0.03143 -1.38238  1.44524
## robin - tree == 0      -0.51500 -1.88234  0.85234
## wagtail - tree == 0    -0.18667 -1.57589  1.20255
## wren - tree == 0       -1.96000 -3.34922 -0.57078
## robin - dunnock == 0   -0.54643 -1.93875  0.84589
## wagtail - dunnock == 0 -0.21810 -1.63190  1.19571
## wren - dunnock == 0    -1.99143 -3.40524 -0.57762
## wagtail - robin == 0    0.32833 -1.03901  1.69567
## wren - robin == 0      -1.44500 -2.81234 -0.07766
## wren - wagtail == 0    -1.77333 -3.16255 -0.38411

5.6 Conclusion

There association between the mean length of a cuckoo’s egg and the foster bird species is an extremely significant (p << 0.001).

The mean length of cuckoo’s eggs in nests of the Eurasian wern are smaller as compared to those from all other bird species in the study:

  • the meadow pipit (adjusted p-value = 0.009, mean difference = -1.4mm, 95% CI [-2.5, -0.2])
  • the tree pipit (adjusted p-value = 0.001, mean difference = -2mm, 95% CI [-3.3, -0.6])
  • the dunnock (adjusted p-value = 0.001, mean difference = -2mm, 95% CI [-3.4, -0.6])
  • the European robin (adjusted p-value = 0.03, mean difference = -1.4mm, 95% CI [-2.8, -0.1])
  • the white wagtail (adjusted p-value = 0.004, mean difference = -1.8mm, 95% CI [-3.2, -0.4])

The mean length of the cuckoo bird’s eggs for all other pairwise comparisons between foster bird species are not statistically significantly different at the 5% level.

LS0tCnRpdGxlOiAiRXhlcmNpc2UgNy4yOiBBTk9WQSBvbiB0aGUgY3Vja29vIGRhdGFzZXQgLSBzb2x1dGlvbiIgICAKYXV0aG9yOiAiTGlldmVuIENsZW1lbnQgYW5kIEplcm9lbiBHaWxpcyIKZGF0ZTogInN0YXRPbWljcywgR2hlbnQgVW5pdmVyc2l0eSAoaHR0cHM6Ly9zdGF0b21pY3MuZ2l0aHViLmlvKSIgIApvdXRwdXQ6CiAgICBodG1sX2RvY3VtZW50OgogICAgICBjb2RlX2Rvd25sb2FkOiB0cnVlICAgIAogICAgICB0aGVtZTogY29zbW8KICAgICAgdG9jOiB0cnVlCiAgICAgIHRvY19mbG9hdDogdHJ1ZQogICAgICBoaWdobGlnaHQ6IHRhbmdvCiAgICAgIG51bWJlcl9zZWN0aW9uczogdHJ1ZQotLS0KCiMgQ3Vja29vIGRhdGFzZXQKClRoZSBjb21tb24gY3Vja29vIGRvZXMgbm90IGJ1aWxkIGl0cyBvd24gbmVzdDogaXQgcHJlZmVycwp0byBsYXkgaXRzIGVnZ3MgaW4gYW5vdGhlciBiaXJkcycgbmVzdC4gSXQgaXMga25vd24sIHNpbmNlIDE4OTIsCnRoYXQgdGhlIHR5cGUgb2YgY3Vja29vIGJpcmQgZWdncyBhcmUgZGlmZmVyZW50IGJldHdlZW4gZGlmZmVyZW50CmxvY2F0aW9ucy4gSW4gYSBzdHVkeSBmcm9tIDE5NDAsIGl0IHdhcyBzaG93biB0aGF0IGN1Y2tvb3MgcmV0dXJuCnRvIHRoZSBzYW1lIG5lc3RpbmcgYXJlYSBlYWNoIHllYXIsIGFuZCB0aGF0IHRoZXkgYWx3YXlzIHBpY2sKdGhlIHNhbWUgYmlyZCBzcGVjaWVzIHRvIGJlIGEgImZvc3RlciBwYXJlbnQiIGZvciB0aGVpciBlZ2dzLgoKT3ZlciB0aGUgeWVhcnMsIHRoaXMgaGFzIGxlYWQgdG8gdGhlIGRldmVsb3BtZW50IG9mIGdlb2dyYXBoaWNhbGx5CmRldGVybWluZWQgc3Vic3BlY2llcyBvZiBjdWNrb29zLiBUaGVzZSBzdWJzcGVjaWVzIGhhdmUgZXZvbHZlZCBpbgpzdWNoIGEgd2F5IHRoYXQgdGhlaXIgZWdncyBsb29rIGFzIHNpbWlsYXIgYXMgcG9zc2libGUgYXMgdGhvc2UKb2YgdGhlaXIgZm9zdGVyIHBhcmVudHMuCgpUaGUgY3Vja29vIGRhdGFzZXQgY29udGFpbnMgaW5mb3JtYXRpb24gb24gMTIwIEN1Y2tvbyBlZ2dzLApvYnRhaW5lZCBmcm9tIHJhbmRvbWx5IHNlbGVjdGVkICJmb3N0ZXIiIG5lc3RzLgpGb3IgdGhlc2UgZWdncywgcmVzZWFyY2hlcnMgaGF2ZSBtZWFzdXJlZCB0aGUgYGxlbmd0aGAgKGluIG1tKQphbmQgZXN0YWJsaXNoZWQgdGhlIGB0eXBlYCAoc3BlY2llcykgb2YgZm9zdGVyIHBhcmVudC4KVGhlIHR5cGUgY29sdW1uIGlzIGNvZGVkIGFzIGZvbGxvd3M6CgotIGB0eXBlPTFgOiBNZWFkb3cgcGlwaXQKLSBgdHlwZT0yYDogVHJlZSBwaXBpdAotIGB0eXBlPTNgOiBEdW5ub2NrCi0gYHR5cGU9NGA6IEV1cm9wZWFuIHJvYmluCi0gYHR5cGU9NWA6IFdoaXRlIHdhZ3RhaWwKLSBgdHlwZT02YDogRXVyYXNpYW4gd3JlbgoKIyBHb2FsCgpUaGUgcmVzZWFyY2hlcnMgd2FudCB0b3Rlc3QgaWYgdGhlIHR5cGUgb2YgZm9zdGVyIHBhcmVudApoYXMgYW4gZWZmZWN0IG9uIHRoZSBhdmVyYWdlIGxlbmd0aCBvZiB0aGUgY3Vja29vIGVnZ3MuCgpJbiB0aGVvcnksIHRoZXkgd2FudCB0byBzdHVkeSB0aGlzIGZvciBhbGwgc2l4IHNwZWNpZXMuClByZXZpb3VzbHksIHdlIGxvb2tlZCBhdCBhIHNpbmdsZSBwYWlyd2lzZSBjb21wYXJpc29uCmJldHdlZW4gdGhlIEV1cm9wZWFuIHJvYmluIGFuZCB0aGUgRXVyYXNpYW4gd3JlbiB3aXRoIGEKdC10ZXN0LiBIZXJlLCB3ZSB3aWxsIGFuYWx5c2UgYWxsIHR5cGVzIHNpbXVsdGFuZW91c2x5CndpdGggQU5PVkEuCgpMb2FkIHRoZSByZXF1aXJlZCBsaWJyYXJpZXMKCmBgYHtyLCBtZXNzYWdlPUZBTFNFfQpsaWJyYXJ5KHRpZHl2ZXJzZSkKYGBgCgojIyBJbXBvcnQgdGhlIGRhdGEKCmBgYHtyLCBtZXNzYWdlPUZBTFNFfQpDdWNrb28gPC0gcmVhZF90c3YoImh0dHBzOi8vcmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbS9zdGF0T21pY3MvUFNMUzIxL2RhdGEvQ3Vja29vLnR4dCIpCmhlYWQoQ3Vja29vKQpgYGAKCiMgRGF0YSB0aWR5aW5nCgpJdCBzZWVtcyB0aGF0IHRoZSBgdHB5ZWAgY29sdW1uIGlzIGEgYGRvdWJsZWAgcmF0aGVyIHRoYW4gYSBgZmFjdG9yYC4gCkxldCdzIGZpeCB0aGlzOgoKYGBge3J9CkN1Y2tvbyA8LSBDdWNrb28gJT4lCiAgbXV0YXRlKHR5cGUgPSBhcy5mYWN0b3IodHlwZSkpICU+JQogIG11dGF0ZSh0eXBlID0gcmVjb2RlKHR5cGUsIAogICAgICAgICAgICIxIiA9ICJtZWFkb3ciLCAKICAgICAgICAgICAiMiIgPSAidHJlZSIsIAogICAgICAgICAgICIzIiA9ICJkdW5ub2NrIiwgIAogICAgICAgICAgICI0IiA9ICJyb2JpbiIsIAogICAgICAgICAgICI1IiA9ICJ3YWd0YWlsIiwgCiAgICAgICAgICAgIjYiID0gIndyZW4iKSkKYGBgCgoKIyBEYXRhIGV4cGxvcmF0aW9uCgpIb3cgbWFueSBiaXJkcyBkbyB3ZSBoYXZlIGZvciBlYWNoIHR5cGU/CgpgYGB7cn0KQ3Vja29vICU+JQogIGNvdW50KHR5cGUpCmBgYAoKVmlzdWFsaXplIHRoZSBkYXRhCgpgYGB7cn0KQ3Vja29vICU+JQogIGdncGxvdChhZXMoeD10eXBlLHk9bGVuZ3RoLGZpbGw9dHlwZSkpICsKICAgIHRoZW1lX2J3KCkgKwogICAgc2NhbGVfZmlsbF9icmV3ZXIocGFsZXR0ZT0iUmRHeSIpICsKICAgIGdlb21fYm94cGxvdCgpICsKICAgIGdlb21faml0dGVyKHdpZHRoID0gMC4yKSArCiAgICBnZ3RpdGxlKCJCb3hwbG90IG9mIHRoZSBsZW5ndGggb2YgZWdncyBwZXIgdHlwZSIpICsKICAgIHlsYWIoImxlbmd0aCAobW0pIikgKyAKICAgIHN0YXRfc3VtbWFyeShmdW49bWVhbiwgZ2VvbT0icG9pbnQiLCBzaGFwZT01LCBzaXplPTMsIGNvbG9yPSJibGFjayIsIGZpbGw9ImJsYWNrIikKYGBgCgojIEFOT1ZBCgpUbyBzdHVkeSBpZiB0aGUgb2JzZXJ2ZWQgZGlmZmVyZW5jZXMgaW4gYXZlcmFnZSBlZ2cgbGVuZ3RoIApiZXR3ZWVuIHRoZSBkaWZmZXJlbnQgZm9zdGVyIGJpcmQgdHlwZXMgYXJlIHNpZ25pZmljYW50LCB3ZSBtYXkgcGVyZm9ybSAKYW4gQU5PVkEuCgojIyBGb3JtdWxhdGUgbnVsbCBhbmQgYWx0ZXJuYXRpdmUgaHlwb3RoZXNpcwoKVGhlIG51bGwgaHlwb3RoZXNpcyBvZiBBTk9WQSBzdGF0ZXMgdGhhdDoKJEgwJDogVGhlIG1lYW4gZWdnIGxlbmd0aCBpcyBlcXVhbCBiZXR3ZWVuIHRoZSBkaWZmZXJlbnQgYmlyZCB0eXBlcy4KClRoZSBhbHRlcm5hdGl2ZSBoeXBvdGhlc2lzIG9mIEFOT1ZBIHN0YXRlcyB0aGF0OgokSEEkOiBUaGUgbWVhbiBlZ2cgbGVuZ3RoIGZvciBhdCBsZWFzdCBvbmUgYmlyZCB0eXBlIGlzIGRpZmZlcmVudCAKZnJvbSB0aGUgbWVhbiBlZ2cgbGVuZ3RoIGluIGF0IGxlYXN0IG9uZSBvdGhlciBiaXJkIHR5cGUuCgojIyBQYXJhbWV0ZXIgZXN0aW1hdGlvbiAKCmBgYHtyfQptb2QxIDwtIGxtKGxlbmd0aH50eXBlLCBDdWNrb28pCmBgYAoKIyMgQ2hlY2sgYXNzdW1wdGlvbnMKCkJlZm9yZSB3ZSBtYXkgcHJvY2VlZCB3aXRoIHRoZSBhbmFseXNpcywgd2UgbXVzdCBtYWtlIHN1cmUgdGhhdCBhbGwKYXNzdW1wdGlvbnMgZm9yIEFOT1ZBIGFyZSBtZXQuIEFOT1ZBIGhhcyB0aHJlZSBhc3N1bXB0aW9uczoKCjEuIFRoZSBvYnNlcnZhdGlvbnMgYXJlIGluZGVwZW5kZW50IG9mIGVhY2ggb3RoZXIgKGluIGFsbCBncm91cHMpCjIuIFRoZSBkYXRhIChsZW5ndGgpIG11c3QgYmUgbm9ybWFsbHkgZGlzdHJpYnV0ZWQgKGluIGFsbCBncm91cHMpCjMuIFRoZSB2YXJpYWJpbGl0eSB3aXRoaW4gYWxsIGdyb3VwcyBpcyBzaW1pbGFyCgpUaGUgZmlyc3QgYXNzdW1wdGlvbiBpcyBtZXQsIGFzIHdlIG1heSBhc3N1bWUgdGhhdCB0aGVyZSBhcmUgbm8Kc3BlY2lmaWMgcGF0dGVybnMgb2YgY29ycmVsYXRpb24gYmV0d2VlbiB0aGUgcmFuZG9tbHkgc2VsZWN0ZWQgbmVzdHMuCgpUbyBjaGVjayB0aGUgbm9ybWFsaXR5IGFzc3VtcHRpb24sIHdlIHdpbGwgdXNlIFFRIHBsb3RzLgoKYGBge3J9CnBsb3QobW9kMSwgd2hpY2ggPSAyKQpgYGAKCmBgYHtyfQpDdWNrb28gJT4lIAogIGdncGxvdChhZXMoc2FtcGxlPWxlbmd0aCkpICsKICBnZW9tX3FxKCkgKwogIGdlb21fcXFfbGluZSgpICsgCiAgZmFjZXRfZ3JpZCh+dHlwZSkKYGBgCgpUaGVyZSBzZWVtIHRvIGJlIG5vIGNsZWFyIGRldmlhdGlvbnMgZnJvbSBub3JtYWxpdHkuCgpUaGUgdGhpcmQgYXNzdW1wdGlvbiBvZiBlcXVhbCB2YXJpYW5jZXMgc2VlbXMgdG8gYmUgbWV0IGJhc2VkIG9uIHRoZQp2aXN1YWxpemF0aW9uIHdpdGggdGhlIGJveHBsb3RzIChzZWUgYWJvdmUpLgoKQXMgc3VjaCwgd2UgbWF5IHByb2NlZWQgd2l0aCB0aGUgQU5PVkEgYW5hbHlzaXMuCgoKIyMjIFNpbXVsYXRlIHRvIHRyYWluIHlvdXIgc2tpbGxzIGluIGFzc2Vzc2luZyBhc3N1bXB0aW9ucyAKCmBgYHtyfQpzZXQuc2VlZCgxMDMxKQpzaWdtYSA8LSBtb2QxICU+JSBzaWdtYSAKZGF0YUhscCA8LSBDdWNrb28KCnNpbU1vZGVscyA8LSBsaXN0KCkKcGxvdExpc3QgPC0gbGlzdCgpCnBsb3RMaXN0UVExIDwtIGxpc3QoKSAKCnBhcihtZnJvdz1jKDMsMykpCmZvciAoaSBpbiAxOjkpCnsKICBub2JzIDwtIEN1Y2tvbyAlPiUgbnJvdwogIGRhdGFIbHAkeVNpbSA8LSBtb2QxJGZpdCArIHJub3JtKG5vYnMsIHNkID0gc2lnbWEpCiAgc2ltTW9kZWxzW1tpXV0gPC0gbG0oeVNpbX50eXBlLCBkYXRhSGxwKQogIHBsb3Qoc2ltTW9kZWxzW1tpXV0sIHdoaWNoID0gMikKICBwbG90TGlzdFtbaV1dIDwtIGRhdGFIbHAgJT4lICAKICAgIGdncGxvdChhZXModHlwZSx5U2ltKSkgKyAKICAgIGdlb21fYm94cGxvdChvdXRsaWVyLnNoYXBlID0gTkEpICsKICAgIGdlb21faml0dGVyKCkKICBwbG90TGlzdFFRMVtbaV1dIDwtIGRhdGFIbHAgJT4lICAKICAgIGZpbHRlcih0eXBlID09ICJtZWFkb3ciKSAgJT4lIAogICAgZ2dwbG90KGFlcyhzYW1wbGUgPSB5U2ltKSkgKyAKICAgIGdlb21fcXEoKSArCiAgICBnZW9tX3FxX2xpbmUoKSArIAogICAgeGxhYigiUVEgZm9yIG1lYWRvdyIpCn0KCmRvLmNhbGwoZ3JpZEV4dHJhOjpncmlkLmFycmFuZ2UsIHBsb3RMaXN0KQpkby5jYWxsKGdyaWRFeHRyYTo6Z3JpZC5hcnJhbmdlLCBwbG90TGlzdFFRMSkKYGBgCgpUaGUgbW9zdCBkZXZpYXRpbmcgUVEtcGxvdCBvZiB0aGUgcmVzaWR1YWxzIGluIHRoZSBuaW5lIHNpbXVsYXRpb24gcnVuczogCmBgYHtyfQpwYXIobWZyb3c9YygxLDEpKQpwbG90KHNpbU1vZGVsc1tbOF1dLCB3aGljaCA9IDIpCmBgYAoKCiMjIEFOT1ZBIG1vZGVsCgpgYGB7cn0KbW9kMV9hbm92YSA8LSBhbm92YShtb2QxKQptb2QxX2Fub3ZhCmBgYAoKVGhlIHAtdmFsdWUgb2YgdGhlIEFOT1ZBIGFuYWx5c2lzIGlzIGV4dHJlbWVseSBzaWduaWZpY2FudAoocC12YWx1ZSA8IDAuMDAxKSwgCnNvIHdlIHJlamVjdCB0aGUgbnVsbCBoeXBvdGhlc2lzIHRoYXQgdGhlIG1lYW4gCmVnZyBsZW5ndGggaXMgZXF1YWwgYmV0d2VlbiB0aGUgZGlmZmVyZW50IGJpcmQgdHlwZXMuCldlIGNhbiBzYXkgdGhhdCB0aGUgbWVhbiBlZ2cgbGVuZ3RoIGlzIHNpZ25pZmljYW50bHkgZGlmZmVyZW50CmJldHdlZW4gYXQgbGVhc3QgdHdvIGJpcmQgdHlwZXMgb24gdGhlIDUlIHNpZ25pZmljYW5jZSBsZXZlbC4KCkJhc2VkIG9uIHRoaXMgYW5hbHlzaXMsIHdlIGRvIG5vdCB5ZXQga25vdyBiZXR3ZWVuIHdoaWNoIHBhcnRpY3VsYXIKYmlyZCB0eXBlcyB0aGVyZSBpcyBhIHNpZ25pZmljYW50IGRpZmZlcmVuY2UuIFRvIHN0dWR5IHRoaXMsIHdlIHdpbGwKcGVyZm9ybSB0aGUgVHVja2V5IHBvc3QtaG9jIGFuYWx5c2lzLgoKIyMgUG9zdC1ob2MgYW5hbHlzaXMKCldlIHdpbGwgcGVyZm9ybSBhIHBvc3QtaG9jIGFuYWx5c2lzLCB0byBsb29rIGF0IHRoZSBkaWZmZXJlbmNlIAppbiBlZ2cgbGVuZ3RoIGJldHdlZW4gZWFjaCBwYWlyd2lzZSBjb21wYXJpc29uIG9mIGJpcmQgdHlwZXMuCkltcG9ydGFudGx5LCB3aXRoIHRoaXMgc3RyYXRlZ3ksIHRoZSBwLXZhbHVlcyB3aWxsIGJlIGNvcnJlY3RseQphZGp1c3RlZCBmb3IgbXVsdGlwbGUgdGVzdGluZy4KClRoZSBudWxsIGh5cG90aGVzaXMgZm9yIGVhY2ggcGFpcndpc2UgdGVzdCBpcyB0aGF0IHRoZXJlIGlzIG5vCmRpZmZlcmVuY2UgaW4gdGhlIG1lYW4gZWdnIGxlbmd0aCBiZXR3ZWVuIGJvdGggYmlyZCB0eXBlcy4KClRoZSBhbHRlcm5hdGl2ZSBoeXBvdGhlc2lzIGZvciBlYWNoIHBhaXJ3aXNlIHRlc3Qgc3RhdGVzIHRoYXQgdGhlcmUKaXMgaW5kZWVkIGEgZGlmZmVyZW5jZSBpbiB0aGUgbWVhbiBlZ2cgbGVuZ3RoIGJldHdlZW4gYm90aCBiaXJkIHR5cGVzLgoKV2Ugd2lsbCBhbHNvIGNhbGN1bGF0ZSB0aGUgY29uZmlkZW5jZSBpbnRlcnZhbCBvbiB0aGUgbWVhbiBkaWZmZXJlbmNlcy4KCmBgYHtyLG1lc3NhZ2U9RkFMU0V9CmxpYnJhcnkobXVsdGNvbXAsIHF1aWV0bHkgPSBUUlVFKQptY3AgPC0gZ2xodChtb2QxLCBsaW5mY3QgPSBtY3AodHlwZSA9ICJUdWtleSIpKQpzdW1tYXJ5KG1jcCkKY29uZmludChtY3ApCmBgYAoKIyMgQ29uY2x1c2lvbgoKVGhlcmUgYXNzb2NpYXRpb24gYmV0d2VlbiB0aGUgbWVhbiBsZW5ndGggb2YgYSBjdWNrb28ncyBlZ2cgYW5kIHRoZSBmb3N0ZXIgYmlyZCBzcGVjaWVzIGlzIGFuIGV4dHJlbWVseSBzaWduaWZpY2FudCAocCA8PCAwLjAwMSkuCgpUaGUgbWVhbiBsZW5ndGggb2YgY3Vja29vJ3MgZWdncyBpbiBuZXN0cyBvZiB0aGUgRXVyYXNpYW4Kd2VybiBhcmUgc21hbGxlciBhcyBjb21wYXJlZCB0byB0aG9zZSBmcm9tIGFsbCBvdGhlciBiaXJkCnNwZWNpZXMgaW4gdGhlIHN0dWR5OgoKLSB0aGUgbWVhZG93IHBpcGl0IChhZGp1c3RlZCBwLXZhbHVlID0gYHIgc3VtbWFyeShtY3ApJHRlc3QkcHZhbHVlc1tuYW1lcyhzdW1tYXJ5KG1jcCkkdGVzdCR0c3RhdCkgPT0gIndyZW4gLSBtZWFkb3ciXSAlPiUgZm9ybWF0KGRpZ2l0cz0xKWAsIG1lYW4gZGlmZmVyZW5jZSA9IGByIHJvdW5kKGNvbmZpbnQobWNwKSRjb25maW50WyJ3cmVuIC0gbWVhZG93IiwxXSwxKWBtbSwgOTUlIENJIFtgciByb3VuZChjb25maW50KG1jcCkkY29uZmludFsid3JlbiAtIG1lYWRvdyIsLTFdLDEpYF0pCi0gdGhlIHRyZWUgcGlwaXQgKGFkanVzdGVkIHAtdmFsdWUgPSBgciBzdW1tYXJ5KG1jcCkkdGVzdCRwdmFsdWVzW25hbWVzKHN1bW1hcnkobWNwKSR0ZXN0JHRzdGF0KSA9PSAid3JlbiAtIHRyZWUiXSAlPiUgZm9ybWF0KGRpZ2l0cz0xKWAsIG1lYW4gZGlmZmVyZW5jZSA9IGByIHJvdW5kKGNvbmZpbnQobWNwKSRjb25maW50WyJ3cmVuIC0gdHJlZSIsMV0sMSlgbW0sIDk1JSBDSSBbYHIgcm91bmQoY29uZmludChtY3ApJGNvbmZpbnRbIndyZW4gLSB0cmVlIiwtMV0sMSlgXSkKLSB0aGUgZHVubm9jayAoYWRqdXN0ZWQgcC12YWx1ZSA9IGByIHN1bW1hcnkobWNwKSR0ZXN0JHB2YWx1ZXNbbmFtZXMoc3VtbWFyeShtY3ApJHRlc3QkdHN0YXQpID09ICJ3cmVuIC0gZHVubm9jayJdICU+JSBmb3JtYXQoZGlnaXRzPTEpYCwgbWVhbiBkaWZmZXJlbmNlID0gYHIgcm91bmQoY29uZmludChtY3ApJGNvbmZpbnRbIndyZW4gLSBkdW5ub2NrIiwxXSwxKWBtbSwgOTUlIENJIFtgciByb3VuZChjb25maW50KG1jcCkkY29uZmludFsid3JlbiAtIGR1bm5vY2siLC0xXSwxKWBdKQotIHRoZSBFdXJvcGVhbiByb2JpbiAoYWRqdXN0ZWQgcC12YWx1ZSA9IGByIHN1bW1hcnkobWNwKSR0ZXN0JHB2YWx1ZXNbbmFtZXMoc3VtbWFyeShtY3ApJHRlc3QkdHN0YXQpID09ICJ3cmVuIC0gcm9iaW4iXSAlPiUgZm9ybWF0KGRpZ2l0cz0xKWAsIG1lYW4gZGlmZmVyZW5jZSA9IGByIHJvdW5kKGNvbmZpbnQobWNwKSRjb25maW50WyJ3cmVuIC0gcm9iaW4iLDFdLDEpYG1tLCA5NSUgQ0kgW2ByIHJvdW5kKGNvbmZpbnQobWNwKSRjb25maW50WyJ3cmVuIC0gcm9iaW4iLC0xXSwxKWBdKQotIHRoZSB3aGl0ZSB3YWd0YWlsIChhZGp1c3RlZCBwLXZhbHVlID0gYHIgc3VtbWFyeShtY3ApJHRlc3QkcHZhbHVlc1tuYW1lcyhzdW1tYXJ5KG1jcCkkdGVzdCR0c3RhdCkgPT0gIndyZW4gLSB3YWd0YWlsIl0gJT4lIGZvcm1hdChkaWdpdHM9MSlgLCBtZWFuIGRpZmZlcmVuY2UgPSBgciByb3VuZChjb25maW50KG1jcCkkY29uZmludFsid3JlbiAtIHdhZ3RhaWwiLDFdLDEpYG1tLCA5NSUgQ0kgW2ByIHJvdW5kKGNvbmZpbnQobWNwKSRjb25maW50WyJ3cmVuIC0gd2FndGFpbCIsLTFdLDEpYF0pCgpUaGUgbWVhbiBsZW5ndGggb2YgdGhlIGN1Y2tvbyBiaXJkJ3MgZWdncyBmb3IgYWxsIG90aGVyIHBhaXJ3aXNlIGNvbXBhcmlzb25zIGJldHdlZW4gZm9zdGVyIGJpcmQgc3BlY2llcyBhcmUgbm90IHN0YXRpc3RpY2FsbHkgc2lnbmlmaWNhbnRseSBkaWZmZXJlbnQgYXQgdGhlIDUlIGxldmVsLgoKCg==