library(readr)
library(dplyr)
Introduction
Kidney transplantation or renal transplantation is the organ transplant of a kidney into a patient who has an end-stage renal disease. Scientists claim that some genes are responsible for a patient’s likelihood of rejecting a kidney after transplantation.
In this project, you are to investigate this claim. You will analyze data that contains gene expression levels of 54675 genes from 282 patients, taken from the study by Einecke et al. (2010).
The data consists of a subset of the original data where only the 25% most variable genes (i.e. 13669 genes) were retained and the number of patients was subsampled to 250.
The data can be found on the GitHub page and can be read in directly using read_csv
. However, the data is still quite large so it’s recommended to download it locally to your computer, for example in a raw-data/
folder and then loaded into R locally.
To do that, just run the code shown below in an R console. The output directory will be relative to your current working directory (which you can check with getwd()
). You can also specify an absolute file path.
# Create raw-data/ folder if it does not exist (you can change this to whatever path you want)
out_path <- "raw-data"
if (!(dir.exists(out_path))) dir.create(out_path)
# Download data (only if doesn't exist locally)
# Delete existing file with `unlink(file.path(out_path, fname))`
# to force re-downloading
fname <- "GSE21374-kidney-data.csv.gz"
if (!file.exists(file.path(out_path, fname))) {
data_url <-
"https://github.com/statOmics/HDA2020/raw/data/GSE21374-kidney-data.csv.gz"
download.file(data_url, destfile = file.path(out_path, fname))
}
The data can then be loaded into R as follows. The first two columns consist of the Patient_ID
and Reject_Status
, which is a binary variable encoding whether the kidney transplant was accepted (0) or rejected (1) for each patient. The other columns contain the microarray expression data for the 13669 genes.
## Assumes data is stored in "raw-data/" folder under current working directory
data_dir <- "raw-data"
kidney_data <- read_csv(
file.path(data_dir, "GSE21374-kidney-data.csv.gz"),
col_types = cols(
.default = col_double(),
Patient_ID = col_character()
)
)
## Glimpse first 10 columns
str(kidney_data[, 1:10])
#> tibble[,10] [250 × 10] (S3: tbl_df/tbl/data.frame)
#> $ Patient_ID : chr [1:250] "GSM533921" "GSM533922" "GSM533923" "GSM533924" ...
#> $ Reject_Status: num [1:250] 0 0 0 0 0 1 1 0 0 0 ...
#> $ X121_at : num [1:250] 0.62 1.94 0.119 -0.798 0.228 ...
#> $ X1255_g_at : num [1:250] 0.815 1.39 -1.291 -0.368 -0.499 ...
#> $ X1294_at : num [1:250] -1.5589 -0.809 1.2385 -0.0565 -1.9907 ...
#> $ X1405_i_at : num [1:250] -0.376 -0.531 0.959 -0.638 -1.429 ...
#> $ X1431_at : num [1:250] -0.5659 -0.6719 -0.0688 -0.7883 -0.5423 ...
#> $ X1552261_at : num [1:250] 1.596 -0.281 0.735 -0.32 -0.972 ...
#> $ X1552269_at : num [1:250] 1.319 1.592 -0.854 -1.321 0.998 ...
#> $ X1552274_at : num [1:250] 1.731 0.139 -0.353 0.349 1.561 ...
## Extract gene expression data as matrix X
X <- kidney_data %>%
dplyr::select(-Patient_ID, -Reject_Status) %>%
as.matrix()
rownames(X) <- kidney_data$Patient_ID
dim(X)
#> [1] 250 13669
str(X)
#> num [1:250, 1:13669] 0.62 1.94 0.119 -0.798 0.228 ...
#> - attr(*, "dimnames")=List of 2
#> ..$ : chr [1:250] "GSM533921" "GSM533922" "GSM533923" "GSM533924" ...
#> ..$ : chr [1:13669] "X121_at" "X1255_g_at" "X1294_at" "X1405_i_at" ...
## Extract Reject_Status column as vector
reject_status <- kidney_data$Reject_Status
names(reject_status) <- kidney_data$Patient_ID
length(reject_status)
#> [1] 250
table(reject_status) # number of 0's (accepts) and 1's (rejects)
#> reject_status
#> 0 1
#> 174 76
We are interested in the following research questions:
How do the genes vary in terms of their gene expression levels? Is the variability associated with kidney rejection? (only to be answered in a data explorative / graphical manner)
Which genes are differentially expressed between the two kidney rejection groups? You must control the FDR at 10%.
Can the kidney rejection be predicted from the gene expressions? What genes are most important in predicting the kidney transplant rejection? How well does the prediction model perform in terms of predicting rejection status?
Note that the response variable is binary, so if you use regression models, you will need to use logistic regression, which models the response with a binomial distribution. This can generally be done in R
by specifying family = "binomial"
in regression functions such as glm
and glmnet
.
Assignment
You must work in groups of four students.
Write a scientific report that answers the research questions related to this study. The report must consist of two parts:
An executive summary of about half a page. This summary contains the answers to the original research questions, and should be written in a non-technical manner (it is meant for researchers without a statistical background).
A technical report that explains in detail how the results were obtained. It’s recommended to prepare this technical report an RMarkdown file. If you choose to use another format, then the R code should be submitted as a separate file (please comment your R code).
The report is expected to be concise, but must evidently be accurate and sufficiently detailed to enable the reader to verify the correctness of the result (i.e. your results must be reproducible). The total length of the report (excluding graphs, R code and possibly appendices) should not be more than three pages. The report should not contain an explanation of the theory behind the statistical methods, and should also not contain the study description given above (you can assume that the reader already knows this).
However, interpretation of the results is key!
Some more specific guidelines:
For the first research question, you may use one of the data exploration tools that we have seen in class. However, you are also free to search the literature for other techniques for data exploration and visualisation (your final mark will not depend on whether you searched the literature or not). You are only asked to explore whether the variability in gene expression levels is associated with rejection status; no need for hypothesis testing.
For the second research question, you are asked to perform hypothesis testing and correct for multiple testing so as to control the FDR at 10%. The full list with differentially expressed genes may be presented in an appendix. Only list the most important results in the body of the report.
For the third research question you are asked to predict rejection status using gene expression levels. You should randomly split the data into a test (30%) and a training (70%) dataset. Make sure you use a seed (set.seed()
function) in R for reproducibility. The following prediction models should be evaluated:
In choosing the number of PCs in PCR, and the \(\gamma\) in the Ridge and Lasso models, you need to use cross validation (CV) on the training dataset. You should use the area under the receiver characteristic curve (AUC) as a performance measure.
Once you have selected the optimal PCR, Ridge and Lasso models, you have to decide with what model you want to continue. For this model you have to determine a good threshold \(c\) for the prediction cut-off that gives a good compromise between sensitivity and specificity.
Use the test data for final performance evaluation in terms of sensitivity and specificity.
Submission
It is recommended (but not mandatory) to prepare your report in RMarkdown. You can render it to either HTML (output: html_document
) or to PDF (output: pdf_document
). In both cases the original .Rmd
file should be included when handing in the assignment. If you don’t use RMarkdown, you should include the .R
file(s) containing your implementation and analysis scripts.
When submitting, please use the following format:
where Name
is your family name. It’s also recommended to mention your full name in the report itself.
Submissions should be done through UFora.
The deadline for submission is 17/12/2020 at 23:59
References
Einecke, Gunilla, Jeff Reeve, Banu Sis, Michael Mengel, Luis Hidalgo, Konrad S Famulski, Arthur Matas, et al. 2010. “A Molecular Classifier for Predicting Future Graft Loss in Late Kidney Transplant Biopsies.” The Journal of Clinical Investigation 120 (6): 1862–72.
LS0tCnRpdGxlOiAiUHJvamVjdCBhc3NpZ25tZW50IgpzdWJ0aXRsZTogIkhpZ2ggRGltZW5zaW9uYWwgRGF0YSBBbmFseXNpcyAyMDIwIgphdXRob3I6ICJBZGFwdGVkIGJ5IE1pbGFuIE1hbGZhaXQiCmRhdGU6ICIxOSBOb3YgMjAyMCIKb3V0cHV0OgogIGh0bWxfZG9jdW1lbnQ6CiAgICBudW1iZXJfc2VjdGlvbnM6IGZhbHNlCiAgcGRmX2RvY3VtZW50OgogICAgbnVtYmVyX3NlY3Rpb25zOiBmYWxzZQpyZWZlcmVuY2VzOgotIGlkOiBlaW5lY2tlMjAxMG1vbGVjdWxhcgogIHR5cGU6IGFydGljbGUtam91cm5hbAogIGF1dGhvcjoKICAtIGZhbWlseTogRWluZWNrZQogICAgZ2l2ZW46IEd1bmlsbGEKICAtIGZhbWlseTogUmVldmUKICAgIGdpdmVuOiBKZWZmCiAgLSBmYW1pbHk6IFNpcwogICAgZ2l2ZW46IEJhbnUKICAtIGZhbWlseTogTWVuZ2VsCiAgICBnaXZlbjogTWljaGFlbAogIC0gZmFtaWx5OiBIaWRhbGdvCiAgICBnaXZlbjogTHVpcwogIC0gZmFtaWx5OiBGYW11bHNraQogICAgZ2l2ZW46IEtvbnJhZCBTCiAgLSBmYW1pbHk6IE1hdGFzCiAgICBnaXZlbjogQXJ0aHVyCiAgLSBmYW1pbHk6IEthc2lza2UKICAgIGdpdmVuOiBCZXJ0CiAgLSBmYW1pbHk6IEthcGxhbgogICAgZ2l2ZW46IEJydWNlCiAgLSBmYW1pbHk6IEhhbGxvcmFuCiAgICBnaXZlbjogUGhpbGlwIEYKICAtIGxpdGVyYWw6IG90aGVycwogIGlzc3VlZDoKICAtIHllYXI6IDIwMTAKICB0aXRsZTogQSBtb2xlY3VsYXIgY2xhc3NpZmllciBmb3IgcHJlZGljdGluZyBmdXR1cmUgZ3JhZnQgbG9zcyBpbiBsYXRlIGtpZG5leSB0cmFuc3BsYW50CiAgICBiaW9wc2llcwogIGNvbnRhaW5lci10aXRsZTogVGhlIEpvdXJuYWwgb2YgY2xpbmljYWwgaW52ZXN0aWdhdGlvbgogIHB1Ymxpc2hlcjogQW0gU29jIENsaW4gSW52ZXN0aWcKICBwYWdlOiAxODYyLTE4NzIKICB2b2x1bWU6ICcxMjAnCiAgaXNzdWU6ICc2JwotLS0KCmBgYHtyIHNldHVwLCBpbmNsdWRlPUZBTFNFLCBjYWNoZT1GQUxTRX0Ka25pdHI6Om9wdHNfY2h1bmskc2V0KAogIGNvbGxhcHNlID0gVFJVRSwKICBjb21tZW50ID0gIiM+IgopCmBgYAoKYGBge3IgbGlicmFyaWVzLCBtZXNzYWdlPUZBTFNFLCB3YXJuaW5nPUZBTFNFfQpsaWJyYXJ5KHJlYWRyKQpsaWJyYXJ5KGRwbHlyKQpgYGAKCioqKgoKIyBJbnRyb2R1Y3Rpb24KCktpZG5leSB0cmFuc3BsYW50YXRpb24gb3IgcmVuYWwgdHJhbnNwbGFudGF0aW9uIGlzIHRoZSBvcmdhbiB0cmFuc3BsYW50IG9mIGEga2lkbmV5IGludG8gYSBwYXRpZW50IHdobyBoYXMgYW4gZW5kLXN0YWdlIHJlbmFsIGRpc2Vhc2UuIFNjaWVudGlzdHMgY2xhaW0gIHRoYXQgc29tZSBnZW5lcyBhcmUgcmVzcG9uc2libGUgZm9yIGEgcGF0aWVudCdzIGxpa2VsaWhvb2Qgb2YgcmVqZWN0aW5nIGEga2lkbmV5IGFmdGVyIHRyYW5zcGxhbnRhdGlvbi4gCgpJbiB0aGlzIHByb2plY3QsIHlvdSBhcmUgdG8gaW52ZXN0aWdhdGUgdGhpcyBjbGFpbS4gWW91IHdpbGwgYW5hbHl6ZSBkYXRhIHRoYXQgY29udGFpbnMgZ2VuZSBleHByZXNzaW9uIGxldmVscyBvZiA1NDY3NSBnZW5lcyBmcm9tIDI4MiBwYXRpZW50cywgdGFrZW4gZnJvbSB0aGUgc3R1ZHkgYnkgQGVpbmVja2UyMDEwbW9sZWN1bGFyLgoKVGhlIGRhdGEgY29uc2lzdHMgb2YgYSBzdWJzZXQgb2YgdGhlIG9yaWdpbmFsIGRhdGEgd2hlcmUgb25seSB0aGUgMjUlIG1vc3QgdmFyaWFibGUgZ2VuZXMgKGkuZS4gMTM2NjkgZ2VuZXMpIHdlcmUgcmV0YWluZWQgYW5kIHRoZSBudW1iZXIgb2YgcGF0aWVudHMgd2FzIHN1YnNhbXBsZWQgdG8gMjUwLgoKVGhlIGRhdGEgY2FuIGJlIGZvdW5kIG9uIHRoZSBHaXRIdWIgcGFnZSBhbmQgY2FuIGJlIHJlYWQgaW4gZGlyZWN0bHkgdXNpbmcgYHJlYWRfY3N2YC4KSG93ZXZlciwgdGhlIGRhdGEgaXMgc3RpbGwgcXVpdGUgbGFyZ2Ugc28gaXQncyByZWNvbW1lbmRlZCB0byBkb3dubG9hZCBpdCBsb2NhbGx5IHRvIHlvdXIgY29tcHV0ZXIsCmZvciBleGFtcGxlIGluIGEgYHJhdy1kYXRhL2AgZm9sZGVyIGFuZCB0aGVuIGxvYWRlZCBpbnRvIFIgbG9jYWxseS4KClRvIGRvIHRoYXQsIGp1c3QgcnVuIHRoZSBjb2RlIHNob3duIGJlbG93IGluIGFuIFIgY29uc29sZS4KVGhlIG91dHB1dCBkaXJlY3Rvcnkgd2lsbCBiZSAqcmVsYXRpdmUgdG8geW91ciBjdXJyZW50IHdvcmtpbmcgZGlyZWN0b3J5KiAod2hpY2ggeW91IGNhbiBjaGVjayB3aXRoIGBnZXR3ZCgpYCkuIFlvdSBjYW4gYWxzbyBzcGVjaWZ5IGFuIGFic29sdXRlIGZpbGUgcGF0aC4KCgpgYGB7ciBkb3dubG9hZC1kYXRhfQojIENyZWF0ZSByYXctZGF0YS8gZm9sZGVyIGlmIGl0IGRvZXMgbm90IGV4aXN0ICh5b3UgY2FuIGNoYW5nZSB0aGlzIHRvIHdoYXRldmVyIHBhdGggeW91IHdhbnQpCm91dF9wYXRoIDwtICJyYXctZGF0YSIKaWYgKCEoZGlyLmV4aXN0cyhvdXRfcGF0aCkpKSBkaXIuY3JlYXRlKG91dF9wYXRoKQoKIyBEb3dubG9hZCBkYXRhIChvbmx5IGlmIGRvZXNuJ3QgZXhpc3QgbG9jYWxseSkKIyBEZWxldGUgZXhpc3RpbmcgZmlsZSB3aXRoIGB1bmxpbmsoZmlsZS5wYXRoKG91dF9wYXRoLCBmbmFtZSkpYAojIHRvIGZvcmNlIHJlLWRvd25sb2FkaW5nCmZuYW1lIDwtICJHU0UyMTM3NC1raWRuZXktZGF0YS5jc3YuZ3oiCmlmICghZmlsZS5leGlzdHMoZmlsZS5wYXRoKG91dF9wYXRoLCBmbmFtZSkpKSB7CiAgZGF0YV91cmwgPC0KICAgICJodHRwczovL2dpdGh1Yi5jb20vc3RhdE9taWNzL0hEQTIwMjAvcmF3L2RhdGEvR1NFMjEzNzQta2lkbmV5LWRhdGEuY3N2Lmd6IgogIGRvd25sb2FkLmZpbGUoZGF0YV91cmwsIGRlc3RmaWxlID0gZmlsZS5wYXRoKG91dF9wYXRoLCBmbmFtZSkpCn0KYGBgCgpUaGUgZGF0YSBjYW4gdGhlbiBiZSBsb2FkZWQgaW50byBSIGFzIGZvbGxvd3MuClRoZSBmaXJzdCB0d28gY29sdW1ucyBjb25zaXN0IG9mIHRoZSBgUGF0aWVudF9JRGAgYW5kIGBSZWplY3RfU3RhdHVzYCwgd2hpY2ggaXMKYSBfX2JpbmFyeSB2YXJpYWJsZV9fIGVuY29kaW5nIHdoZXRoZXIgdGhlIGtpZG5leSB0cmFuc3BsYW50IHdhcyBhY2NlcHRlZCAoMCkgb3IKcmVqZWN0ZWQgKDEpIGZvciBlYWNoIHBhdGllbnQuClRoZSBvdGhlciBjb2x1bW5zIGNvbnRhaW4gdGhlIG1pY3JvYXJyYXkgZXhwcmVzc2lvbiBkYXRhIGZvciB0aGUgMTM2NjkgZ2VuZXMuCgpgYGB7ciBsb2FkLWRhdGF9CiMjIEFzc3VtZXMgZGF0YSBpcyBzdG9yZWQgaW4gInJhdy1kYXRhLyIgZm9sZGVyIHVuZGVyIGN1cnJlbnQgd29ya2luZyBkaXJlY3RvcnkKZGF0YV9kaXIgPC0gInJhdy1kYXRhIgpraWRuZXlfZGF0YSA8LSByZWFkX2NzdigKICBmaWxlLnBhdGgoZGF0YV9kaXIsICJHU0UyMTM3NC1raWRuZXktZGF0YS5jc3YuZ3oiKSwKICBjb2xfdHlwZXMgPSBjb2xzKAogICAgLmRlZmF1bHQgPSBjb2xfZG91YmxlKCksCiAgICBQYXRpZW50X0lEID0gY29sX2NoYXJhY3RlcigpCiAgKQopCgojIyBHbGltcHNlIGZpcnN0IDEwIGNvbHVtbnMKc3RyKGtpZG5leV9kYXRhWywgMToxMF0pCgojIyBFeHRyYWN0IGdlbmUgZXhwcmVzc2lvbiBkYXRhIGFzIG1hdHJpeCBYClggPC0ga2lkbmV5X2RhdGEgJT4lIAogIGRwbHlyOjpzZWxlY3QoLVBhdGllbnRfSUQsIC1SZWplY3RfU3RhdHVzKSAlPiUgCiAgYXMubWF0cml4KCkKcm93bmFtZXMoWCkgPC0ga2lkbmV5X2RhdGEkUGF0aWVudF9JRApkaW0oWCkKc3RyKFgpCgojIyBFeHRyYWN0IFJlamVjdF9TdGF0dXMgY29sdW1uIGFzIHZlY3RvcgpyZWplY3Rfc3RhdHVzIDwtIGtpZG5leV9kYXRhJFJlamVjdF9TdGF0dXMKbmFtZXMocmVqZWN0X3N0YXR1cykgPC0ga2lkbmV5X2RhdGEkUGF0aWVudF9JRApsZW5ndGgocmVqZWN0X3N0YXR1cykKdGFibGUocmVqZWN0X3N0YXR1cykgIyBudW1iZXIgb2YgMCdzIChhY2NlcHRzKSBhbmQgMSdzIChyZWplY3RzKQpgYGAKCldlIGFyZSBpbnRlcmVzdGVkIGluIHRoZSBmb2xsb3dpbmcgcmVzZWFyY2ggcXVlc3Rpb25zOgoKLSBIb3cgZG8gdGhlIGdlbmVzIHZhcnkgaW4gdGVybXMgb2YgdGhlaXIgZ2VuZSBleHByZXNzaW9uIGxldmVscz8gSXMgdGhlIHZhcmlhYmlsaXR5IGFzc29jaWF0ZWQgd2l0aCBraWRuZXkgcmVqZWN0aW9uPyAob25seSB0byBiZSBhbnN3ZXJlZCBpbiBhIGRhdGEgZXhwbG9yYXRpdmUgLyBncmFwaGljYWwgbWFubmVyKQotIFdoaWNoIGdlbmVzIGFyZSBkaWZmZXJlbnRpYWxseSBleHByZXNzZWQgYmV0d2VlbiB0aGUgdHdvIGtpZG5leSByZWplY3Rpb24gZ3JvdXBzPyBZb3UgbXVzdCBjb250cm9sIHRoZSBGRFIgYXQgMTAlLiAKLSBDYW4gdGhlIGtpZG5leSByZWplY3Rpb24gIGJlIHByZWRpY3RlZCBmcm9tIHRoZSBnZW5lIGV4cHJlc3Npb25zPyBXaGF0IGdlbmVzICBhcmUgbW9zdCBpbXBvcnRhbnQgaW4gcHJlZGljdGluZyB0aGUga2lkbmV5IHRyYW5zcGxhbnQgcmVqZWN0aW9uPyBIb3cgd2VsbCBkb2VzIHRoZSBwcmVkaWN0aW9uIG1vZGVsIHBlcmZvcm0gaW4gdGVybXMgb2YgcHJlZGljdGluZyByZWplY3Rpb24gc3RhdHVzPwoKICBOb3RlIHRoYXQgdGhlIHJlc3BvbnNlIHZhcmlhYmxlIGlzICpiaW5hcnkqLCBzbyBpZiB5b3UgdXNlIHJlZ3Jlc3Npb24gbW9kZWxzLAogIHlvdSB3aWxsIG5lZWQgdG8gdXNlICAqbG9naXN0aWMgcmVncmVzc2lvbiosIHdoaWNoIG1vZGVscyB0aGUgcmVzcG9uc2Ugd2l0aCBhCiAgYmlub21pYWwgZGlzdHJpYnV0aW9uLgogIFRoaXMgY2FuIGdlbmVyYWxseSBiZSBkb25lIGluIGBSYCBieSBzcGVjaWZ5aW5nIGBmYW1pbHkgPSAiYmlub21pYWwiYCBpbiByZWdyZXNzaW9uCiAgZnVuY3Rpb25zIHN1Y2ggYXMgYGdsbWAgYW5kIGBnbG1uZXRgLgoKCiMgQXNzaWdubWVudAoKX19Zb3UgbXVzdCB3b3JrIGluIGdyb3VwcyBvZiBmb3VyIHN0dWRlbnRzLl9fCgpXcml0ZSBhIHNjaWVudGlmaWMgcmVwb3J0IHRoYXQgYW5zd2VycyB0aGUgcmVzZWFyY2ggcXVlc3Rpb25zIHJlbGF0ZWQgdG8gdGhpcyBzdHVkeS4KVGhlIHJlcG9ydCBtdXN0IGNvbnNpc3Qgb2YgdHdvIHBhcnRzOgoKLSBBbiBleGVjdXRpdmUgc3VtbWFyeSBvZiBhYm91dCBoYWxmIGEgcGFnZS4gVGhpcyBzdW1tYXJ5IGNvbnRhaW5zIHRoZSBhbnN3ZXJzIHRvIHRoZSBvcmlnaW5hbCByZXNlYXJjaCBxdWVzdGlvbnMsIGFuZCBzaG91bGQgYmUgd3JpdHRlbiBpbiBhIG5vbi10ZWNobmljYWwgbWFubmVyIChpdCBpcyBtZWFudCBmb3IgcmVzZWFyY2hlcnMgd2l0aG91dCBhIHN0YXRpc3RpY2FsIGJhY2tncm91bmQpLgogCi0gQSB0ZWNobmljYWwgcmVwb3J0IHRoYXQgZXhwbGFpbnMgaW4gZGV0YWlsIGhvdyB0aGUgcmVzdWx0cyB3ZXJlIG9idGFpbmVkLgogIEl0J3MgcmVjb21tZW5kZWQgdG8gcHJlcGFyZSB0aGlzIHRlY2huaWNhbCByZXBvcnQgYW4gX19STWFya2Rvd25fXyBmaWxlLgogIElmIHlvdSBjaG9vc2UgdG8gdXNlIGFub3RoZXIgZm9ybWF0LCB0aGVuIHRoZSBSIGNvZGUgc2hvdWxkIGJlIHN1Ym1pdHRlZCAKICBhcyBhIHNlcGFyYXRlIGZpbGUgKHBsZWFzZSBjb21tZW50IHlvdXIgUiBjb2RlKS4KClRoZSByZXBvcnQgaXMgZXhwZWN0ZWQgdG8gYmUgY29uY2lzZSwgYnV0IG11c3QgZXZpZGVudGx5IGJlIGFjY3VyYXRlIGFuZCBzdWZmaWNpZW50bHkgZGV0YWlsZWQgdG8gZW5hYmxlIHRoZSByZWFkZXIgdG8gdmVyaWZ5IHRoZSBjb3JyZWN0bmVzcyBvZiB0aGUgcmVzdWx0IChpLmUuIHlvdXIgcmVzdWx0cyBtdXN0IGJlIHJlcHJvZHVjaWJsZSkuClRoZSB0b3RhbCBsZW5ndGggb2YgdGhlIHJlcG9ydCAoZXhjbHVkaW5nIGdyYXBocywgUiBjb2RlIGFuZCBwb3NzaWJseSBhcHBlbmRpY2VzKSBzaG91bGQgbm90IGJlIG1vcmUgdGhhbiB0aHJlZSBwYWdlcy4gClRoZSByZXBvcnQgc2hvdWxkIG5vdCBjb250YWluIGFuIGV4cGxhbmF0aW9uIG9mIHRoZSB0aGVvcnkgYmVoaW5kIHRoZSBzdGF0aXN0aWNhbCBtZXRob2RzLCBhbmQgc2hvdWxkIGFsc28gbm90IGNvbnRhaW4gdGhlIHN0dWR5IGRlc2NyaXB0aW9uIGdpdmVuIGFib3ZlICh5b3UgY2FuIGFzc3VtZSB0aGF0IHRoZSByZWFkZXIgYWxyZWFkeSBrbm93cyB0aGlzKS4KCkhvd2V2ZXIsICppbnRlcnByZXRhdGlvbiogb2YgdGhlIHJlc3VsdHMgaXMga2V5IQoKU29tZSBtb3JlIHNwZWNpZmljIGd1aWRlbGluZXM6CgotIEZvciB0aGUgZmlyc3QgcmVzZWFyY2ggcXVlc3Rpb24sIHlvdSBtYXkgdXNlIG9uZSBvZiB0aGUgZGF0YSBleHBsb3JhdGlvbiB0b29scyB0aGF0IHdlIGhhdmUgc2VlbiBpbiBjbGFzcy4gSG93ZXZlciwgeW91IGFyZSBhbHNvIGZyZWUgdG8gc2VhcmNoIHRoZSBsaXRlcmF0dXJlIGZvciBvdGhlciB0ZWNobmlxdWVzIGZvciBkYXRhIGV4cGxvcmF0aW9uIGFuZCB2aXN1YWxpc2F0aW9uICh5b3VyIGZpbmFsIG1hcmsgd2lsbCBub3QgZGVwZW5kIG9uIHdoZXRoZXIgeW91IHNlYXJjaGVkIHRoZSBsaXRlcmF0dXJlIG9yIG5vdCkuCllvdSBhcmUgb25seSBhc2tlZCB0byAqZXhwbG9yZSogd2hldGhlciB0aGUgdmFyaWFiaWxpdHkgaW4gZ2VuZSBleHByZXNzaW9uIGxldmVscyBpcyBhc3NvY2lhdGVkIHdpdGggcmVqZWN0aW9uIHN0YXR1czsgbm8gbmVlZCBmb3IgaHlwb3RoZXNpcyB0ZXN0aW5nLgogCi0gRm9yIHRoZSBzZWNvbmQgcmVzZWFyY2ggcXVlc3Rpb24sIHlvdSBhcmUgYXNrZWQgdG8gcGVyZm9ybSBoeXBvdGhlc2lzIHRlc3RpbmcgYW5kIGNvcnJlY3QgZm9yIG11bHRpcGxlIHRlc3Rpbmcgc28gYXMgdG8gY29udHJvbCB0aGUgRkRSIGF0IDEwJS4gVGhlIGZ1bGwgbGlzdCB3aXRoIGRpZmZlcmVudGlhbGx5IGV4cHJlc3NlZCBnZW5lcyBtYXkgYmUgcHJlc2VudGVkIGluIGFuIGFwcGVuZGl4LiBPbmx5IGxpc3QgdGhlIG1vc3QgaW1wb3J0YW50IHJlc3VsdHMgaW4gdGhlIGJvZHkgb2YgdGhlIHJlcG9ydC4gCiAKLSBGb3IgdGhlIHRoaXJkIHJlc2VhcmNoIHF1ZXN0aW9uIHlvdSBhcmUgYXNrZWQgdG8gcHJlZGljdCByZWplY3Rpb24gc3RhdHVzIHVzaW5nIGdlbmUgZXhwcmVzc2lvbiBsZXZlbHMuIAogIFlvdSBzaG91bGQgcmFuZG9tbHkgc3BsaXQgdGhlIGRhdGEgaW50byBhIHRlc3QgKDMwJSkgYW5kIGEgdHJhaW5pbmcgKDcwJSkgZGF0YXNldC4KICBNYWtlIHN1cmUgeW91IHVzZSBhIHNlZWQgKGBzZXQuc2VlZCgpYCBmdW5jdGlvbikgaW4gUiBmb3IgcmVwcm9kdWNpYmlsaXR5LgogIFRoZSBmb2xsb3dpbmcgcHJlZGljdGlvbiBtb2RlbHMgc2hvdWxkIGJlIGV2YWx1YXRlZDoKICAKICAqIFByaW5jaXBhbCBDb21wb25lbnQgUmVncmVzc2lvbiAoUENSKQogICAgCiAgKiBSaWRnZSBSZWdyZXNzaW9uCgogICogTGFzc28gcmVncmVzc2lvbgoKSW4gY2hvb3NpbmcgdGhlIG51bWJlciBvZiBQQ3MgIGluIFBDUiwgYW5kIHRoZSAkXGdhbW1hJCBpbiB0aGUgUmlkZ2UgIGFuZCBMYXNzbyBtb2RlbHMsIHlvdSBuZWVkIHRvIHVzZSBjcm9zcyB2YWxpZGF0aW9uIChDVikgb24gdGhlIHRyYWluaW5nIGRhdGFzZXQuCllvdSBzaG91bGQgdXNlIHRoZSBfX2FyZWEgdW5kZXIgdGhlIHJlY2VpdmVyIGNoYXJhY3RlcmlzdGljIGN1cnZlIChBVUMpX18gYXMgYSBwZXJmb3JtYW5jZSBtZWFzdXJlLgoKT25jZSB5b3UgaGF2ZSBzZWxlY3RlZCB0aGUgb3B0aW1hbCBQQ1IsIFJpZGdlIGFuZCBMYXNzbyBtb2RlbHMsIHlvdSBoYXZlIHRvIGRlY2lkZSB3aXRoIHdoYXQgbW9kZWwgeW91IHdhbnQgdG8gY29udGludWUuIEZvciB0aGlzIG1vZGVsIHlvdSBoYXZlIHRvIGRldGVybWluZSBhIGdvb2QgdGhyZXNob2xkICRjJCBmb3IgdGhlIHByZWRpY3Rpb24gY3V0LW9mZiB0aGF0IGdpdmVzIGEgZ29vZCBjb21wcm9taXNlIGJldHdlZW4gc2Vuc2l0aXZpdHkgYW5kIHNwZWNpZmljaXR5LiAKClVzZSB0aGUgdGVzdCBkYXRhIGZvciBmaW5hbCBwZXJmb3JtYW5jZSBldmFsdWF0aW9uIGluIHRlcm1zIG9mICpzZW5zaXRpdml0eSogYW5kICpzcGVjaWZpY2l0eSouIAoKCiMgU3VibWlzc2lvbgoKSXQgaXMgcmVjb21tZW5kZWQgKGJ1dCBub3QgbWFuZGF0b3J5KSB0byBwcmVwYXJlIHlvdXIgcmVwb3J0IGluIF9fUk1hcmtkb3duX18uCllvdSBjYW4gcmVuZGVyIGl0IHRvIGVpdGhlciBIVE1MIChgb3V0cHV0OiBodG1sX2RvY3VtZW50YCkgb3IgdG8gUERGIChgb3V0cHV0OiBwZGZfZG9jdW1lbnRgKS4KSW4gYm90aCBjYXNlcyB0aGUgb3JpZ2luYWwgYC5SbWRgIGZpbGUgc2hvdWxkIGJlIGluY2x1ZGVkIHdoZW4gaGFuZGluZyBpbiB0aGUgYXNzaWdubWVudC4KSWYgeW91IGRvbid0IHVzZSBSTWFya2Rvd24sIHlvdSBzaG91bGQgaW5jbHVkZSB0aGUgYC5SYCBmaWxlKHMpIGNvbnRhaW5pbmcgeW91ciBpbXBsZW1lbnRhdGlvbiBhbmQgYW5hbHlzaXMgc2NyaXB0cy4KCldoZW4gc3VibWl0dGluZywgcGxlYXNlIHVzZSB0aGUgZm9sbG93aW5nIGZvcm1hdDoKCiogSFctTmFtZTEtTmFtZTItTmFtZTMtTmFtZTQuW3BkZnxodG1sXQoKKiBIVy1OYW1lMS1OYW1lMi1OYW1lMy1OYW1lNC5SW21kXQoKd2hlcmUgYE5hbWVgIGlzIHlvdXIgX19mYW1pbHkgbmFtZV9fLgpJdCdzIGFsc28gcmVjb21tZW5kZWQgdG8gbWVudGlvbiB5b3VyIGZ1bGwgbmFtZSBpbiB0aGUgcmVwb3J0IGl0c2VsZi4KClN1Ym1pc3Npb25zIHNob3VsZCBiZSBkb25lIF9fdGhyb3VnaCBbVUZvcmFdKGh0dHBzOi8vdWZvcmEudWdlbnQuYmUvZDJsL2hvbWUvMjIxMjEyKV9fLgoKPHNwYW4gc3R5bGU9ImNvbG9yOnJlZCI+X19UaGUgZGVhZGxpbmUgZm9yIHN1Ym1pc3Npb24gaXMgMTcvMTIvMjAyMCBhdCAyMzo1OV9fPC9zcGFuPgoKCgojIFJlZmVyZW5jZXMgey19Cg==