Creative Commons License

This is part of the online course Proteomics Data Analysis 2021 (PDA21)

Outline

  1. Introduction

  2. Preprocessing

    • Log-transformation
    • Filtering
    • Normalization
    • Summarization

Note, that the R-code is included for learners who are aiming to develop R/markdown scripts to automate their quantitative proteomics data analyses. According to the target audience of the course we either work with a graphical user interface (GUI) in a R/shiny App msqrob2gui (e.g. Proteomics Bioinformatics course of the EBI and the Proteomics Data Analysis course at the Gulbenkian institute) or with R/markdowns scripts (e.g. Bioinformatics Summer School at UCLouvain or the Statistical Genomics Course at Ghent University).


1 Intro: Challenges in Label-Free Quantitative Proteomics

1.1 MS-based workflow

  • Peptide Characteristics

    • Modifications
    • Ionisation Efficiency: huge variability
    • Identification
      • Misidentification \(\rightarrow\) outliers
      • MS\(^2\) selection on peptide abundance
      • Context depending missingness
      • Non-random missingness

\(\rightarrow\) Unbalanced pepide identifications across samples and messy data


1.2 Level of quantification

  • MS-based proteomics returns peptides: pieces of proteins

  • Quantification commonly required on the protein level


1.3 Label-free Quantitative Proteomics Data Analysis Workflows


1.4 CPTAC Spike-in Study

  • Same trypsin-digested yeast proteome background in each sample

  • Trypsin-digested Sigma UPS1 standard: 48 different human proteins spiked in at 5 different concentrations (treatment A-E)

  • Samples repeatedly run on different instruments in different labs

  • After MaxQuant search with match between runs option

    • 41% of all proteins are quantified in all samples
    • 6.6% of all peptides are quantified in all samples

\(\rightarrow\) vast amount of missingness

1.5 Maxquant output


2 Import the data in R

2.1 Data infrastructure

Click to see background on data infrastructure used in R to store proteomics data

  • We use the QFeatures package that provides the infrastructure to
    • store,
    • process,
    • manipulate and
    • analyse quantitative data/features from mass spectrometry experiments.
  • It is based on the SummarizedExperiment and MultiAssayExperiment classes.
Conceptual representation of a `SummarizedExperiment` object.  Assays contain information on the measured omics features (rows) for different samples (columns). The `rowData` contains information on the omics features, the `colData` contains information on the samples, i.e. experimental design etc.

Conceptual representation of a SummarizedExperiment object. Assays contain information on the measured omics features (rows) for different samples (columns). The rowData contains information on the omics features, the colData contains information on the samples, i.e. experimental design etc.

  • Assays in a QFeatures object have a hierarchical relation:

    • proteins are composed of peptides,
    • themselves produced by spectra
    • relations between assays are tracked and recorded throughout data processing
Conceptual representation of a `QFeatures` object and the aggregative relation between different assays.

Conceptual representation of a QFeatures object and the aggregative relation between different assays.

2.2 Import data in R

2.2.1 Load libraries

Click to see code

library(tidyverse)
library(limma)
library(QFeatures)
library(msqrob2)
library(plotly)
library(ggplot2)

2.2.2 Read data

Click to see background and code

  1. We use a peptides.txt file from MS-data quantified with maxquant that contains MS1 intensities summarized at the peptide level.
peptidesFile <- "https://raw.githubusercontent.com/statOmics/PDA21/data/quantification/fullCptacDatasSetNotForTutorial/peptides.txt"
  1. Maxquant stores the intensity data for the different samples in columnns that start with Intensity. We can retreive the column names with the intensity data with the code below:
ecols <- grep("Intensity\\.", names(read.delim(peptidesFile)))
  1. Read the data and store it in QFeatures object
pe <- readQFeatures(
  table = peptidesFile,
  fnames = 1,
  ecol = ecols,
  name = "peptideRaw", sep="\t")

2.2.3 Explore object

Click to see background and code

  • The rowData contains information on the features (peptides) in the assay. E.g. Sequence, protein, …
rowData(pe[["peptideRaw"]])
## DataFrame with 11466 rows and 143 columns
##                          Sequence N.term.cleavage.window C.term.cleavage.window
##                       <character>            <character>            <character>
## AAAAGAGGAGDSGDAVTK  AAAAGAGGAG...          EHQHDEQKAA...          DSGDAVTKIG...
## AAAALAGGK               AAAALAGGK          QQLSKAAKAA...          AAALAGGKKS...
## AAAALAGGKK             AAAALAGGKK          QQLSKAAKAA...          AALAGGKKSK...
## AAADALSDLEIK        AAADALSDLE...          MPKETPSKAA...          ALSDLEIKDS...
## AAADALSDLEIKDSK     AAADALSDLE...          MPKETPSKAA...          DLEIKDSKSN...
## ...                           ...                    ...                    ...
## YYSIYDLGNNAVGLAK    YYSIYDLGNN...          VGDAFLRKYY...          NNAVGLAKAI...
## YYTFNGPNYNENETIR    YYTFNGPNYN...          FKDGSYPKYY...          YNENETIRHI...
## YYTITEVATR             YYTITEVATR          QEWDINERYY...          TITEVATRAK...
## YYTVFDRDNNR         YYTVFDRDNN...          LGDVFIGRYY...          VFDRDNNRVG...
## YYTVFDRDNNRVGFAEAAR YYTVFDRDNN...          LGDVFIGRYY...          VGFAEAARL_...
##                     Amino.acid.before First.amino.acid Second.amino.acid
##                           <character>      <character>       <character>
## AAAAGAGGAGDSGDAVTK                  K                A                 A
## AAAALAGGK                           K                A                 A
## AAAALAGGKK                          K                A                 A
## AAADALSDLEIK                        K                A                 A
## AAADALSDLEIKDSK                     K                A                 A
## ...                               ...              ...               ...
## YYSIYDLGNNAVGLAK                    K                Y                 Y
## YYTFNGPNYNENETIR                    K                Y                 Y
## YYTITEVATR                          R                Y                 Y
## YYTVFDRDNNR                         R                Y                 Y
## YYTVFDRDNNRVGFAEAAR                 R                Y                 Y
##                     Second.last.amino.acid Last.amino.acid Amino.acid.after
##                                <character>     <character>      <character>
## AAAAGAGGAGDSGDAVTK                       T               K                I
## AAAALAGGK                                G               K                K
## AAAALAGGKK                               K               K                S
## AAADALSDLEIK                             I               K                D
## AAADALSDLEIKDSK                          S               K                S
## ...                                    ...             ...              ...
## YYSIYDLGNNAVGLAK                         A               K                A
## YYTFNGPNYNENETIR                         I               R                H
## YYTITEVATR                               T               R                A
## YYTVFDRDNNR                              N               R                V
## YYTVFDRDNNRVGFAEAAR                      A               R                L
##                       A.Count   R.Count   N.Count   D.Count   C.Count   Q.Count
##                     <integer> <integer> <integer> <integer> <integer> <integer>
## AAAAGAGGAGDSGDAVTK          7         0         0         2         0         0
## AAAALAGGK                   5         0         0         0         0         0
## AAAALAGGKK                  5         0         0         0         0         0
## AAADALSDLEIK                4         0         0         2         0         0
## AAADALSDLEIKDSK             4         0         0         3         0         0
## ...                       ...       ...       ...       ...       ...       ...
## YYSIYDLGNNAVGLAK            2         0         2         1         0         0
## YYTFNGPNYNENETIR            0         1         4         0         0         0
## YYTITEVATR                  1         1         0         0         0         0
## YYTVFDRDNNR                 0         2         2         2         0         0
## YYTVFDRDNNRVGFAEAAR         3         3         2         2         0         0
##                       E.Count   G.Count   H.Count   I.Count   L.Count   K.Count
##                     <integer> <integer> <integer> <integer> <integer> <integer>
## AAAAGAGGAGDSGDAVTK          0         5         0         0         0         1
## AAAALAGGK                   0         2         0         0         1         1
## AAAALAGGKK                  0         2         0         0         1         2
## AAADALSDLEIK                1         0         0         1         2         1
## AAADALSDLEIKDSK             1         0         0         1         2         2
## ...                       ...       ...       ...       ...       ...       ...
## YYSIYDLGNNAVGLAK            0         2         0         1         2         1
## YYTFNGPNYNENETIR            2         1         0         1         0         0
## YYTITEVATR                  1         0         0         1         0         0
## YYTVFDRDNNR                 0         0         0         0         0         0
## YYTVFDRDNNRVGFAEAAR         1         1         0         0         0         0
##                       M.Count   F.Count   P.Count   S.Count   T.Count   W.Count
##                     <integer> <integer> <integer> <integer> <integer> <integer>
## AAAAGAGGAGDSGDAVTK          0         0         0         1         1         0
## AAAALAGGK                   0         0         0         0         0         0
## AAAALAGGKK                  0         0         0         0         0         0
## AAADALSDLEIK                0         0         0         1         0         0
## AAADALSDLEIKDSK             0         0         0         2         0         0
## ...                       ...       ...       ...       ...       ...       ...
## YYSIYDLGNNAVGLAK            0         0         0         1         0         0
## YYTFNGPNYNENETIR            0         1         1         0         2         0
## YYTITEVATR                  0         0         0         0         3         0
## YYTVFDRDNNR                 0         1         0         0         1         0
## YYTVFDRDNNRVGFAEAAR         0         2         0         0         1         0
##                       Y.Count   V.Count   U.Count    Length Missed.cleavages
##                     <integer> <integer> <integer> <integer>        <integer>
## AAAAGAGGAGDSGDAVTK          0         1         0        18                0
## AAAALAGGK                   0         0         0         9                0
## AAAALAGGKK                  0         0         0        10                1
## AAADALSDLEIK                0         0         0        12                0
## AAADALSDLEIKDSK             0         0         0        15                1
## ...                       ...       ...       ...       ...              ...
## YYSIYDLGNNAVGLAK            3         1         0        16                0
## YYTFNGPNYNENETIR            3         0         0        16                0
## YYTITEVATR                  2         1         0        10                0
## YYTVFDRDNNR                 2         1         0        11                1
## YYTVFDRDNNRVGFAEAAR         2         2         0        19                2
##                          Mass      Proteins Leading.razor.protein
##                     <numeric>   <character>           <character>
## AAAAGAGGAGDSGDAVTK   1445.675 sp|P38915|...         sp|P38915|...
## AAAALAGGK             728.418 sp|Q3E792|...         sp|Q3E792|...
## AAAALAGGKK            856.513 sp|Q3E792|...         sp|Q3E792|...
## AAADALSDLEIK         1215.635 sp|P09938|...         sp|P09938|...
## AAADALSDLEIKDSK      1545.789 sp|P09938|...         sp|P09938|...
## ...                       ...           ...                   ...
## YYSIYDLGNNAVGLAK      1759.88 sp|P07267|...         sp|P07267|...
## YYTFNGPNYNENETIR      1993.88 sp|Q00955|...         sp|Q00955|...
## YYTITEVATR            1215.61 sp|P38891|...         sp|P38891|...
## YYTVFDRDNNR           1461.66 P07339ups|...         P07339ups|...
## YYTVFDRDNNRVGFAEAAR   2263.08 P07339ups|...         P07339ups|...
##                     Start.position End.position Unique..Groups.
##                          <integer>    <integer>     <character>
## AAAAGAGGAGDSGDAVTK              97          114             yes
## AAAALAGGK                       13           21             yes
## AAAALAGGKK                      13           22             yes
## AAADALSDLEIK                     9           20             yes
## AAADALSDLEIKDSK                  9           23             yes
## ...                            ...          ...             ...
## YYSIYDLGNNAVGLAK               388          403             yes
## YYTFNGPNYNENETIR              1275         1290             yes
## YYTITEVATR                     311          320             yes
## YYTVFDRDNNR                    225          235             yes
## YYTVFDRDNNRVGFAEAAR            225          243             yes
##                     Unique..Proteins.     Charges        PEP     Score
##                           <character> <character>  <numeric> <numeric>
## AAAAGAGGAGDSGDAVTK                yes           2 1.1843e-05    82.942
## AAAALAGGK                          no           2 7.4562e-06   134.810
## AAAALAGGKK                         no           2 3.3094e-09   143.730
## AAADALSDLEIK                      yes           2 9.1593e-23   182.230
## AAADALSDLEIKDSK                   yes           3 1.5319e-04    73.927
## ...                               ...         ...        ...       ...
## YYSIYDLGNNAVGLAK                  yes           2 7.7415e-37   174.240
## YYTFNGPNYNENETIR                  yes           2 4.2208e-21   147.750
## YYTITEVATR                        yes           2 1.3566e-04   109.160
## YYTVFDRDNNR                       yes           2 6.1425e-04   110.930
## YYTVFDRDNNRVGFAEAAR               yes           3 8.9859e-04    59.728
##                     Identification.type.6A_1 Identification.type.6A_2
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...                 By MS/MS
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                By matchin...            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6A_3 Identification.type.6A_4
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...                 By MS/MS
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...                 By MS/MS
## AAADALSDLEIK                   By matchin...                 By MS/MS
## AAADALSDLEIKDSK                By matchin...                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...                 By MS/MS
## YYTFNGPNYNENETIR               By matchin...                 By MS/MS
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6A_5 Identification.type.6A_6
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS                 By MS/MS
## YYTFNGPNYNENETIR                    By MS/MS                 By MS/MS
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6A_7 Identification.type.6A_8
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS                 By MS/MS
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                        By MS/MS            By matchin...
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6A_9 Identification.type.6B_1
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS            By matchin...
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...                 By MS/MS
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6B_2 Identification.type.6B_3
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                        By MS/MS            By matchin...
## AAADALSDLEIKDSK                By matchin...            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6B_4 Identification.type.6B_5
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS            By matchin...
## YYTFNGPNYNENETIR                    By MS/MS                 By MS/MS
## YYTITEVATR                          By MS/MS                 By MS/MS
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6B_6 Identification.type.6B_7
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...                 By MS/MS
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR                    By MS/MS            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6B_8 Identification.type.6B_9
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS                 By MS/MS
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                   By matchin...            By matchin...
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6C_1 Identification.type.6C_2
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...                 By MS/MS
## AAADALSDLEIK                        By MS/MS            By matchin...
## AAADALSDLEIKDSK                By matchin...            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6C_3 Identification.type.6C_4
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...                 By MS/MS
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                By matchin...                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...                 By MS/MS
## YYTFNGPNYNENETIR               By matchin...                 By MS/MS
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6C_5 Identification.type.6C_6
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS                 By MS/MS
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6C_7 Identification.type.6C_8
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS            By matchin...
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                   By matchin...                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...                 By MS/MS
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6C_9 Identification.type.6D_1
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                           By MS/MS            By matchin...
## AAAALAGGKK                          By MS/MS            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6D_2 Identification.type.6D_3
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                   By matchin...            By matchin...
## AAADALSDLEIKDSK                     By MS/MS            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS                 By MS/MS
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6D_4 Identification.type.6D_5
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                          By MS/MS            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS                 By MS/MS
## YYTFNGPNYNENETIR                    By MS/MS                 By MS/MS
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6D_6 Identification.type.6D_7
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK                  By MS/MS            By matchin...
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...                 By MS/MS
## AAADALSDLEIK                        By MS/MS            By matchin...
## AAADALSDLEIKDSK                By matchin...                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS            By matchin...
## YYTFNGPNYNENETIR                    By MS/MS            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...                 By MS/MS
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6D_8 Identification.type.6D_9
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                          By MS/MS            By matchin...
## YYTVFDRDNNR                         By MS/MS            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6E_1 Identification.type.6E_2
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6E_3 Identification.type.6E_4
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...                 By MS/MS
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                   By matchin...                 By MS/MS
## AAADALSDLEIKDSK                     By MS/MS            By matchin...
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...                 By MS/MS
## YYTFNGPNYNENETIR               By matchin...                 By MS/MS
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                    By matchin...                 By MS/MS
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6E_5 Identification.type.6E_6
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                      By matchin...            By matchin...
## AAAALAGGKK                     By matchin...            By matchin...
## AAADALSDLEIK                        By MS/MS            By matchin...
## AAADALSDLEIKDSK                     By MS/MS                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK                    By MS/MS                 By MS/MS
## YYTFNGPNYNENETIR                    By MS/MS                 By MS/MS
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                         By MS/MS            By matchin...
## YYTVFDRDNNRVGFAEAAR            By matchin...                 By MS/MS
##                     Identification.type.6E_7 Identification.type.6E_8
##                                  <character>              <character>
## AAAAGAGGAGDSGDAVTK             By matchin...            By matchin...
## AAAALAGGK                           By MS/MS                 By MS/MS
## AAAALAGGKK                          By MS/MS                 By MS/MS
## AAADALSDLEIK                        By MS/MS                 By MS/MS
## AAADALSDLEIKDSK                By matchin...                 By MS/MS
## ...                                      ...                      ...
## YYSIYDLGNNAVGLAK               By matchin...            By matchin...
## YYTFNGPNYNENETIR               By matchin...            By matchin...
## YYTITEVATR                     By matchin...            By matchin...
## YYTVFDRDNNR                         By MS/MS                 By MS/MS
## YYTVFDRDNNRVGFAEAAR            By matchin...            By matchin...
##                     Identification.type.6E_9 Experiment.6A_1 Experiment.6A_2
##                                  <character>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK             By matchin...              NA               1
## AAAALAGGK                           By MS/MS              NA               1
## AAAALAGGKK                          By MS/MS              NA               1
## AAADALSDLEIK                        By MS/MS               1               1
## AAADALSDLEIKDSK                     By MS/MS               1               1
## ...                                      ...             ...             ...
## YYSIYDLGNNAVGLAK               By matchin...              NA              NA
## YYTFNGPNYNENETIR                    By MS/MS              NA              NA
## YYTITEVATR                     By matchin...               1              NA
## YYTVFDRDNNR                         By MS/MS              NA              NA
## YYTVFDRDNNRVGFAEAAR            By matchin...              NA              NA
##                     Experiment.6A_3 Experiment.6A_4 Experiment.6A_5
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA               1               1
## AAAALAGGK                         2               1               1
## AAAALAGGKK                       NA               1              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                  NA               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA               1               1
## YYTFNGPNYNENETIR                 NA               1               1
## YYTITEVATR                        1              NA              NA
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6A_6 Experiment.6A_7 Experiment.6A_8
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1               1               1
## AAAALAGGK                         1               2               1
## AAAALAGGKK                        1               1               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1              NA              NA
## YYTFNGPNYNENETIR                  1               1              NA
## YYTITEVATR                        1               1              NA
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6A_9 Experiment.6B_1 Experiment.6B_2
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1              NA              NA
## AAAALAGGK                         1               1               1
## AAAALAGGKK                        1              NA               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1              NA               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA              NA              NA
## YYTFNGPNYNENETIR                  1              NA              NA
## YYTITEVATR                       NA               1               1
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6B_3 Experiment.6B_4 Experiment.6B_5
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA              NA               1
## AAAALAGGK                         1               2               1
## AAAALAGGKK                        1               1              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                  NA               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA               1               1
## YYTFNGPNYNENETIR                 NA               1               1
## YYTITEVATR                        1               1               1
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6B_6 Experiment.6B_7 Experiment.6B_8
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1              NA               1
## AAAALAGGK                        NA               2               1
## AAAALAGGKK                       NA               1               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1              NA              NA
## YYTFNGPNYNENETIR                  1               1              NA
## YYTITEVATR                        1              NA               1
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6B_9 Experiment.6C_1 Experiment.6C_2
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1              NA              NA
## AAAALAGGK                         2              NA               1
## AAAALAGGKK                        1              NA               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA              NA              NA
## YYTFNGPNYNENETIR                 NA              NA              NA
## YYTITEVATR                       NA               1               1
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6C_3 Experiment.6C_4 Experiment.6C_5
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA               1               1
## AAAALAGGK                         2               2              NA
## AAAALAGGKK                       NA               1              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA               1               1
## YYTFNGPNYNENETIR                 NA               1               1
## YYTITEVATR                        1               1              NA
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6C_6 Experiment.6C_7 Experiment.6C_8
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1               1               1
## AAAALAGGK                        NA               2               1
## AAAALAGGKK                       NA               1               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1              NA              NA
## YYTFNGPNYNENETIR                  1               1               1
## YYTITEVATR                        1              NA               1
## YYTVFDRDNNR                       1              NA               1
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6C_9 Experiment.6D_1 Experiment.6D_2
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1              NA              NA
## AAAALAGGK                         1              NA               1
## AAAALAGGKK                        1              NA              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA              NA              NA
## YYTFNGPNYNENETIR                  1              NA              NA
## YYTITEVATR                        1              NA               1
## YYTVFDRDNNR                      NA              NA              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6D_3 Experiment.6D_4 Experiment.6D_5
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA               1               1
## AAAALAGGK                         1               1               1
## AAAALAGGKK                       NA               1              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA               1               1
## YYTFNGPNYNENETIR                 NA               1               1
## YYTITEVATR                        1               1               1
## YYTVFDRDNNR                      NA               1               1
## YYTVFDRDNNRVGFAEAAR              NA               1              NA
##                     Experiment.6D_6 Experiment.6D_7 Experiment.6D_8
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1               1              NA
## AAAALAGGK                        NA               2               1
## AAAALAGGKK                       NA               1               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1               1              NA
## YYTFNGPNYNENETIR                  1               1               1
## YYTITEVATR                        1              NA               1
## YYTVFDRDNNR                       1               1               1
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6D_9 Experiment.6E_1 Experiment.6E_2
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA              NA               1
## AAAALAGGK                         2              NA               1
## AAAALAGGKK                        1              NA              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                 NA              NA              NA
## YYTFNGPNYNENETIR                  1              NA              NA
## YYTITEVATR                       NA              NA               1
## YYTVFDRDNNR                       1               1              NA
## YYTVFDRDNNRVGFAEAAR              NA              NA              NA
##                     Experiment.6E_3 Experiment.6E_4 Experiment.6E_5
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK               NA              NA               1
## AAAALAGGK                         2               2               1
## AAAALAGGKK                       NA               1              NA
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1               1               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1               1               1
## YYTFNGPNYNENETIR                 NA               1               1
## YYTITEVATR                        1               1               1
## YYTVFDRDNNR                       1               1               1
## YYTVFDRDNNRVGFAEAAR              NA               1               1
##                     Experiment.6E_6 Experiment.6E_7 Experiment.6E_8
##                           <integer>       <integer>       <integer>
## AAAAGAGGAGDSGDAVTK                1              NA              NA
## AAAALAGGK                        NA               2               2
## AAAALAGGKK                       NA               1               1
## AAADALSDLEIK                      1               1               1
## AAADALSDLEIKDSK                   1              NA               1
## ...                             ...             ...             ...
## YYSIYDLGNNAVGLAK                  1              NA              NA
## YYTFNGPNYNENETIR                  1               1               1
## YYTITEVATR                       NA              NA              NA
## YYTVFDRDNNR                       1               1               1
## YYTVFDRDNNRVGFAEAAR               1               1               1
##                     Experiment.6E_9 Intensity     Reverse Potential.contaminant
##                           <integer> <numeric> <character>           <character>
## AAAAGAGGAGDSGDAVTK               NA   1190800                                  
## AAAALAGGK                         1 280990000                                  
## AAAALAGGKK                        1  33360000                                  
## AAADALSDLEIK                      1  54622000                                  
## AAADALSDLEIKDSK                   1  18910000                                  
## ...                             ...       ...         ...                   ...
## YYSIYDLGNNAVGLAK                 NA   2145900                                  
## YYTFNGPNYNENETIR                  1   5608800                                  
## YYTITEVATR                       NA  13034000                                  
## YYTVFDRDNNR                       1   8702500                                  
## YYTVFDRDNNRVGFAEAAR               1   2391100                                  
##                            id Protein.group.IDs Mod..peptide.IDs  Evidence.IDs
##                     <integer>       <character>      <character>   <character>
## AAAAGAGGAGDSGDAVTK          0               859                0 0;1;2;3;4;...
## AAAALAGGK                   1               230                1 24;25;26;2...
## AAAALAGGKK                  2               230                2 74;75;76;7...
## AAADALSDLEIK                3               229                3 99;100;101...
## AAADALSDLEIKDSK             4               229                4 144;145;14...
## ...                       ...               ...              ...           ...
## YYSIYDLGNNAVGLAK        11461               196            12240 331367;331...
## YYTFNGPNYNENETIR        11462              1254            12241 331384;331...
## YYTITEVATR              11463               854            12242 331411;331...
## YYTVFDRDNNR             11464                34            12243 331439;331...
## YYTVFDRDNNRVGFAEAAR     11465                34            12244 331455;331...
##                         MS.MS.IDs Best.MS.MS Oxidation..M..site.IDs MS.MS.Count
##                       <character>  <integer>            <character>   <integer>
## AAAAGAGGAGDSGDAVTK  0;1;2;3;4;...          0                                 10
## AAAALAGGK           10;11;12;1...         21                                 18
## AAAALAGGKK          30;31;32;3...         31                                 21
## AAADALSDLEIK        51;52;53;5...         72                                 29
## AAADALSDLEIKDSK     85;86;87;8...         94                                 32
## ...                           ...        ...                    ...         ...
## YYSIYDLGNNAVGLAK    169138;169...     169147                                 13
## YYTFNGPNYNENETIR    169151;169...     169159                                 14
## YYTITEVATR          169165;169...     169173                                 12
## YYTVFDRDNNR         169177;169...     169180                                  7
## YYTVFDRDNNRVGFAEAAR        169184     169184                                  1
  • The colData contains information on the samples
colData(pe)
## DataFrame with 45 rows and 0 columns
  • No information is stored yet on the design.
pe %>% colnames
## CharacterList of length 1
## [["peptideRaw"]] Intensity.6A_1 Intensity.6A_2 ... Intensity.6E_9
  • Note, that the sample names include the spike-in condition.

  • They also end on a number.

    • 1-3 is from lab 1,
    • 4-6 from lab 2 and
    • 7-9 from lab 3.
  • We update the colData with information on the design

colData(pe)$lab <- rep(rep(paste0("lab",1:3),each=3),5) %>% as.factor
colData(pe)$condition <- pe[["peptideRaw"]] %>% colnames %>% substr(12,12) %>% as.factor
colData(pe)$spikeConcentration <- rep(c(A = 0.25, B = 0.74, C = 2.22, D = 6.67, E = 20),each = 9)
  • We explore the colData again
colData(pe)
## DataFrame with 45 rows and 3 columns
##                     lab condition spikeConcentration
##                <factor>  <factor>          <numeric>
## Intensity.6A_1     lab1         A               0.25
## Intensity.6A_2     lab1         A               0.25
## Intensity.6A_3     lab1         A               0.25
## Intensity.6A_4     lab2         A               0.25
## Intensity.6A_5     lab2         A               0.25
## ...                 ...       ...                ...
## Intensity.6E_5     lab2         E                 20
## Intensity.6E_6     lab2         E                 20
## Intensity.6E_7     lab3         E                 20
## Intensity.6E_8     lab3         E                 20
## Intensity.6E_9     lab3         E                 20


3 Preprocessing

3.1 Log-transformation

3.1.1 Explore the data with plots

Peptide AALEELVK from spiked-in UPS protein P12081. We only show data from lab1.

Click to see code to make plot

subset <- pe["AALEELVK",colData(pe)$lab=="lab1"]
plotWhyLog <- data.frame(concentration = colData(subset)$spikeConcentration,
           y = assay(subset[["peptideRaw"]]) %>% c
           ) %>% 
  ggplot(aes(concentration, y)) +
  geom_point() +
  xlab("concentration (fmol/l)") +
  ggtitle("peptide AALEELVK in lab1")

plotWhyLog

  • Variance increases with the mean \(\rightarrow\) Multiplicative error structure
Click to see code to make plot

plotLog <- data.frame(concentration = colData(subset)$spikeConcentration,
           y = assay(subset[["peptideRaw"]]) %>% c
           ) %>% 
  ggplot(aes(concentration, y)) +
  geom_point() + 
  scale_x_continuous(trans='log2') + 
  scale_y_continuous(trans='log2') +
  xlab("concentration (fmol/l)") +
  ggtitle("peptide AALEELVK in lab1 with axes on log scale")

plotLog

  • Data seems to be homoscedastic on log-scale \(\rightarrow\) log transformation of the intensity data
  • In quantitative proteomics analysis on \(\log_2\)

\(\rightarrow\) Differences on a \(\log_2\) scale: \(\log_2\) fold changes

\[ \log_2 B - \log_2 A = \log_2 \frac{B}{A} = \log FC_\text{B - A} \] \[ \begin{array} {l} log_2 FC = 1 \rightarrow FC = 2^1 =2\\ log_2 FC = 2 \rightarrow FC = 2^2 = 4\\ \end{array} \]

3.1.2 log-transformation of the data

Click to see code to log-transfrom the data

  • We calculate how many non zero intensities we have for each peptide and this can be useful for filtering.
rowData(pe[["peptideRaw"]])$nNonZero <- rowSums(assay(pe[["peptideRaw"]]) > 0)
  • Peptides with zero intensities are missing peptides and should be represent with a NA value rather than 0.
pe <- zeroIsNA(pe, "peptideRaw") # convert 0 to NA
  • Logtransform data with base 2
pe <- logTransform(pe, base = 2, i = "peptideRaw", name = "peptideLog")


3.2 Filtering

  • Reverse sequences
  • Only identified by modification site (only modified peptides detected)
  • Razor peptides: non-unique peptides assigned to the protein group with the most other peptides
  • Contaminants
  • Peptides few identifications
  • Proteins that are only identified with one or a few peptides

Filtering does not induce bias if the criterion is independent from the downstream data analysis!

Click to see code to filter the data

  1. Handling overlapping protein groups

In our approach a peptide can map to multiple proteins, as long as there is none of these proteins present in a smaller subgroup.

pe <- filterFeatures(pe, ~ Proteins %in% smallestUniqueGroups(rowData(pe[["peptideLog"]])$Proteins))
  1. Remove reverse sequences (decoys) and contaminants

We now remove the contaminants, peptides that map to decoy sequences, and proteins which were only identified by peptides with modifications.

pe <- filterFeatures(pe,~Reverse != "+")
pe <- filterFeatures(pe,~ Potential.contaminant != "+")
  1. Drop peptides that were only identified in one sample

We keep peptides that were observed at last twice.

pe <- filterFeatures(pe,~ nNonZero >=2)
nrow(pe[["peptideLog"]])
## [1] 10478
We keep 10478 peptides upon filtering.


3.3 Normalization

Click to see code to make plot

densityConditionD <- pe[["peptideLog"]][,colData(pe)$condition=="D"] %>% 
  assay %>%
  as.data.frame() %>%
  gather(sample, intensity) %>% 
  mutate(lab = colData(pe)[sample,"lab"]) %>%
  ggplot(aes(x=intensity,group=sample,color=lab)) + 
    geom_density() +
    ggtitle("condition D")

densityLab2 <- pe[["peptideLog"]][,colData(pe)$lab=="lab2"] %>% 
  assay %>%
  as.data.frame() %>%
  gather(sample, intensity) %>% 
  mutate(condition = colData(pe)[sample,"condition"]) %>%
  ggplot(aes(x=intensity,group=sample,color=condition)) + 
    geom_density() +
    ggtitle("lab2")

densityConditionD
## Warning: Removed 39179 rows containing non-finite values (stat_density).

densityLab2
## Warning: Removed 44480 rows containing non-finite values (stat_density).

- Even in very clean synthetic dataset (same background, only 48 UPS proteins can be different) the marginal peptide intensity distribution across samples can be quite distinct

  • Considerable effects between and within labs for replicate samples
  • Considerable effects between samples with different spike-in concentration

\(\rightarrow\) Normalization is needed


3.3.1 Mean or median?

  • Miller and Fishkin (1997) reported that over a period of 30 years males would like to have on average 64.3 partners and females 2.8.

  • Miller and Fishkin (1997) reported that the median number of partners someone would like to have over a period of 30 years males is 1 for both males and females.

Mean is very sensitive to outliers!


3.3.2 Normalization of the data by median centering

\[y_{ip}^\text{norm} = y_{ip} - \hat\mu_i\] with \(\hat\mu_i\) the median intensity over all observed peptides in sample \(i\).

Click to see R-code to normalize the data

pe <- normalize(pe, 
                i = "peptideLog", 
                name = "peptideNorm", 
                method = "center.median")

3.3.3 Plots of normalized data

Click to see code to make plot

densityConditionDNorm <- pe[["peptideNorm"]][,colData(pe)$condition=="D"] %>% 
  assay %>%
  as.data.frame() %>%
  gather(sample, intensity) %>% 
  mutate(lab = colData(pe)[sample,"lab"]) %>%
  ggplot(aes(x=intensity,group=sample,color=lab)) + 
    geom_density() +
    ggtitle("condition D")

densityLab2Norm <- pe[["peptideNorm"]][,colData(pe)$lab=="lab2"] %>% 
  assay %>%
  as.data.frame() %>%
  gather(sample, intensity) %>% 
  mutate(condition = colData(pe)[sample,"condition"]) %>%
  ggplot(aes(x=intensity,group=sample,color=condition)) + 
    geom_density() +
    ggtitle("lab2")

densityConditionDNorm
## Warning: Removed 39179 rows containing non-finite values (stat_density).

densityLab2Norm
## Warning: Removed 44480 rows containing non-finite values (stat_density).

  • Upon normalization the marginal distributions of the peptide intensities across samples are much more comparable
  • We still see deviations
  • This can be due to technical variability
  • In micro-array literature, quantile normalisation is used to force the median and all other quantiles to be equal across samples
  • In proteomics quantile normalisation often introduces artifacts due to a difference in missing peptides across samples
  • More advanced methods should be developed for normalizing proteomics data
  • If there are differences in the width of the marginal distributions of the data across samples. They can also be standardized by using a robust estimator for location and scale, i.e.  \[y_{ip}^\text{norm} = \frac{y_{ip} - \mu_i}{s_i}\]

3.4 Summarization

  • We illustrate summarization issues using a subset of the cptac study (Lab 2, condition A and E) for a spiked protein (UPS P12081).
Click to see code to make plot

summaryPlot <- pe[["peptideNorm"]][
    rowData(pe[["peptideNorm"]])$Proteins == "P12081ups|SYHC_HUMAN_UPS",
    colData(pe)$lab=="lab2"&colData(pe)$condition %in% c("A","E")] %>%
  assay %>%
  as.data.frame %>%
  rownames_to_column(var = "peptide") %>%
  gather(sample, intensity, -peptide) %>% 
  mutate(condition = colData(pe)[sample,"condition"]) %>%
  ggplot(aes(x = peptide, y = intensity, color = sample, group = sample, label = condition), show.legend = FALSE) +
  geom_line(show.legend = FALSE) +
  geom_text(show.legend = FALSE) +
  theme_minimal() +
  theme(axis.text.x = element_text(angle = 90, vjust = 0.5, hjust = 1)) +
  xlab("Peptide") + 
  ylab("Intensity (log2)")

summaryPlot
## Warning: Removed 10 row(s) containing missing values (geom_path).
## Warning: Removed 90 rows containing missing values (geom_text).

We observe:

  • intensities from multiple peptides for each protein in a sample
  • Strong peptide effect -Unbalanced peptide identification
  • Pseudo-replication: peptide intensities from a particular protein in the same sample are correlated, i.e. they more alike than peptide intensities from a particular protein between samples.

\(\rightarrow\) Summarize all peptide intensities from the same protein in a sample into a single protein expression value

Commonly used methods are

  • Mean summarization \[ y_{ip}=\beta_i^\text{samp} + \epsilon_{ip} \]

  • Median summarization

  • Maxquant’s maxLFQ summarization (in protein groups file)

  • Model based summarization: \[ y_{ip}=\beta_i^\text{samp} + \beta_p^\text{pep} + \epsilon_{ip} \]

Click to see R-code to normalize the data

We use the standard sumarization in aggregateFeatures, which is robust model based summarization.

pe <- aggregateFeatures(pe,
    i = "peptideNorm", 
    fcol = "Proteins", 
    na.rm = TRUE,
    name = "protein")
## Your quantitative and row data contain missing values. Please read the
## relevant section(s) in the aggregateFeatures manual page regarding the
## effects of missing values on data aggregation.

Other summarization methods can be implemented by using the fun argument in the aggregateFeatures function.

  • fun = MsCoreUtils::medianPolish() to fits an additive model (two way decomposition) using Tukey’s median polish_ procedure using stats::medpolish()

  • fun = MsCoreUtils::robustSummary() to calculate a robust aggregation using MASS::rlm() (default)

  • fun = base::colMeans() to use the mean of each column

  • fun = matrixStats::colMedians() to use the median of each column

  • fun = base::colSums() to use the sum of each column


4 Exercise

  1. We will evaluate different summarization methods (Maxquant maxLFQ, median and robust model based) in the tutorial session before discussing on their advantages/disadvantages.
  2. Can you anticipate on potential problems related to the summarization?

5 Software & code

  • Our R/Bioconductor package msqrob2 can be used in R markdown scripts or with a GUI/shinyApp in the msqrob2gui package.

  • The GUI is intended as a introduction to the key concepts of proteomics data analysis for users who have no experience in R.

  • However, learning how to code data analyses in R markdown scripts is key for open en reproducible science and for reporting your proteomics data analyses and interpretation in a reproducible way.

  • More information on our tools can be found in our papers (Goeminne, Gevaert, and Clement 2016), (Goeminne et al. 2020) and (Sticker et al. 2020). Please refer to our work when using our tools.

5.1 Code

  1. Data infrastructure

  2. Import proteomics data

  3. Preprocessing

    • Log-transformation

    • Filtering

    • Normalisation

    • Summarization

5.2 Data analysis with the GUI/shinyApp msqrob2gui

References

Goeminne, L. J. E., A. Sticker, L. Martens, K. Gevaert, and L. Clement. 2020. “MSqRob Takes the Missing Hurdle: Uniting Intensity- and Count-Based Proteomics.” Anal Chem 92 (9): 6278–87.

Goeminne, L. J., K. Gevaert, and L. Clement. 2016. “Peptide-level Robust Ridge Regression Improves Estimation, Sensitivity, and Specificity in Data-dependent Quantitative Label-free Shotgun Proteomics.” Mol Cell Proteomics 15 (2): 657–68.

Sticker, A., L. Goeminne, L. Martens, and L. Clement. 2020. “Robust Summarization and Inference in Proteome-wide Label-free Quantification.” Mol Cell Proteomics 19 (7): 1209–19.

LS0tCnRpdGxlOiAiU3RhdGlzdGljYWwgTWV0aG9kcyBmb3IgUXVhbnRpdGF0aXZlIE1TLWJhc2VkIFByb3Rlb21pY3M6IFBhcnQgSS4gUHJlcHJvY2Vzc2luZyIKYXV0aG9yOiAiTGlldmVuIENsZW1lbnQiCmRhdGU6ICJbc3RhdE9taWNzXShodHRwczovL3N0YXRvbWljcy5naXRodWIuaW8pLCBHaGVudCBVbml2ZXJzaXR5IgpvdXRwdXQ6CiAgICBodG1sX2RvY3VtZW50OgogICAgICBjb2RlX2Rvd25sb2FkOiB0cnVlCiAgICAgIHRoZW1lOiBjb3NtbwogICAgICB0b2M6IHRydWUKICAgICAgdG9jX2Zsb2F0OiB0cnVlCiAgICAgIGhpZ2hsaWdodDogdGFuZ28KICAgICAgbnVtYmVyX3NlY3Rpb25zOiB0cnVlCiAgICBwZGZfZG9jdW1lbnQ6CiAgICAgIHRvYzogdHJ1ZQogICAgICBudW1iZXJfc2VjdGlvbnM6IHRydWUKbGlua2NvbG9yOiBibHVlCnVybGNvbG9yOiBibHVlCmNpdGVjb2xvcjogYmx1ZQoKYmlibGlvZ3JhcGh5OiBtc3Fyb2IyLmJpYgogICAgICAKLS0tCgo8YSByZWw9ImxpY2Vuc2UiIGhyZWY9Imh0dHBzOi8vY3JlYXRpdmVjb21tb25zLm9yZy9saWNlbnNlcy9ieS1uYy1zYS80LjAiPjxpbWcgYWx0PSJDcmVhdGl2ZSBDb21tb25zIExpY2Vuc2UiIHN0eWxlPSJib3JkZXItd2lkdGg6MCIgc3JjPSJodHRwczovL2kuY3JlYXRpdmVjb21tb25zLm9yZy9sL2J5LW5jLXNhLzQuMC84OHgzMS5wbmciIC8+PC9hPgoKVGhpcyBpcyBwYXJ0IG9mIHRoZSBvbmxpbmUgY291cnNlIFtQcm90ZW9taWNzIERhdGEgQW5hbHlzaXMgMjAyMSAoUERBMjEpXShodHRwczovL3N0YXRvbWljcy5naXRodWIuaW8vUERBMjEvKQoKPGlmcmFtZSB3aWR0aD0iNTYwIiBoZWlnaHQ9IjMxNSIKc3JjPSJodHRwczovL3d3dy55b3V0dWJlLmNvbS9lbWJlZC9WNklrLVNPUm9LbyIKZnJhbWVib3JkZXI9IjAiCnN0eWxlPSJkaXNwbGF5OiBibG9jazsgbWFyZ2luOiBhdXRvOyIKYWxsb3c9ImF1dG9wbGF5OyBlbmNyeXB0ZWQtbWVkaWEiIGFsbG93ZnVsbHNjcmVlbj48L2lmcmFtZT4KCi0gW1BsYXlsaXN0IFBEQSBQcmVwcm9jZXNzaW5nXShodHRwczovL3d3dy55b3V0dWJlLmNvbS9wbGF5bGlzdD9saXN0PVBMWkgxaFA4X0xiSkpYUWVRX0tZRE5vcS1BWHlGQkc2blgpCgojIE91dGxpbmUgey19CgoxLiBJbnRyb2R1Y3Rpb24KCjIuIFByZXByb2Nlc3NpbmcKCiAgICAtIExvZy10cmFuc2Zvcm1hdGlvbgogICAgLSBGaWx0ZXJpbmcKICAgIC0gTm9ybWFsaXphdGlvbgogICAgLSBTdW1tYXJpemF0aW9uCgpOb3RlLCB0aGF0IHRoZSBSLWNvZGUgaXMgaW5jbHVkZWQgZm9yIGxlYXJuZXJzIHdobyBhcmUgYWltaW5nIHRvIGRldmVsb3AgUi9tYXJrZG93biBzY3JpcHRzIHRvIGF1dG9tYXRlIHRoZWlyIHF1YW50aXRhdGl2ZSBwcm90ZW9taWNzIGRhdGEgYW5hbHlzZXMuCkFjY29yZGluZyB0byB0aGUgdGFyZ2V0IGF1ZGllbmNlIG9mIHRoZSBjb3Vyc2Ugd2UgZWl0aGVyIHdvcmsgd2l0aCBhIGdyYXBoaWNhbCB1c2VyIGludGVyZmFjZSAoR1VJKSBpbiBhIFIvc2hpbnkgQXBwIG1zcXJvYjJndWkgKGUuZy4gUHJvdGVvbWljcyBCaW9pbmZvcm1hdGljcyBjb3Vyc2Ugb2YgdGhlIEVCSSBhbmQgdGhlIFByb3Rlb21pY3MgRGF0YSBBbmFseXNpcyBjb3Vyc2UgYXQgdGhlIEd1bGJlbmtpYW4gaW5zdGl0dXRlKSBvciB3aXRoIFIvbWFya2Rvd25zIHNjcmlwdHMgKGUuZy4gQmlvaW5mb3JtYXRpY3MgU3VtbWVyIFNjaG9vbCBhdCBVQ0xvdXZhaW4gb3IgdGhlIFN0YXRpc3RpY2FsIEdlbm9taWNzIENvdXJzZSBhdCBHaGVudCBVbml2ZXJzaXR5KS4gCgotLS0KCiMgSW50cm86IENoYWxsZW5nZXMgaW4gTGFiZWwtRnJlZSBRdWFudGl0YXRpdmUgUHJvdGVvbWljcwoKCjxpZnJhbWUgd2lkdGg9IjU2MCIgaGVpZ2h0PSIzMTUiCnNyYz0iaHR0cHM6Ly93d3cueW91dHViZS5jb20vZW1iZWQvM3dfSzRTX0dVekUiCmZyYW1lYm9yZGVyPSIwIgpzdHlsZT0iZGlzcGxheTogYmxvY2s7IG1hcmdpbjogYXV0bzsiCmFsbG93PSJhdXRvcGxheTsgZW5jcnlwdGVkLW1lZGlhIiBhbGxvd2Z1bGxzY3JlZW4+PC9pZnJhbWU+CgoKIyMgTVMtYmFzZWQgd29ya2Zsb3cKCmBgYHtyIGVjaG89RkFMU0V9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCIuL2ZpZ3VyZXMvUHJvdGVvbWljc1dvcmtmbG93LnBuZyIpCmBgYAogIAotIFBlcHRpZGUgQ2hhcmFjdGVyaXN0aWNzCiAgCiAgLSBNb2RpZmljYXRpb25zCiAgLSBJb25pc2F0aW9uIEVmZmljaWVuY3k6IGh1Z2UgdmFyaWFiaWxpdHkKICAtIElkZW50aWZpY2F0aW9uCiAgICAtIE1pc2lkZW50aWZpY2F0aW9uICRccmlnaHRhcnJvdyQgb3V0bGllcnMKICAgIC0gTVMkXjIkIHNlbGVjdGlvbiBvbiBwZXB0aWRlIGFidW5kYW5jZQogICAgLSBDb250ZXh0IGRlcGVuZGluZyBtaXNzaW5nbmVzcwogICAgLSBOb24tcmFuZG9tIG1pc3NpbmduZXNzCgokXHJpZ2h0YXJyb3ckIFVuYmFsYW5jZWQgcGVwaWRlIGlkZW50aWZpY2F0aW9ucyBhY3Jvc3Mgc2FtcGxlcyBhbmQgbWVzc3kgZGF0YQoKLS0tCgojIyBMZXZlbCBvZiBxdWFudGlmaWNhdGlvbgoKLSBNUy1iYXNlZCBwcm90ZW9taWNzIHJldHVybnMgcGVwdGlkZXM6IHBpZWNlcyBvZiBwcm90ZWlucwoKYGBge3IgZWNobz1GQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoIi4vZmlndXJlcy9jaGFsbGVuZ2VzX3BlcHRpZGVzLnBuZyIpCmBgYAoKLSBRdWFudGlmaWNhdGlvbiBjb21tb25seSByZXF1aXJlZCBvbiB0aGUgcHJvdGVpbiBsZXZlbAoKYGBge3IgZWNobz1GQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoIi4vZmlndXJlcy9jaGFsbGVuZ2VzX3Byb3RlaW5zLnBuZyIpCmBgYAoKLS0tCgojIyBMYWJlbC1mcmVlIFF1YW50aXRhdGl2ZSBQcm90ZW9taWNzIERhdGEgQW5hbHlzaXMgV29ya2Zsb3dzCgpgYGB7ciBlY2hvPUZBTFNFfQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiLi9maWd1cmVzL3Byb3Rlb21pY3NEYXRhQW5hbHlzaXMucG5nIikKYGBgCgotLS0KCiMjIENQVEFDIFNwaWtlLWluIFN0dWR5Cgo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkLzZWNUdpQTRDZzJVIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKYGBge3IgZWNobz1GQUxTRSwgb3V0LndpZHRoPSI1MCUifQprbml0cjo6aW5jbHVkZV9ncmFwaGljcygiLi9maWd1cmVzL2NwdGFjTGF5b3V0THVkZ2VyLnBuZyIpCmBgYAoKLSBTYW1lIHRyeXBzaW4tZGlnZXN0ZWQgeWVhc3QgcHJvdGVvbWUgYmFja2dyb3VuZCBpbiBlYWNoIHNhbXBsZQotIFRyeXBzaW4tZGlnZXN0ZWQgU2lnbWEgVVBTMSBzdGFuZGFyZDogNDggZGlmZmVyZW50IGh1bWFuIHByb3RlaW5zIHNwaWtlZCBpbiBhdCA1IGRpZmZlcmVudCBjb25jZW50cmF0aW9ucyAodHJlYXRtZW50IEEtRSkgCi0gU2FtcGxlcyByZXBlYXRlZGx5IHJ1biBvbiBkaWZmZXJlbnQgaW5zdHJ1bWVudHMgaW4gZGlmZmVyZW50IGxhYnMKLSBBZnRlciBNYXhRdWFudCBzZWFyY2ggd2l0aCBtYXRjaCBiZXR3ZWVuIHJ1bnMgb3B0aW9uCgogIC0gNDFcJSBvZiBhbGwgcHJvdGVpbnMgYXJlIHF1YW50aWZpZWQgaW4gYWxsIHNhbXBsZXMKICAtIDYuNlwlIG9mIGFsbCBwZXB0aWRlcyBhcmUgcXVhbnRpZmllZCBpbiBhbGwgc2FtcGxlcwoKJFxyaWdodGFycm93JCB2YXN0IGFtb3VudCBvZiBtaXNzaW5nbmVzcwoKCiMjIE1heHF1YW50IG91dHB1dAoKYGBge3IgZWNobz1GQUxTRX0Ka25pdHI6OmluY2x1ZGVfZ3JhcGhpY3MoIi4vZmlndXJlcy9tYXhxdWFudE91dHB1dERpci5wbmciKQpgYGAKCi0tLQoKCiMgSW1wb3J0IHRoZSBkYXRhIGluIFIgCgojIyBEYXRhIGluZnJhc3RydWN0dXJlCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGJhY2tncm91bmQgb24gZGF0YSBpbmZyYXN0cnVjdHVyZSB1c2VkIGluIFIgdG8gc3RvcmUgcHJvdGVvbWljcyBkYXRhIDwvc3VtbWFyeT48cD4KLSBXZSB1c2UgdGhlIGBRRmVhdHVyZXNgIHBhY2thZ2UgdGhhdCBwcm92aWRlcyB0aGUgaW5mcmFzdHJ1Y3R1cmUgdG8KICAtIHN0b3JlLCAgCiAgLSBwcm9jZXNzLCAKICAtIG1hbmlwdWxhdGUgYW5kIAogIC0gYW5hbHlzZSBxdWFudGl0YXRpdmUgZGF0YS9mZWF0dXJlcyBmcm9tIG1hc3Mgc3BlY3Ryb21ldHJ5CmV4cGVyaW1lbnRzLiAKCi0gSXQgaXMgYmFzZWQgb24gdGhlIGBTdW1tYXJpemVkRXhwZXJpbWVudGAgYW5kCmBNdWx0aUFzc2F5RXhwZXJpbWVudGAgY2xhc3Nlcy4gCgoKCmBgYHtyIGZpZy5jYXAgPSAiQ29uY2VwdHVhbCByZXByZXNlbnRhdGlvbiBvZiBhIGBTdW1tYXJpemVkRXhwZXJpbWVudGAgb2JqZWN0LiAgQXNzYXlzIGNvbnRhaW4gaW5mb3JtYXRpb24gb24gdGhlIG1lYXN1cmVkIG9taWNzIGZlYXR1cmVzIChyb3dzKSBmb3IgZGlmZmVyZW50IHNhbXBsZXMgKGNvbHVtbnMpLiBUaGUgYHJvd0RhdGFgIGNvbnRhaW5zIGluZm9ybWF0aW9uIG9uIHRoZSBvbWljcyBmZWF0dXJlcywgdGhlIGBjb2xEYXRhYCBjb250YWlucyBpbmZvcm1hdGlvbiBvbiB0aGUgc2FtcGxlcywgaS5lLiBleHBlcmltZW50YWwgZGVzaWduIGV0Yy4iLCBlY2hvPUZBTFNFLCBvdXQud2lkdGg9IjgwJSJ9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCIuL2ZpZ3VyZXMvU0UucG5nIikKYGBgCgotIEFzc2F5cyBpbiBhIFFGZWF0dXJlcyBvYmplY3QgaGF2ZSBhCmhpZXJhcmNoaWNhbCByZWxhdGlvbjogCiAgCiAgLSBwcm90ZWlucyBhcmUgY29tcG9zZWQgb2YgcGVwdGlkZXMsIAogIC0gdGhlbXNlbHZlcyBwcm9kdWNlZCBieSBzcGVjdHJhCiAgLSByZWxhdGlvbnMgYmV0d2VlbiBhc3NheXMgYXJlIHRyYWNrZWQgYW5kIHJlY29yZGVkIHRocm91Z2hvdXQgZGF0YSBwcm9jZXNzaW5nCgpgYGB7ciBmZWF0dXJlc3Bsb3QsIGZpZy5jYXAgPSAiQ29uY2VwdHVhbCByZXByZXNlbnRhdGlvbiBvZiBhIGBRRmVhdHVyZXNgIG9iamVjdCBhbmQgdGhlIGFnZ3JlZ2F0aXZlIHJlbGF0aW9uIGJldHdlZW4gZGlmZmVyZW50IGFzc2F5cy4iLCBlY2hvID0gRkFMU0V9CnBhcihtYXIgPSBjKDAsIDAsIDAsIDApKQpwbG90KE5BLCB4bGltID0gYygwLCAxMiksIHlsaW0gPSBjKDAsIDIwKSwKICAgICB4YXh0ID0gIm4iLCB5YXh0ID0gIm4iLAogICAgIHhsYWIgPSAiIiwgeWxhYiA9ICIiLCBidHkgPSAibiIpCmZvciAoaSBpbiAwOjcpCiAgICByZWN0KDAsIGksIDMsIGkrMSwgY29sID0gImxpZ2h0Z3JleSIsIGJvcmRlciA9ICJ3aGl0ZSIpCmZvciAoaSBpbiA4OjEyKQogICAgcmVjdCgwLCBpLCAzLCBpKzEsIGNvbCA9ICJzdGVlbGJsdWUiLCBib3JkZXIgPSAid2hpdGUiKQpmb3IgKGkgaW4gMTM6MTgpCiAgICByZWN0KDAsIGksIDMsIGkrMSwgY29sID0gIm9yYW5nZSIsIGJvcmRlciA9ICJ3aGl0ZSIpCmZvciAoaSBpbiAxOSkKICAgIHJlY3QoMCwgaSwgMywgaSsxLCBjb2wgPSAiZGFya2dyZXkiLCBib3JkZXIgPSAid2hpdGUiKQpmb3IgKGkgaW4gNTo3KQogICAgcmVjdCg1LCBpLCA4LCBpKzEsIGNvbCA9ICJsaWdodGdyZXkiLCBib3JkZXIgPSAid2hpdGUiKQpmb3IgKGkgaW4gODoxMCkKICAgIHJlY3QoNSwgaSwgOCwgaSsxLCBjb2wgPSAic3RlZWxibHVlIiwgYm9yZGVyID0gIndoaXRlIikKZm9yIChpIGluIDExOjEzKQogICAgcmVjdCg1LCBpLCA4LCBpKzEsIGNvbCA9ICJvcmFuZ2UiLCBib3JkZXIgPSAid2hpdGUiKQpmb3IgKGkgaW4gMTQpCiAgICByZWN0KDUsIGksIDgsIGkrMSwgY29sID0gImRhcmtncmV5IiwgYm9yZGVyID0gIndoaXRlIikKcmVjdCg5LCA4LCAxMiwgOCsxLCBjb2wgPSAibGlnaHRncmV5IiwgYm9yZGVyID0gIndoaXRlIikKcmVjdCg5LCA5LCAxMiwgOSsxLCBjb2wgPSAic3RlZWxibHVlIiwgYm9yZGVyID0gIndoaXRlIikKcmVjdCg5LCAxMCwgMTIsIDEwKzEsIGNvbCA9ICJvcmFuZ2UiLCBib3JkZXIgPSAid2hpdGUiKQpyZWN0KDksIDExLCAxMiwgMTErMSwgY29sID0gImRhcmtncmV5IiwgYm9yZGVyID0gIndoaXRlIikKc2VnbWVudHMoMywgOCwgNSwgOCwgbHR5ID0gImRhc2hlZCIpCnNlZ21lbnRzKDMsIDYsIDUsIDcsIGx0eSA9ICJkYXNoZWQiKQpzZWdtZW50cygzLCA0LCA1LCA2LCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMCwgNSwgNSwgbHR5ID0gImRhc2hlZCIpCnNlZ21lbnRzKDMsIDEwLCA1LCA5LCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMTEsIDUsIDEwLCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMTMsIDUsIDExLCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMTQsIDUsIDEyLCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMTYsIDUsIDEzLCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMTksIDUsIDE0LCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoMywgMjAsIDUsIDE1LCBsdHkgPSAiZGFzaGVkIikKc2VnbWVudHMoOCwgNSwgOSwgOCwgbHR5ID0gImRhc2hlZCIpCnNlZ21lbnRzKDgsIDgsIDksIDksIGx0eSA9ICJkYXNoZWQiKQpzZWdtZW50cyg4LCAxMSwgOSwgMTAsIGx0eSA9ICJkYXNoZWQiKQpzZWdtZW50cyg4LCAxNCwgOSwgMTEsIGx0eSA9ICJkYXNoZWQiKQpzZWdtZW50cyg4LCAxNSwgOSwgMTIsIGx0eSA9ICJkYXNoZWQiKQpgYGAKCjwvcD48L2RldGFpbHM+CgojIyBJbXBvcnQgZGF0YSBpbiBSCgoKIyMjIExvYWQgbGlicmFyaWVzIAoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIHRvIHNlZSBjb2RlIDwvc3VtbWFyeT48cD4KYGBge3IsIHdhcm5pbmc9RkFMU0UsIG1lc3NhZ2U9RkFMU0V9CmxpYnJhcnkodGlkeXZlcnNlKQpsaWJyYXJ5KGxpbW1hKQpsaWJyYXJ5KFFGZWF0dXJlcykKbGlicmFyeShtc3Fyb2IyKQpsaWJyYXJ5KHBsb3RseSkKbGlicmFyeShnZ3Bsb3QyKQpgYGAKPC9wPjwvZGV0YWlscz4KCiMjIyBSZWFkIGRhdGEgCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGJhY2tncm91bmQgYW5kIGNvZGUgPC9zdW1tYXJ5PjxwPgoxLiBXZSB1c2UgYSBwZXB0aWRlcy50eHQgZmlsZSBmcm9tIE1TLWRhdGEgcXVhbnRpZmllZCB3aXRoIG1heHF1YW50IHRoYXQgCmNvbnRhaW5zIE1TMSBpbnRlbnNpdGllcyBzdW1tYXJpemVkIGF0IHRoZSBwZXB0aWRlIGxldmVsLiAKYGBge3J9CnBlcHRpZGVzRmlsZSA8LSAiaHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL3N0YXRPbWljcy9QREEyMS9kYXRhL3F1YW50aWZpY2F0aW9uL2Z1bGxDcHRhY0RhdGFzU2V0Tm90Rm9yVHV0b3JpYWwvcGVwdGlkZXMudHh0IgpgYGAKCjIuIE1heHF1YW50IHN0b3JlcyB0aGUgaW50ZW5zaXR5IGRhdGEgZm9yIHRoZSBkaWZmZXJlbnQgc2FtcGxlcyBpbiBjb2x1bW5ucyB0aGF0IHN0YXJ0IHdpdGggSW50ZW5zaXR5LiBXZSBjYW4gcmV0cmVpdmUgdGhlIGNvbHVtbiBuYW1lcyB3aXRoIHRoZSBpbnRlbnNpdHkgZGF0YSB3aXRoIHRoZSBjb2RlIGJlbG93OiAKCmBgYHtyfQplY29scyA8LSBncmVwKCJJbnRlbnNpdHlcXC4iLCBuYW1lcyhyZWFkLmRlbGltKHBlcHRpZGVzRmlsZSkpKQpgYGAKCjMuIFJlYWQgdGhlIGRhdGEgYW5kIHN0b3JlIGl0IGluICBRRmVhdHVyZXMgb2JqZWN0IAoKYGBge3J9CnBlIDwtIHJlYWRRRmVhdHVyZXMoCiAgdGFibGUgPSBwZXB0aWRlc0ZpbGUsCiAgZm5hbWVzID0gMSwKICBlY29sID0gZWNvbHMsCiAgbmFtZSA9ICJwZXB0aWRlUmF3Iiwgc2VwPSJcdCIpCmBgYAo8L3A+PC9kZXRhaWxzPgoKIyMjIEV4cGxvcmUgb2JqZWN0Cgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGJhY2tncm91bmQgYW5kIGNvZGUgPC9zdW1tYXJ5PjxwPgotIFRoZSByb3dEYXRhIGNvbnRhaW5zIGluZm9ybWF0aW9uIG9uIHRoZSBmZWF0dXJlcyAocGVwdGlkZXMpIGluIHRoZSBhc3NheS4gRS5nLiBTZXF1ZW5jZSwgcHJvdGVpbiwgLi4uCgpgYGB7cn0Kcm93RGF0YShwZVtbInBlcHRpZGVSYXciXV0pCmBgYAoKLSBUaGUgY29sRGF0YSBjb250YWlucyBpbmZvcm1hdGlvbiBvbiB0aGUgc2FtcGxlcwoKYGBge3J9IApjb2xEYXRhKHBlKQpgYGAKCi0gTm8gaW5mb3JtYXRpb24gaXMgc3RvcmVkIHlldCBvbiB0aGUgZGVzaWduLiAKCgpgYGB7cn0gCnBlICU+JSBjb2xuYW1lcwpgYGAKCi0gTm90ZSwgdGhhdCB0aGUgc2FtcGxlIG5hbWVzIGluY2x1ZGUgdGhlIHNwaWtlLWluIGNvbmRpdGlvbi4gCi0gVGhleSBhbHNvIGVuZCBvbiBhIG51bWJlci4gCiAgCiAgLSAxLTMgaXMgZnJvbSBsYWIgMSwgCiAgLSA0LTYgZnJvbSBsYWIgMiBhbmQgCiAgLSA3LTkgZnJvbSBsYWIgMy4gCgotIFdlIHVwZGF0ZSB0aGUgY29sRGF0YSB3aXRoIGluZm9ybWF0aW9uIG9uIHRoZSBkZXNpZ24KCmBgYHtyfQpjb2xEYXRhKHBlKSRsYWIgPC0gcmVwKHJlcChwYXN0ZTAoImxhYiIsMTozKSxlYWNoPTMpLDUpICU+JSBhcy5mYWN0b3IKY29sRGF0YShwZSkkY29uZGl0aW9uIDwtIHBlW1sicGVwdGlkZVJhdyJdXSAlPiUgY29sbmFtZXMgJT4lIHN1YnN0cigxMiwxMikgJT4lIGFzLmZhY3Rvcgpjb2xEYXRhKHBlKSRzcGlrZUNvbmNlbnRyYXRpb24gPC0gcmVwKGMoQSA9IDAuMjUsIEIgPSAwLjc0LCBDID0gMi4yMiwgRCA9IDYuNjcsIEUgPSAyMCksZWFjaCA9IDkpCmBgYAoKLSBXZSBleHBsb3JlIHRoZSBjb2xEYXRhIGFnYWluCgpgYGB7cn0KY29sRGF0YShwZSkKYGBgCgo8L3A+PC9kZXRhaWxzPgoKLS0tCgojIFByZXByb2Nlc3NpbmcKCiMjIExvZy10cmFuc2Zvcm1hdGlvbgoKPGlmcmFtZSB3aWR0aD0iNTYwIiBoZWlnaHQ9IjMxNSIKc3JjPSJodHRwczovL3d3dy55b3V0dWJlLmNvbS9lbWJlZC9BUHNkU2J1MVJCTSIKZnJhbWVib3JkZXI9IjAiCnN0eWxlPSJkaXNwbGF5OiBibG9jazsgbWFyZ2luOiBhdXRvOyIKYWxsb3c9ImF1dG9wbGF5OyBlbmNyeXB0ZWQtbWVkaWEiIGFsbG93ZnVsbHNjcmVlbj48L2lmcmFtZT4KCiMjIyBFeHBsb3JlIHRoZSBkYXRhIHdpdGggcGxvdHMKClBlcHRpZGUgQUFMRUVMVksgZnJvbSBzcGlrZWQtaW4gVVBTIHByb3RlaW4gUDEyMDgxLiAKV2Ugb25seSBzaG93IGRhdGEgZnJvbSBsYWIxLgoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIHRvIHNlZSBjb2RlIHRvIG1ha2UgcGxvdCA8L3N1bW1hcnk+PHA+CmBgYHtyfQpzdWJzZXQgPC0gcGVbIkFBTEVFTFZLIixjb2xEYXRhKHBlKSRsYWI9PSJsYWIxIl0KcGxvdFdoeUxvZyA8LSBkYXRhLmZyYW1lKGNvbmNlbnRyYXRpb24gPSBjb2xEYXRhKHN1YnNldCkkc3Bpa2VDb25jZW50cmF0aW9uLAogICAgICAgICAgIHkgPSBhc3NheShzdWJzZXRbWyJwZXB0aWRlUmF3Il1dKSAlPiUgYwogICAgICAgICAgICkgJT4lIAogIGdncGxvdChhZXMoY29uY2VudHJhdGlvbiwgeSkpICsKICBnZW9tX3BvaW50KCkgKwogIHhsYWIoImNvbmNlbnRyYXRpb24gKGZtb2wvbCkiKSArCiAgZ2d0aXRsZSgicGVwdGlkZSBBQUxFRUxWSyBpbiBsYWIxIikKYGBgCjwvcD48L2RldGFpbHM+CgpgYGB7cn0KcGxvdFdoeUxvZwpgYGAKCi0gVmFyaWFuY2UgaW5jcmVhc2VzIHdpdGggdGhlIG1lYW4KJFxyaWdodGFycm93JCBNdWx0aXBsaWNhdGl2ZSBlcnJvciBzdHJ1Y3R1cmUgCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGNvZGUgdG8gbWFrZSBwbG90IDwvc3VtbWFyeT48cD4KYGBge3J9CnBsb3RMb2cgPC0gZGF0YS5mcmFtZShjb25jZW50cmF0aW9uID0gY29sRGF0YShzdWJzZXQpJHNwaWtlQ29uY2VudHJhdGlvbiwKICAgICAgICAgICB5ID0gYXNzYXkoc3Vic2V0W1sicGVwdGlkZVJhdyJdXSkgJT4lIGMKICAgICAgICAgICApICU+JSAKICBnZ3Bsb3QoYWVzKGNvbmNlbnRyYXRpb24sIHkpKSArCiAgZ2VvbV9wb2ludCgpICsgCiAgc2NhbGVfeF9jb250aW51b3VzKHRyYW5zPSdsb2cyJykgKyAKICBzY2FsZV95X2NvbnRpbnVvdXModHJhbnM9J2xvZzInKSArCiAgeGxhYigiY29uY2VudHJhdGlvbiAoZm1vbC9sKSIpICsKICBnZ3RpdGxlKCJwZXB0aWRlIEFBTEVFTFZLIGluIGxhYjEgd2l0aCBheGVzIG9uIGxvZyBzY2FsZSIpCmBgYAo8L3A+PC9kZXRhaWxzPgoKYGBge3J9CnBsb3RMb2cKYGBgCgotIERhdGEgc2VlbXMgdG8gYmUgaG9tb3NjZWRhc3RpYyBvbiBsb2ctc2NhbGUgJFxyaWdodGFycm93JCBsb2cgdHJhbnNmb3JtYXRpb24gb2YgdGhlIGludGVuc2l0eSBkYXRhCi0gSW4gcXVhbnRpdGF0aXZlIHByb3Rlb21pY3MgYW5hbHlzaXMgb24gJFxsb2dfMiQgCgokXHJpZ2h0YXJyb3ckIERpZmZlcmVuY2VzIG9uIGEgJFxsb2dfMiQgc2NhbGU6ICRcbG9nXzIkIGZvbGQgY2hhbmdlcwoKJCQKXGxvZ18yIEIgLSBcbG9nXzIgQSA9IFxsb2dfMiBcZnJhY3tCfXtBfSA9IFxsb2cgRkNfXHRleHR7QiAtIEF9CiQkCiQkIApcYmVnaW57YXJyYXl9IHtsfQpsb2dfMiBGQyA9IDEgXHJpZ2h0YXJyb3cgRkMgPSAyXjEgPTJcXApsb2dfMiBGQyA9IDIgXHJpZ2h0YXJyb3cgRkMgPSAyXjIgPSA0XFwKXGVuZHthcnJheX0KJCQKCgojIyMgbG9nLXRyYW5zZm9ybWF0aW9uIG9mIHRoZSBkYXRhIAoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIHRvIHNlZSBjb2RlIHRvIGxvZy10cmFuc2Zyb20gdGhlIGRhdGEgPC9zdW1tYXJ5PjxwPgotIFdlIGNhbGN1bGF0ZSBob3cgbWFueSBub24gemVybyBpbnRlbnNpdGllcyB3ZSBoYXZlIGZvciBlYWNoIHBlcHRpZGUgYW5kIHRoaXMgY2FuIGJlIHVzZWZ1bCBmb3IgZmlsdGVyaW5nLgoKYGBge3J9CnJvd0RhdGEocGVbWyJwZXB0aWRlUmF3Il1dKSRuTm9uWmVybyA8LSByb3dTdW1zKGFzc2F5KHBlW1sicGVwdGlkZVJhdyJdXSkgPiAwKQpgYGAKCgotIFBlcHRpZGVzIHdpdGggemVybyBpbnRlbnNpdGllcyBhcmUgbWlzc2luZyBwZXB0aWRlcyBhbmQgc2hvdWxkIGJlIHJlcHJlc2VudAp3aXRoIGEgYE5BYCB2YWx1ZSByYXRoZXIgdGhhbiBgMGAuCgpgYGB7cn0KcGUgPC0gemVyb0lzTkEocGUsICJwZXB0aWRlUmF3IikgIyBjb252ZXJ0IDAgdG8gTkEKYGBgCgotIExvZ3RyYW5zZm9ybSBkYXRhIHdpdGggYmFzZSAyCgpgYGB7cn0KcGUgPC0gbG9nVHJhbnNmb3JtKHBlLCBiYXNlID0gMiwgaSA9ICJwZXB0aWRlUmF3IiwgbmFtZSA9ICJwZXB0aWRlTG9nIikKYGBgCjwvcD48L2RldGFpbHM+CgotLS0KCiMjIEZpbHRlcmluZwoKPGlmcmFtZSB3aWR0aD0iNTYwIiBoZWlnaHQ9IjMxNSIKc3JjPSJodHRwczovL3d3dy55b3V0dWJlLmNvbS9lbWJlZC8zelZGcWRUbEJfYyIKZnJhbWVib3JkZXI9IjAiCnN0eWxlPSJkaXNwbGF5OiBibG9jazsgbWFyZ2luOiBhdXRvOyIKYWxsb3c9ImF1dG9wbGF5OyBlbmNyeXB0ZWQtbWVkaWEiIGFsbG93ZnVsbHNjcmVlbj48L2lmcmFtZT4KCi0gUmV2ZXJzZSBzZXF1ZW5jZXMKLSBPbmx5IGlkZW50aWZpZWQgYnkgbW9kaWZpY2F0aW9uIHNpdGUgKG9ubHkgbW9kaWZpZWQgcGVwdGlkZXMgZGV0ZWN0ZWQpCi0gUmF6b3IgcGVwdGlkZXM6IG5vbi11bmlxdWUgcGVwdGlkZXMgYXNzaWduZWQgdG8gdGhlIHByb3RlaW4gZ3JvdXAgd2l0aCB0aGUgbW9zdCBvdGhlciBwZXB0aWRlcyAKLSBDb250YW1pbmFudHMKLSBQZXB0aWRlcyBmZXcgaWRlbnRpZmljYXRpb25zCi0gUHJvdGVpbnMgdGhhdCBhcmUgb25seSBpZGVudGlmaWVkIHdpdGggb25lIG9yIGEgZmV3IHBlcHRpZGVzCgpGaWx0ZXJpbmcgZG9lcyBub3QgaW5kdWNlIGJpYXMgaWYgdGhlIGNyaXRlcmlvbiBpcyBpbmRlcGVuZGVudCBmcm9tIHRoZSBkb3duc3RyZWFtIGRhdGEgYW5hbHlzaXMhCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGNvZGUgdG8gZmlsdGVyIHRoZSBkYXRhIDwvc3VtbWFyeT48cD4KCjEuIEhhbmRsaW5nIG92ZXJsYXBwaW5nIHByb3RlaW4gZ3JvdXBzCgpJbiBvdXIgYXBwcm9hY2ggYSBwZXB0aWRlIGNhbiBtYXAgdG8gbXVsdGlwbGUgcHJvdGVpbnMsIGFzIGxvbmcgYXMgdGhlcmUgaXMKbm9uZSBvZiB0aGVzZSBwcm90ZWlucyBwcmVzZW50IGluIGEgc21hbGxlciBzdWJncm91cC4KCmBgYHtyfQpwZSA8LSBmaWx0ZXJGZWF0dXJlcyhwZSwgfiBQcm90ZWlucyAlaW4lIHNtYWxsZXN0VW5pcXVlR3JvdXBzKHJvd0RhdGEocGVbWyJwZXB0aWRlTG9nIl1dKSRQcm90ZWlucykpCmBgYAoKMi4gUmVtb3ZlIHJldmVyc2Ugc2VxdWVuY2VzIChkZWNveXMpIGFuZCBjb250YW1pbmFudHMKCldlIG5vdyByZW1vdmUgdGhlIGNvbnRhbWluYW50cywgcGVwdGlkZXMgdGhhdCBtYXAgdG8gZGVjb3kgc2VxdWVuY2VzLCBhbmQgcHJvdGVpbnMKd2hpY2ggd2VyZSBvbmx5IGlkZW50aWZpZWQgYnkgcGVwdGlkZXMgd2l0aCBtb2RpZmljYXRpb25zLgoKYGBge3J9CnBlIDwtIGZpbHRlckZlYXR1cmVzKHBlLH5SZXZlcnNlICE9ICIrIikKcGUgPC0gZmlsdGVyRmVhdHVyZXMocGUsfiBQb3RlbnRpYWwuY29udGFtaW5hbnQgIT0gIisiKQpgYGAKCjMuIERyb3AgcGVwdGlkZXMgdGhhdCB3ZXJlIG9ubHkgaWRlbnRpZmllZCBpbiBvbmUgc2FtcGxlCgpXZSBrZWVwIHBlcHRpZGVzIHRoYXQgd2VyZSBvYnNlcnZlZCBhdCBsYXN0IHR3aWNlLgoKYGBge3J9CnBlIDwtIGZpbHRlckZlYXR1cmVzKHBlLH4gbk5vblplcm8gPj0yKQpucm93KHBlW1sicGVwdGlkZUxvZyJdXSkKYGBgCgpXZSBrZWVwIGByIG5yb3cocGVbWyJwZXB0aWRlTG9nIl1dKWAgcGVwdGlkZXMgdXBvbiBmaWx0ZXJpbmcuCjwvcD48L2RldGFpbHM+CgotLS0KCiMjIE5vcm1hbGl6YXRpb24gCgo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkLzlxQ0pFa2RTU2hJIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIHRvIHNlZSBjb2RlIHRvIG1ha2UgcGxvdCA8L3N1bW1hcnk+PHA+CgpgYGB7cn0KZGVuc2l0eUNvbmRpdGlvbkQgPC0gcGVbWyJwZXB0aWRlTG9nIl1dWyxjb2xEYXRhKHBlKSRjb25kaXRpb249PSJEIl0gJT4lIAogIGFzc2F5ICU+JQogIGFzLmRhdGEuZnJhbWUoKSAlPiUKICBnYXRoZXIoc2FtcGxlLCBpbnRlbnNpdHkpICU+JSAKICBtdXRhdGUobGFiID0gY29sRGF0YShwZSlbc2FtcGxlLCJsYWIiXSkgJT4lCiAgZ2dwbG90KGFlcyh4PWludGVuc2l0eSxncm91cD1zYW1wbGUsY29sb3I9bGFiKSkgKyAKICAgIGdlb21fZGVuc2l0eSgpICsKICAgIGdndGl0bGUoImNvbmRpdGlvbiBEIikKCmRlbnNpdHlMYWIyIDwtIHBlW1sicGVwdGlkZUxvZyJdXVssY29sRGF0YShwZSkkbGFiPT0ibGFiMiJdICU+JSAKICBhc3NheSAlPiUKICBhcy5kYXRhLmZyYW1lKCkgJT4lCiAgZ2F0aGVyKHNhbXBsZSwgaW50ZW5zaXR5KSAlPiUgCiAgbXV0YXRlKGNvbmRpdGlvbiA9IGNvbERhdGEocGUpW3NhbXBsZSwiY29uZGl0aW9uIl0pICU+JQogIGdncGxvdChhZXMoeD1pbnRlbnNpdHksZ3JvdXA9c2FtcGxlLGNvbG9yPWNvbmRpdGlvbikpICsgCiAgICBnZW9tX2RlbnNpdHkoKSArCiAgICBnZ3RpdGxlKCJsYWIyIikKYGBgCjwvcD48L2RldGFpbHM+CgpgYGB7cn0KZGVuc2l0eUNvbmRpdGlvbkQKYGBgCgpgYGB7cn0KZGVuc2l0eUxhYjIKYGBgCi0gRXZlbiBpbiB2ZXJ5IGNsZWFuIHN5bnRoZXRpYyBkYXRhc2V0IChzYW1lIGJhY2tncm91bmQsIG9ubHkgNDggVVBTCnByb3RlaW5zIGNhbiBiZSBkaWZmZXJlbnQpIHRoZSBtYXJnaW5hbCBwZXB0aWRlIGludGVuc2l0eSBkaXN0cmlidXRpb24KYWNyb3NzIHNhbXBsZXMgY2FuIGJlIHF1aXRlIGRpc3RpbmN0CgotIENvbnNpZGVyYWJsZSBlZmZlY3RzIGJldHdlZW4gYW5kIHdpdGhpbiBsYWJzIGZvciByZXBsaWNhdGUgc2FtcGxlcwotIENvbnNpZGVyYWJsZSBlZmZlY3RzIGJldHdlZW4gc2FtcGxlcyB3aXRoIGRpZmZlcmVudCBzcGlrZS1pbgpjb25jZW50cmF0aW9uCgokXHJpZ2h0YXJyb3ckIE5vcm1hbGl6YXRpb24gaXMgbmVlZGVkCgotLS0KCgojIyMgTWVhbiBvciBtZWRpYW4/CgotIE1pbGxlciBhbmQgRmlzaGtpbiAoMTk5NykgcmVwb3J0ZWQgdGhhdCBvdmVyIGEgcGVyaW9kIG9mIDMwIHllYXJzIG1hbGVzIHdvdWxkIGxpa2UgdG8gaGF2ZSBvbiBhdmVyYWdlIDY0LjMgcGFydG5lcnMgYW5kIGZlbWFsZXMgMi44LiAKCgo8ZGV0YWlscz48c3VtbWFyeT4gPC9zdW1tYXJ5PjxwPgotIE1pbGxlciBhbmQgRmlzaGtpbiAoMTk5NykgcmVwb3J0ZWQgdGhhdCB0aGUgbWVkaWFuIG51bWJlciBvZiBwYXJ0bmVycyBzb21lb25lIHdvdWxkIGxpa2UgdG8gaGF2ZSBvdmVyIGEgcGVyaW9kIG9mIDMwIHllYXJzIG1hbGVzIGlzIDEgZm9yIGJvdGggbWFsZXMgYW5kIGZlbWFsZXMuIAo8L3A+PC9kZXRhaWxzPgoKPGRldGFpbHM+PHN1bW1hcnk+IDwvc3VtbWFyeT48cD4KTWVhbiBpcyB2ZXJ5IHNlbnNpdGl2ZSB0byBvdXRsaWVycyEKCmBgYHtyIGVjaG89RkFMU0V9CmtuaXRyOjppbmNsdWRlX2dyYXBoaWNzKCIuL2ZpZ3VyZXMvcGFydG5lcnMucG5nIikKYGBgCjwvcD48L2RldGFpbHM+CgoKLS0tCgojIyMgTm9ybWFsaXphdGlvbiBvZiB0aGUgZGF0YSBieSBtZWRpYW4gY2VudGVyaW5nCgokJHlfe2lwfV5cdGV4dHtub3JtfSA9IHlfe2lwfSAtIFxoYXRcbXVfaSQkIAp3aXRoICRcaGF0XG11X2kkIHRoZSBtZWRpYW4gaW50ZW5zaXR5IG92ZXIgYWxsIG9ic2VydmVkIHBlcHRpZGVzIGluIHNhbXBsZSAkaSQuCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIFItY29kZSB0byBub3JtYWxpemUgdGhlIGRhdGEgPC9zdW1tYXJ5PjxwPgpgYGB7cn0KcGUgPC0gbm9ybWFsaXplKHBlLCAKICAgICAgICAgICAgICAgIGkgPSAicGVwdGlkZUxvZyIsIAogICAgICAgICAgICAgICAgbmFtZSA9ICJwZXB0aWRlTm9ybSIsIAogICAgICAgICAgICAgICAgbWV0aG9kID0gImNlbnRlci5tZWRpYW4iKQpgYGAKPC9wPjwvZGV0YWlscz4KCiMjIyBQbG90cyBvZiBub3JtYWxpemVkIGRhdGEKCgo8ZGV0YWlscz48c3VtbWFyeT4gQ2xpY2sgdG8gc2VlIGNvZGUgdG8gbWFrZSBwbG90IDwvc3VtbWFyeT48cD4KYGBge3J9CmRlbnNpdHlDb25kaXRpb25ETm9ybSA8LSBwZVtbInBlcHRpZGVOb3JtIl1dWyxjb2xEYXRhKHBlKSRjb25kaXRpb249PSJEIl0gJT4lIAogIGFzc2F5ICU+JQogIGFzLmRhdGEuZnJhbWUoKSAlPiUKICBnYXRoZXIoc2FtcGxlLCBpbnRlbnNpdHkpICU+JSAKICBtdXRhdGUobGFiID0gY29sRGF0YShwZSlbc2FtcGxlLCJsYWIiXSkgJT4lCiAgZ2dwbG90KGFlcyh4PWludGVuc2l0eSxncm91cD1zYW1wbGUsY29sb3I9bGFiKSkgKyAKICAgIGdlb21fZGVuc2l0eSgpICsKICAgIGdndGl0bGUoImNvbmRpdGlvbiBEIikKCmRlbnNpdHlMYWIyTm9ybSA8LSBwZVtbInBlcHRpZGVOb3JtIl1dWyxjb2xEYXRhKHBlKSRsYWI9PSJsYWIyIl0gJT4lIAogIGFzc2F5ICU+JQogIGFzLmRhdGEuZnJhbWUoKSAlPiUKICBnYXRoZXIoc2FtcGxlLCBpbnRlbnNpdHkpICU+JSAKICBtdXRhdGUoY29uZGl0aW9uID0gY29sRGF0YShwZSlbc2FtcGxlLCJjb25kaXRpb24iXSkgJT4lCiAgZ2dwbG90KGFlcyh4PWludGVuc2l0eSxncm91cD1zYW1wbGUsY29sb3I9Y29uZGl0aW9uKSkgKyAKICAgIGdlb21fZGVuc2l0eSgpICsKICAgIGdndGl0bGUoImxhYjIiKQpgYGAKPC9wPjwvZGV0YWlscz4KCmBgYHtyfQpkZW5zaXR5Q29uZGl0aW9uRE5vcm0KYGBgCgpgYGB7cn0KZGVuc2l0eUxhYjJOb3JtCmBgYAoKLSBVcG9uIG5vcm1hbGl6YXRpb24gdGhlIG1hcmdpbmFsIGRpc3RyaWJ1dGlvbnMgb2YgdGhlIHBlcHRpZGUgaW50ZW5zaXRpZXMgYWNyb3NzIHNhbXBsZXMgYXJlIG11Y2ggbW9yZSBjb21wYXJhYmxlCi0gV2Ugc3RpbGwgc2VlIGRldmlhdGlvbnMKLSBUaGlzIGNhbiBiZSBkdWUgdG8gdGVjaG5pY2FsIHZhcmlhYmlsaXR5Ci0gSW4gbWljcm8tYXJyYXkgbGl0ZXJhdHVyZSwgcXVhbnRpbGUgbm9ybWFsaXNhdGlvbiBpcyB1c2VkIHRvIGZvcmNlIHRoZSBtZWRpYW4gYW5kIGFsbCBvdGhlciBxdWFudGlsZXMgdG8gYmUgZXF1YWwgYWNyb3NzIHNhbXBsZXMKLSBJbiBwcm90ZW9taWNzIHF1YW50aWxlIG5vcm1hbGlzYXRpb24gb2Z0ZW4gaW50cm9kdWNlcyBhcnRpZmFjdHMgZHVlIHRvIGEgZGlmZmVyZW5jZSBpbiBtaXNzaW5nIHBlcHRpZGVzIGFjcm9zcyBzYW1wbGVzIAotIE1vcmUgYWR2YW5jZWQgbWV0aG9kcyBzaG91bGQgYmUgZGV2ZWxvcGVkIGZvciBub3JtYWxpemluZyBwcm90ZW9taWNzIGRhdGEKLSBJZiB0aGVyZSBhcmUgZGlmZmVyZW5jZXMgaW4gdGhlIHdpZHRoIG9mIHRoZSBtYXJnaW5hbCBkaXN0cmlidXRpb25zIG9mIHRoZSBkYXRhIGFjcm9zcyBzYW1wbGVzLiBUaGV5IGNhbiBhbHNvIGJlIHN0YW5kYXJkaXplZCBieSB1c2luZyBhIHJvYnVzdCBlc3RpbWF0b3IgZm9yIGxvY2F0aW9uIGFuZCBzY2FsZSwgaS5lLiAKJCR5X3tpcH1eXHRleHR7bm9ybX0gPSBcZnJhY3t5X3tpcH0gLSBcbXVfaX17c19pfSQkIAoKLS0tCgojIyBTdW1tYXJpemF0aW9uIAoKPGlmcmFtZSB3aWR0aD0iNTYwIiBoZWlnaHQ9IjMxNSIKc3JjPSJodHRwczovL3d3dy55b3V0dWJlLmNvbS9lbWJlZC9OVVBKV1BKZ1FVYyIKZnJhbWVib3JkZXI9IjAiCnN0eWxlPSJkaXNwbGF5OiBibG9jazsgbWFyZ2luOiBhdXRvOyIKYWxsb3c9ImF1dG9wbGF5OyBlbmNyeXB0ZWQtbWVkaWEiIGFsbG93ZnVsbHNjcmVlbj48L2lmcmFtZT4KCi0gV2UgaWxsdXN0cmF0ZSBzdW1tYXJpemF0aW9uIGlzc3VlcyB1c2luZyBhIHN1YnNldCBvZiB0aGUgY3B0YWMgc3R1ZHkgKExhYiAyLCBjb25kaXRpb24gQSBhbmQgRSkgZm9yIGEgc3Bpa2VkIHByb3RlaW4gKFVQUyBQMTIwODEpLgoKPGRldGFpbHM+PHN1bW1hcnk+IENsaWNrIHRvIHNlZSBjb2RlIHRvIG1ha2UgcGxvdCA8L3N1bW1hcnk+PHA+CmBgYHtyIHBsb3QgPSBGQUxTRX0Kc3VtbWFyeVBsb3QgPC0gcGVbWyJwZXB0aWRlTm9ybSJdXVsKICAgIHJvd0RhdGEocGVbWyJwZXB0aWRlTm9ybSJdXSkkUHJvdGVpbnMgPT0gIlAxMjA4MXVwc3xTWUhDX0hVTUFOX1VQUyIsCiAgICBjb2xEYXRhKHBlKSRsYWI9PSJsYWIyIiZjb2xEYXRhKHBlKSRjb25kaXRpb24gJWluJSBjKCJBIiwiRSIpXSAlPiUKICBhc3NheSAlPiUKICBhcy5kYXRhLmZyYW1lICU+JQogIHJvd25hbWVzX3RvX2NvbHVtbih2YXIgPSAicGVwdGlkZSIpICU+JQogIGdhdGhlcihzYW1wbGUsIGludGVuc2l0eSwgLXBlcHRpZGUpICU+JSAKICBtdXRhdGUoY29uZGl0aW9uID0gY29sRGF0YShwZSlbc2FtcGxlLCJjb25kaXRpb24iXSkgJT4lCiAgZ2dwbG90KGFlcyh4ID0gcGVwdGlkZSwgeSA9IGludGVuc2l0eSwgY29sb3IgPSBzYW1wbGUsIGdyb3VwID0gc2FtcGxlLCBsYWJlbCA9IGNvbmRpdGlvbiksIHNob3cubGVnZW5kID0gRkFMU0UpICsKICBnZW9tX2xpbmUoc2hvdy5sZWdlbmQgPSBGQUxTRSkgKwogIGdlb21fdGV4dChzaG93LmxlZ2VuZCA9IEZBTFNFKSArCiAgdGhlbWVfbWluaW1hbCgpICsKICB0aGVtZShheGlzLnRleHQueCA9IGVsZW1lbnRfdGV4dChhbmdsZSA9IDkwLCB2anVzdCA9IDAuNSwgaGp1c3QgPSAxKSkgKwogIHhsYWIoIlBlcHRpZGUiKSArIAogIHlsYWIoIkludGVuc2l0eSAobG9nMikiKQpgYGAKPC9wPjwvZGV0YWlscz4KCmBgYHtyfQpzdW1tYXJ5UGxvdApgYGAKCldlIG9ic2VydmU6ICAKCi0gaW50ZW5zaXRpZXMgZnJvbSBtdWx0aXBsZSBwZXB0aWRlcyBmb3IgZWFjaCBwcm90ZWluIGluIGEgc2FtcGxlCi0gU3Ryb25nIHBlcHRpZGUgZWZmZWN0IAotVW5iYWxhbmNlZCBwZXB0aWRlIGlkZW50aWZpY2F0aW9uCi0gUHNldWRvLXJlcGxpY2F0aW9uOiBwZXB0aWRlIGludGVuc2l0aWVzIGZyb20gYSBwYXJ0aWN1bGFyIHByb3RlaW4gaW4gdGhlIHNhbWUgc2FtcGxlIGFyZSBjb3JyZWxhdGVkLCBpLmUuIHRoZXkgbW9yZSBhbGlrZSB0aGFuIHBlcHRpZGUgaW50ZW5zaXRpZXMgZnJvbSBhIHBhcnRpY3VsYXIgcHJvdGVpbiBiZXR3ZWVuIHNhbXBsZXMuIAoKCiRccmlnaHRhcnJvdyQgU3VtbWFyaXplIGFsbCBwZXB0aWRlIGludGVuc2l0aWVzIGZyb20gdGhlIHNhbWUgcHJvdGVpbiBpbiBhIHNhbXBsZSBpbnRvIGEgc2luZ2xlIHByb3RlaW4gZXhwcmVzc2lvbiB2YWx1ZQoKQ29tbW9ubHkgdXNlZCBtZXRob2RzIGFyZSAKCi0gTWVhbiBzdW1tYXJpemF0aW9uCiQkCnlfe2lwfT1cYmV0YV9pXlx0ZXh0e3NhbXB9ICsgXGVwc2lsb25fe2lwfQokJAoKLSBNZWRpYW4gc3VtbWFyaXphdGlvbgotIE1heHF1YW50J3MgbWF4TEZRIHN1bW1hcml6YXRpb24gKGluIHByb3RlaW4gZ3JvdXBzIGZpbGUpCi0gTW9kZWwgYmFzZWQgc3VtbWFyaXphdGlvbjogCiQkCnlfe2lwfT1cYmV0YV9pXlx0ZXh0e3NhbXB9ICsgXGJldGFfcF5cdGV4dHtwZXB9ICsgXGVwc2lsb25fe2lwfQokJAoKCjxkZXRhaWxzPjxzdW1tYXJ5PiBDbGljayB0byBzZWUgUi1jb2RlIHRvIG5vcm1hbGl6ZSB0aGUgZGF0YSA8L3N1bW1hcnk+PHA+CldlIHVzZSB0aGUgc3RhbmRhcmQgc3VtYXJpemF0aW9uIGluIGFnZ3JlZ2F0ZUZlYXR1cmVzLCB3aGljaCBpcwpyb2J1c3QgbW9kZWwgYmFzZWQgc3VtbWFyaXphdGlvbi4gCgpgYGB7cix3YXJuaW5nPUZBTFNFfQpwZSA8LSBhZ2dyZWdhdGVGZWF0dXJlcyhwZSwKICAgIGkgPSAicGVwdGlkZU5vcm0iLCAKICAgIGZjb2wgPSAiUHJvdGVpbnMiLCAKICAgIG5hLnJtID0gVFJVRSwKICAgIG5hbWUgPSAicHJvdGVpbiIpCmBgYAoKT3RoZXIgc3VtbWFyaXphdGlvbiBtZXRob2RzIGNhbiBiZSBpbXBsZW1lbnRlZCBieSB1c2luZyB0aGUgYGZ1bmAgYXJndW1lbnQgaW4gdGhlIGBhZ2dyZWdhdGVGZWF0dXJlc2AgZnVuY3Rpb24uIAoKLSBgZnVuID0gTXNDb3JlVXRpbHM6Om1lZGlhblBvbGlzaCgpYCB0byBmaXRzIGFuIGFkZGl0aXZlIG1vZGVsICh0d28gd2F5IGRlY29tcG9zaXRpb24pIHVzaW5nIFR1a2V5J3MgbWVkaWFuIHBvbGlzaF8gcHJvY2VkdXJlIHVzaW5nIHN0YXRzOjptZWRwb2xpc2goKQoKLSBgZnVuID0gTXNDb3JlVXRpbHM6OnJvYnVzdFN1bW1hcnkoKWAgdG8gY2FsY3VsYXRlIGEgcm9idXN0IGFnZ3JlZ2F0aW9uIHVzaW5nIE1BU1M6OnJsbSgpIChkZWZhdWx0KQoKLSBgZnVuID0gYmFzZTo6Y29sTWVhbnMoKWAgdG8gdXNlIHRoZSBtZWFuIG9mIGVhY2ggY29sdW1uCgotIGBmdW4gPSBtYXRyaXhTdGF0czo6Y29sTWVkaWFucygpYCB0byB1c2UgdGhlIG1lZGlhbiBvZiBlYWNoIGNvbHVtbgoKLSBgZnVuID0gYmFzZTo6Y29sU3VtcygpYCB0byB1c2UgdGhlIHN1bSBvZiBlYWNoIGNvbHVtbgoKPC9wPjwvZGV0YWlscz4KCi0tLQoKIyBFeGVyY2lzZQoKMS4gV2Ugd2lsbCBldmFsdWF0ZSBkaWZmZXJlbnQgc3VtbWFyaXphdGlvbiBtZXRob2RzIChNYXhxdWFudCBtYXhMRlEsIG1lZGlhbiBhbmQgcm9idXN0IG1vZGVsIGJhc2VkKSBpbiB0aGUgdHV0b3JpYWwgc2Vzc2lvbiBiZWZvcmUgZGlzY3Vzc2luZyBvbiB0aGVpciBhZHZhbnRhZ2VzL2Rpc2FkdmFudGFnZXMuCjIuIENhbiB5b3UgYW50aWNpcGF0ZSBvbiBwb3RlbnRpYWwgcHJvYmxlbXMgcmVsYXRlZCB0byB0aGUgc3VtbWFyaXphdGlvbj8KCi0tLQoKIyBTb2Z0d2FyZSAmIGNvZGUKCi0gT3VyIFIvQmlvY29uZHVjdG9yIHBhY2thZ2UgW21zcXJvYjJdKGh0dHBzOi8vd3d3LmJpb2NvbmR1Y3Rvci5vcmcvcGFja2FnZXMvcmVsZWFzZS9iaW9jL2h0bWwvbXNxcm9iMi5odG1sKSBjYW4gYmUgdXNlZCBpbiBSIG1hcmtkb3duIHNjcmlwdHMgb3Igd2l0aCBhIEdVSS9zaGlueUFwcCBpbiB0aGUgW21zcXJvYjJndWldKGh0dHBzOi8vZ2l0aHViLmNvbS9zdGF0T21pY3MvbXNxcm9iMmd1aSkgcGFja2FnZS4KCi0gVGhlIEdVSSBpcyBpbnRlbmRlZCBhcyBhIGludHJvZHVjdGlvbiB0byB0aGUga2V5IGNvbmNlcHRzIG9mIHByb3Rlb21pY3MgZGF0YSBhbmFseXNpcyBmb3IgdXNlcnMgd2hvIGhhdmUgbm8gZXhwZXJpZW5jZSBpbiBSLiAKCi0gSG93ZXZlciwgbGVhcm5pbmcgaG93IHRvIGNvZGUgZGF0YSBhbmFseXNlcyBpbiBSIG1hcmtkb3duIHNjcmlwdHMgaXMga2V5IGZvciBvcGVuIGVuIHJlcHJvZHVjaWJsZSBzY2llbmNlIGFuZCBmb3IgcmVwb3J0aW5nIHlvdXIgcHJvdGVvbWljcyBkYXRhIGFuYWx5c2VzIGFuZCBpbnRlcnByZXRhdGlvbiBpbiBhIHJlcHJvZHVjaWJsZSB3YXkuIAoKCi0gTW9yZSBpbmZvcm1hdGlvbiBvbiBvdXIgdG9vbHMgY2FuIGJlIGZvdW5kIGluIG91ciBwYXBlcnMgW0Bnb2VtaW5uZTIwMTZdLCBbQGdvZW1pbm5lMjAyMF0gYW5kIFtAc3RpY2tlcjIwMjBdLiBQbGVhc2UgcmVmZXIgdG8gb3VyIHdvcmsgd2hlbiB1c2luZyBvdXIgdG9vbHMuIAoKCgojIyBDb2RlIAoKMS4gRGF0YSBpbmZyYXN0cnVjdHVyZQo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkL04weUlaMVlCMmFRIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKMi4gSW1wb3J0IHByb3Rlb21pY3MgZGF0YQo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkL0RQSHRyR2o4c2trIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKMy4gUHJlcHJvY2Vzc2luZwoKICAgIC0gICBMb2ctdHJhbnNmb3JtYXRpb24KPGlmcmFtZSB3aWR0aD0iNTYwIiBoZWlnaHQ9IjMxNSIKc3JjPSJodHRwczovL3d3dy55b3V0dWJlLmNvbS9lbWJlZC9heERUMnRKTkx3TSIKZnJhbWVib3JkZXI9IjAiCnN0eWxlPSJkaXNwbGF5OiBibG9jazsgbWFyZ2luOiBhdXRvOyIKYWxsb3c9ImF1dG9wbGF5OyBlbmNyeXB0ZWQtbWVkaWEiIGFsbG93ZnVsbHNjcmVlbj48L2lmcmFtZT4KCiAgICAtIEZpbHRlcmluZwo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkL2NJd0F4Zko1Q0dvIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKICAgIC0gTm9ybWFsaXNhdGlvbgo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkLzlxQ0pFa2RTU2hJIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPgoKICAgIC0gU3VtbWFyaXphdGlvbgo8aWZyYW1lIHdpZHRoPSI1NjAiIGhlaWdodD0iMzE1IgpzcmM9Imh0dHBzOi8vd3d3LnlvdXR1YmUuY29tL2VtYmVkL2ZXcTRyTXhtSlRVIgpmcmFtZWJvcmRlcj0iMCIKc3R5bGU9ImRpc3BsYXk6IGJsb2NrOyBtYXJnaW46IGF1dG87IgphbGxvdz0iYXV0b3BsYXk7IGVuY3J5cHRlZC1tZWRpYSIgYWxsb3dmdWxsc2NyZWVuPjwvaWZyYW1lPiAgICAKCiMjIERhdGEgYW5hbHlzaXMgd2l0aCB0aGUgR1VJL3NoaW55QXBwIG1zcXJvYjJndWkKCjxpZnJhbWUgd2lkdGg9IjU2MCIgaGVpZ2h0PSIzMTUiCnNyYz0iaHR0cHM6Ly93d3cueW91dHViZS5jb20vZW1iZWQvRjVkMEUtVTRyT00iCmZyYW1lYm9yZGVyPSIwIgpzdHlsZT0iZGlzcGxheTogYmxvY2s7IG1hcmdpbjogYXV0bzsiCmFsbG93PSJhdXRvcGxheTsgZW5jcnlwdGVkLW1lZGlhIiBhbGxvd2Z1bGxzY3JlZW4+PC9pZnJhbWU+ICAgCgojIFJlZmVyZW5jZXM=