Data Partitioning

---

… or, Chopping our Data Into Pieces

Data Partitioning

---

flowchart LR
  A[(All data)] --> B[/Training/]
  A --> C[/Testing/]
  B --> D[Resample]
  D --> D1[/Analysis/]
  D --> D2[/Assessment/]
  D1 --> E{{Logistic Regr}}
  D1 --> F{{Decision Tree}}
  D1 --> G{{Random Forest}}
  E --> H{Select a Model}
  F --> H
  G --> H
  D2 ---> H
  H --> I[Final Model Fit]
  B --> I
  I --> J([Verify Model Performance])
  C ------> J
  
  classDef dataNode fill:#4a90d9,stroke:#2c5f8a,color:#fff
  classDef modelNode fill:#e8f5e9,stroke:#2e7d32,color:#000
  classDef decisionNode fill:#fff3e0,stroke:#e65100,color:#000
  classDef box fill:#eeeeee,stroke:#eeeeee,color:#000
  classDef hidden display:none

  class A,B,C,D1,D2 dataNode
  class E,F,G modelNode
  class H decisionNode
  class LoopBox box

Training on Data

A model is a mathematical function with adjustable “knobs” (parameters)

Training the model involves showing it a lot of data, from which it optimizes settings of the “knobs” to best represent the underlying patterns, weights, and relationships

flowchart LR
  A(Input) --> B(Model)
  B --> C(Prediction)
  C --> D(Error)
  D --> E{Adjust} --> B

What If the Model Just… Memorizes?

Overfitting: A model that aces every training example but fails on new data

Analogy: That student who memorizes every past exam question but can’t answer a novel problem

What If the Model Just… Memorizes?

Overfitting: A model that aces every training example but fails on new data

Analogy: That student who memorizes every past exam question but can’t answer a novel problem

Partitioning Your Data

• Training* set - what we use to develop our model (~85%)
  • Analysis* set - what the model learns from (~70%)
  • Assessment* set - for model tuning, evaluation, selection (~15%)
• Test set - a sealed envelope, opened only once to test performance of the final model (~15%)

flowchart LR
  A[(All data)] --> B[/Training/]
  A --> C[/Testing/]
  B --> D[Resample]
  D --> D1[/Analysis/]
  D --> D2[/Assessment/]
  
  classDef dataNode fill:#4a90d9,stroke:#2c5f8a,color:#fff
  classDef modelNode fill:#e8f5e9,stroke:#2e7d32,color:#000
  classDef decisionNode fill:#fff3e0,stroke:#e65100,color:#000
  classDef hidden display:none

  class A,B,C,D1,D2 dataNode
  class E,F,G modelNode
  class H decisionNode

Analogy: Assessment = practice exams; Test = the real exam, taken once

* NOTE: Different sources use different terms for these partitions; for example, some call them “development” (for “training”), “training” (for “analysis”), and “validation” (for “assessment”) sets. The tidymodels documentation uses “testing”, “analysis”, and “assessment” so that’s what we’re using here!

A model is only as good as its ability to predict data it hasn’t yet seen.

Time to Make Some Models!

We may want to test different models: different sets of predictors, different algorithms

We train each model using the Analysis data and then compare each model’s predictive power using the Assessment set

flowchart LR
  A[(All data)] --> B[/Training/]
  A --> C[/Testing/]
  B --> D[Resample]
  D --> D1[/Analysis/]
  D --> D2[/Assessment/]
  D1 --> E{{Logistic Regr}}
  D1 --> F{{Decision Tree}}
  D1 --> G{{Random Forest}}
  E --> H{Select a Model}
  F --> H
  G --> H
  D2 ---> H

  classDef dataNode fill:#4a90d9,stroke:#2c5f8a,color:#fff
  classDef modelNode fill:#e8f5e9,stroke:#2e7d32,color:#000
  classDef decisionNode fill:#fff3e0,stroke:#e65100,color:#000
  classDef box fill:#ffffff,stroke:#ffffff,color:#ffffff
  classDef hidden display:none

  class A,B,C,D1,D2 dataNode
  class E,F,G modelNode
  class H decisionNode
  class LoopBox box

Model Selection!

After identifying the model structure with the best predictive performance, we use the entire Training set to fit the model one last time, to fine-tune model parameters…

And at long last, we use the Test set to understand how well our final model performs.

flowchart LR
  A[(All data)] --> B[/Training/]
  A --> C[/Testing/]
  B --> D[Resample]
  D --> D1[/Analysis/]
  D --> D2[/Assessment/]
  D1 --> E{{Logistic Regr}}
  D1 --> F{{Decision Tree}}
  D1 --> G{{Random Forest}}
  E --> H{Select a Model}
  F --> H
  G --> H
  D2 ---> H
  H --> I[Final Model Fit]
  B --> I
  I --> J([Verify Model Performance])
  C ------> J
  
  classDef dataNode fill:#4a90d9,stroke:#2c5f8a,color:#fff
  classDef modelNode fill:#e8f5e9,stroke:#2e7d32,color:#000
  classDef decisionNode fill:#fff3e0,stroke:#e65100,color:#000
  classDef box fill:#eeeeee,stroke:#eeeeee,color:#000
  classDef hidden display:none

  class A,B,C,D1,D2 dataNode
  class E,F,G modelNode
  class H decisionNode
  class LoopBox box

… The Result: Fame and Fortune!