Similar presentations:
Topic10_Case_Group3_Read_the_Map
1.
iTPUGroup 3: Read the Map
Three real charts - find the sweet spot, the traps and the next step
Breakout room 3 | 12 minutes work + 3 minutes presenting | Real Titanic results
2.
iTPUGROUP 3 | MISSION
Your mission, roles and timing
You are map readers. Each chart is a 'map' of real tuning results. Find the best place, the dangerous places, and
decide where you would search NEXT.
Driver
Speaker
Checker
Time keeper
shares the slide / writes the
report
presents for 1 minute at the
end
checks every number and
direction
watches the clock, keeps the
pace
0-2 min: how to read a map
2-10 min: maps 1, 2 and 3
10-12 min: fill in the report
All maps use 5-fold CV accuracy on the 712 Titanic training passengers. Baseline (always 'died'): 61.7%.
Topic 10 | Breakout room 3 | Read the Map
2
3.
iTPUGROUP 3 | HOW TO READ A MAP
Three rules for reading tuning charts
1. Find the peak
2. Measure the gap
3. Look at the edges
the best CV score - NOT the best
train score
train minus CV. A big gap =
overfitting
the baseline value = the model
learned nothing; a best value at
the edge = search further
Write your answers with NUMBERS from the chart: e.g. "at C = 10 and gamma = 0.1 the CV is ..."
Topic 10 | Breakout room 3 | Read the Map
3
4.
iTPUGROUP 3 | MAP 1
Map 1: SVM - train accuracy vs CV accuracy
TRAIN accuracy %
READ THE MAP
CV accuracy % (5-fold)
gamma ->
0.001
0.01
0.1
1
10
C=0.01
61.7
61.7
61.7
61.7
61.7
C=0.1
61.7
73.9
81.0
77.6
62.3
gamma ->
0.001
0.01
0.1
1
10
C=0.01
61.7
61.7
61.7
61.7
61.7
C=0.1
61.7
73.5
80.3
74.5
61.7
C=1
76.2
79.5
83.7
88.6
92.2
C=1
76.0
79.4
82.3
81.6
72.8
C=10
78.9
80.9
85.8
90.5
94.1
C=10
78.9
80.9
83.2
79.6
72.6
C=100
79.4
84.2
88.4
92.6
95.8
C=100
79.4
82.6
80.8
77.5
70.2
C=1000
81.0
85.0
89.6
94.0
96.4
C=1000
80.9
82.0
79.5
75.4
68.7
1
Where is the best CV score? Give C,
gamma and the value.
2
Find the cell with the BIGGEST train CV gap. What is happening there?
3
The whole C = 0.01 row shows the
same number. What is the model
doing?
4
You may search 9 more settings. Which
C and gamma values would you try?
darker = higher
Topic 10 | Breakout room 3 | Read the Map
4
5.
iTPUGROUP 3 | MAP 2
Map 2: Gradient boosting - step size and number of trees
learning_rate = 0.01
train
learning_rate = 0.1
CV
train
learning_rate = 1
CV
train
100%
100%
100%
95%
95%
95%
90%
90%
90%
85%
85%
85%
80%
80%
80%
75%
75%
75%
70%
70%
70%
65%
65%
65%
60%
60%
60%
10
50
100
300 1000
trees
10
50
100
trees
300 1000
10
50
READ THE MAP
CV
100
300 1000
1
For EACH learning rate: how many
trees give the best CV? What is that
CV?
2
learning_rate 0.1: after which number
of trees does overfitting start? Use
numbers.
3
Why is learning_rate 0.01 with 10
trees so bad?
4
Recommend ONE setting for this data
and explain why.
trees
orange = train, teal = CV
Topic 10 | Breakout room 3 | Read the Map
5
6.
iTPUGROUP 3 | MAP 3
Map 3: The race - best CV so far, 36 trials each
best CV accuracy so far
Grid
Random
READ THE MAP
Optuna (Bayesian)
83,5%
1
At which trial did each strategy FIRST
reach 82%?
2
Imagine ONE trial takes 1 hour. How
many hours did Random and Optuna
need to reach 82.3%?
3
Grid was best at trial 1. Is grid really
the best method? What if its list were
in another order?
4
Final difference between the lines vs
the fold noise (~2 points): is there a
real winner?
83,0%
82,5%
82,0%
81,5%
81,0%
80,5%
80,0%
79,5%
79,0%
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36
trial number
Topic 10 | Breakout room 3 | Read the Map
6
7.
iTPUGROUP 3 | HINTS
Stuck? Three hints
Hint for map 1
Hint for map 2
Hint for map 3
61.7% is the baseline. And the good
cells form a diagonal band - where
does it go?
Overfitting = the train line goes UP
while the CV line goes DOWN.
Read where each line jumps up for
the first time above the 82% grid
line.
Always answer with NUMBERS from the chart.
Topic 10 | Breakout room 3 | Read the Map
7
8.
iTPUGROUP 3 | REPORT
Report template - fill it in, then the Speaker presents (1 minute)
Item
Your answer
Map 1: best cell, biggest gap, the C = 0.01 row,
next search
Map 2: best trees per learning rate, start of
overfitting
Map 2: our recommended setting
Map 3: first trial >= 82%, hours needed, real
winner?
One sentence: what did the maps teach us
about tuning?
Speaker: 1 minute. Say your BEST answer first, then ONE surprise you found.
Topic 10 | Breakout room 3 | Read the Map
8