Clustering & decision trees

Monday, June 22, 2026

NoteToday: Monday June 22

Before class

• Perusall Reading (Canvas): Data Computing §18.2 (clustering)

Plan for today

• Quiz 4 (rescheduled from Friday).
• Supervised vs. unsupervised learning.
• k-means and hierarchical clustering.
• Decision trees / CART.
• In-class activity.

Helpful links

• Syllabus · Schedule · DC Ch. 18

Big picture

Two kinds of learning

	Unsupervised	Supervised
You have…	features only, no labels	features and a known answer
Goal	find structure / groups	predict the answer for new cases
Examples	clustering (k-means, hierarchical)	decision trees (CART)
“Right answer”?	no; you judge usefulness	yes; you can measure accuracy

Today, we’ll be using the Palmer penguins body measurements to explore supervised and unsupervised learning techniques.

Penguins data

peng |>
  select(species, bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g) |>
  glimpse()

Rows: 342
Columns: 5
$ species           <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Adel…
$ bill_length_mm    <dbl> 39.1, 39.5, 40.3, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0…
$ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, 19.3, 20.6, 17.8, 19.6, 18.1, 20.2…
$ flipper_length_mm <int> 181, 186, 195, 193, 190, 181, 195, 193, 190, 186, 18…
$ body_mass_g       <int> 3750, 3800, 3250, 3450, 3650, 3625, 4675, 3475, 4250…

For clustering we’ll hide species and see if we can “rediscover” these categories. For trees we’ll use species as the answer to predict.

Clustering

Overview

Clustering puts rows into groups so that rows in a group are similar and rows in different groups are different, using only the feature columns.

• No labels go into the model, i.e., you don’t tell it “species.”
• “Similar” means close in distance once features are on a comparable scale.
• How we measure distance depends on the method.

Warning

Scale first. Body mass (~3000-6000 g) would swamp bill depth (~13-22 mm) in any distance calculation. We almost always want to standardize so every feature gets an equal vote.

X <- peng |>
  select(bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g) |>
  scale() # subtract mean, divide by sd --> unit-less

round(colMeans(X), 14) # column means = zero

   bill_length_mm     bill_depth_mm flipper_length_mm       body_mass_g 
                0                 0                 0                 0 

apply(X, 2, var) # column variances = one

   bill_length_mm     bill_depth_mm flipper_length_mm       body_mass_g 
                1                 1                 1                 1 

k-means clustering

kmeans() finds k cluster centers and assigns each point to its nearest one, iterating until things stop moving. k is a hyperparameter you pick.

set.seed(184) # random algorithm; set a seed so results are reproducible
km <- kmeans(X, centers = 3, nstart = 25) # nstart = try 25 random starts, keep best
km$size # how many penguins per cluster

[1] 132  87 123

nstart matters here: k-means depends on random starting centers, so we try several and keep the best.

• set.seed() makes it reproducible (we’ll get the same results if we re-run our code).

Clustering results

Did it recover the species categories?

We clustered without species. Now compare the clusters to the real species:

Did it recover the species categories?

Each cluster is mostly one species, found with no labels. (Gentoo separates cleanly; the other two overlap a bit.)

How many clusters? The elbow

There’s no “correct” k. A common heuristic is to plot total within-cluster spread as k grows, and look for the elbow where adding clusters stops helping. (Uses purrr::map_dbl() from last week.)

tibble(k = 1:8) |>
  mutate(wss = map_dbl(k, ~ kmeans(X, centers = .x, nstart = 10)$tot.withinss)) |>
  ggplot(aes(k, wss)) +
  geom_line() + geom_point(size = 2) +
  scale_x_continuous(breaks = 1:8) +
  labs(x = "k (number of clusters)", y = "Error",
       title = "Elbow around k = 3")

Here, “Error” is measured by the total within-cluster sum of squares:

\[\text{Error}_k = \sum_{i=1}^{N} ||\mathbf{x}_i - \boldsymbol{\mu}_{c(i)}||^2\]

Equivalently,

\[\text{Error}_k = \sum_{j=1}^{k} \sum_{\mathbf{x}_i \in C_j} ||\mathbf{x}_i - \boldsymbol{\mu}_j||^2\]

Hierarchical clustering

Another approach is to not pick k up front. Instead, you can start with every point alone, then repeatedly merge the two closest groups, and record the whole tree (a dendrogram).

d  <- dist(X) # pairwise distances
hc <- hclust(d, method = "complete")
plot(hc, labels = FALSE, main = "Dendrogram", xlab = "", sub = "")
rect.hclust(hc, k = 3, border = "#7C0A02") # cut into 3 groups

Height = how different two groups were when merged. Cut the tree at a height to get clusters, then you choose after seeing the structure.

Hierarchical: cut and compare

clusters_h <- cutree(hc, k = 3)
table(cluster = clusters_h, species = peng$species)

       species
cluster Adelie Chinstrap Gentoo
      1    151        14      0
      2      0         0    123
      3      0        54      0

k-means needs k up front and is fast on big data; hierarchical shows the whole nesting structure and lets you decide k afterward.

Decision trees (CART)

Now we have labels

Switch to supervised: use the measurements to predict species, and get a model a human can read.

A decision tree asks a sequence of yes/no questions about the features, splitting the data at each step, until each leaf is (mostly) one class. CART = Classification And Regression Trees.

fit <- rpart(
  species ~ bill_length_mm + bill_depth_mm + flipper_length_mm + body_mass_g,
  data = peng, method = "class"
)

Read the tree

plot(fit, margin = 0.1)
text(fit, use.n = TRUE, cex = 0.9)

Follow a penguin from the top: at each node, answer the question and go left if yes. The leaf is the predicted species. Notice it often needs only two measurements, which is much more interpretable.

How does it pick the splits?

At each step the tree tries every feature and every cut point and keeps the split that makes the two sides most “pure” (closest to a single class).

• Purity is measured by impurity (Gini index): low impurity = the node is almost all one species.

• It splits greedily, top-down, then stops when splits no longer help enough.

Tip

This is why trees are easy to explain: each split is just a threshold on one variable, like flipper_length_mm < 206.

How good is it?

Because species is known, we can score predictions with a confusion table:

pred <- predict(fit, peng, type = "class")
table(prediction = pred, actual = peng$species)

           actual
prediction  Adelie Chinstrap Gentoo
  Adelie       145         5      0
  Chinstrap      6        63      1
  Gentoo         0         0    122

mean(pred == peng$species) # overall accuracy

[1] 0.9649123

Visualize with a heatmap

Warning

This is accuracy on the same data the tree learned from, which is overly optimistic. A more realistic evaluation uses a held-out test set (the model must perform on data it has never seen). Deep trees can overfit, i.e., memorize the training data, fail on new data.

Trees vs. clustering

Clustering (unsupervised)

• No labels available
• Question: “what natural groups exist?”
• Output: a group ID per row
• Judged by usefulness, not accuracy
• e.g. customer segments, document topics

Decision tree (supervised)

• Labels available
• Question: “predict this known outcome”
• Output: a prediction + a readable rule
• Judged by accuracy on held-out data
• e.g. spam / not-spam, species, churn

An unsupervised cluster is not a verified truth, it’s a hypothesis you still have to check.

Activity

In-class activity (~20 min)

Work in day24-activity.R (or a fresh script) on the penguins data.

1. Cluster. Standardize the four numeric measurements, run kmeans() with centers = 2, then 4. How does table(cluster, species) change vs. k = 3?
2. Elbow. Rebuild the elbow plot using only two features. Does the elbow still say 3?
3. Tree. Fit a tree predicting species from only bill_length_mm and bill_depth_mm. Plot it and report accuracy. Did dropping two features hurt much?
4. Discuss. Which method would you prefer if the dataset had no species column at all? Why?

Wrap-up & looking ahead

What we covered

• Unsupervised (clustering) finds groups with no labels; supervised (trees) predicts a known label.
• k-means: pick k, scale features, nstart, read the elbow.
• Hierarchical: merge-closest, read the dendrogram, cut afterward.
• Trees: greedy splits on single features \(\rightarrow\) readable rules; score on held-out data, beware overfitting.

Upcoming

• Tue Jun 23: using R with LLMs · HW 5 due 11:59 p.m.
• Wed Jun 24: case study · Checkpoint 2 due

Reach out

Stuck? Office hours, or email me (cgc5478@psu.edu) or Xinyue (xpw5228@psu.edu).

Reference

• DC Ch. 18
• Syllabus · Schedule