Two kinds of learning

Today, we’ll be using the Palmer penguins body measurements to explore supervised and unsupervised learning techniques.

Penguins data

peng |>
  select(species, bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g) |>
  glimpse()

Rows: 342
Columns: 5
$ species           <fct> Adelie, Adelie, Adelie, Adelie, Adelie, Adelie, Adel…
$ bill_length_mm    <dbl> 39.1, 39.5, 40.3, 36.7, 39.3, 38.9, 39.2, 34.1, 42.0…
$ bill_depth_mm     <dbl> 18.7, 17.4, 18.0, 19.3, 20.6, 17.8, 19.6, 18.1, 20.2…
$ flipper_length_mm <int> 181, 186, 195, 193, 190, 181, 195, 193, 190, 186, 18…
$ body_mass_g       <int> 3750, 3800, 3250, 3450, 3650, 3625, 4675, 3475, 4250…

For clustering we’ll hide species and see if we can “rediscover” these categories. For trees we’ll use species as the answer to predict.

Overview

Clustering puts rows into groups so that rows in a group are similar and rows in different groups are different, using only the feature columns.

• No labels go into the model, i.e., you don’t tell it “species.”
• “Similar” means close in distance once features are on a comparable scale.
• How we measure distance depends on the method.

X <- peng |>
  select(bill_length_mm, bill_depth_mm, flipper_length_mm, body_mass_g) |>
  scale() # subtract mean, divide by sd --> unit-less

round(colMeans(X), 14) # column means = zero

   bill_length_mm     bill_depth_mm flipper_length_mm       body_mass_g 
                0                 0                 0                 0 

apply(X, 2, var) # column variances = one

   bill_length_mm     bill_depth_mm flipper_length_mm       body_mass_g 
                1                 1                 1                 1 

k-means clustering

kmeans() finds k cluster centers and assigns each point to its nearest one, iterating until things stop moving. k is a hyperparameter you pick.

set.seed(184) # random algorithm; set a seed so results are reproducible
km <- kmeans(X, centers = 3, nstart = 25) # nstart = try 25 random starts, keep best
km$size # how many penguins per cluster

[1] 132  87 123

nstart matters here: k-means depends on random starting centers, so we try several and keep the best.

• set.seed() makes it reproducible (we’ll get the same results if we re-run our code).

Clustering results

Did it recover the species categories?

We clustered without species. Now compare the clusters to the real species:

Did it recover the species categories?

Each cluster is mostly one species, found with no labels. (Gentoo separates cleanly; the other two overlap a bit.)

How many clusters? The elbow

There’s no “correct” k. A common heuristic is to plot total within-cluster spread as k grows, and look for the elbow where adding clusters stops helping. (Uses purrr::map_dbl() from last week.)

tibble(k = 1:8) |>
  mutate(wss = map_dbl(k, ~ kmeans(X, centers = .x, nstart = 10)$tot.withinss)) |>
  ggplot(aes(k, wss)) +
  geom_line() + geom_point(size = 2) +
  scale_x_continuous(breaks = 1:8) +
  labs(x = "k (number of clusters)", y = "Error",
       title = "Elbow around k = 3")

Here, “Error” is measured by the total within-cluster sum of squares:

\[\text{Error}_k = \sum_{i=1}^{N} ||\mathbf{x}_i - \boldsymbol{\mu}_{c(i)}||^2\]

Equivalently,

\[\text{Error}_k = \sum_{j=1}^{k} \sum_{\mathbf{x}_i \in C_j} ||\mathbf{x}_i - \boldsymbol{\mu}_j||^2\]

Hierarchical clustering

Another approach is to not pick k up front. Instead, you can start with every point alone, then repeatedly merge the two closest groups, and record the whole tree (a dendrogram).

d  <- dist(X) # pairwise distances
hc <- hclust(d, method = "complete")
plot(hc, labels = FALSE, main = "Dendrogram", xlab = "", sub = "")
rect.hclust(hc, k = 3, border = "#7C0A02") # cut into 3 groups

Height = how different two groups were when merged. Cut the tree at a height to get clusters, then you choose after seeing the structure.

Hierarchical: cut and compare

clusters_h <- cutree(hc, k = 3)
table(cluster = clusters_h, species = peng$species)

       species
cluster Adelie Chinstrap Gentoo
      1    151        14      0
      2      0         0    123
      3      0        54      0

k-means needs k up front and is fast on big data; hierarchical shows the whole nesting structure and lets you decide k afterward.

Now we have labels

Switch to supervised: use the measurements to predict species, and get a model a human can read.

A decision tree asks a sequence of yes/no questions about the features, splitting the data at each step, until each leaf is (mostly) one class. CART = Classification And Regression Trees.

fit <- rpart(
  species ~ bill_length_mm + bill_depth_mm + flipper_length_mm + body_mass_g,
  data = peng, method = "class"
)

Read the tree

plot(fit, margin = 0.1)
text(fit, use.n = TRUE, cex = 0.9)

Follow a penguin from the top: at each node, answer the question and go left if yes. The leaf is the predicted species. Notice it often needs only two measurements, which is much more interpretable.

How does it pick the splits?

At each step the tree tries every feature and every cut point and keeps the split that makes the two sides most “pure” (closest to a single class).

• Purity is measured by impurity (Gini index): low impurity = the node is almost all one species.

• It splits greedily, top-down, then stops when splits no longer help enough.

How good is it?

Because species is known, we can score predictions with a confusion table:

pred <- predict(fit, peng, type = "class")
table(prediction = pred, actual = peng$species)

           actual
prediction  Adelie Chinstrap Gentoo
  Adelie       145         5      0
  Chinstrap      6        63      1
  Gentoo         0         0    122

mean(pred == peng$species) # overall accuracy

[1] 0.9649123

Trees vs. clustering

An unsupervised cluster is not a verified truth, it’s a hypothesis you still have to check.

In-class activity (~20 min)

Work in day24-activity.R (or a fresh script) on the penguins data.

1. Cluster. Standardize the four numeric measurements, run kmeans() with centers = 2, then 4. How does table(cluster, species) change vs. k = 3?
2. Elbow. Rebuild the elbow plot using only two features. Does the elbow still say 3?
3. Tree. Fit a tree predicting species from only bill_length_mm and bill_depth_mm. Plot it and report accuracy. Did dropping two features hurt much?
4. Discuss. Which method would you prefer if the dataset had no species column at all? Why?

Wrap-up & looking ahead