The Great Readsby · a working piece

Random Forest Classification

A classification algorithm.

Press START and see it live.

The algorithm starts with a table of penguin data. Each row represents one penguin, while the columns contain information about it, such as its flipper length, bill length, and body mass. One column also gives the penguin's species. The algorithm uses the measurements to learn how to predict that species.

Every penguin in the table, placed by two of its measurements and coloured by species.

A decision tree sorts the penguins by asking simple yes-or-no questions. For example, it might ask whether a penguin's flipper is longer than a certain length. Penguins that meet the condition are placed in one group, while the rest are placed in another.

It tries different questions and chooses the one that separates the species best.

One column, every penguin on it, and the cut that separates the species best.

One tree can learn the training data too closely, including its random quirks. So instead, the algorithm builds many trees, each using a slightly different random sample of the penguins.

Some penguins may appear more than once in a sample, while others are left out. The ones left out can be used to check how well that tree works.

One tree's sample. Dark = in the sample, and darker still if drawn more than once. Pale = left out, so it can test this tree.

The trees are also made different in another way. At each step, a tree is only allowed to consider a few randomly chosen columns. It then chooses the best question it can from those columns, even if another column could have produced a better split.

So different trees may ask different questions, even when looking at similar penguins.

At this question the tree may use only the two columns in colour.

It splits, then does the same on both sides, and keeps going until a group holds one species only, or is too small to split, or the tree has grown as deep as it is allowed.

A real tree from this data. Each box is a group of penguins; the colours are the species in it. Boxes with a thick outline are where the tree stopped.

Then every tree is shown a penguin it never saw, and each one says which species it thinks it is. The answer with the most votes wins.

Because each tree saw different rows and different columns, they do not all get the same penguins wrong. That is the whole point of building many.

sample the rows → grow a tree → repeat → vote

One penguin, and the vote of every tree that never saw it. Squares with an outline are the trees that got it wrong.

What we are doing

the objective

Grow — decision trees on —. Each tree gets its own sample of the rows, and at every question may use only — of the columns, chosen at random.

  1. Sample the rows. Draw as many rows as the table has, at random, with replacement. About a third are left out, and those become that tree's test set. This is the bootstrap.
  2. Find the question. Out of the few columns it is allowed, try every value each could be cut at, and keep the cut that leaves the two sides least mixed. How mixed a group is, is measured by Gini impurity.
  3. Do it again on both sides, until a group holds one species, or is too small, or the tree is as deep as it may go.
  4. Vote. Every tree is shown the rows it never saw and says which species it thinks each one is. The answer with the most votes wins.

The claim we are checking: the forest is right because its trees are wrong about different rows. If that holds, the forest will be right far more often than the average tree in it, and any two trees will disagree with each other a great deal.

the table
Rows read—
Rows usable—
Predicting—
Columns in use—
the columns · click one to drop it
the settings

The first 2 trees are shown question by question at reading speed. The rest are built at full speed.

Data · Palmer Penguins (Horst, Hill & Gorman), CC0. Bundled here as data/penguins.csv. Trees, bootstrap, Gini and out-of-bag scoring are all computed in this page — no library.

Random Forest Classification
speed
1 · the cut it is looking for—

Each row is one column the tree may use at this node. The curve shows, for every possible cut, how much more separated the two sides would be than the node is now. The highest point is the cut it takes. Hover the curve.

2 · the tree being grown—

Every box is a group of penguins. The colours show which species are in it, so a box of one colour holds one species. A thick outline means the tree stopped there. Hover any box.

3 · the forest so far—

One drawing per finished tree. Each tree has a different shape because each saw a different sample. The bar underneath is how often that tree alone is correct. Click a tree to open it.

rows · dark = in this tree's bag, pale = out of bag —
tree

—

—forest, out of bag
—average single tree

—

Click any row in the strip above to see how the trees voted on it.

—mean pairwise agreement