One-hot encoding turns a column of labels, such as a cat's favourite fish, into numbers a machine learning model can use without inventing an order that isn't there. It is one of the first steps in preparing most datasets, and getting it wrong makes a model worse without any error message.
Why models need numbers
A machine learning model is maths: it multiplies inputs by weights and measures distances between rows, and that only works on numbers. A column like 'favourite fish', with values 'tuna', 'salmon' and 'sardine', is categorical data: each value is a label for a group, not an amount of anything.
Before training, every categorical column has to become numbers somehow. The question is how to do that without changing what the data means.
The problem with numbering the categories
The obvious fix is to give each label a number: tuna is 1, salmon is 2, sardine is 3. This is called label encoding (or integer encoding), and it takes one line of code. It also tells the model things that aren't true.
To a model, 3 is bigger than 1, and 2 sits halfway between them. So it now 'knows' that sardine is more than tuna, and that salmon is the average of the two. None of that is in the data. The numbers are just labels, not rankings.
You can see the damage in two common kinds of model:
- Linear models learn one weight per input column. With a single fish column, the fish's effect on the prediction is that weight times 1, 2 or 3. Salmon's effect is forced to sit exactly between tuna's and sardine's, whatever the real data says.
- Distance-based models, such as k-nearest neighbours, compare rows by how far apart their values are. Tuna (1) is one step from salmon (2) but two steps from sardine (3), so the model treats tuna and sardine as the least alike, with nothing in the data to support it.
Tree-based models, such as random forests, split on thresholds rather than multiplying, so arbitrary numbers hurt them less. They still split the codes in order, so one split can group tuna with salmon, but never tuna with sardine.
Wondering how this was made? How we make these