How a neural network learns
The short answer
A neural network learns from examples. It starts with random connections, so its guesses are mostly wrong. For each example its answer is compared with the right one, the error is passed back through the network, and every connection is nudged a little in the direction that shrinks it. Repeated thousands of times, the nudges add up: each neuron draws one straight line, and together they draw the border the examples follow.
Chapters and transcript
- 0:00
Blue inside, orange outside. Nobody tells it the rule.
120 dots: blue ones inside, orange ones in a ring around them. The network is never told this rule; it only sees the dots.
- 0:09
A new network starts with random connections.
Two inputs, the position of a dot, feed six neurons, which feed one output. Every connection starts with a random weight, drawn here as its thickness.
- 0:16
So its first guesses are mostly wrong.
The colours behind the dots are the network's guesses. With random weights it gets 78 of the 120 dots wrong.
- 0:23
One dot goes in. The signal flows forward.
One blue dot goes in. Its signal flows through the connections, and the output answers 0.16 on a scale from orange (0) to blue (1).
- 0:29
It measures how wrong it was.
The right answer is 1, blue. The gap between the two is the error.
- 0:34
The error flows back. Every connection is nudged.
The error is passed back through the network, and every connection is nudged in the direction that shrinks it.
- 0:39
Same dot again: a little less wrong.
Shown the same dot again, the network answers 0.42. Still wrong, but less wrong. This one step is made larger than the others so it can be seen.
- 0:45
Now repeat for every dot, thousands of times.
The network then takes 4,000 small steps, each on one dot picked at random.
- 1:05
Each nudge is tiny. Together they find the rule.
No single step does much, but together they move its guesses until no dot is wrong. Every number on screen comes from this training run, computed in your browser.
- 1:29
Each neuron only knows one straight line.
Each of the six neurons can only split the plane with one straight line.
- 1:38
Together, they draw the border.
Combined by the output, the six lines bend into one border around the blue dots.
- 1:46
Dots it has never seen? It gets them right.
Twelve new dots it never trained on are all coloured correctly.
- 1:55
It learned the rule from examples alone.
Nobody wrote the rule down. The network found it from examples and its own mistakes.
Sources
Every figure in the film comes from these. The animation is illustrative.