ABC Tools

How AI Learns to Drive with Neuroevolution

After reading this you will understand how a genetic algorithm evolves the weights of a tiny neural network until a population of cars laps a race track, and you will be able to reproduce the fitness numbers by hand and read the best-fitness curve without fooling yourself.

What neuroevolution is, with one concrete run

Neuroevolution trains a neural network by copying, mutating and selecting whole networks instead of computing gradients. Each car on this track is driven by a fixed neural network: five raycast distance sensors feed a 5→6→2 multilayer perceptron whose two outputs are steering and throttle. The network shape never changes. Only the numbers inside it (the weights and biases) change from generation to generation.

Count the parameters. The first layer maps 5 inputs to 6 hidden units: that is 5 \times 6 = 30 weights plus 6 biases. The second layer maps 6 hidden units to 2 outputs: 6 \times 2 = 12 weights plus 2 biases. The whole genome is 30 + 6 + 12 + 2 = 50 real numbers. Evolution searches this 50-dimensional space.

The hook is the first run. Generation 1 draws its weights at random, so the cars veer into the first wall within a second. By generation 30 or so on default settings, the leader takes clean laps. Nobody wrote a steering rule. Selection on total fitness did all the work.

This is not gradient-based reinforcement learning. No rewards are backpropagated and no value function is learned. The only signal that crosses generations is which genomes survived, so the credit assignment is coarse: a good genome is kept whole, a bad one is discarded whole.

When to reach for it, and when not

Neuroevolution earns its place when the reward is sparse, delayed or non-differentiable, and when the network is small. Progress around a track is exactly that kind of signal: you only know a driving policy was good after the car has driven for several seconds, and there is no clean derivative of "distance travelled" with respect to a steering weight.

It stops being the right tool when the network is large. A genome of 50 numbers is comfortable. A genome of 50 million numbers, as in a modern vision model, is not: random mutation in that many dimensions almost never improves anything, and each fitness evaluation is expensive. Gradient methods scale to those sizes because the gradient points in a useful direction; blind mutation does not.

So treat this demo as an honest picture of the algorithm on a tiny problem, not as evidence about production-scale training. The mechanics are real. The scale is not.

The three operators that make a generation

Each new generation is built from the old one with three operators.

Elitism
Copy the top k genomes unchanged into the next generation. This guarantees the best fitness never drops, because the current champion is always still present.
Tournament selection
To pick a parent, draw t random cars and keep the fittest of them. Larger t means stronger pressure toward the current best, at the cost of variety.
Gaussian mutation
Add a small random number to each weight, drawn from a normal distribution with mean 0 and standard deviation \sigma. This is the only source of new behavior.

Mutation is the operator you tune first. Its rule is one line per weight:

w_i' = w_i + \sigma \cdot z_i, \quad z_i \sim \mathcal{N}(0, 1)

Here w_i is one weight of a parent, z_i is a fresh standard normal draw (mean 0, variance 1), \sigma is the mutation strength you set with the slider, and w_i' is the child's weight. Do this for all 50 numbers to make one child.

The size of \sigma sets the explore/exploit balance. With \sigma = 0.05, a typical step is small, so children stay close to a good parent and progress is steady but slow. With \sigma = 0.5, steps are large: you explore widely but often destroy a good driver. About 68% of Gaussian draws land within \pm \sigma and about 95% within \pm 2\sigma, so \sigma = 0.2 means roughly 95% of weight changes are smaller than 0.4 in size.

Fitness: what "good driving" actually measures

Fitness here is the angle travelled around the closed course, measured through checkpoints. Think of the track as a loop of checkpoints. A car's fitness is how far around the loop it has progressed, counted continuously so that being halfway between two checkpoints counts as half.

There is a stall rule: a car that makes no progress for a few seconds is killed early. Without it, standing still would be a safe local optimum, because a stopped car never crashes and a crashed car scores no more. The stall rule removes that hiding place and forces evolution to reward motion.

Fitness is total progress, not speed or smoothness. A jerky car that grinds along a wall can outscore an elegant car that spins out early. Do not read the leader's style as "understanding". It is whatever collection of 50 numbers happened to travel furthest.

Reproducing one generation with the default population

Use the demo defaults: a population of 100 cars, mutation \sigma = 0.1, elitism keeping the top 2, tournament size 5. Suppose after generation 5 the recorded fitnesses (in checkpoint units, higher is better) for the top six cars are as below.

  1. Rank the cars by fitness: 41.2, 39.8, 37.5, 36.9, 30.1, 28.4. The best fitness for this generation is 41.2.
  2. Elitism copies the top 2 genomes (41.2 and 39.8) unchanged into generation 6. These 2 seats are filled with no mutation.
  3. Fill the remaining 98 seats by tournament selection plus mutation. For one child, draw 5 cars at random, say the ones scoring 36.9, 28.4, 30.1, 39.8, 12.0. The winner is 39.8.
  4. Mutate the winner's 50 weights with \sigma = 0.1. If one weight was 0.740 and its draw was z = 1.3, the child's weight is 0.740 + 0.1 \times 1.3 = 0.870.
  5. Because the elite 41.2 genome survives untouched, generation 6's best fitness is at least 41.2. It can only rise or stay flat, never fall.

Reading the best-fitness curve

The most useful plot is best fitness per generation. Because of elitism it is monotone non-decreasing: a flat portion means the champion has not been beaten, and a step up means a mutated child overtook it. A typical default run looks like the curve below: fast early gains, then a plateau near the first full lap.

Best fitness rises fast, then flattens as cars near one full lap (fitness about 190). The marker at generation 9 is roughly where cars first clear the opening corners.

A plateau does not always mean "solved". It can mean the population has converged: every car is a near-copy of one genome, so tournament selection keeps picking clones and mutation cannot find anything better nearby. If the plateau sits well below a full lap, raise \sigma to widen the search, or raise population size so more variants are tried each generation.

Watch mutation reshape the search

The single most instructive knob is \sigma. Too small and the population barely moves; too large and good drivers get scrambled every generation. The widget below lets you feel that trade-off directly.

With mutation \sigma = 0.02 the best-fitness curve climbs slowly and steadily, often stalling below a full lap. With \sigma = 0.5 it lurches: big jumps up followed by long flats, because most children are worse than their parents and only elitism holds the line. A middle value near 0.1 to 0.2 reaches a full lap fastest.

Common mistakes when reading this demo

The first mistake is trusting one run. Evolution is stochastic: the random seed sets the initial 100 genomes and every mutation draw. One run might solve the track in 22 generations and the next in 45. Judge settings by several runs, not one lucky one.

The second mistake is over-reading generalisation. The five range sensors make the policy almost track-independent, because the network only ever sees distances to walls, not absolute positions. Evolved drivers often transfer to a fresh track at once. But "often" is not "always": press the new-track button several times and you will see the odd track where the driver clips a corner it never met before.

The third mistake is cranking \sigma to escape a plateau and then leaving it there. A large \sigma helps you jump out of a stuck population, but once the population is exploring again, that same large step keeps damaging good drivers. Raise it to break the plateau, then lower it to refine.

Related tools on this site

If you want to see credit assignment done with explicit rewards and a value estimate rather than blind selection, the Q-Learning Gridworld shows an agent learning a value for each state by trial and error. For the pure exploration/exploitation trade-off stripped of any driving, the Multi-Armed Bandit Lab compares ε-greedy, UCB1 and Thompson sampling on the same problem. And for a close cousin of this method, the CartPole Balancing AI uses the cross-entropy method, which also evolves a policy by keeping the best samples and resampling near them.

Frequently asked questions

Why does the best-fitness curve never go down?

Because of elitism. The top genomes are copied unchanged into the next generation, so the current champion is always still competing. The best fitness can only stay flat or rise. The average fitness, by contrast, can drop when many mutated children turn out worse than their parents.

Is this reinforcement learning?

It solves a reinforcement learning problem (act to maximise cumulative reward) but not with reinforcement learning algorithms. No rewards are backpropagated and no value function is learned. Selection on total fitness does the credit assignment, which is why it is called neuroevolution.

What should I change if progress stalls below a full lap?

Raise mutation \sigma first to widen the search around the converged population. If that is not enough, raise population size so more distinct variants are evaluated each generation. Both cost speed, so use the fast-forward option to run more physics steps per frame.

Why keep the network so small at 50 weights?

Random mutation searches every dimension at once, and its chance of improving a genome falls sharply as dimensions grow. Fifty numbers is small enough that a few mutations per generation regularly land improvements. Millions of numbers would need gradients, not mutation, to make progress in reasonable time.

Do the evolved cars actually understand the track?

No. A car is 50 numbers that map five distance readings to a steering and throttle value. It reacts to how far the walls are, nothing more. That reactive policy is exactly why it transfers to new tracks: it never encoded the old track's shape in the first place.

Try it yourself in the tool

Related tools

Q-Learning Gridworld

Paint a gridworld and watch a Q-learning agent discover the best path by trial and error.

CartPole Balancing AI

Balance the pole yourself, then watch the cross-entropy method evolve a policy that never drops it.

Multi-Armed Bandit Lab

Race ε-greedy, UCB1 and Thompson sampling on the same bandit and compare their regret.

Backpropagation Step-by-Step

Walk through one forward and backward pass of a tiny neural network, number by number.

Decision Tree Builder

Grow a CART decision tree split by split and watch the partition map and tree diagram build together.

Diffusion Model Toy

Train a tiny denoising diffusion model and watch it turn pure noise back into a 2D shape.