Topics Technology and the internet

How do self-driving cars see the road?

Self-driving cars see the road through several kinds of sensors at once, then use software to turn the raw signals into a picture of lanes, vehicles, people and signs. Cameras read color, lane lines and traffic lights. Radar measures how far away objects are and how fast they move. Many systems also use lidar, which fires pulses of laser light and times their return to build a 3D map of the surroundings.

What makes it interesting is that no single sensor is trusted on its own. Cameras struggle in glare and darkness, radar has coarse detail, and lidar can be troubled by heavy rain or fog. Software blends the streams, labels what it sees, and predicts where a cyclist or a merging truck will be a moment from now. Companies disagree about which mix of sensors is best, and that debate is still open.

An episode would walk through each sensor, then the harder part: deciding what to do with what was seen, including the rare situations nobody planned for. The hosts on bre are AI, so they can get details wrong, and you can press Talk to ask about any step as it goes by.

What a bre episode would cover

An outline of the episode bre would make for this question. Every episode is written fresh when you ask, so yours will differ.

  1. Cameras: the eyes that read the worldHow ordinary cameras spot lane lines, signs and traffic lights, and why glare, night and bad weather make them harder to trust.
  2. Radar: distance and speed through rainRadio waves bounce off objects and reveal how far and how fast they are moving, even when the view is poor, though with little fine detail.
  3. Lidar: building a 3D map from laser lightPulses of light timed on the way back produce a cloud of points that outlines cars, curbs and pedestrians. Some makers use it, others avoid it.
  4. Sensor fusion: combining imperfect viewsSoftware merges the sensor streams so that one sensor's blind spot can be covered by another's strength.
  5. Recognizing and predictingTrained neural networks label objects, then the system estimates where each one is heading. This is where odd, rare situations cause trouble.
  6. Maps and positioningDetailed maps and satellite positioning help the car know where it is, while the sensors handle what has changed since the map was made.
  7. What still goes wrongUnusual objects, construction zones and weather remain hard, and different levels of automation still expect a human to be ready in some cases.

How the episode might open

A sample exchange between two of bre’s AI hosts, bre and Arlo. Both are AI; this is written by AI, as every bre episode is.

  1. breAI host

    Picture a car at a four-way stop at dusk. A kid on a scooter, a delivery van half blocking the lane, a stop sign with a sticker on it. A human takes that in without thinking. How does a car even start?

  2. ArloAI host

    Sensors. Lots of them.

  3. breAI host

    Right, but different kinds. Cameras, radar, and in some cars lidar. Each one sees the scene in a different way, and each one has a weakness.

  4. ArloAI host

    So why not just cameras? Humans manage with two eyes.

  5. breAI host

    That's the argument companies are actually having. Some say cameras plus good software is enough. Others add radar and lidar as backup. I don't think anyone has fully settled it.

  6. ArloAI host

    Who counted that? Which side is safer?

  7. breAI host

    Fair. I don't know of a clean answer, and we should be careful with any claim that sounds tidy. Let's start with what each sensor can and can't do.

  8. ArloAI host

    Fine. Cameras first. What do they miss?

Questions people also ask

Do all self-driving cars use lidar?
No. Many robotaxi systems use lidar alongside cameras and radar, while some carmakers rely mostly on cameras and radar. The companies disagree about which approach is best, and the debate has not been settled.
Can self-driving cars see in the dark or in fog?
Partly. Radar and lidar do not depend on daylight, so they help at night. Heavy rain, snow and fog can still degrade cameras and lidar, which is one reason systems combine several sensors and sometimes limit where they operate.
How does a self-driving car know what an object is?
Software called a neural network is trained on huge numbers of labeled examples, so it learns to tell cars, people, bikes and signs apart. It can still be fooled by unusual objects it has rarely seen.
Are self-driving cars fully autonomous today?
Mostly not. Some services run driverless in limited areas, but many consumer cars offer driver assistance that requires a person to pay attention. The levels of automation describe how much the human must still do.

Related topics

More: all 300 topics, technology and the internet, or the longer reads on /learn.

bre’s hosts are AI, and every episode is generated, so they can be wrong: check anything that matters. This page outlines what an episode would cover. It is for interest and learning, not medical, financial or legal advice.