How Self-Driving Cars Work: Sensors, AI, and the Levels of Autonomous Driving
A comprehensive explanation of how autonomous vehicles work — the sensor suite, perception algorithms, decision-making systems, the SAE levels of autonomy, major players, and the remaining technical and regulatory challenges.
What Is an Autonomous Vehicle?
An autonomous vehicle (AV) — commonly called a self-driving car — is a vehicle capable of sensing its environment and navigating without human input. Rather than relying on a human driver to process the environment and control steering, acceleration, and braking, autonomous vehicles use a combination of sensors, computational hardware, and artificial intelligence software to perform these tasks.
Fully autonomous vehicles represent one of the most complex engineering challenges of the 21st century: they must operate safely across an essentially infinite variety of real-world conditions — rain, snow, construction zones, unpredictable pedestrians, edge-case traffic situations — all in real time, at speed, with near-zero tolerance for error.
The Sensor Suite: How AVs See the World
Autonomous vehicles build a real-time model of their environment using multiple complementary sensor types:
LiDAR (Light Detection and Ranging)
LiDAR emits pulses of laser light and measures the time for each pulse to return after reflecting off objects, creating a precise 3D point cloud of the surrounding environment at ranges up to 200 meters. LiDAR provides highly accurate depth measurements regardless of ambient light conditions and is particularly effective at detecting object boundaries and geometry. High-resolution LiDAR units can produce 300,000+ data points per second.
A key debate in the industry: Waymo, Cruise, and most robotaxi companies consider LiDAR essential. Tesla has argued that cameras alone (without LiDAR) are sufficient, since human drivers navigate using only cameras (eyes).
Cameras
Arrays of cameras (typically 8–12 per vehicle) capture visual information — lane markings, traffic signals, road signs, and color information that LiDAR and radar cannot provide. Camera data is processed by deep learning convolutional neural networks trained on millions of labeled images.
Radar
Millimeter-wave radar measures the range and velocity of objects using Doppler frequency shifts. Unlike cameras and LiDAR, radar penetrates fog, rain, and snow effectively, making it valuable in adverse weather conditions. Radar provides less spatial resolution than LiDAR but excels at measuring relative velocity — critical for highway adaptive cruise control.
Ultrasonic Sensors
Short-range sensors (typically 0.3–5 meters) used primarily for parking assistance and low-speed maneuvering in tight spaces.
GPS / High-Definition Maps
Consumer GPS is accurate to ~3–5 meters — insufficient for lane-level autonomous driving. AVs use high-precision GPS (RTK) accurate to centimeters, combined with high-definition maps that encode lane positions, speed limits, traffic sign locations, and road geometry. The vehicle localizes itself within the HD map by matching its real-time sensor data to the pre-mapped environment.
The Autonomous Driving Stack
Raw sensor data is processed through a software pipeline called the autonomous driving stack:
1. Perception
Sensor fusion algorithms combine data from LiDAR, cameras, and radar into a unified 3D model of the environment. Deep learning object detection networks classify and track surrounding objects — vehicles, pedestrians, cyclists, traffic cones — and predict their trajectories over the next few seconds. This is one of the hardest problems in AV development: reliably detecting unusual objects (a mattress on the highway, a child on a bicycle, a horse) that were not well-represented in training data.
2. Localization
The system determines the vehicle's precise position — not just GPS location but its exact position within the lane, including heading and velocity — by matching sensor data against HD maps and integrating inertial measurement unit (IMU) data.
3. Prediction
A prediction module models the likely future behavior of all detected objects. Will that pedestrian at the curb step into the street? Will the adjacent vehicle change lanes? Prediction models must account for social conventions, traffic rules, and context — a car stopped behind a school bus behaves differently from one stopped at a red light.
4. Planning
Given the predicted states of all objects, the planning module determines the vehicle's path and speed over the next several seconds. It must balance safety, comfort, traffic law compliance, and progress toward the destination. Planner modules typically use a combination of rule-based systems and learned models.
5. Control
The control module converts the planned path into low-level commands (steering angle, throttle, brake) sent to the vehicle's actuators, executing the plan at millisecond timescales.
SAE Levels of Automation
| Level | Name | Description | Human Role | Examples |
|---|---|---|---|---|
| 0 | No Automation | Human drives; system may provide warnings | Full control | Most older cars |
| 1 | Driver Assistance | System controls either steering OR speed | Monitors and controls the other | Cruise control, lane keep assist |
| 2 | Partial Automation | System controls both steering AND speed | Must monitor at all times, hands on wheel | Tesla Autopilot, GM Super Cruise |
| 3 | Conditional Automation | System handles all driving in defined conditions | Must be ready to take over on request | Mercedes Drive Pilot (limited geofenced areas) |
| 4 | High Automation | No human needed in defined conditions/geofence | Passenger; no need to monitor | Waymo One (Phoenix, San Francisco) |
| 5 | Full Automation | No human needed anywhere, any condition | None | Does not yet exist commercially |
Major Players and Current Status (2024–2025)
- Waymo (Alphabet): Operating the world's most advanced commercial robotaxi service (Level 4) in Phoenix and San Francisco. As of 2024, Waymo One completes over 100,000 paid driverless trips per week. Uses a full sensor suite including LiDAR, radar, and cameras.
- Tesla: ~4 million vehicles with Autopilot/FSD (Full Self-Driving). Despite the branding, FSD as of 2024 remains Level 2 — Tesla internally describes it as "supervised." Uses cameras only (no LiDAR) and end-to-end neural networks trained on fleet data.
- Cruise (GM): Robotaxi service; faced significant operational setbacks in 2023 following a pedestrian incident in San Francisco and subsequent permit suspension.
- Baidu Apollo Go: China's leading AV operator; largest commercial robotaxi fleet globally by number of trips in some periods.
Key Challenges
- Edge cases ("long tail"): An AV must handle the rare, unexpected situations that occur infrequently but are statistically inevitable across millions of miles. The breadth of possible scenarios is essentially unbounded.
- Adverse weather: Rain, snow, and fog degrade camera and LiDAR performance significantly.
- V2X (Vehicle-to-Everything): Communication between vehicles and infrastructure (traffic lights, construction warnings) could dramatically improve AV decision-making but requires large-scale deployment of new infrastructure.
- Regulation: A patchwork of national and state/regional regulations creates fragmented frameworks for AV deployment.
- Liability: In Level 4+, who is legally responsible for accidents — the passenger, the manufacturer, or the software developer?