Among other things, techniques are described for receiving, from at least one sensor of a vehicle, sensor data associated with a surface along a path to be traveled by a vehicle; using a surface classifier to determine a classification of the surface based on the sensor data; determining, based on the classification of the surface, drivability properties of the surface; planning, based on the drivability properties of the surface, a behavior of the vehicle when driving near the surface or on the surface; and controlling the vehicle based on the planned behavior.