A vision system of a vehicle includes a camera disposed at a vehicle and having a field of view that encompasses first and second regions exterior of the vehicle. The camera captures image data that includes a first set of image data representative of the first region of the field of view and a second set of image data representative of the second region of the field of view. An image processor processes the first set of image data to detect a pedestrian present in the first region based on a first predetermined orientation of a pedestrian. The image processor processes the second set of image data to detect a pedestrian present in the second region based on a second predetermined orientation of a pedestrian.