A vehicular rearview mirror control system includes a plurality of rearview mirror assemblies having variable reflectance mirror reflective elements. An electronic control unit receives image data captured by a rear backup camera of the vehicle. The rearview mirror control system, responsive to processing a first subset of received image data captured by the rear backup camera, determines ambient light rearward of the vehicle. The rearview mirror control system, responsive to processing a second subset of received image data captured by the rear backup camera, determines glare light emanating from a headlight of another vehicle following the equipped vehicle. The first subset of received image data is different than the second subset of received image data. The rearview mirror control system generates an output to control dimming of respective ones of the mirror reflective elements of the plurality of rearview mirror assemblies based on location of the determined glare light.