A video image data reduction system through the use of object tracking reduces information in a video signal having a having a plurality of frames received from a motion video camera with a field of view. Moving objects in the field of view of the motion video camera are detected. These objects are selected if they are of interest. This is determined either by comparison of the detected object against predetermined characteristics of objects of interest or by manual user selection. A data set for each frame of the plurality of frames in the video signal is created based on the selected moving objects. If a frame contains no selected objects then the data set for that frame is marked as empty.