The present technology relates to an image processing device, an image processing method, an endoscope system, and a program that enable motion detection with higher accuracy. The endoscope system includes: a light source capable of performing a pulsed-light emission; a light-source control unit that controls the light source such that the light source performs the pulsed-light emission a plurality of times in an exposure time period of each captured image; an imaging unit that takes the captured images; and a motion detection unit that detects a motion of a photographic subject in the captured images. The present technology is applicable to the endoscope system.