A remote control device, which controls any of a plurality of apparatuses by transmitting a remote control signal for controlling a target apparatus in response to a user operation, detects a position of the remote control device and generates self-position information indicating the position, and specifies one of the apparatuses that is the closest in distance from the remote control device, as the target apparatus, based on the generated self-position information and a predetermined criterion. When the remote control device detects a user operation, the remote control device transmits a remote control signal corresponding to the detected user operation.