Implementations generally relate to determining camera motion. In one implementation, a method includes capturing a first image and a second image of a physical scene with a camera in respective first and second positions. The method further includes determining first and second image points from the respective first and second image points. The method further includes determining first and second directions of gravity relative to the camera in the respective first and second positions. The method further includes determining a motion of the camera between the capturing of the first image and the capturing of the second image, wherein the determining of the motion of the camera is based at least in part on the first image points, the second image points, the first direction of gravity, and the second direction of gravity.