The methods and systems herein enable relative movement-based seatbelt use detection. Image data of an occupant of a vehicle is received over time. The image data is then input into a machine-learned model or other module that determines whether relative movement between one or more portions of the occupant and corresponding portions of a seatbelt are less than a threshold amount. If the relative movement is less than the threshold amount, an indication of seatbelt misuse (e.g., a fake seatbelt) is output to a vehicle component. By basing the indication of seatbelt misuse on relative movement between the occupant and the seatbelt, seatbelt use detection may be improved.