Difference calculation based on partial position is described. In a representative aspect, a method of video processing comprising: calculating, during a conversion between a current block of video and a bitstream representation of the current block, differences between two reference blocks associated with the current block or differences between two reference sub-blocks associated with a sub-block within the current block based on representative positions of the reference blocks or representative positions of the reference sub-blocks; and performing the conversion based on the differences.