US Patent 11973973 Prediction refinement based on optical flow

A method of video processing includes determining a refined prediction sample P′(x,y) at a position (x,y) in a video block by modifying a prediction sample P(x,y) at the position (x,y) with a first gradient component Gx(x, y) in a first direction estimated at the position (x,y) and a second gradient component Gy(x, y) in a second direction estimated at the position (x,y) and a first motion displacement Vx(x,y) estimated for the position (x,y) and a second motion displacement Vy(x,y) estimated for the position (x,y), where x and y are integer numbers, and performing a conversion between the video block and a bitstream representation of the video block using a reconstructed sample value Rec(x,y) at the position (x,y) that is obtained based on the refined prediction sample P′(x,y) and a residue sample value Res(x,y).