A video processing method is disclosed to include: determining that a coding mode of a current video block is an affine mode; determining, for the current video block, an applicability mode of a prediction refinement optical flow (PROF) procedure with the affine mode according to a rule that specifies a condition for applying the PROF procedure; and performing, based on the determining, a conversion between the current video block and a coded representation of the video; wherein, during the PROF procedure, one or more initial predictions for the current video block are computed at a sub-block level and refined using an optical flow calculation.