A signal processing apparatus comprises a decoding unit configured to generate a decoded image by decoding lossy compressed image data, and a restoration processing unit configured to perform image restoration processing on the decoded image. The restoration processing unit determines whether or not to perform the restoration processing for each of blocks in the decoded image in accordance with specific image information, and for a block on which it is determined that the restoration processing is to be performed, performs the restoration processing on the basis of an inference made using a coefficient learned in advance.