A color conversion section in an image processing apparatus converts input colors at positions of interest in input color document data to a limited number of colors, on the basis of page layout information that is updated on a page-by-page basis and delivered from a page layout analysis section, a color-assignment table that is updated on a page-by-page basis and delivered from a color-assignment table generating section, and color category attribute information relating to the positions of interest in the input color document data, which is delivered from a color categorizing section.