An imaging means captures a color image of lumber. An image processing means obtains the color distribution of the color image captured by the imaging means, compares the obtained color distribution with a predetermined color distribution of normal lumber, judges the obtained color distribution as an abnormal one when it is deviated from the color distribution of normal lumber by a predetermined value or more, and detects a defect of the lumber having the abnormal color distribution deviated by a value larger than the predetermined value in an area on the surface of the lumber captured by the imaging means.