A cutting tool assembly includes a mandrel member and a cutting tool. The mandrel member has a shaft rod and a slide sleeve fitted around the shaft rod. At least one alignment pin is mounted in the shaft rod and at least one engagement pin is disposed in the slide sleeve. At least one locking member is disposed between the shaft rod and the slide sleeve. The cutting tool has a shank body and a tool bit. At least one alignment sink is disposed on the shank body. When assembled, the shank body is plugged into the shaft rod to align the alignment pin with the alignment sink so as to release the slide sleeve from the engagement of the engagement pin, permitting the slide sleeve to move to a locking position where the locking member is latched with the cutting tool.