A tool holding device is suitable to have an application with a power tool. The tool holding device includes a toolholder for receiving a bit. A sleeve is disposed outside the toolholder and is operable to move between a first position which allows for removal of the bit and a second position which allows the bit to be securely retained in the cavity of the toolholder. A returning member is disposed circumferentially between the toolholder and the second section of the sleeve for returning the sleeve to a neutral position between the first and second positions. Moreover, the portion of the sleeve which does not surround the returning member abuts the toolholder.