A cap assembly structure has a sliding groove for attaching to the visor of a cap and a sliding anchor block attached to an accessory. The sliding groove, which has an inserting slot opening for receiving an I-shaped engaging slipper of the sliding anchor block, extends from the inserting slot opening end up to the other end such that the sliding groove has at least one ledge formed with a plurality of serrated notches for locking with the I-shaped engaging slipper to enable the wearer to assemble, disassemble, move and replace the accessories. The accessories can also be kept in holding elements when they are not in use.