A modular garment system includes an attachment system for attaching multiple garments together. In one embodiment, the garment system includes an upper body garment that is attachable to a lower body garment. The upper body garment includes a plurality of connectors that are aligned with a plurality of retaining members on the lower body garment. The retaining members on the lower body garment, for instance, may be placed on top of existing belt loops. The retaining members may include a flap, while the connectors may have a loop-like structure. In order to connect the upper body garment to the lower body garment, the flap may be inserted into the loop-like structure and secured to the lower body garment. The upper body garment can also include a draw cord that can be used to further secure the upper body garment against the lower body garment.