A caster assembly includes a base body, a tubular member, a retaining member and two wheel units. The base body includes an inner surrounding surface, a through hole surrounded by the inner surrounding surface, and a positioning slot extending along a line tangent to the inner surrounding surface and communicating with the through hole. The tubular member is inserted into the through hole. The retaining member is inserted into the positioning slot to retain the tubular member within the through hole. The wheel units are respectively and rotatably sleeved on two lateral connection portions of the tubular member.