A stepless retractable handle assembly for luggage includes a handle grip and two retractable bars connected to the handle grip in such a manner that when pressed a press-button in the handle grip, two transmission plates in the handle grip are forced by the press-button to move transversely in reverse directions, and two connection shafts are forced by the transmission plates to move vertically in the respective retractable bars in biasing respective anchor clamps away from the inside walls of the outer tubes of the respective retractable bars for allowing adjustment of the height of respective inner tubes of the retractable bars. When released the press-button after the adjustment, the anchor clamps are returned to abut against the inside walls of the respective outer tubes, locking the inner tubes in the adjusted position.