An information handling system peripheral, such as a mouse, accepts a cable connector in a cable port, such a USB Type-C cable connector and port. In one embodiment, the cable connector has a wireless dongle that inserts substantially flush with the cable port so that external torsional forces do not operate against the port. A magnetic toggle ejector translates an ejection force by operation of like poles to push a portion of the magnet against the cable connector and provide a grasp point for end user fingers to grasp the cable connector.