This invention includes an autonomous driving system for automobiles, comprising: one or more common electronic communication ports of autonomous driving (communication ports) that are built-in on the automobiles; and one or more universal autonomous driving portable controllers (portable controllers) that are to be plugged-in to the said communication ports that are built-in on the automobiles. The interfaces of the communication ports and the portable controllers are both standardized such that the portable controllers can be plugged-in universally to all of the automobiles that are equipped with the built-in communication ports. The communication ports comprise electronic communication of all relevant electronic control units (ECUs) and feedback information of the automobiles, dedicated for the said portable controllers to communicate with and to control the automobiles. In addition to the portable controllers, the communication ports comprise a buffer that is designed to execute a short duration of controls to make emergency stops, in case of loss of connection with the portable controllers due to accidents or other failure conditions. The portable controllers comprise a central control unit (CCU), and a plurality of sensors and processors, and a plurality of data storages, and a plurality of data links, and a Global Positioning System (GPS). The portable controllers have standardized interfaces that match with that of the communication ports. The invention disclosed herein enables all automobiles to be ready for autonomous driving with minimal cost, provided that the said communication ports are adapted to the automobiles. The said portable controllers integrate all the hardware and software relevant to autonomous driving as standalone devices which can share the components, simplify the systems, reduce parasitic material and components, and most importantly, will be safer when multiple sensors and processors that are based on different physics are grouped together to detect objects and environment conditions. A method of compound sensor clustering (CSC) is introduced herein. The CSC method makes the sensors and processors to self-organize and to address real-world driving conditions. The portable controllers can be mass-produced as standard consumer electronics at lower cost. The portable controllers can also be more easily updated with the latest technologies since that they are standalone devices, which would be otherwise hard to achieve when the hardware and software are built-in permanently as part of the automobiles. The invention disclosed herein is more efficient, since that the portable controllers can be plugged-in to the automobiles when there are needs for autonomous driving, comparing with current methods of integrating autonomous driving control hardware and software that are built-in to automobiles permanently, which may not be used for autonomous driving frequently. The system also decouples the liability from automotive manufactures in case of accidents. The portable controllers can be insured by insurance companies independently, much like insuring human drivers.