A system for collaborative analysis from different processes on different data sources. The system uses a unique approach to lightweight temporary data structures in order to allow communication of interim results among processes, and construction of semantically appropriate reports. The data structures are generated in near real time and their lightweight nature supports massive scaling, including many diverse streaming inputs.