Patent attributes
The technology disclosed relates to a system that provides exactly-once processing of stream data. The system includes a queue manager which receives a stream of data. The system establishes aggregation intermediation checkpoints during processing of the received data. To do this, the system partitions delivery of the data stream at offsets, saves partition demarcation offsets at the end of processing windows, and saves intermediate aggregation results to a distributed file system with a window identifier (abbreviated ID) that correlates the offsets and the aggregation results. At each checkpoint, the intermediate aggregation results can be initially saved on at least one write-ahead log (abbreviated WAL) on the distributed file system and, post-saving, persisted to storage in accordance with a fault tolerance scheme.
