US Patent 11782982 Offset-based watermarks for data stream processing

Aspects of the present disclosure relate to watermarks and watermarking techniques for data streaming pipelines. Time stamp and offset timeline data is shared by computing instances along the pipeline to enable improved watermarking of the data stream through the pipeline. The improved watermarks enable better determination of completeness for the data stream and improve materialization of the results. The watermarking techniques can include periodically publishing watermark data by processing units of a vertex, fetching a merged watermark for a vertex by a vertex, and/or watching a data storage for the watermark data for events. Consensus algorithms can be used to maintain consensus among vertices for the watermark data.