US Patent 9471585 Decentralized de-duplication techniques for largescale data streams

A local de-duplication table for at least a particular partition of a data stream is instantiated at a particular ingestion node of a multi-tenant stream management service. A submission request indicating a data record of the partition is received at the ingestion node. In response to a determination that (a) the submission request was received within a de-duplication time window corresponding to the particular partition, and (b) the local de-duplication table does not indicate that the data record is a duplicate, a write operation to store the data record at one or more storage locations of the stream management system is initiated.