US Patent 8001329 Speculative stream scanning

A system and method for partitioning a data stream into tokens includes steps or acts of: receiving the data stream; setting a partition scanner to a beginning point in the data stream; identifying likely token boundaries in the data stream using the partition scanner; partitioning the data stream according to the likely token boundaries as determined by the partition scanner, wherein each partition of the partitioned data stream bounded by the likely token boundaries comprises a chunk; and passing the chunk to a next available token scanner, one chunk per token scanner, for identifying at least one actual token within each chunk.