US Patent 8954399 Data de-duplication for information storage systems

Methods and systems for identifying candidates for de-duplication are disclosed herein. An example system may include: a de-duplication lister that coordinates with a kernel module to identify a unit of data for de-duplication and calculates a signature of the unit of data; a de-duplication queue that manages a de-duplication queue; a de-duplication tracker that maintains a de-duplication data structure including a plurality of entries corresponding to units of data and searches the de-duplication data structure to determine whether the signature of the identified unit of data matches the signature of at least one of the entries in the de-duplication data structure; a de-duplication maintainer that reviews the entries of the de-duplication data structure and removes obsolete entries from the de-duplication data structure; and a de-duplication engine that coordinates with the kernel module to perform de-duplication operations on the de-duplication queue.