Efficient data storage system
A system and method are disclosed for providing efficient data storage. A data stream comprising a plurality of data segments is received. The system determines whether one of the plurality of data segments has been stored previously using a summary in a low latency memory; in the event that the data segment is determined not to have been stored previously, assigning an identifier to the data segment.
1. A method for storing data comprising:
receiving a data stream comprising a plurality of data segments;
determining whether one of the plurality of data segments has been stored previously using a summary in a low latency memory, wherein the low latency memory is random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM);
in the event that the data segment is determined not to have been stored previously, assigning an identifier to the data segment; and
generating segment information for each of the plurality of data segments, wherein the segment information is a space efficient, probabilistic summary of segment information.
2. A computer program product for storing data, the computer program product being embodied in a computer readable medium and comprising computer instructions for:
receiving a data stream comprising a plurality of data segments;
determining whether one of the plurality of data segments has been stored previously using a summary in a low latency memory, wherein the low latency memory is random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM);
in the event that the data segment is determined not to have been stored previously, assigning an identifier to the data segment; and generating segment information for each of the plurality of data segments, wherein the segment information is a space efficient, probabilistic summary of segment information.
3. A data storage device comprising:
an input interface adapted to receive a data stream comprising a plurality of data segments; and
a segment redundancy check engine configured to determine whether one of the plurality of data segments has been stored previously using a summary in a low latency memory, wherein the low latency memory is random access memory (RAM), dynamic random access memory (DRAM), or synchronous dynamic random access memory (SDRAM); in the event that the data segment is determined not to have been stored previously, assign an identifier to the data segment; and generate segment information for each of the plurality of data segments, wherein the segment information is a space efficient, probabilistic summary of segment information.