Chunk compression in a deduplication aware client environment
View Patent ↗Techniques and mechanisms described herein facilitate the transmission of a data stream to a networked storage system. According to various embodiments, a data stream may be parsed to identify one or more uncompressed data chunks for transmission to a networked storage system. Each uncompressed data chunk may be compressed to produce a respective compressed data chunk. Each compressed data chunk may be transmitted to the networked storage system via a network for storage at the networked storage system.
1. A method comprising:
at a client device comprising a processor and memory, parsing a data stream to identify one or more uncompressed data chunks for transmission to a networked storage system;
identifying an amount of available computing resources at the client device;
when the identified amount of available computing resources meets or exceeds a designated computing resource availability compression threshold, compressing each uncompressed data chunk to produce a respective compressed data chunk, and transmitting each compressed data chunk to the networked storage system via a network for storage at the networked storage system; and
when the identified amount of available computing resources is below the designated computing resource availability compression threshold, encrypting each uncompressed data chunk prior to transmission, and transmitting each encrypted, uncompressed data chunk to the networked storage system via the network;
wherein one or more uncompressed data chunks are identified via a rolling hash parsing technique operable to generate at least some identical chunks when parsing different but overlapping data streams.
2. The method recited in claim 1 , the method further comprising:
identifying a respective fingerprint for each uncompressed data chunk by applying a hash function to the uncompressed data chunk.
3. The method recited in claim 2 , the method further comprising:
determining whether the uncompressed data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.
4. The method recited in claim 3 , the method further comprising:
transmitting a block map update request message to the networked storage system, the block map update request message including information for updating a block map at the networked storage system, the block map identifying a designated memory location at which each data compressed chunk is stored at the networked storage system.
5. The method recited in claim 3 , wherein determining whether the uncompressed chunk is stored at the networked storage system further comprises receiving a fingerprint status message from the networked storage system, the fingerprint status message indicating whether the chunk associated with the fingerprint is stored at the networked storage system.
6. The method recited in claim 1 , the method further comprising:
encrypting each compressed data chunk prior to transmission.
7. The method recited in claim 1 , wherein the networked storage system is operable to store deduplicated data based on storage requests received via the network.
8. The method recited in claim 1 , the data stream is generated at the client device via a network storage protocol selected from the group consisting of: the Network File System (NFS) protocol, the Common Internet File System (CIFS) protocol, and the Open Storage (OST) protocol.
9. A device comprising:
memory operable to store one or more uncompressed data chunks for transmission to a networked storage system, the one or more uncompressed data chunks identified by parsing a data stream;
a processor operable to identify an amount of available computing resources at the device and:
compress each uncompressed data chunk to produce a respective compressed data chunk when the identified amount of available computing resources meets or exceeds a designated computing resource availability compression threshold; and
encrypt each uncompressed data chunk prior to transmission when the identified amount of available computing resources is below the designated computing resource availability compression threshold; and
a communications interface operable to:
transmit, via a network for storage at the networked storage system, each compressed data chunk to the networked storage system when the identified amount of available computing resources meets or exceeds the designated computing resource availability compression threshold; and
transmit each encrypted, uncompressed data chunk to the networked storage system via the network when the identified amount of available computing resources is below the designated computing resource availability compression threshold;
wherein one or more uncompressed data chunks are identified via a rolling hash parsing technique operable to generate at least some identical chunks when parsing different but overlapping data streams.
10. The device recited in claim 9 , wherein the processor is further operable to:
identify a respective fingerprint for each uncompressed data chunk by applying a hash function to the uncompressed data chunk.
11. The device recited in claim 10 , wherein the processor is further operable to:
determine whether the uncompressed data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.
12. One or more non-transitory computer readable media having instructions stored thereon for performing a method, the method comprising:
at a client device comprising a processor and memory, parsing a data stream to identify one or more uncompressed data chunks for transmission to a networked storage system;
identifying an amount of available computing resources at the client device;
when the identified amount of available computing resources meets or exceeds a designated computing resource availability compression threshold,
compressing each uncompressed data chunk to produce a respective compressed data chunk; and
transmitting each compressed data chunk to the networked storage system via a network for storage at the networked storage system; and
when the identified amount of available computing resources is below the designated computing resource availability compression threshold,
encrypting each uncompressed data chunk prior to transmission, and
transmitting each encrypted, uncompressed data chunk to the networked storage system via the network;
wherein one or more uncompressed data chunks are identified via a rolling hash parsing technique operable to generate at least some identical chunks when parsing different but overlapping data streams.