Technique selection in a deduplication aware client environment
View Patent ↗Techniques and mechanisms described herein facilitate the transmission of a data stream to a networked storage system. According to various embodiments, a determination may be made as to whether an amount of available computing resources at a client device meets or exceeds a computing resource availability threshold at the client device. A processing operation on a data stream may be performed at the client device to produce a pre-processed data stream when the amount of available computing resources meets or exceeds the computing resource availability threshold. The pre-processed data stream may be transmitted to a networked storage system for storage via a network. The networked storage system may be operable to store deduplicated data for retrieval via the network.
1. A method comprising:
determining whether an amount of available computing resources at a client device comprising a processor and memory meets or exceeds a computing resource availability threshold at the client device;
performing a processing operation on a data stream at the client device to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by:
determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;
determining that the hash value for the first chunk qualifies as a chunk boundary; and
in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and
transmitting the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.
2. The method recited in claim 1 , wherein the processing operation comprises:
deduplicating the data stream.
3. The method recited in claim 1 , wherein the processing operation comprises compressing the data stream.
4. The method recited in claim 1 , wherein the processing operation comprises encrypting the data stream.
5. The method recited in claim 1 , wherein the processing operation comprises:
for one or more of the identified data chunks, performing an operation selected from the group consisting of: deduplicating the identified data chunk, compressing the identified data chunk, and encrypting the identified data chunk.
6. The method recited in claim 5 , wherein the processing operation further comprises:
determining a respective fingerprint for each identified data chunk by applying a hash function to the identified data chunk.
7. The method recited in claim 6 , wherein the processing operation further comprises:
determining whether each identified data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.
8. The method recited in claim 7 , wherein determining whether the identified chunk is stored at the networked storage system further comprises:
receiving a fingerprint status message from the networked storage system, the fingerprint status message indicating whether the chunk associated with the fingerprint is stored at the networked storage system.
9. The method recited in claim 5 , wherein the processing operation further comprises:
transmitting a block map update request message to the networked storage system, the block map update request message including information for updating a block map at the networked storage system, the block map identifying a designated memory location at which each data encrypted chunk is stored at the networked storage system.
10. The method recited in claim 1 , the method further comprising:
identifying an amount of available computing resources at the networked storage system.
11. The method recited in claim 1 , wherein the data stream is generated at the client device via a network storage protocol selected from the group consisting of: the Network File System (NFS) protocol, the Common Internet File System (CIFS) protocol, and the Open Storage (OST) protocol.
12. A device comprising:
memory operable to store a data stream;
a processor operable to determine whether an amount of available computing resources at a client device meets or exceeds a computing resource availability threshold at the client device and perform a processing operation on the data stream to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by:
determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;
determining that the hash value for the first chunk qualifies as a chunk boundary; and
in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and
a communications interface operable to transmit the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.
13. The device recited in claim 12 , wherein the processing operation is an operation selected from the group consisting of: deduplicating the data stream, encrypting the data stream, and compressing the data stream.
14. The device recited in claim 12 , wherein the processing operation further comprises:
determining a respective fingerprint for each identified data chunk by applying a hash function to the identified data chunk.
15. The device recited in claim 14 , wherein the processing operation further comprises:
determining whether each identified data chunk is stored at a networked storage system by transmitting each fingerprint to the networked storage system via the network.
16. The device recited in claim 15 , wherein determining whether the identified chunk is stored at the networked storage system further comprises:
receiving a fingerprint status message from the networked storage system, the fingerprint status message indicating whether the chunk associated with the fingerprint is stored at the networked storage system.
17. One or more non-transitory computer readable media having instructions stored thereon to perform operations comprising:
determining whether an amount of available computing resources at a client device comprising a processor and memory meets or exceeds a computing resource availability threshold at the client device;
performing a processing operation on a data stream at the client device to produce a pre-processed data stream when it is determined that the amount of available computing resources meets or exceeds the computing resource availability threshold, wherein the processing operating comprises parsing the data stream to identify one or more data chunks, and wherein the one or more data chunks are identified via a designated rolling hash parsing technique operable to identify at least some identical chunks when parsing different but overlapping data streams, wherein the one or more data chunks are identified by:
determining, using the rolling hash parsing technique, a hash value for a first chunk of the at least some identical chunks;
determining that the hash value for the first chunk qualifies as a chunk boundary; and
in response to determining that the hash value for the first chunk qualifies as the chunk boundary, determining that the data stream has reached a chunk boundary; and
transmitting the pre-processed data stream for storage to a networked storage system via a network, the networked storage system operable to store deduplicated data for retrieval via the network, the networked storage system operable to parse data streams via the designated rolling hash parsing technique.