Method and device for deduplication
Embodiments of the present disclosure provide a method and device for deduplication. Specifically, the method may comprise obtaining a property of a file stream, the property of a file stream including a file type or a magic number identifying a format of a protocol or a file. The method further includes in response to receiving an I/O request for a data block of the file stream, assigning a deduplication level to the I/O request based on the property of the file stream. Moreover, the method further includes deduplicating the data block of the file stream based on the deduplication level assigned to the I/O request. In addition, a corresponding device and computer program product are provided.
1. A method of deduplication, comprising:
obtaining a property of a file stream, the property of the file stream including a file type or a magic number identifying a format of a protocol or a file;
in response to receiving an I/O request for a data block of the file stream, assigning a deduplication level to the I/O request based on the property of the file stream at least in part by determining whether the file stream is compressed based on the property of the file stream and assigning the deduplication level based on whether the file stream is compressed, wherein the assigning of the deduplication level to the I/O request includes, in response to a determination that the file stream is uncompressed, assigning a third deduplication level to the I/O request; and
deduplicating the data block of the file stream based on the deduplication level assigned to the I/O request, at least in part by,
in response to the assigned deduplication level being a first deduplication level, skipping deduplication of the data block of the file stream,
in response to the assigned deduplication level being a second deduplication level, deduplicating the data block of the file stream based on a state of an available computing resource, and
in response to the assigned deduplication level being the third deduplication level, deduplicating the data block of the file stream.
2. The method according to claim 1 , wherein assigning the deduplication level comprises:
in response to a determination that the file stream is compressed, assigning the first deduplication level to the I/O request.
3. The method according to claim 1 , wherein assigning the deduplication level comprises:
in response to failing to determine whether the file stream is compressed, assigning the second deduplication level to the I/O request.
4. The method according to claim 1 , wherein deduplicating the data block of the file stream based on the state of the available computing resource further comprises:
in response to an amount of the available computing resource being above a predetermined threshold, deduplicating the data block of the file stream; and
in response to the amount of the available computing resource being below the predetermined threshold, skipping deduplication of the data block of the file stream.
5. The method according to claim 1 , further comprising:
wherein assigning the deduplication level further comprises, in response to failing to determine whether the file stream is compressed, assigning a second deduplication level to the I/O request; and
wherein deduplicating the data block of the file stream further comprises, in response to the deduplication level assigned to the I/O request being the second deduplication level, skipping deduplication of the data block of the file stream regardless of a state of an available computing resource.
6. The method according to claim 1 , further comprising:
after assigning the deduplication level to the I/O request, calculating a fingerprint of the I/O request, the fingerprint identifying the data block for the I/O request;
storing the fingerprint in a cache; and
in response to a fingerprint of a subsequent I/O request after the I/O request hitting the fingerprint stored in the cache, assigning a deduplication level to the subsequent I/O request, the assigned deduplication level being the same as that of the I/O request corresponding to the hit fingerprint in the cache.
7. The method of claim 6 , further comprising:
maintaining the cache such that more fingerprints of I/O requests assigned a highest deduplication level are stored in the cache than fingerprints of other I/O requests.
8. The method of claim 7 , further comprising:
maintaining in the cache a predetermined ratio between fingerprints of I/O requests assigned the highest deduplication level and fingerprints of I/O requests that are assigned at least one other deduplication level.
9. An electronic device, comprising:
at least one processor; and
at least one memory coupled to the at least one processor and having machine-executable instructions stored thereon which, when executed by the at least one processor, causing the device to perform acts comprising:
obtaining a property of a file stream, the property of the file stream including a file type or a magic number identifying a format of a protocol or a file;
in response to receiving an I/O request for a data block of the file stream, assigning a deduplication level to the I/O request based on the property of the file stream at least in part by determining whether the file stream is compressed based on the property of the file stream and assigning the deduplication level based on whether the file stream is compressed, wherein the assigning of the deduplication level to the I/O request includes, in response to a determination that the file stream is uncompressed, assigning a third deduplication level to the I/O request; and
deduplicating the data block of the file stream based on the deduplication level assigned to the I/O request, at least in part by,
in response to the assigned deduplication level being a first deduplication level, skipping deduplication of the data block of the file stream,
in response to the assigned deduplication level being a second deduplication level, deduplicating the data block of the file stream based on a state of an available computing resource, and
in response to the assigned deduplication level being the third deduplication level, deduplicating the data block of the file stream.
10. The device according to claim 9 , wherein assigning the deduplication level comprises:
in response to a determination that the file stream is compressed, assigning the first deduplication level to the I/O request.
11. The device according to claim 9 , wherein assigning the deduplication level comprises:
in response to failing to determine whether the file stream is compressed, assigning the second deduplication level to the I/O request.
12. The device according to claim 9 , wherein deduplicating the data block of the file stream based on the state of the available computing resource further comprises:
in response to an amount of the available computing resource being above a predetermined threshold, deduplicating the data block of the file stream; and
in response to the amount of the available computing resource being below the predetermined threshold, skipping deduplication of the data block of the file stream.
13. The device according to claim 9 , further comprising:
wherein assigning the deduplication level further comprises, in response to failing to determine whether the file stream is compressed, assigning a second deduplication level to the I/O request; and
wherein deduplicating the data block of the file stream further comprises, in response to the deduplication level assigned to the I/O request being the second deduplication level, skipping deduplication of the data block of the file stream regardless of a state of an available computing resource.
14. The device according to claim 9 , wherein the acts further comprise:
after assigning the deduplication level to the I/O request, calculating a fingerprint of the I/O request, the fingerprint identifying the data block for the I/O request;
storing the fingerprint in a cache; and
in response to a fingerprint of a subsequent I/O request after the I/O request hitting the fingerprint stored in the cache, assigning a deduplication level to the subsequent I/O request, the assigned deduplication level being the same as that of the I/O request corresponding to the hit fingerprint in the cache.
15. A computer program product for deduplication, the computer program product comprising:
a non-transitory computer readable medium encoded with computer-executable code, the code configured for the execution of:
obtaining a property of a file stream, the property of the file stream including a file type or a magic number identifying a format of a protocol or a file;
in response to receiving an I/O request for a data block of the file stream, assigning a deduplication level to the I/O request based on the property of the file stream at least in part by determining whether the file stream is compressed based on the property of the file stream and assigning the deduplication level based on whether the file stream is compressed, wherein the assigning of the deduplication level to the I/O request includes, in response to a determination that the file stream is uncompressed, assigning a third deduplication level to the I/O request; and
deduplicating the data block of the file stream based on the deduplication level assigned to the I/O request, at least in part by,
in response to the assigned deduplication level being a first deduplication level, skipping deduplication of the data block of the file stream,
in response to the assigned deduplication level being a second deduplication level, deduplicating the data block of the file stream based on a state of an available computing resource, and
in response to the assigned deduplication level being the third deduplication level, deduplicating the data block of the file stream.