Granted Patent B2

US 10,572,153 · App. 15/621,960 · Granted Feb 25, 2020

Inventors: Ashish Singhai (Los Altos, CA); Ajith Kumar Battaje (Karnataka, IN); Sandeep Sharma (Karnataka, IN); Saurabh Manchanda (Delhi, IN)

Assignee: Western Digital Technologies, Inc.

G06F3/0608G06F3/064G06F3/0641G06F3/0673G06F12/023G06F12/0292

Abstract

Claims (70)

1. A system comprising:

a non-transitory storage medium for storing and providing access to data;

an interface processor coupled to an application, the interface processor configured to receive a read compressed request from the application; and

a controller logic coupled to an output of the interface processor, the controller logic configured to, responsive to receiving the read compressed request:

retrieve a compressed payload from the non-transitory storage medium;

generate a data container having a data container format and including the compressed payload;

determine attributes of a fixed size metadata;

determine attributes of a variable size metadata;

attach, as metadata to the data container, the fixed size metadata and the variable size metadata; and

return the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application for integrity verification.

2. The system of claim 1 , wherein; the fixed size metadata includes data that is opaque to the application; and the variable size metadata includes data interpretable by the application.

3. The system of claim 1 , wherein the metadata includes validity check information for verifying an integrity of the compressed payload, the validity check information for verifying the integrity of the compressed payload being configured to be interpreted by the application.

4. The system of claim 1 , wherein the metadata includes validity check information for verifying an integrity of an uncompressed form of the compressed payload, the validity check information for verifying the integrity the uncompressed form of the compressed payload being configured to be opaque to the application.

5. The system of claim 1 , wherein the controller logic is further configured to, prior to receiving the read compressed request:

encode data into the compressed payload using data deduplication; and

store the compressed payload in the non-transitory storage medium.

6. The system of claim 5 , wherein encoding the data into the compressed payload includes:

processing a data block of the data, the data block comprising a plurality of pieces of data;

determining whether a second piece of data of the plurality of pieces of data has a first threshold level of similarity to a first piece of data of the plurality of pieces of data;

determining whether the second piece of data has a second threshold level of similarity to a reference block, the reference block representing a piece of data that is not in the data block; and

encoding the data block based on the determination that the second piece of data has the first threshold level of similarity to the first piece of data and the determination that the second piece of data has the second threshold level of similarity to the reference block.

7. The system of claim 5 , wherein;

the interface processor is executable to receive a query from the application for a stream of data, the stream of data including compressed data;

the controller logic is executable to;

determine an oldest reference set for the compressed data, the oldest reference set including at least one reference block;

insert the at least one reference block of the oldest reference set into the variable size metadata; and

construct one or more second data containers using data blocks that refer to the oldest reference set; and

the interface processor is executable to transmit the one or more second data containers to the application.

8. The system of claim 7 , wherein:

the controller logic is further executable to, in response to all data blocks referring to the oldest reference set being inserted into the one or more second data containers;

determine a next oldest reference set, the next oldest reference set including at least one second reference block;

insert the at least one second reference block of the next oldest reference set into the variable size metadata; and

insert data blocks that refer to the next oldest reference set into the one or more second data containers.

9. The system of claim 1 , wherein the fixed size metadata includes data that is interpretable by the application to allow the application to interpret the variable size metadata.

10. A method comprising:

receiving, by an interface processor coupled to an application, a read compressed request from the application;

responsive to receiving the read compressed request, retrieving, by a controller logic coupled to an output of the interface processor, a compressed payload from a non-transitory storage medium;

generating a data container having a data container format and including the compressed payload;

determining attributes of a fixed size metadata;

determining attributes of a variable size metadata;

attaching, as metadata to the data container, the fixed size metadata and the variable size metadata; and

returning the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application for integrity verification.

11. The method of claim 10 , wherein; the fixed size metadata includes data that is opaque to the application; and the variable size metadata includes data interpretable by the application.

12. The method of claim 10 , wherein the metadata includes validity check information for verifying an integrity of the compressed payload, the validity check information for verifying the integrity of the compressed payload being configured to be interpreted by the application.

13. The method of claim 10 , wherein the metadata includes validity check information for verifying an integrity of an uncompressed form of the compressed payload, the validity check information for verifying the integrity the uncompressed form of the compressed payload being configured to be opaque to the application.

14. The method of claim 10 , further comprising:

prior to receiving the read compressed request, encoding, by the controller logic, data into a compressed payload using data deduplication; and

storing the compressed payload in the non-transitory storage medium.

15. The method of claim 14 , wherein encoding the data into the compressed payload includes:

processing a data block of the data, the data block comprising a plurality of pieces of data;

determining whether a second piece of data of the plurality of pieces of data has a first threshold level of similarity to a first piece of data of the plurality of pieces of data;

determining whether the second piece of data has a second threshold level of similarity to a reference block, the reference block representing a piece of data that is not in the data block; and

encoding the data block based on the determination that the second piece of data has the first threshold level of similarity to the first piece of data and the determination that the second piece of data has the second threshold level of similarity to the reference block.

16. The method of claim 14 , further comprising;

querying, by the application, the interface processor for a stream of data, the stream of data including compressed data;

determining, by the controller logic, an oldest reference set for the compressed data, the oldest reference set including at least one reference block;

inserting, by the controller logic, the at least one reference block of the oldest reference set into the variable size metadata;

constructing, by the controller logic, one or more second data containers using data blocks that refer to the oldest reference set; and

transmitting, by the interface processor, the one or more second data containers to the application.

17. The method of claim 16 , further comprising:

in response to all data blocks referring to the oldest reference set being inserted into the one or more second data containers;

determining, by the controller logic, a next oldest reference set, the next oldest reference set including at least one second reference block;

inserting, by the controller logic, the at least one second reference block of the next oldest reference set into the variable size metadata; and

inserting, by the controller logic, data blocks that refer to the next oldest reference set into the one or more second data containers.

18. The method of claim 10 , wherein the fixed size metadata includes data that is interpretable by the application to allow the application to interpret the variable size metadata.

19. A system comprising:

means for storing and providing access to data;

means for receiving a read compressed request from an application; and

means for receiving the read compressed request, and responsive to receiving the read compressed request, retrieving a compressed payload from a non-transitory storage medium, generating a data container having a data container format and including the compressed payload, determining attributes of a fixed size metadata, determining attributes of a variable size metadata, attaching, as metadata to the data container, the fixed size metadata and the variable size metadata, and returning the data container having the data container format and including the compressed payload, the fixed size metadata, and the variable size metadata to the application for integrity verification.

20. The system of claim 19 , wherein; the fixed size metadata includes data that is opaque to the application; and the variable size metadata includes data interpretable by the application.

Assignments (10)

Continuity (2)

Cited By (5)

US 12,206,671 US 12,260,084 US 12,373,393 US 12,554,718 US 12,704,960