Method and system for in-memory metadata reduction in cloud storage system
Methods and systems for in-memory metadata reduction in cloud storage system are provided. According to an aspect, a method comprises receiving a first command to write a data stream to a storage device; writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and generating an index for individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment. Individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous, and the index indicates the information including at least: an offset value of the individual record in the individual fragment; the pre-set logical size of the individual record; and a pre-set physical size of the individual record.
1. A method comprising:
receiving a first command to write a data stream to a storage device;
writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and
generating an index for an individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment, the individual fragment including a plurality of groups, an individual group having a same number of individual records and all individual records in the individual group being continuous.
2. The method of claim 1 , wherein individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous.
3. The method of claim 2 , wherein the index indicates the information including at least:
an offset value of an individual record in the individual fragment;
a pre-set logical size of the individual record; and
a pre-set physical size of the individual record.
4. The method of claim 3 , further comprising:
receiving a second command to read the data stream from the storage device;
mapping the logical addresses of the plurality of fragments into the physical addresses based at least in part on the offset value of the individual record in the individual fragment, the pre-set logical size of the individual record, and the pre-set physical size of the individual record; and
read the data stream from the storage device according to the physical addresses of the plurality of fragments.
5. The method of claim 3 , further comprising:
receiving a third command to delete a portion of individual records from the individual fragment;
labeling the portion of individual records as invalid records; and
converting the index of the individual fragment to include a parameter associated with the invalid records.
6. The method of claim 5 , further comprising:
mapping the logical addresses of the plurality of fragments into the physical addresses based at least in part on the offset value of the individual record in the individual fragment, the pre-set logical size of the individual record, the pre-set physical size of the individual record, and the parameter associated with the invalid records.
7. The method of claim 1 , wherein the index indicates the information including at least:
a first offset value of an individual record in the individual fragment;
a group record count;
a pre-set logical size of the individual record;
a pre-set physical size of the individual record; and
second offset values associated with individual groups.
8. The method of claim 7 , further comprising:
mapping the logical addresses of the plurality of fragments into the physical addresses based at least in part on the first offset value of the individual record in the individual fragment, the group record count, the pre-set logical size of the individual record, the pre-set physical size of the individual record, and the second offset values associated with individual groups.
9. A system comprising:
one or more processors, and
memory communicatively coupled to the one or more processors, the memory storing computer-executable modules executable by the one or more processors that, when executed by the one or more processors, perform actions including:
receiving a first command to write a data stream to a storage device;
writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and
generating an index for an individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment, the individual fragment including a plurality of groups, an individual group having a same number of individual records and all individual records in the individual group being continuous.
10. The system of claim 9 , wherein individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous, and the index indicates the information including at least:
an offset value of an individual record in the individual fragment;
a pre-set logical size of the individual record; and
a pre-set physical size of the individual record.
11. The system of claim 10 , wherein the actions further comprises:
receiving a second command to read the data stream from the storage device;
mapping the logical addresses of the plurality of fragments into the physical addresses based at least in part on the offset value of the individual record in the individual fragment, the pre-set logical size of the individual record, and the pre-set physical size of the individual record; and
read the data stream from the storage device according to the physical addresses of the plurality of fragments.
12. The system of claim 10 , wherein the actions further comprises:
receiving a third command to delete a portion of individual records from the individual fragment;
labeling the portion of individual records as invalid records; and
converting the index of the individual fragment to include a parameter associated with the invalid records.
13. The system of claim 12 , wherein the actions further comprises:
mapping the logical addresses of the plurality of fragments into the physical addresses based at least in part on the offset value of the individual record in the individual fragment, the pre-set logical size of the individual record, the pre-set physical size of the individual record, and the parameter associated with the invalid records.
14. The system of claim 9 , wherein the index indicates the information including at least:
a first offset value of an individual record in the individual fragment;
a group record count;
a pre-set logical size of the individual record;
a pre-set physical size of the individual record; and
second offset values associated with individual groups.
15. One or more non-transitory computer-readable storage media storing computer-readable instructions executable by one or more processors of a video compression system, that when executed by the one or more processors, cause the one or more processors to perform actions comprising:
receiving a first command to write a data stream to a storage device;
writing the data stream into a plurality of fragments having logical addresses corresponding to physical addresses on the storage device; and
generating an index for an individual fragment of the plurality of fragments, the index indicating information to locate the physical addresses of the individual fragment, the individual fragment including a plurality of groups, an individual group having a same number of individual records and all individual records in the individual group being continuous.
16. The one or more non-transitory computer-readable storage media of claim 15 , wherein individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous, and the index indicates the information including at least:
an offset value of an individual record in the individual fragment;
a pre-set logical size of the individual record; and
a pre-set physical size of the individual record.
17. The one or more non-transitory computer-readable storage media of claim 15 , wherein the actions further comprises:
receiving a third command to delete a portion of individual records from the individual fragment;
labeling the portion of individual records as invalid records; and
converting the index of the individual fragment to include a parameter associated with the invalid records.
18. The one or more non-transitory computer-readable storage media of claim 15 , wherein the index indicates the information including at least:
a first offset value of an individual record in the individual fragment;
a group record count;
a pre-set logical size of the individual record;
a pre-set physical size of the individual record; and
second offset values associated with individual groups.
19. The system of claim 9 , wherein individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous.
20. The one or more non-transitory computer-readable storage media of claim 15 , wherein individual records in the individual fragment have a same pre-set logical size and all individual records in the individual fragment are continuous.