B-file abstraction for efficiently archiving self-expiring data
View Patent ↗Systems and methods are provided for data processing and storage management. In an illustrative implementation an exemplary computing environment comprises at least one data store, a data processing and storage management engine (B-File engine) and at least one instruction set to instruct the B-File engine to process and/or store data according to a selected data processing and storage management paradigm. In an illustrative operation, the illustrative B-File engine can generate a B-File comprising multiple buckets and store sample items in a random bucket according to a selected distribution. When the size of the B-FILE grows to reach a selected threshold (e.g., maximum available space), the B-File engine can shrink the B-File by discarding the largest bucket. Additionally, the B-File engine can append data to existing buckets and explicitly cluster data when erasing data such that data can be deleted together into the same flash block.
1. A method comprising:
receiving data items for storage on a solid state persistent storage device having multiple storage blocks;
creating a file having multiple storage buckets for storing the data items;
determining different expiration intensities for the data items, wherein the expiration intensities of the data items are determined by one or more attributes of the data items;
assigning the data items to different assigned storage buckets of the file based at least on the expiration intensities;
designating different in-memory page buffers for the assigned storage buckets and temporarily storing the data items in the in-memory page buffers for the assigned storage buckets;
flushing the data items from the in-memory page buffers to the solid state persistent storage device by appending the data items to previously-stored data items such that data items within a given assigned storage bucket are clustered together in corresponding blocks of the solid state persistent storage device; and
erasing storage blocks of an expiring storage bucket on the solid state persistent storage device via one or more erase operations, wherein the expiring storage bucket is selected for the erasing based at least on the expiration intensity of corresponding expiring data items stored in the expiring storage bucket.
2. The method of claim 1 , wherein the one or more attributes of the data items used to establish the expiration intensities include at least one non-temporal attribute.
3. The method of claim 2 , wherein the at least one non-temporal attribute used to establish the expiration intensities is a priority of the data items.
4. The method of claim 3 , wherein, as the data items are received, the data items are appended within the corresponding blocks without overwriting existing data in the corresponding blocks.
5. The method of claim 1 , wherein the data items are flushed as the in-memory page buffers become full.
6. The method of claim 5 , wherein the erasing comprises erasing all storage blocks that store data items from the expiring storage bucket.
7. A system comprising:
a non-volatile solid state storage device having multiple storage blocks;
a memory device; and
a control device configured to:
obtain data items for storage on the non-volatile solid state storage device, the data items having associated attributes;
establish different expiration intensities for the data items using the associated attributes of the data items;
assign the data items to different assigned storage buckets of a file based at least on the different expiration intensities;
designate different in-memory page buffers for the assigned storage buckets and temporarily store the data items in the in-memory page buffers on the memory device;
flush the data items from the in-memory page buffers to the non-volatile solid state storage device by appending the data items to previously-stored data items that are already stored on the non-volatile solid state storage device, wherein the data items and the previously-stored data items are clustered together by storage bucket assignments in individual blocks of the non-volatile solid state storage device; and
erase certain storage blocks of the solid state storage device that store expiring data items from an expiring storage bucket based at least on corresponding expiration intensities of the expiring data items.
8. The system of claim 7 , wherein the control device comprises a hardware processing unit, a solid state storage device controller, or both.
9. The system of claim 7 , wherein the control device is configured to:
monitor sizes of the multiple storage buckets over time; and
as particular storage buckets reach a specified bucket size, designate the particular storage buckets for erasing.
10. The system of claim 7 , wherein the memory device is volatile.
11. The system of claim 10 , wherein the control device is further configured to:
maintain a first in-memory page buffer of first data items assigned to a first bucket and a second in-memory page buffer of second data items assigned to a second bucket;
at a first time when the first in-memory page buffer becomes full, flush the first in-memory page buffer of first data items to the non-volatile solid state storage device by appending the first data items to other first data items assigned to the first bucket that are already stored on the non-volatile solid state storage device; and
at a second time when the second in-memory page buffer becomes full, flush the second in-memory page buffer of second data items to the non-volatile solid state storage device by appending the second data items to other second data items assigned to the second bucket that are already stored on the non-volatile solid state storage device.
12. The system of claim 10 , wherein the control device is further configured to:
chain corresponding blocks of a first individual bucket together using pointers; and
maintain an in-memory block pointer that points to a particular block of the first individual bucket.
13. A system comprising:
a non-volatile solid state storage device having multiple storage blocks;
a hardware processing unit; and
a memory device storing computer-readable instructions that cause the hardware processing unit to:
obtain data items for storage on the non-volatile solid state storage device, the data items having associated attributes;
establish different expiration intensities for the data items using the associated attributes of the data items;
assign the data items to different assigned storage buckets based at least on the different expiration intensities;
designate different in-memory page buffers for the assigned storage buckets and temporarily store the data items in the in-memory page buffers on the memory device;
flush the data items from the in-memory page buffers to the non-volatile solid state storage device by appending the data items to previously-stored data items that are already stored on the non-volatile solid state storage device, wherein the data items and the previously-stored data items for a given storage bucket are clustered together in individual blocks of the non-volatile solid state storage device; and
erase individual storage blocks that store expiring data items from an expiring storage bucket based at least on corresponding expiration intensities of the expiring data items, the individual storage blocks being erased via one or more block erase operations.
14. The system of claim 13 , wherein the computer-readable instructions cause the hardware processing unit to:
repeatedly make determinations that a size of a file containing the different storage buckets has reached a threshold; and
repeatedly erase the individual storage blocks of the expiring storage bucket responsive to the determinations.
15. The system of claim 13 , wherein the computer-readable instructions cause the hardware processing unit to:
append first data items of a first storage bucket to other first data items of the first storage bucket on the non-volatile solid state storage device without overwriting any of the other first data items.
16. The system of claim 13 , wherein the different storage buckets have different sizes.
17. The system of claim 13 , wherein the one or more block erase operations are performed such that all data items with the same expiration intensity are deleted together.
18. The system of claim 13 , wherein the non-volatile solid state storage device is a flash storage device.
19. The system of claim 13 , wherein the computer-readable instructions cause the hardware processing unit to:
flush the in-memory page buffers of the data items as individual in-memory page buffers reach one page worth of data.
20. The system of claim 19 , wherein each in-memory page buffers stores only particular data items that each share a particular expiration intensity for an associated storage bucket.