Organizing present and future reads from a tiered streaming data storage layer
Implementations are provided herein for organizing present and future reads from a tiered streaming data storage layer. Implementations allow for access to multi-tiered streaming data organized in different append-only segments, some of which may be related to each other. Streaming data can be read from fast local tier 1 storage, streaming data can be retrieved from fold tier 2 storage, and registrations can be made to read streaming data that has not yet been written to the storage layer.
1. A method comprising:
receiving a lookup request from a caller, wherein the lookup request includes a segment, an offset, and a maximum length, wherein the offset identifies a starting position of a block of data corresponding to the segment and the maximum length corresponds to a number of bytes; and
establishing an iterator based on the segment and the offset, wherein the iterator returns one of the following to the caller for each iteration of a period based on the maximum length:
a next set of data responsive to the lookup request wherein the next set of data is stored in a cache memory of a receiving device of the lookup request;
a storage read result; or
a future read result.
2. The method of claim 1 , further comprising:
receiving a notification from the caller for a storage read based on the iterator return;
reading a set of data responsive to the storage read from tier 2 storage;
storing the set of data in the cache memory; and
returning the set of data to the caller.
3. The method of claim 1 , further comprising:
receiving a notification from the caller for a future read based on the iterator return;
registering the future read in a read index;
in response to the read index indicating that a segment associated with the future read has a new append, determining a set of data responsive to the future read; and
returning the set of data to the caller.
4. The method of claim 3 , further comprising:
pausing the iterator in response to registering the future read in the read index, wherein determining the set of data responsive to the future read restarts the iterator.
5. The method of claim 1 , wherein the segment identifier and the offset are a key and a cache location is a value in a key value database.
6. The method of claim 1 , wherein the iterator returns the future read result based on a current read offset reaching an end of the segment.
7. A system comprising at least one storage device and at least one hardware processor configured to:
receive a lookup request from a caller, wherein the lookup request includes a segment, an offset, and a maximum length, wherein the offset identifies a starting position of a block of data corresponding to the segment and the maximum length corresponds to a number of bytes; and
establish an iterator based on the segment and the offset, wherein the iterator returns one of the following to the caller for each iteration of a period based on the maximum length:
a next set of data responsive to the lookup request wherein the next set of data is stored in a cache memory of a receiving device of the lookup request;
a storage read result; or
a future read result.
8. The system of claim 7 , further configured to:
receive a notification from the caller for a storage read based on the iterator return;
read a set of data responsive to the storage read from tier 2 storage;
store the set of data in the cache memory; and
return the set of data to the caller.
9. The system of claim 7 , further configured to:
receiving a notification from the caller for a future read based on the iterator return;
registering the future read in a read index;
in response to the read index indicating that a segment associated with the future read has a new append, determining a set of data responsive to the future read; and
returning the set of data to the caller.
10. The system of claim 9 , further configured to:
pausing the iterator in response to registering the future read in the read index, wherein determining the set of data responsive to the future read restarts the iterator.
11. The system of claim 7 , wherein the segment identifier and the offset are a key and a cache location is a value in a key value database.
12. The system of claim 7 , wherein the iterator returns the future read result based on a current read offset reaching an end of the segment.
13. A non-transitory computer readable medium with program instructions stored thereon to perform the following acts:
receiving a lookup request from a caller, wherein the lookup request includes a segment, an offset, and a maximum length, wherein the offset identifies a starting position of a block of data corresponding to the segment and the maximum length corresponds to a number of bytes; and
establishing an iterator based on the segment and the offset, wherein the iterator returns one of the following to the caller for each iteration of a period based on the maximum length:
a next set of data responsive to the lookup request wherein the next set of data is stored in a cache memory of a receiving device of the lookup request;
a storage read result; or
a future read result.
14. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further perform the following acts:
receiving a notification from the caller for a storage read based on the iterator return;
reading a set of data responsive to the storage read from tier 2 storage;
storing the set of data in the cache memory; and
returning the set of data to the caller.
15. The non-transitory computer readable medium of claim 13 , with program instructions stored thereon to further perform the following acts:
receiving a notification from the caller for a future read based on the iterator return;
registering the future read in a read index;
in response to the read index indicating that a segment associated with the future read has a new append, determining a set of data responsive to the future read; and
returning the set of data to the caller.
16. The non-transitory computer readable medium of claim 15 , with program instructions stored thereon to further perform the following acts:
pausing the iterator in response to registering the future read in the read index, wherein determining the set of data responsive to the future read restarts the iterator.
17. The non-transitory computer readable medium of claim 13 , wherein the segment identifier and the offset are a key and a cache location is a value in a key value database.
18. The non-transitory computer readable medium of claim 13 , wherein the iterator returns the future read result based on a current read offset reaching an end of the segment.