Containerized storage stream microservice
A containerized stream microservice is described. The containerized stream microservice is configured to provide the functionality of volume presentation along with all related interactions including the receipt and processing of IO requests and related services. The containerized stream microservice preferably implements stream metadata in the management of storage operations, and interacts with a store to provide underlying data storage. The store, which may also be referred to as a data store, is where underlying data is stored in a persistent manner. In one example, the store is an object store.
1. A method for data representation comprising:
receiving write transactions directed to a given volume; and
retaining stream metadata corresponding to the write transactions, the stream metadata identifying a sequence of the write transactions according to the order in which the write transactions are received for the given volume, and identifying respective location information for each of the write transactions, the location information including object identifiers respectively corresponding to the write transactions, and the location information including offsets respectively corresponding to each of the write transactions.
2. The method of claim 1 , further comprising:
associating a stream image to the stream metadata, the stream image providing a representation of the data in the given volume for a point in time by referencing a non-contiguous subset of the sequence of write transactions referenced in the stream metadata.
3. The method of claim 1 , wherein the object identifiers include a first object identifier corresponding to a current object.
4. The method of claim 3 , further comprising:
determining whether a current write transaction would exceed a capacity of the current object;
identifying a next object and updating the stream metadata to reference the next object for the current write transaction, where it is determined that the current write transaction would exceed the capacity of the current object.
5. The method of claim 3 , further comprising:
determining where a current write transaction would exceed a capacity of the current object; and
updating the stream metadata to reference the current object for the current write transaction, where it is determined that the current write transaction would not exceed the capacity of the current object.
6. The method of claim 1 , wherein receiving the write transactions and retaining the stream metadata are performed in a containerized microservice.
7. A non-transitory computer readable medium storing program code for providing a containerized stream microservice, the program code being executable by a processor to perform operations comprising:
receiving write transactions directed to a given volume; and
retaining stream metadata corresponding to the write transactions, the stream metadata identifying a sequence of the write transactions according to the order in which the write transactions are received for the given volume, and identifying respective location information for each of the write transactions, the location information including object identifiers respectively corresponding to the write transactions, and the location information including offsets respectively corresponding to each of the write transactions.
8. The non-transitory computer readable medium of claim 7 , wherein the operations further comprise:
associating a stream image to the stream metadata, the stream image providing a representation of the data in the given volume for a point in time by referencing a non-contiguous subset of the sequence of write transactions referenced in the stream metadata.
9. The non-transitory computer readable medium of claim 7 , wherein the object identifiers include a first object identifier corresponding to a current object.
10. The non-transitory computer readable medium of claim 9 , wherein the operations further comprise:
determining whether a current write transaction would exceed a capacity of the current object;
identifying a next object and updating the stream metadata to reference the next object for the current write transaction, where it is determined that the current write transaction would exceed the capacity of the current object.
11. The non-transitory computer readable medium of claim 9 , wherein the operations further comprise:
determining where a current write transaction would exceed a capacity of the current object; and
updating the stream metadata to reference the current object for the current write transaction, where it is determined that the current write transaction would not exceed the capacity of the current object.
12. The non-transitory computer readable medium of claim 7 , wherein receiving the write transactions and retaining the stream metadata are performed in a containerized microservice.
13. An apparatus comprising:
a processing platform including an operating system and hosting an application container and a containerized stream microservice, the containerized stream microservice being configured to:
receive write transactions directed to a given volume by the application container; and
retain stream metadata corresponding to the write transactions, the stream metadata identifying a sequence of the write transactions according to the order in which the write transactions are received for the given volume, and identifying respective location information for each of the write transactions, the location information including object identifiers respectively corresponding to the write transactions, and the location information including offsets respectively corresponding to each of the write transactions.
14. The apparatus of claim 13 , wherein the object identifiers include a first object identifier corresponding to a current object.
15. The apparatus of claim 14 , wherein the operations further comprise:
determining whether a current write transaction would exceed a capacity of the current object;
identifying a next object and updating the stream metadata to reference the next object for the current write transaction, where it is determined that the current write transaction would exceed the capacity of the current object.
16. The apparatus of claim 14 , wherein the operations further comprise:
determining where a current write transaction would exceed a capacity of the current object; and
updating the stream metadata to reference the current object for the current write transaction, where it is determined that the current write transaction would not exceed the capacity of the current object.
17. The apparatus of claim 13 , wherein receiving the write transactions and retaining the stream metadata are performed in a containerized microservice.