Dynamic memory buffering using containers
Dynamic memory allocation is provided using containers and overlay snaps. A first container is generated to receive a first data stream, and an associated overlay snap is generated to which the data of the first container is transmitted. In response to a storage capacity of the first container reaching a threshold, a second container is generated to receive a second data stream. A second overlay snap associated with the second container is generated, and data of the second container is transmitted to the second overlay snap.
1. A computer-implemented method, comprising:
receiving a number of data streams, including a first data stream received between an initial time to a first time and a second data stream received between the first time and a second time;
generating a first container to receive the first data stream, wherein the first container has a first storage capacity;
generating a first overlay snap associated with the first container by generating an initial snapshot of logical storage volume, and transmitting data of the first container to the first overlay snap;
in response to the first storage capacity reaching a threshold, generating a second container to receive the second data stream, wherein the second container has a second storage capacity; and
generating a second overlay snap associated with the second container by generating a second initial snapshot of a second logical storage volume, and transmitting data of the second container to the second overlay snap.
2. The computer-implemented method of claim 1 wherein the first overlay snap is further generated by writing the first data stream to the first overlay snap, generating first pointers indicating locations of new data written to the first overlay snap, and generating first null pointers for locations that do not have new data written to the first overlay snap, wherein the first overlay snap comprises the first pointers and the first null pointers, and wherein the second overlay snap is further generated by writing the second data stream to the second overlay snap, generating second pointers indicating locations of new data written to the second overlay snap, and generating second null pointers for locations that do not have new data written to the second overlay snap, wherein the second overlay snap comprises the second pointers and the second null pointers.
3. The computer-implemented method of claim 1 wherein the threshold is selected to allow for the second container to be generated before the first storage capacity of the first container becomes full.
4. The computer-implemented method of claim 1 , the method further comprising:
attaching the first overlay snap to a storage volume to generate a first snapshot, wherein the first snapshot records information of a storage volume at a first point in time; and
after attaching the first overlay snap, attaching the second overlay snap to the storage volume to generate a second snapshot, wherein the second snapshot records information of the storage volume at a second point in time.
5. The computer-implemented method of claim 1 , wherein the first overlay snap and the second overlay snap are applied to a storage volume in an order determined by an order of the first and second containers being generated.
6. The computer-implemented method of claim 5 , wherein the first overlay snap is applied to the storage volume in response to completion of the data of the first container being transmitted to the first overlay snap.
7. The computer-implemented method of claim 6 , wherein completion of data of the first container being transmitted to the first overlay snap includes resolving any errors in the data.
8. An apparatus comprising:
electronic hardware circuitry, including a processor and memory configured to:
receive a number of data streams, including a first data stream received between an initial time to a first time and a second data stream received between the first time and a second time;
generate a first container to receive the first data stream, wherein the first container has a first storage capacity;
generate a first overlay snap associated with the first container by generating an initial snapshot of a logical storage volume, and transmit data of the first container to the first overlay snap;
in response to the first storage capacity reaching a threshold, generate a second container to receive the second data stream, wherein the second container has a second storage capacity; and
generate a second overlay snap associated with the second container by generating a second initial snapshot of a second logical storage volume, and transmit data of the second container to the second overlay snap.
9. The apparatus of claim 8 wherein the first overlay snap is further generated by writing the first data stream to the first overlay snap, generating first pointers indicating locations of new data written to the first overlay snap, and generating first null pointers for locations that do not have new data written to the first overlay snap, wherein the first overlay snap comprises the first pointers and the first null pointers, and wherein the second overlay snap is further generated by writing the second data stream to the second overlay snap, generating second pointers indicating locations of new data written to the second overlay snap, and generating second null pointers for locations that do not have new data written to the second overlay snap, wherein the second overlay snap comprises the second pointers and the second null pointers.
10. The apparatus of claim 8 wherein the threshold is selected to allow for the second container to be generated before the first storage capacity of the first container becomes full.
11. The apparatus of claim 8 , the electronic hardware circuitry further configured to:
attach the first overlay snap to a storage volume to generate a first snapshot, wherein the first snapshot records information of a storage volume at a first point in time; and
after attaching the first overlay snap, attach the second overlay snap to the storage volume to generate a second snapshot, wherein the second snapshot records information of the storage volume at a second point in time.
12. The apparatus of claim 8 wherein the first overlay snap and the second overlay snap are applied to a storage volume in an order determined by an order of the first and second containers being generated.
13. The apparatus of claim 12 wherein the first overlay snap is applied to the storage volume in response to completion of the data of the first container being transmitted to the first overlay snap.
14. The apparatus of claim 13 wherein completion of data of the first container being transmitted to the first overlay snap includes resolving any errors in the data.
15. An article comprising:
a non-transitory computer-readable medium that stores computer-executable instructions, the instructions causing a machine to:
receive a number of data streams, including a first data stream received between an initial time to a first time and a second data stream received between the first time and a second time;
generate a first container to receive the first data stream, wherein the first container has a first storage capacity;
generate a first overlay snap associated with the first container by generating an initial snapshot of a logical storage volume, and transmit data of the first container to the first overlay snap;
in response to the first storage capacity reaching a threshold, generate a second container to receive the second data stream, wherein the second container has a second storage capacity; and
generate a second overlay snap associated with the second container by generating a second initial snapshot of a second logical storage volume, and transmit data of the second container to the second overlay snap.
16. The article of claim 15 wherein the first overlay snap is further generated by writing the first data stream to the first overlay snap, generating first pointers indicating locations of new data written to the first overlay snap, and generating first null pointers for locations that do not have new data written to the first overlay snap, wherein the first overlay snap comprises the first pointers and the first null pointers, and wherein the second overlay snap is further generated by writing the second data stream to the second overlay snap, generating second pointers indicating locations of new data written to the second overlay snap, and generating second null pointers for locations that do not have new data written to the second overlay snap, wherein the second overlay snap comprises the second pointers and the second null pointers.
17. The article of claim 15 wherein the threshold is selected to allow for the second container to be generated before the first storage capacity of the first container becomes full.
18. The article of claim 15 , the instructions further causing the machine to:
attach the first overlay snap to a storage volume to generate a first snapshot, wherein the first snapshot records information of a storage volume at a first point in time; and
after attaching the first overlay snap, attach the second overlay snap to the storage volume to generate a second snapshot, wherein the second snapshot records information of the storage volume at a second point in time.
19. The article of claim 15 wherein the first overlay snap and the second overlay snap are applied to a storage volume in an order determined by an order of the first and second containers being generated.
20. The article of claim 19 wherein the first overlay snap is applied to the storage volume in response to completion of the data of the first container being transmitted to the first overlay snap.
21. The article of claim 20 wherein completion of data of the first container being transmitted to the first overlay snap includes resolving any errors in the data.