Virtual data streams in a data streaming platform
An example method includes receiving, by a data streaming platform from a client, a write request to write data to a data stream; storing, by the data streaming platform and based on the receiving the write request, the data in a particular partition of a plurality of partitions of the data stream and on a particular cluster of a plurality of clusters of the data streaming platform; translating, by the data streaming platform, an actual offset specifying the particular cluster and the particular partition into a virtual offset defining an ordering of the data relative to other data in the data stream stored across the plurality of partitions and the plurality of clusters; and storing, by the data streaming platform, a mapping of the virtual offset to the actual offset.
1. A method comprising:
receiving, by a data streaming platform from a client, a write request to write data to a data stream;
storing, by the data streaming platform and based on the receiving the write request, the data in a particular partition of a plurality of partitions of the data stream and on a particular cluster of a plurality of clusters of the data streaming platform;
translating, by the data streaming platform, an actual offset specifying the particular cluster and the particular partition into a virtual offset defining an ordering of the data relative to other data in the data stream stored across the plurality of partitions and the plurality of clusters; and
storing, by the data streaming platform, a mapping of the virtual offset to the actual offset.
2. The method of claim 1 , further comprising providing, to the client, the virtual offset.
3. The method of claim 1 , further comprising adding, based on an increase in client requests to the data stream, an additional partition to the plurality of partitions of the data stream on an additional cluster of the plurality of clusters.
4. The method of claim 1 , further comprising:
initiating a new data stream;
migrating the data stream to the new data stream; and
remapping the virtual offset to a new actual offset of the new data stream corresponding to the data.
5. The method of claim 4 , wherein:
the initiating is based on a decrease in client requests to the data stream; and
the new data stream comprises a new plurality of partitions having fewer partitions than the plurality of partitions.
6. The method of claim 4 , wherein:
the plurality of clusters are located in an availability zone; and
the new data stream comprises a new plurality of partitions on an additional plurality of clusters in an additional availability zone different from the availability zone.
7. The method of claim 6 , wherein the initiating is based on a failure of a cluster in the availability zone.
8. The method of claim 6 , wherein the initiating is based on a determining a resource optimization by migrating the data stream to the additional availability zone.
9. The method of claim 1 , further comprising:
detecting a failure in the particular cluster;
adding an additional partition to the plurality of partitions of the data stream on an additional cluster of the plurality of clusters;
migrating the particular partition to the additional partition; and
remapping the virtual offset to a new actual offset on the additional partition corresponding to the data.
10. The method of claim 1 , wherein the virtual offset comprises a tuple specifying a slice of the data stream over a particular time period and a particular offset.
11. The method of claim 10 , further comprising:
providing, to the client, a lease associated with the slice of the data stream;
initiating a new slice of the data stream corresponding to a new time period; and
restricting write requests from the client to the slice of the data stream upon at least one of:
an expiration of the lease; or
a relinquishing of the lease by the client acknowledging the new slice.
12. The method of claim 11 , further comprising:
restricting, upon the at least one of the expiration of the lease or the relinquishing of the lease by the client and any additional clients writing to the data stream, write requests to the slice of the data stream; and
defining, based on the restricting, an endpoint of the slice.
13. The method of claim 1 , wherein the virtual offset comprises a tuple specifying the data stream, the particular partition, and a particular offset.
14. The method of claim 1 , further comprising:
receiving, from an additional client, a read request to read the data from the data stream, the data specified by the virtual offset;
translating, based on the receiving the read request, the virtual offset into the actual offset; and
providing, to the additional client and based on the actual offset, the data.
15. A method comprising:
receiving, by a data streaming platform from a client, a read request to read data from a data stream, the data specified by a virtual offset defining an ordering of the data relative to other data in the data stream;
translating, by the data streaming platform, the virtual offset into an actual offset specifying a particular partition of a plurality of partitions of the data stream and a particular cluster of a plurality of clusters of the data streaming platform; and
providing, by the data streaming platform to the client and based on the actual offset, the data.
16. The method of claim 15 , wherein the virtual offset comprises a tuple specifying the data stream, the particular partition, and a particular offset.
17. The method of claim 15 , wherein:
the data stream comprises a plurality of slices associated with time periods of the data stream; and
the plurality of slices comprises a current slice and a previous slice for which write requests are restricted and an endpoint is defined.
18. The method of claim 17 , wherein:
the read request is for data at the endpoint of the previous slice; and
the method further comprises receiving, subsequent to the providing the data and from the client, an additional read request to read additional data from the data stream, the additional data specified by an additional virtual offset that specifies the additional data at a start of the current slice.
19. The method of claim 17 , wherein:
the read request is for data at a current end of the current slice; and
the method further comprises directing the client to wait for additional data to be written to the data stream or a defining of an endpoint of the current slice.
20. A non-transitory, computer-readable medium storing computer instructions that, when executed, direct one or more processors of one or more computing devices to perform a process comprising:
receiving, by a data streaming platform from a client, a write request to write data to a data stream;
storing, by the data streaming platform and based on the receiving the write request, the data in a particular partition of a plurality of partitions of the data stream and on a particular cluster of a plurality of clusters of the data streaming platform;
translating, by the data streaming platform, an actual offset specifying the particular cluster and the particular partition into a virtual offset defining an ordering of the data relative to other data in the data stream stored across the plurality of partitions and the plurality of clusters; and
storing, by the data streaming platform, a mapping of the virtual offset to the actual offset.