Optimizing data warehouse utilization by a data ingestion pipeline
Optimizing data warehouse utilization by a data ingestion pipeline, including: gathering information describing usage of a plurality of data warehouses; receiving, from a data ingestion pipeline, a request for a data warehouse to receive data from the data ingestion pipeline; selecting, based on the information, the data warehouse from the plurality of data warehouses; and providing, to the data ingestion pipeline, an indication of the data warehouse.
1. A method of optimizing data warehouse utilization by a data ingestion pipeline, the method comprising:
gathering information describing usage of a plurality of data warehouses;
receiving, from a data ingestion pipeline, a request for a data warehouse to receive data from the data ingestion pipeline;
selecting, based on the information, the data warehouse from the plurality of data warehouses; and
providing, to the data ingestion pipeline, an indication of the data warehouse.
2. The method of claim 1 , wherein selecting the data warehouse comprises selecting, as the data warehouse, a previously assigned data warehouse associated with one or more attributes of the request.
3. The method of claim 2 , wherein the one or more attributes of the request comprises a partition identifier for an event stream.
4. The method of claim 2 , wherein the one or more attributes of the request comprises a customer identifier.
5. The method of claim 1 , wherein selecting the data warehouse comprises selecting the data warehouse from the plurality of data warehouses as having a threshold number of query slots to satisfy the request.
6. The method of claim 5 , further comprising storing an assignment of the data warehouse.
7. The method of claim 1 , wherein selecting the data warehouse comprises selecting, in response to no data warehouse having a threshold number of query slots to satisfy the request, the data warehouse as having a highest number of open query slots.
8. The method of claim 1 , further comprising modifying a resource allocation for the plurality of data warehouses in response to no data warehouse having a threshold number of query slots.
9. The method of claim 8 , wherein the resource allocation includes a size of one or more data warehouses.
10. The method of claim 8 , wherein the resource allocation includes a size of one or more data warehouses.
11. A method of optimizing data warehouse utilization by a data ingestion pipeline, the method comprising:
gathering performance metrics from a data ingestion pipeline associated with a plurality of data warehouses;
identifying, based on the performance metrics, one or more misfit data warehouse assignments; and
modify a size of one or more data warehouses corresponding to the one or more misfit data warehouse assignments.
12. The method of claim 11 , wherein the performance metrics comprise one or more latencies for storing data by the data ingestion pipeline into the plurality of data warehouses.
13. The method of claim 11 , wherein the performance metrics comprise one or more request frequencies from the data ingestion pipeline to a warehouse manager.
14. The method of claim 11 , wherein the performance metrics comprise one or more backlogs of the data ingestion pipeline.
15. The method of claim 11 , further comprising:
determining whether the plurality of data warehouses fall with one or more performance bounds; and
modifying one or more data warehouse assignments responsive to one or more of the data warehouses falling outside the one or more performance bounds.
16. The method of claim 15 , wherein the one or more performance bounds comprise a target utilization bound.
17. The method of claim 15 , wherein the one or more performance bounds comprise a latency to request frequency bound.
18. A computer program product for optimizing data warehouse utilization by a data ingestion pipeline, the computer program product disposed on a non-transitory computer readable medium, the
computer program product including computer program instructions configurable to carry out the steps of:
gathering information describing usage of a plurality of data warehouses;
receiving, from a data ingestion pipeline, a request for a data warehouse to receive data from the data ingestion pipeline;
selecting, based on the information, the data warehouse from the plurality of data warehouses; and
providing, to the data ingestion pipeline, an indication of the data warehouse.
19. The computer program product of claim 18 , wherein selecting the data warehouse comprises
selecting, as the data warehouse, a previously assigned data warehouse associated with one or more attributes of the request.
20. The computer program product of claim 19 , wherein the one or more attributes of the request comprises a partition identifier for an event stream.