Methods and systems for generating recommendations in cloud-based data warehousing system
Methods, systems, devices, and computer-readable media used by a cloud data management system for collecting data from accounts hosted by a cloud-based data warehousing system on different cloud platforms or in different cloud regions of a cloud platform. Collection of data in the multi-cloud platform and/or multi-cloud region environments may be facilitated by the on-demand creation of one or more data collection accounts. Based on the collected data, one or more recommendations, notifications, or alerts associated with usage of the data warehousing system may be generated.
1. A method comprising:
receiving, by a first computing device associated with a first data manager account hosted in a first cloud region of a cloud-based data warehouse system, a request to collect data from one or more client accounts associated with a first client and hosted by the cloud-based data warehouse system;
sending, by the first computing device and to a second computing device associated with a first client account, from the one or more client accounts associated with the first client, first instructions for the second computing device to share, with a second data manager account hosted in a second cloud region of the cloud-based data warehouse system, data from the first client account;
sending, by the first computing device and to a third computing device associated with the second data manager account, second instructions for the third computing device to replicate the shared data to the first data manager account, upon receiving an indication that the shared data is accessible;
receiving, by the first computing device, an indication that the shared data has been replicated;
storing, by the first computing device, based on the indication that the shared data has been replicated, the shared data in a database associated with the first data manager account;
causing an analysis of the stored shared data to determine one or more new data warehouse resources since a previous request to collect the data;
retrieving, by the first computing device, configuration information associated with a plurality of recommendation algorithms associated with the first client, wherein the configuration information indicates an execution frequency for each of the plurality of recommendation algorithms;
retrieving, by the first computing device, information indicating a last execution date for each of the plurality of recommendation algorithms;
determining, by the first computing device and based on the configuration information, the execution frequency of each of the plurality of recommendation algorithms, and the last execution date for each of the plurality of recommendation algorithms, that a first recommendation algorithm of the plurality of recommendation algorithms is scheduled for execution;
causing, by the first computing device and using the stored shared data, execution of the first recommendation algorithm;
receiving, by the first computing device and based on the execution of the first recommendation algorithm, at least one recommendation associated with a first data warehouse associated with the first client, wherein the at least one recommendation comprising a recommendation of a set of operating parameters to be associated with the one or more new data warehouse resources; and
outputting, by the first computing device, the at least one recommendation.
2. The method of claim 1 , wherein the second instructions further comprise instructions for the third computing device to:
create a second data warehouse comprising at least one database for receiving the shared data; and
drop the second data warehouse after replicating the shared data to the first data manager account.
3. The method of claim 1 , wherein causing execution of the first recommendation algorithm comprises:
causing an analysis of the stored shared data to determine one or more load metrics associated with usage of the first data warehouse during a period of time; and
generating, based on the one or more load metrics, a recommended schedule for operating the first data warehouse with different operating parameters for different periods of time, and
wherein receiving the at least one recommendation comprises receiving the recommended schedule for operating the first data warehouse.
4. The method of claim 1 , wherein causing execution of the first recommendation algorithm comprises:
determining, based on the configuration information, one or more dependencies associated with the first recommendation algorithm, wherein the one or more dependencies comprise one or more second recommendation algorithms;
determining, based on the configuration information, a sequence of execution of the one or more second recommendation algorithms; and
causing, prior to execution of the first recommendation algorithm, in the determined sequence, and using the stored shared data, execution of the one or more second recommendation algorithms.
5. The method of claim 1 , wherein the stored shared data comprises information associated with computing resource usage of the first data warehouse during a period of time, and
wherein execution of the first recommendation algorithm comprises analyzing the information associated with the computing resource usage of the first data warehouse during the period of time to generate the at least one recommendation.
6. The method of claim 1 , further comprising:
causing, based on receiving the at least one recommendation, one or more of:
modification of one or more operating parameters associated with operating the first data warehouse,
setting of a schedule for operating the first data warehouse, or
modification of a size of the first data warehouse.
7. The method of claim 1 , further comprising:
determining, based on the configuration information, a second recommendation algorithm;
causing, using the stored shared data, execution of the second recommendation algorithm, wherein the second recommendation algorithm is configured to:
analyze the stored shared data to determine a metric associated with usage of a first computing resource associated with the first data warehouse; and
based on the metric, determine that usage of the first computing resource exceeds a threshold; and
transmitting, based on the execution of the second recommendation algorithm and to a user device, a notification indicating a usage spike associated with the first computing resource.
8. A first computing device associated with a first data manager account hosted in a first cloud region of a cloud-based data warehouse system, wherein the first computing device comprises:
one or more processors; and memory storing instructions that, when executed by the one or more processors, cause the first computing device to:
receive a request to collect data from one or more client accounts associated with a first client and hosted by the cloud-based data warehouse system;
send, to a second computing device associated with a first client account, from the one or more client accounts associated with the first client, first instructions to share, with a second data manager account hosted in a second cloud region of the cloud-based data warehouse system, data from the first client account;
send, to a third computing device associated with the second data manager account, second instructions to replicate the shared data to the first data manager account, upon receipt of an indication that the shared data is accessible;
receive an indication that the shared data has been replicated;
store, based on the indication that the shared data has been replicated, the shared data in a database associated with the first data manager account;
cause an analysis of the stored shared data to determine one or more new data warehouse resources since a previous request to collect the data;
retrieve configuration information associated with a plurality of recommendation algorithms associated with the first client, wherein the configuration information indicates an execution frequency for each of the plurality of recommendation algorithms;
retrieve information indicating a last execution date for each of the plurality of recommendation algorithms;
determine, based on the configuration information, the execution frequency of each of the plurality of recommendation algorithms, and the last execution date for each of the plurality of recommendation algorithms, that a first recommendation algorithm of the plurality of recommendation algorithms is scheduled for execution;
cause, using the stored shared data, execution of the first recommendation algorithm;
receive, based on the execution of the first recommendation algorithm, at least one recommendation associated with a first data warehouse associated with the first client, wherein the at least one recommendation comprises a recommendation of a set of operating parameters to be associated with the one or more new data warehouse resources; and
output the at least one recommendation.
9. The first computing device of claim 8 , wherein the second instructions further comprise instructions for the third computing device to:
create a second data warehouse comprising at least one database for receiving the shared data; and
drop the second data warehouse after replicating the shared data to the first data manager account.
10. The first computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the execution of the first recommendation algorithm by causing the first computing device to:
cause an analysis of the stored shared data to determine one or more load metrics associated with usage of the first data warehouse during a period of time; and
generate, based on the one or more load metrics, a recommended schedule for operating the first data warehouse with different operating parameters for different periods of time, and
wherein the instructions, when executed by the one or more processors, cause the receipt of the at least one recommendation by causing the first computing device to receive the recommended schedule for operating the first data warehouse.
11. The first computing device of claim 8 , wherein the instructions, when executed by the one or more processors, cause the execution of the first recommendation algorithm by causing the first computing device to:
determine, based on the configuration information, one or more dependencies associated with the first recommendation algorithm, wherein the one or more dependencies comprise one or more second recommendation algorithms;
determine, based on the configuration information, a sequence of execution of the one or more second recommendation algorithms; and
cause, prior to execution of the first recommendation algorithm, in the determined sequence, and using the stored shared data, execution of the one or more second recommendation algorithms.
12. The first computing device of claim 8 , wherein the stored shared data comprises information associated with computing resource usage of the first data warehouse during a period of time, and wherein the instructions, when executed by the one or more processors, cause the execution of the first recommendation algorithm by causing the first computing device to analyze the information associated with the computing resource usage of the first data warehouse during the period of time to generate the at least one recommendation.
13. The first computing device of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the first computing device to:
cause, based on receipt of the at least one recommendation, one or more of:
modification of one or more operating parameters associated with operating the first data warehouse,
setting of a schedule for operating the first data warehouse, or
modification of a size of the first data warehouse.
14. The first computing device of claim 8 , wherein the instructions, when executed by the one or more processors, further cause the first computing device:
determine, based on the configuration information, a second recommendation algorithm;
cause, using the stored shared data, execution of the second recommendation algorithm, wherein the second recommendation algorithm is configured to:
analyze the stored shared data to determine a metric associated with usage of a first computing resource associated with the first data warehouse; and
based on the metric, determine that usage of the first computing resource exceeds a threshold; and
transmit, based on the execution of the second recommendation algorithm and to a user device, a notification indicating a usage spike associated with the first computing resource.
15. A cloud data management system comprising:
a first computing device associated with a first data manager account hosted in a first cloud region of a cloud-based data warehouse system;
a second computing device associated with a first client account, from one or more client accounts associated with a first client;
a third computing device associated with a second data manager account;
wherein the first computing device comprises:
one or more processors; and
memory storing instructions that, when executed by the one or more processors, cause the first computing device to:
receive a request to collect data from the one or more client accounts associated with the first client and hosted by the cloud-based data warehouse system;
send, to the second computing device, first instructions for the second computing device to share, with the second data manager account hosted in a second cloud region of the cloud-based data warehouse system, data from the first client account;
send, to the third computing device, second instructions for the third computing device to replicate the shared data to the first data manager account, upon receipt of an indication that the shared data is accessible;
receive an indication that the shared data has been replicated;
store, based on the indication that the shared data has been replicated, the shared data in a database associated with the first data manager account;
cause an analysis of the stored shared data to determine one or more new data warehouse resources since a previous request to collect the data;
retrieve configuration information associated with a plurality of recommendation algorithms associated with the first client, wherein the configuration information indicates an execution frequency for each of the plurality of recommendation algorithms;
retrieve information indicating a last execution date for each of the plurality of recommendation algorithms;
determine, based on the configuration information, the execution frequency of each of the plurality of recommendation algorithms, and the last execution date for each of the plurality of recommendation algorithms, that a first recommendation algorithm of the plurality of recommendation algorithms is scheduled for execution;
cause, using the stored shared data, execution of the first recommendation algorithm;
receive, based on the execution of the first recommendation algorithm, at least one recommendation associated with a first data warehouse associated with the first client, wherein the at least one recommendation comprises a recommendation of a set of operating parameters to be associated with the one or more new data warehouse resources; and
output the at least one recommendation.
16. The cloud data management system of claim 15 , wherein the second instructions further comprise instruction for the third computing device to:
create a second data warehouse comprising at least one database for receiving the shared data; and
drop the second data warehouse after replicating the shared data to the first data manager account.
17. The cloud data management system of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the first computing device to:
cause an analysis of the stored shared data to determine one or more load metrics associated with usage of the first data warehouse during a period of time; and
generate, based on the one or more load metrics, a recommended schedule for operating the first data warehouse with different operating parameters for different periods of time, and
wherein the first computing device is configured to receive the at least one recommendation by causing the first computing device to receive the recommended schedule for operating the first data warehouse.
18. The cloud data management system of claim 15 , wherein the stored shared data comprises information associated with computing resource usage of the first data warehouse during a period of time, and
wherein the first computing device is configured to cause the execution of the first recommendation algorithm by causing the first computing device to analyze the information associated with the computing resource usage of the first data warehouse during the period of time to generate the at least one recommendation.
19. The cloud data management system of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the first computing device to:
cause, based on receipt of the at least one recommendation, one or more of:
modification of one or more operating parameters associated with operating the first data warehouse,
setting of a schedule for operating the first data warehouse, or
modification of a size of the first data warehouse.
20. The cloud data management system of claim 15 , wherein the instructions, when executed by the one or more processors, further cause the first computing device to:
determine, based on the configuration information, a second recommendation algorithm;
cause, using the stored shared data, execution of the second recommendation algorithm, wherein the second recommendation algorithm is configured to:
analyze the stored shared data to determine a metric associated with usage of a first computing resource associated with the first data warehouse; and
based on the metric, determine that usage of the first computing resource exceeds a threshold; and
transmit based on the execution of the second recommendation algorithm and to a user device, a notification indicating a usage spike associated with the first computing resource.