Automated workload sizer for data platform service
In general, techniques are described for an automated workload sizer for data platform service. A computing device comprising a memory and processing circuitry may be configured to perform the techniques. The memory may store metadata indicative of data storage by a third-party service provider for a client of the data platform. The processing circuitry may execute an automated workload sizer. The automated workload sizer may process of the metadata indicative of the storage of the data by the third-party service provider to obtain an estimate of providing data platform services by the data platform for the data stored by the third-party service, and output of the estimate of providing the data platform services.
1 . A method comprising:
obtaining, by processing circuitry of a computing device that executes a data platform and from a third-party service provider, metadata indicative of data storage by the third-party service provider for a client of the data platform during a first time period;
extrapolating, by an automated workload sizer executed by the processing circuitry, the metadata indicative of the data storage by the third-party service provider to obtain a forecast of the data storage by the third-party service provider for the client over a second time period greater than the first time period, wherein extrapolating the metadata to obtain the forecast comprises:
generating, by a seasonal autoregressive integrated moving averaging model of a machine learning model of the automated workload sizer, a first forecast of estimated data storage required by the client for the third-party service provider; and
generating, by a long-short term memory model of the machine learning model of the automated workload sizer, based on the first forecast of estimated data storage, a second forecast of estimated data storage required by the client for the third-party service provider;
combining, by the automated workload sizer, the first forecast and the second forecast to generate an estimate of providing data platform services by the data platform for the data storage by the third-party service provider; and
outputting, by the automated workload sizer, the estimate of providing the data platform services.
2 . The method of claim 1 , wherein obtaining the metadata comprises executing a script that interfaces with the third-party service provider via an application programming interface exposed by the third-party service provider.
3 . The method of claim 1 , wherein the metadata indicates one or more of:
a number of objects stored for each account supported by the third-party service provider for the client of the data platform;
an amount of storage available for each account supported by the third-party service provider for the client of the data platform;
a frequency with which the objects are stored for each account supported by the third-party service provider for the client of the data platform;
a number of sites provided by the third-party service provider that perform the data storage for the client of the data platform;
a location of the sites provided by the third-party service provider;
a cost of providing the data storage of the objects by the third-party service provider for the client of the data platform;
a number of active accounts; and
a number of active workloads per each of the active accounts.
4 . The method of claim 3 ,
wherein the metadata indicates one or more of:
the number of objects stored for each account supported by the third-party service provider for the client of the data platform;
the frequency with which the objects are stored for each account supported by the third-party service provider for the client of the data platform; and
the cost of providing the data storage of the objects by the third-party service provider for the client of the data platform, and
wherein the objects include one or more of files, texts, chat messages, and electronic messages.
5 . The method of claim 1 , wherein the estimate includes one or more of:
a cluster sizing for providing the data platform services that estimates a number of nodes used by the data platform to support the data platform services for the data storage by the third-party service provider;
one or more regions of the data platform services to be utilized for the data platform;
an approximate cost of providing the data platform services; and
a schedule for performing the data platform services.
6 . The method of claim 1 , wherein the data platform services comprise one or more of backup of the data storage, archiving of the data storage, and snapshotting of the data storage.
7 . The method of claim 1 , further comprising determining, by the automated workload sizer, whether an alert indicating when to move the client between different clusters of the data platform is to be generated.
8 . The method of claim 7 , further comprising determining, by the automated workload sizer, whether the alert is to be generated periodically.
9 . The method of claim 1 , wherein the estimate comprises a prediction of whether providing the data platform services by the data platform for the data storage by the third-party service provider satisfies a Service-level Agreement (SLA) of the client.
10 . The method of claim 1 , wherein the metadata indicative of the data storage by the third-party service provider for the client comprises a capacity provided by the third-party service, an amount of data stored by the third-party service provider for the client, and an amount of time such data is stored.
11 . A computing device comprising:
storage media comprising instructions; and
processing circuitry in communication with the storage media, wherein the instructions cause the processing circuitry to:
obtain, from a third-party service provider, metadata indicative of data storage by the third-party service provider for a client of a data platform during a first time period;
execute an automated workload sizer configured to:
extrapolate the metadata indicative of the data storage by the third-party service provider to obtain a forecast of the data storage by the third-party service provider for the client over a second time period greater than the first time period, wherein to extrapolate the metadata to obtain the forecast, the automated workload sizer is configured to:
execute a seasonal autoregressive integrated moving averaging model of a machine learning model configured to generate a first forecast of estimated data storage required by the client for the third-party service provider; and
execute a long-short term memory model of the machine learning model configured to generate, based on the first forecast of estimated data storage, a second forecast of estimated data storage required by the client for the third-party service provider; and
combine the first forecast and the second forecast to generate an estimate of providing data platform services by the data platform for the data storage by the third-party service provider; and
output the estimate of providing the data platform services.
12 . The computing device of claim 11 , wherein the automated workload sizer is configured to execute a script that interfaces with the third-party service provider via an application programming interface exposed by the third-party service provider.
13 . The computing device of claim 11 , wherein the metadata indicates one or more of:
a number of objects stored for each account supported by the third-party service provider for the client of the data platform;
an amount of storage available for each account supported by the third-party service provider for the client of the data platform;
a frequency with which the objects are stored for each account supported by the third-party service provider for the client of the data platform;
a number of sites provided by the third-party service provider that perform the data storage for the client of the data platform;
a location of the sites provided by the third-party service provider;
a cost of providing the data storage of the objects by the third-party service provider for the client of the data platform;
a number of active accounts; and
a number of active workloads per each of the active accounts.
14 . The computing device of claim 13 ,
wherein the metadata indicates one or more of:
the number of objects stored for each account supported by the third-party service provider for the client of the data platform;
the frequency with which the objects are stored for each account supported by the third-party service provider for the client of the data platform; and
the cost of providing the data storage of the objects by the third-party service provider for the client of the data platform, and
wherein the objects include one or more of files, texts, chat messages, and electronic messages.
15 . The computing device of claim 11 , wherein the estimate includes one or more of:
a cluster sizing for providing the data platform services that estimates a number of nodes used by the data platform to support the data platform services for the data storage by the third-party service provider;
one or more regions of the data platform services to be utilized for the data platform;
an approximate cost of providing the data platform services; and
a schedule for performing the data platform services.
16 . Non-transitory, computer-readable storage media having instructions stored thereon that, when executed, cause one or more processors to:
obtain, from a third-party service provider, metadata indicative of data storage by the third-party service provider for a client of a data platform during a first time period;
execute an automated workload sizer configured to:
extrapolate the metadata indicative of the data storage by the third-party service provider to obtain a forecast of the data storage by the third-party service provider for the client over a second time period greater than the first time period, wherein to extrapolate the metadata to obtain the forecast, the automated workload sizer is configured to:
execute a seasonal autoregressive integrated moving averaging model of a machine learning model configured to generate a first forecast of estimated data storage required by the client for the third-party service provider; and
execute a long-short term memory model of the machine learning model configured to generate, based on the first forecast of estimated data storage, a second forecast of estimated data storage required by the client for the third-party service provider; and
combine the first forecast and the second forecast to generate an estimate of providing data platform services by the data platform for the data storage by the third-party service provider; and
output the estimate of providing the data platform services.