SYSTEMS AND METHODS FOR ADAPTIVE API RULE MANAGEMENT
A method and associated system for retrieving data associated with an online marketplace, including determining, based on a search of a configuration table, a configuration associated with an online marketplace channel API; determining, based on a search of a customer query table, one or more customers requiring data retrieval; generating one or more required time periods for data retrieval based in part on one or more settings of the configuration associated with the online marketplace channel API; dividing, based on the one or more settings of the configuration associated with the online marketplace channel API, the one or more required time periods into a plurality of time subperiods compliant with the online marketplace channel API; retrieving data from the online marketplace channel API in batches based on the plurality of time subperiods compliant with the online marketplace channel API; and storing the retrieved data.
1 . A method for retrieving data associated with an online marketplace, comprising:
under control of one or more processors configured with executable instructions,
determining, based on a search of a configuration table, a configuration associated with an online marketplace channel API, the configuration table including one or more data fields for one or more online marketplace channel current or updated APIs;
determining, based on a search of a customer query table, one or more customers requiring data retrieval;
generating one or more required time periods for data retrieval based in part on one or more settings of the configuration associated with the online marketplace channel API;
dividing, based on the one or more settings of the configuration associated with the online marketplace channel API, the one or more required time periods into a plurality of time subperiods compliant with the online marketplace channel API;
retrieving data from the online marketplace channel API in batches based on the plurality of time subperiods compliant with the online marketplace channel API; and
storing the retrieved data.
2 . The method of claim 1 , wherein the one or more data fields for one or more online marketplace channel current or updated APIs are one or more of a channel name, a data ingestion call alias, a listing of all fields of data extracted from an online marketplace channel API endpoint, a local storage location, an SQL query listing of customers to be processed for data retrieval, a duration for data retrieval, and a data retrieval speed.
3 . The method of claim 2 , wherein the one or more data fields for one or more online marketplace channel current or updated APIs include a channel name, a data ingestion call alias, and a local storage location.
4 . The method of claim 1 , wherein generating one or more required time periods for data retrieval includes retrieving a full data history and a partial data history for establishing configuration settings.
5 . The method of claim 4 , wherein the configuration settings include a period name, a period start date, a period end date, and a number of dates between the period start date and the period end date.
6 . The method of claim 1 , further comprising storing an execution log of all data retrievals.
7 . The method of claim 6 , wherein the execution log includes metrics relating to all data retrievals, the metrics including one or more of a percentage of data retrieval successes, a percentage of data retrieval failures, a number of data retrieval executions, a percentage of errors, and a data retrieval duration.
8 . The method of claim 7 , wherein the errors in the percentage of errors include authentication errors, unknown errors, throttling errors, token service expiration errors, and timeout errors.
9 . The method of claim 6 , wherein the execution log is displayed on a single centralized dashboard.
10 . The method of claim 1 , wherein the retrieved data is stored in a local infrastructure of a digital marketing company.
11 . A system for retrieving data associated with an online marketplace, comprising:
one or more processors;
one or more computer-readable media; and
one or more modules maintained on the one or more computer-readable media that, when executed by the one or more processors, cause the one or more processors to perform operations including:
determining, based on a search of a configuration table, a configuration associated with an online marketplace channel API, the configuration table including one or more data fields for one or more online marketplace channel current or updated APIs;
determining, based on a search of a customer query table, one or more customers requiring data retrieval;
generating one or more required time periods for data retrieval based in part on one or more settings of the configuration associated with the online marketplace channel API;
dividing, based on the one or more settings of the configuration associated with the online marketplace channel API, the one or more required time periods into a plurality of time subperiods compliant with the online marketplace channel API;
retrieving data from the online marketplace channel API in batches based on the plurality of time subperiods compliant with the online marketplace channel API; and
storing the retrieved data.
12 . The system of claim 11 , wherein the one or more data fields for one or more online marketplace channel current or updated APIs are one or more of a channel name, a data ingestion call alias, a listing of all fields of data extracted from an online marketplace channel API endpoint, a local storage location, an SQL query listing of customers to be processed for data retrieval, a duration for data retrieval, and a data retrieval speed.
13 . The system of claim 12 , wherein the one or more data fields for one or more online marketplace channel current or updated APIs include a channel name, a data ingestion call alias, and a local storage location.
14 . The system of claim 11 , wherein generating one or more required time periods for data retrieval includes retrieving a full data history and a partial data history for establishing configuration settings.
15 . The system of claim 14 , wherein the configuration settings include a period name, a period start date, a period end date, and a number of dates between the period start date and the period end date.
16 . The system of claim 11 , further comprising an additional operation of:
storing an execution log of all data retrievals.
17 . The system of claim 16 , wherein the execution log includes metrics relating to all data retrievals, the metrics including one or more of a percentage of data retrieval successes, a percentage of data retrieval failures, a number of data retrieval executions, a percentage of errors, and a data retrieval duration.
18 . The system of claim 17 , wherein the errors in the percentage of errors include authentication errors, unknown errors, throttling errors, token service expiration errors, and timeout errors.
19 . The system of claim 16 , wherein the execution log is displayed on a single centralized dashboard.
20 . The system of claim 11 , wherein the retrieved data is stored in a local infrastructure of a digital marketing company.