Synchronizing state in large-scale distribution systems
A processing client may receive a dataset replacement indication that indicates an upcoming synchronized data replacement in which processing clients are to synchronously replace a first synchronized dataset with a second synchronized dataset as well as a request period during which the processing clients are to request, at randomized times, the second synchronized dataset. The processing client may transmit a query requesting the second synchronized dataset. The processing client may receive a response including the second synchronized dataset and an indication of an expiration time for the first synchronized dataset that indicates when the processing clients are to replace the first synchronized dataset. The processing client may store, at the processing client prior to the expiration time, the second synchronized dataset and the indication of the expiration time. The processing client may replace, in response to the expiration time occurring, the first synchronized dataset with the second synchronized dataset.
1 . A method for synchronized data replacement, comprising:
receiving, by a first processing client from a data source, a dataset replacement indication that is distributed to a plurality of processing clients, the dataset replacement indication indicating an upcoming synchronized data replacement in which each processing client of the plurality of processing clients are to synchronously replace a first synchronized dataset with a second synchronized dataset and a request period during which each processing client of the plurality of processing clients are to request, at randomized times within the request period, the second synchronized dataset;
based at least in part on the dataset replacement indication and at a first randomized time within the request period, transmitting, by the first processing client to the data source, a query requesting the second synchronized dataset that is to replace the first synchronized dataset;
receiving, based at least in part on transmitting the query, a response from the data source comprising the second synchronized dataset and an indication of an expiration time for the first synchronized dataset, the expiration time indicating when the plurality of processing clients are to synchronously replace the first synchronized dataset with the second synchronized dataset;
storing, at the first processing client prior to the expiration time for the first synchronized dataset and based at least in part on receiving the response, the second synchronized dataset and the indication of the expiration time for the first synchronized dataset; and
replacing, at the first processing client and in response to the expiration time for the first synchronized dataset occurring, the first synchronized dataset with the second synchronized dataset.
2 . The method of claim 1 , wherein the first synchronized dataset and the second synchronized dataset each describe a product category tree of an online marketplace and a plurality of sales tax rates for respective product categories of the product category tree, the method further comprising:
receiving an indication of a first product category within the product category tree of the online marketplace;
identifying a first sales tax rate for the first product category indicated by the first synchronized dataset prior to the expiration time for the first synchronized dataset or a second sales tax rate for the first product category indicated by the second synchronized dataset after the expiration time; and
transmitting, to the online marketplace, an indication of one of the first sales tax rate or the second sales tax rate, or an indication of a calculated amount of sales tax for a purchase price of an item calculated using one of the first sales tax rate or the second sales tax rate.
3 . The method of claim 1 , wherein the first synchronized dataset describes a product category tree having a first category structure for an online marketplace and the second synchronized dataset describes the product category tree having a second category structure that differs from the first category structure, the method further comprising:
receiving an indication of a product available from the online marketplace;
identifying in which product category of a plurality of product categories of the product category tree the product is classified using one of the first synchronized dataset prior to the expiration time for the first synchronized dataset or the second synchronized dataset after the expiration time; and
transmitting, to the online marketplace, an indication of the identified product category or data associated with the identified product category.
4 . The method of claim 3 , wherein the data indicates a sales tax rate for products categorized within the identified product category, a geo-location to which the online marketplace ships products categorized within the identified product category, a delivery estimate for products categorized within the identified product category, a shipping cost estimate for products categorized within the identified product category, or any combination thereof.
5 . The method of claim 1 , wherein the first synchronized dataset describes a product category tree having a first category structure for an online marketplace and the second synchronized dataset describes the product category tree having a second category structure that differs from the first category structure, the method further comprising:
receiving a search query via an online marketplace;
mapping one or more terms in the search query to one or more product categories of the product category tree using one of the first synchronized dataset prior to the expiration time of the first synchronized dataset or the second synchronized dataset after the expiration time; and
returning an indication of one or more listings from the one or more product categories based at least in part on the one or more terms in the search query.
6 . The method of claim 1 , further comprising:
periodically transmitting one or more additional queries requesting one or more additional synchronized datasets; and
monitoring for one or more responses to the one or more additional queries.
7 . The method of claim 1 , wherein the first synchronized dataset describes a product category tree associated with an online marketplace and the second synchronized dataset describes a modified version of the product category tree.
8 . The method of claim 1 , wherein the first synchronized dataset describes online marketplace metadata associated with an online marketplace and the second synchronized dataset describes a modified version of the online marketplace metadata.
9 . The method of claim 1 , further comprising:
synchronizing a clock of the first processing client with one or more additional clocks associated with one or more second processing clients of the plurality of processing clients.
10 . The method of claim 1 , wherein replacing the first synchronized dataset with the second synchronized dataset comprises:
adjusting at least one pointer in memory implemented by a storage device from the first synchronized dataset to the second synchronized dataset.
11 . An apparatus for synchronized data replacement, comprising:
at least one processor;
at least one memory coupled with the at least one processor; and
instructions stored in the at least one memory and executable by the at least one processor to cause the apparatus to perform operations comprising:
receiving, from a data source, a dataset replacement indication that is distributed to a plurality of processing clients, the dataset replacement indication indicating an upcoming synchronized data replacement in which each processing client of the plurality of processing clients are to synchronously replace a first synchronized dataset with a second synchronized dataset and a request period during which each processing client of the plurality of processing clients are to request, at randomized times within the request period, the second synchronized dataset;
based at least in part on the dataset replacement indication and at a first randomized time within the request period, transmitting, to the data source, a query requesting the second synchronized dataset that is to replace the first synchronized dataset;
receiving, based at least in part on transmitting the query, a response from the data source comprising the second synchronized dataset and an indication of an expiration time for the first synchronized dataset, the expiration time indicating when the plurality of processing clients are to synchronously replace the first synchronized dataset with the second synchronized dataset;
storing, at a first processing client prior to the expiration time for the first synchronized dataset and based at least in part on receiving the response, the second synchronized dataset and the indication of the expiration time for the first synchronized dataset; and
replacing , at the first processing client and in response to the expiration time for the first synchronized dataset occurring, the first synchronized dataset with the second synchronized dataset.
12 . The apparatus of claim 11 , wherein the first synchronized dataset and the second synchronized dataset each describe a product category tree of an online marketplace and a plurality of sales tax rates for respective product categories of the product category tree, and the operations further comprise:
receiving an indication of a first product category within the product category tree of the online marketplace;
identifying a first sales tax rate for the first product category indicated by the first synchronized dataset prior to the expiration time for the first synchronized dataset or a second sales tax rate for the first product category indicated by the second synchronized dataset after the expiration time; and
transmitting, to the online marketplace, an indication of one of the first sales tax rate or the second sales tax rate, or an indication of a calculated amount of sales tax for a purchase price of an item calculated using one of the first sales tax rate or the second sales tax rate.
13 . The apparatus of claim 11 , wherein the first synchronized dataset describes a product category tree having a first category structure for an online marketplace and the second synchronized dataset describes the product category tree having a second category structure that differs from the first category structure, and the operations further comprise:
receiving an indication of a product available from the online marketplace;
identifying in which product category of a plurality of product categories of the product category tree the product is classified using one of the first synchronized dataset prior to the expiration time for the first synchronized dataset or the second synchronized dataset after the expiration time; and
transmitting, to the online marketplace, an indication of the identified product category or data associated with the identified product category.
14 . The apparatus of claim 13 , wherein the data indicates a sales tax rate for products categorized within the identified product category, a geo-location to which the online marketplace ships products categorized within the identified product category, a delivery estimate for products categorized within the identified product category, a shipping cost estimate for products categorized within the identified product category, or any combination thereof.
15 . The apparatus of claim 11 , wherein the first synchronized dataset describes a product category tree having a first category structure for an online marketplace and the second synchronized dataset describes the product category tree having a second category structure that differs from the first category structure, and the operations further comprise:
receiving a search query via an online marketplace;
mapping one or more terms in the search query to one or more product categories of the product category tree using one of the first synchronized dataset prior to the expiration time of the first synchronized dataset or the second synchronized dataset after the expiration time; and
returning an indication of one or more listings from the one or more product categories based at least in part on the one or more terms in the search query.
16 . The apparatus of claim 11 , wherein the operations further -comprise:
periodically transmitting one or more additional queries requesting one or more additional synchronized datasets; and
monitoring for one or more responses to the one or more additional queries.
17 . The apparatus of claim 11 , wherein the first synchronized dataset describes a product category tree associated with an online marketplace and the second synchronized dataset describes a modified version of the product category tree.
18 . The apparatus of claim 11 , wherein the first synchronized dataset describes online marketplace metadata associated with an online marketplace and the second synchronized dataset describes a modified version of the online marketplace metadata.
19 . The apparatus of claim 11 , wherein the operations further comprise:
synchronizing a clock of the first processing client with one or more additional clocks associated with one or more second processing clients of the plurality of processing clients.
20 . A non-transitory computer-readable medium storing instructions that, when executable by at least one processor, causes the at least one processor to perform operations comprising:
receiving, from a data source, a dataset replacement indication that is distributed to a plurality of processing clients, the dataset replacement indication indicating an upcoming synchronized data replacement in which each processing client of the plurality of processing clients are to synchronously replace a first synchronized dataset with a second synchronized dataset and a request period during which each processing client of the plurality of processing clients are to request, at randomized times within the request period, the second synchronized dataset;
based at least in part on the dataset replacement indication and at a first randomized time within the request period, transmitting, to the data source, a query requesting the second synchronized dataset that is to replace the first synchronized dataset;
receiving, based at least in part on transmitting the query, a response from the data source comprising the second synchronized dataset and an indication of an expiration time for the first synchronized dataset, the expiration time indicating when the plurality of processing clients are to synchronously replace the first synchronized dataset with the second synchronized dataset;
storing, at a first processing client prior to the expiration time for the first synchronized dataset and based at least in part on receiving the response, the second synchronized dataset and the indication of the expiration time for the first synchronized dataset; and
replacing, at the first processing client and in response to the expiration time for the first synchronized dataset occurring, the first synchronized dataset with the second synchronized dataset.