Large scale data ingestion
Disclosed are some implementations of systems, apparatus, methods and computer program products for performing data ingestion. A request received from a client device is processed, where the request has a payload including a plurality of data items pertaining to a data object. An identifier associated with the data object is generated. Information including the data items is obtained from the request. The information including the data items is stored in association with the identifier in a first data store, where the first data store is not a relational database. The data items and identifier are obtained from the first data store and stored in a second data store, where the second data store is a relational database. A query including the identifier is processed to obtain the identifier from the query. It is determined whether the identifier obtained from the query is in the first data store. One or more of the data items associated with the identifier are retrieved from the first data store according to a result of determining whether the identifier obtained from the query is in the first data store.
1. A system comprising:
a database system implemented using a server system including one or more hardware processors, the one or more hardware processors configurable to cause:
processing a request received from a client device, the request having a payload including a plurality of data items pertaining to a data object;
generating an identifier associated with the data object;
obtaining information including the data items from the request;
designating the data items and the identifier for storage in a non-relational database and in a relational database;
storing the information including the data items in association with the identifier in a first data store, the first data store being the non-relational database;
obtaining the data items and the identifier from the first data store;
storing the obtained data items in association with the identifier in a second data store, the second data store being the relational database;
processing a query including the identifier to obtain the identifier from the query;
determining whether the identifier obtained from the query is stored in the first data store; and
retrieving one or more of the data items associated with the identifier from the first data store or the second data store according to a result of the determining whether the identifier obtained from the query is stored in the first data store.
2. The system as recited in claim 1 , the one or more hardware processors further configurable to cause:
after storing the obtained data items in association with the identifier in the second data store, deleting the data items and the identifier from the first data store.
3. The system as recited in claim 2 , wherein deleting the data items and the identifier from the first data store is performed by a background process.
4. The system as recited in claim 1 , the one or more hardware processors further configurable to cause:
storing the identifier in a cache; and
after storing the obtained data items in association with the identifier in the second data store, deleting the identifier from the cache;
wherein determining whether the identifier obtained from the query is stored in the first data store includes ascertaining whether the identifier is stored in the cache.
5. The system as recited in claim 4 , wherein deleting the identifier from the cache is performed by a background process.
6. The system as recited in claim 1 , wherein:
storing the information includes storing the payload; and
obtaining the data items and the identifier from the first data store includes parsing the payload.
7. The system as recited in claim 1 , wherein obtaining the data items and the identifier from the first data store and storing the obtained data items in association with the identifier in the second data store are performed by a background process.
8. A computer program product comprising a non-transitory computer-readable medium storing computer-readable program code capable of being executed by one or more processors, the program code comprising computer-readable instructions configurable to cause:
processing a request received from a client device, the request having a payload including a plurality of data items pertaining to a data object;
generating an identifier associated with the data object;
obtaining information including the data items from the request;
designating the data items and the identifier for storage in a non-relational database and in a relational database;
storing the information including the data items in association with the identifier in a first data store, the first data store being the non-relational database;
obtaining the data items and the identifier from the first data store;
storing the obtained data items in association with the identifier in a second data store, the second data store being the relational database;
processing a query including the identifier to obtain the identifier from the query;
determining whether the identifier obtained from the query is stored in the first data store; and
retrieving one or more of the data items associated with the identifier from the first data store or the second data store according to a result of the determining whether the identifier obtained from the query is stored in the first data store.
9. The computer program product as recited in claim 8 , the program code comprising computer-readable instructions further configurable to cause:
after storing the obtained data items in association with the identifier in the second data store, deleting the data items and the identifier from the first data store.
10. The computer program product as recited in claim 9 , wherein deleting the data items and the identifier from the first data store is performed by a background process.
11. The computer program product as recited in claim 8 , the program code comprising computer-readable instructions further configurable to cause:
storing the identifier in a cache; and
after storing the obtained data items in association with the identifier in the second data store, deleting the identifier from the cache;
wherein determining whether the identifier obtained from the query is stored in the first data store includes ascertaining whether the identifier is stored in the cache.
12. The computer program product as recited in claim 11 , wherein deleting the identifier from the cache is performed by a background process.
13. The computer program product as recited in claim 8 , wherein:
storing the information includes storing the payload; and
obtaining the data items and the identifier from the first data store includes parsing the payload.
14. The computer program product as recited in claim 8 , wherein obtaining the data items and the identifier from the first data store and storing the obtained data items in association with the identifier in the second data store are performed by a background process.
15. A method, comprising:
processing a request received from a client device, the request having a payload including a plurality of data items pertaining to a data object;
generating an identifier associated with the data object;
obtaining information including the data items from the request;
designating the data items and the identifier for storage in a non-relational database and in a relational database;
storing the information including the data items in association with the identifier in a first data store, the first data store being the non-relational database;
obtaining the data items and the identifier from the first data store;
storing the obtained data items in association with the identifier in a second data store, the second data store being the relational database;
processing a query including the identifier to obtain the identifier from the query;
determining whether the identifier obtained from the query is stored in the first data store; and
retrieving one or more of the data items associated with the identifier from the first data store or the second data store according to a result of the determining whether the identifier obtained from the query is stored in the first data store.
16. The method as recited in claim 15 , further comprising:
after storing the obtained data items in association with the identifier in the second data store, deleting the data items and the identifier from the first data store.
17. The method as recited in claim 16 , wherein deleting the data items and the identifier from the first data store is performed by a background process.
18. The method as recited in claim 15 , further comprising:
storing the identifier in a cache; and
after storing the obtained data items in association with the identifier in the second data store, deleting the identifier from the cache;
wherein determining whether the identifier obtained from the query is stored in the first data store includes ascertaining whether the identifier is stored in the cache.
19. The method as recited in claim 18 , wherein deleting the identifier from the cache is performed by a background process.
20. The method as recited in claim 15 , wherein:
storing the information includes storing the payload; and
obtaining the data items and the identifier from the first data store includes parsing the payload.