Partitioned distributed database systems, devices, and methods
A client system can improve processing speeds by executing queries locally. A remote database can have a partitioned, columnar database. Queries from the client can be executed on the remote database, and blocks of a partition of the partitioned, columnar database can be sent to the client. Once the client has enough blocks to form the partition, the client can execute queries that reference the partition. The queries can be executed using templates for specialized expression evaluators.
1. A computer-implemented method for efficiently querying a partitioned columnar database, the computer-implemented method comprising:
receiving, by a client system comprising a client storage device, a client output device, and one or more computer processors in communication with the client storage device and the client output device, from a server system, a plurality of blocks of data, wherein the plurality of bocks of data comprise a subset of data associated with a cluster of records in a server partition of a partitioned columnar database of the server system;
using, by the client system, the plurality of blocks of data to construct and store a local partition in the client storage device, wherein the local partition comprises a subset of data corresponding to some or all of the records of the cluster of records in the server partition;
executing, by the client system, a first query against the local partition stored in the client storage device;
determining, by the client system, that a complete result in response to the first query requires data not present in records of the local partition;
transmitting, from the client system to the server system, a second query to be used against the server partition to return the data not present in the records of the local partition;
prior to receiving a response to the second query, presenting, via the client output device, a partial result based on data present in the records of the local partition;
receiving, by the client system from the server system, a response to the second query, the response to the second query comprising the data not present in the records of the local partition; and
after receiving the response to the second query, presenting, via the client output device, the complete result, wherein the complete result is based on a combination of the data present in the records of the local partition and the data not present in the records of the local partition.
2. The computer-implemented method of claim 1 , wherein the data not present in records of the local partition comprises data that is at least 10 times larger than the data stored in any field of the local partition.
3. The computer-implemented method of claim 1 , wherein the data not present in records of the local partition comprises picture data.
4. The computer-implemented method of claim 1 , wherein the data not present in records of the local partition comprises video data.
5. The computer-implemented method of claim 1 , further comprising:
determining, by the client system, that a new query is related to a separate cluster of records stored in a separate partition of the partitioned columnar database of the server system; and
transmitting, from the client system to the sever system, the new query for execution against the separate partition.
6. The computer-implemented method of claim 5 , further comprising:
receiving, by the client system from the server system, a response to the new query, the response to the new query comprising sufficient data to present a complete result; and
presenting, via the client output device, a complete result based on the response to the new query.
7. The computer-implemented method of claim 6 , further comprising:
receiving, by the client system from the server system, a second plurality of blocks of data, wherein the second plurality of blocks of data comprise a subset of data associated with the separate cluster of records; and
using, by the client system, the second plurality of blocks of data to construct and store a second local partition in the client storage device, wherein the second local partition comprises a subset of data corresponding to some or all of the records of the separate cluster of records in the server partition, such that further new queries related to the separate cluster of records can be executed locally to generate at least partial results based on data present in records of the second local partition.
8. The computer-implemented method of claim 1 , further comprising storing, by the server system in one or more server storage devices of the server system, the partitioned columnar database.
9. The computer-implemented method of claim 1 , wherein the cluster of records in the server partition are associated with a plurality of inventory items.
10. The computer-implemented method of claim 9 , wherein the plurality of inventory items comprises vehicles.
11. A computer readable, non-transitory storage medium having a computer program stored thereon for causing a suitably programmed computer system to process by one or more processors computer-program code by performing a method for efficiently querying a partitioned columnar database when the computer program is executed on the suitably programmed computer system, the method comprising:
receiving, by a client system comprising a client storage device, a client output device, and one or more computer processors in communication with the client storage device and the client output device, from a server system, a plurality of blocks of data, wherein the plurality of bocks of data comprise a subset of data associated with a cluster of records in a server partition of a partitioned columnar database of the server system;
using, by the client system, the plurality of blocks of data to construct and store a local partition in the client storage device, wherein the local partition comprises a subset of data corresponding to some or all of the records of the cluster of records in the server partition;
executing, by the client system, a first query against the local partition stored in the client storage device;
determining, by the client system, that a complete result in response to the first query requires data not present in records of the local partition;
transmitting, from the client system to the server system, a second query to be used against the server partition to return the data not present in the records of the local partition;
prior to receiving a response to the second query, presenting, via the client output device, a partial result based on data present in the records of the local partition;
receiving, by the client system from the server system, a response to the second query, the response to the second query comprising the data not present in the records of the local partition; and
after receiving the response to the second query, presenting, via the client output device, the complete result, wherein the complete result is based on a combination of the data present in the records of the local partition and the data not present in the records of the local partition.
12. The computer readable, non-transitory storage medium of claim 11 , wherein the data not present in records of the local partition comprises data that is at least 10 times larger than the data stored in any field of the local partition.
13. The computer readable, non-transitory storage medium of claim 11 , wherein the data not present in records of the local partition comprises picture data.
14. The computer readable, non-transitory storage medium of claim 11 , wherein the data not present in records of the local partition comprises video data.
15. The computer readable, non-transitory storage medium of claim 11 , the method further comprising:
determining, by the client system, that a new query is related to a separate cluster of records stored in a separate partition of the partitioned columnar database of the server system; and
transmitting, from the client system to the sever system, the new query for execution against the separate partition.
16. The computer readable, non-transitory storage medium of claim 15 , the method further comprising:
receiving, by the client system from the server system, a response to the new query, the response to the new query comprising sufficient data to present a complete result; and
presenting, via the client output device, a complete result based on the response to the new query.
17. The computer readable, non-transitory storage medium of claim 16 , the method further comprising:
receiving, by the client system from the server system, a second plurality of blocks of data, wherein the second plurality of blocks of data comprise a subset of data associated with the separate cluster of records; and
using, by the client system, the second plurality of blocks of data to construct and store a second local partition in the client storage device, wherein the second local partition comprises a subset of data corresponding to some or all of the records of the separate cluster of records in the server partition, such that further new queries related to the separate cluster of records can be executed locally to generate at least partial results based on data present in records of the second local partition.
18. The computer readable, non-transitory storage medium of claim 11 , the method further comprising storing, by the server system in one or more server storage devices of the server system, the partitioned columnar database.
19. The computer readable, non-transitory storage medium of claim 11 , wherein the cluster of records in the server partition are associated with a plurality of inventory items.
20. The computer readable, non-transitory storage medium of claim 19 , wherein the plurality of inventory items comprises vehicles.