Computer data distribution architecture
Described are methods, systems and computer readable media for computer data distribution architecture.
1. A computer database system with a plurality of memory devices optimized for ordered data and read-dominated workloads, the system comprising:
one or more processors;
computer readable storage coupled to the one or more processors, the computer readable storage having stored thereon instructions that, when executed by the one or more processors, cause the one or more processors to perform operations including:
storing at a first data server computer first data in a first memory device in a first column-oriented configuration;
storing at a second data server computer second data in a second memory device in a second column-oriented configuration;
storing at a third data server computer the third data in a third memory device in a third column-oriented configuration;
executing, at a query server computer, a database query, the query server computer comprising a query server memory device and being coupled to the first, second, and third data server computers, the executing comprising:
providing a table data cache proxy operable to optimize performance of computer data access to the first, second, and third data server computers by:
requesting a latest table state from two or more of the first, second, and third data server computers;
filtering the latest table state to generate a complete, non-duplicative composite table state that includes one of data or metadata from the two or more of the first, second, and third data server computers, wherein the non-duplicative composite table state includes table locations, and wherein the table data cache proxy is operable to filter table locations to be distributed to one or more remote query processors to interleave table locations from the first, second, and third data server computers and distribute subsequent data access across the first, second, and third data server computers; and
providing, from the table data cache proxy a single, non-composite service for table location;
accessing a table comprising first and second columns, one or more first rows, and one or more second rows, wherein a given location for the table is provided by the single, non-composite service;
storing one or more first location identifiers indicating where column data of the first column is stored and one or more second location identifiers indicating where column data of the second column is stored,
the one or more first location identifiers indicating that data of the one or more first rows of the first column is stored in the first memory device of the first data server computer,
the one or more second location identifiers indicating that data of the one or more first rows of the second column is stored in the second memory device of the second data server computer and that data of the one or more second rows of the second column is stored in the third memory device of the third data server computer; and
storing a table index comprising, for each location identifier of the one or more first location identifiers and the one or more second location identifiers, an index identifier indicating a valid portion of the data at the location indicated by the location identifier, wherein the valid portion of the data includes data corresponding to the rows of the table and excludes data not corresponding to the rows of the table, such that data corresponding to a combination of the first or second location identifiers and the index identifier comprises the table,
wherein the first memory device, the second memory device, and the third memory device are all different from each other, and
wherein the first memory device includes shared memory and the second memory device is faster than the third memory device.
2. The system of claim 1 , wherein the executing further comprises:
requesting one or more blocks of data of the first column from the first data server; and
requesting one or more blocks of data of the second column from the second or third data server computers.
3. The system of claim 1 , wherein the executing further comprises:
requesting one or more blocks of data of the second column from the second data server computer; and
requesting one or more blocks of data of the second column from the third data server computers.
4. The system of claim 1 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first or second data server computers; and
requesting one or more blocks of data of the one or more second rows from the third data server computer.
5. The system of claim 1 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first data server computer; and
requesting one or more blocks of data of the one or more first rows from the second data server computer.
6. The system of claim 1 , wherein the first memory device is a transitory memory device and the second memory device is a persistent memory device.
7. The system of claim 1 , wherein the one or more first location identifiers indicate that data of the one or more second rows of the first column is stored in the query server memory device of the query server computer.
8. A method optimized for ordered data and read-dominated workloads of a computer data system with a plurality of memory devices, the method comprising:
storing at a first data server computer first data in a first memory device in a first column-oriented configuration;
storing at a second data server computer second data in a second memory device in a second column-oriented configuration;
storing at a third data server computer the third data in a third memory device in a third column-oriented configuration;
executing, at a query server computer, a database query, the query server computer comprising a query server memory device and being coupled to the first, second, and third data server computers, the executing comprising:
providing a table data cache proxy operable to optimize performance of computer data access to the first, second, and third data server computers by:
requesting a latest table state from two or more of the first, second, and third data server computers;
filtering the latest table state to generate a complete, non-duplicative composite table state that includes one of data or metadata from the two or more of the first, second, and third data server computers, wherein the non-duplicative composite table state includes table locations, and wherein the table data cache proxy is operable to filter table locations to be distributed to one or more remote query processors to interleave table locations from the first, second, and third data server computers and distribute subsequent data access across the first, second, and third data server computers; and
providing, from the table data cache proxy a single, non-composite service for table location;
accessing a table comprising first and second columns, one or more first rows, and one or more second rows, wherein a given location for the table is provided by the single, non-composite service,
storing one or more first location identifiers indicating where column data of the first column is stored and one or more second location identifiers indicating where column data of the second column is stored,
the one or more first location identifiers indicating that data of the one or more first rows of the first column is stored in the first memory device of the first data server computer,
the one or more second location identifiers indicating that data of the one or more first rows of the second column is stored in the second memory of the second data server computer and that data of the one or more second rows of the second column is stored in the third memory of the third data server computer, and
storing a table index comprising, for each location identifier of the one or more first location identifiers and the one or more second location identifiers, an index identifier indicating a valid portion of the data at the location indicated by the location identifier, wherein the valid portion of the data includes data corresponding to the rows of the table and excludes data not corresponding to the rows of the table, such that data corresponding to a combination of the first or second location identifiers and the index identifier comprises the table, and
wherein the first memory device includes shared memory and the second memory device is faster than the third memory device.
9. The method of claim 8 , wherein the executing further comprises:
requesting one or more blocks of data of the first column from the first data server computer; and
requesting one or more blocks of data of the second column from the second or third data server computer.
10. The method of claim 8 , wherein the executing further comprises:
requesting one or more blocks of data of the second column from the second data server computer; and
requesting one or more blocks of data of the second column from the third data server computer.
11. The method of claim 8 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first or second data server computers; and
requesting one or more blocks of data of the one or more second rows from the third data server computer.
12. The method of claim 8 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first data server computer; and
requesting one or more blocks of data of the one or more first rows from the second data server computer.
13. The method of claim 8 , wherein the first memory device is a transitory memory device and the second memory device is a persistent memory device.
14. The method of claim 8 , wherein the first memory device, the second memory device, and the third memory device are all different from each other.
15. The method of claim 8 , wherein the one or more first location identifiers indicate that data of the one or more second rows of the first column is stored in the query server memory device of the query server computer.
16. A nontransitory computer readable medium having stored thereon software instructions that, when executed by one or more processors, cause the processors to perform operations including:
storing at a first data server first data in a first memory device in a first column-oriented configuration;
storing at a second data server second data in a second memory device in a second column-oriented configuration;
storing at a third data server the third data in a third memory device in a third column-oriented configuration;
executing, at a query server, a database query, the query server comprising a query server memory device and being coupled to the first, second, and third data servers, the executing comprising:
providing a table data cache proxy operable to optimize performance of computer data access to the first, second, and third data server computers by:
requesting a latest table state from two or more of the first, second, and third data server computers;
filtering the latest table state to generate a complete, non-duplicative composite table state that includes one of data or metadata from the two or more of the first, second, and third data server computers, wherein the non-duplicative composite table state includes table locations, and wherein the table data cache proxy is operable to filter table locations to be distributed to one or more remote query processors to interleave table locations from the first, second, and third data server computers and distribute subsequent data access across the first, second, and third data server computers; and
providing, from the table data cache proxy a single, non-composite service for table location;
accessing a table comprising first and second columns, one or more first rows, and one or more second rows, wherein a given location for the table is provided by the single, non-composite service,
storing one or more first location identifiers indicating where column data of the first column is stored and one or more second location identifiers indicating where column data of the second column is stored,
the one or more first location identifiers indicating that data of the one or more first rows of the first column is stored in the first memory device of the first data server,
the one or more second location identifiers indicating that data of the one or more first rows of the second column is stored in the second memory device of the second data server and that data of the one or more second rows of the second column is stored in the third memory device of the third data server, and
storing a table index comprising, for each location identifier of the one or more first location identifiers and the one or more second location identifiers, an index identifier indicating a valid portion of the data at the location indicated by the location identifier, wherein the valid portion of the data includes data corresponding to the rows of the table and excludes data not corresponding to the rows of the table, such that data corresponding to a combination of the first or second location identifiers and the index identifier comprises the table, and
wherein the first memory device includes shared memory and the second memory device is faster than the third memory device.
17. The computer readable medium of claim 16 , wherein the executing further comprises:
requesting one or more blocks of data of the first column from the first data server; and
requesting one or more blocks of data of the second column from the second or third data server.
18. The computer readable medium of claim 16 , wherein the executing further comprises:
requesting one or more blocks of data of the second column from the second data server; and
requesting one or more blocks of data of the second column from the third data server.
19. The computer readable medium of claim 16 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first or second data server; and
requesting one or more blocks of data of the one or more second rows from the third data server.
20. The computer readable medium of claim 16 , wherein the executing further comprises:
requesting one or more blocks of data of the one or more first rows from the first data server; and
requesting one or more blocks of data of the one or more first rows from the second data server.
21. The computer readable medium of claim 16 , wherein the first memory device is a transitory memory device and the second memory device is a persistent memory device.
22. The computer readable medium of claim 16 , wherein the first memory device, the second memory device, and the third memory device are all different from each other.
23. The computer readable medium of claim 15 , wherein the one or more first location identifiers indicating that data of the one or more second rows of the first column is stored in the query server memory device of the query server.