Storage efficient multimaps for processing database queries
A database system uses a hash table for processing database queries. The system stores keys mapped to data values in the hash table. For processing a set of records using the hash table, the system partially sorts the records to match the order of keys of the hash table. The partial storing of the records results in improving the performance of a cache storing the hash table. The hash table may be a multimap that maps a key to multiple data values. The system stores data values mapped to a key in a vector that stores all data values mapped to the key in contiguous memory. The system stores a pointer to the vector in association with the key in the multimap. The system optimizes the multimap to store the data value instead of the pointer if there is a single data value associated with a key.
1 . A computer-implemented method for executing database queries using a multimap, wherein the multimap is a data structure that stores one or more key pointer pairs, each of the one or more key-pointer pair includes key associated with a pointer that points to a vector, the vector includes a plurality of elements, at least one of the plurality of elements stores a data value associated with the key, and at least one of the plurality of elements stores a size value representing a number of data values stored in the vector, the computer-implemented method comprising:
receiving, by a database system, a database query for processing data stored in one or more database tables;
receiving a first set of records based on the one or more database tables;
repeating for each record of the first set of records:
determining a key and a data value corresponding to the record,
identifying a vector mapped to the key,
adding the data value to the vector, wherein the data value is stored in a next available contiguous memory location after a previously stored data value in the vector,
updating a size value representing a number of data values stored in the vector based on the adding of the data value, and
storing in the multimap a pointer to the vector in association with the key, wherein the pointer references a memory location of the vector that stores the size value;
receiving a second set of records based on the one or more database tables;
repeating for each record of the second set of records:
determining a key corresponding to the record,
accessing a pointer corresponding to the key from the multimap,
accessing data values stored in a vector pointed at by the pointer, and
determining one or more result values based on the data values accessed; and
sending a result of processing the database query based on result values determined using the multimap.
2 . The computer-implemented method of claim 1 , wherein the size value representing the number of data values is stored in the vector as an element adjacent to a last data value stored in the vector.
3 . The computer-implemented method of claim 1 , further comprising:
determining that the vector is full based on the size value;
allocating a new vector that is larger than the vector; and
copying the data values of the vector to the new vector.
4 . The computer-implemented method of claim 3 , wherein each vector stores a number of elements that is a power of two, wherein determining that the vector is full comprises comparing the size value with a power of two.
5 . The computer-implemented method of claim 1 , wherein accessing data values stored in a vector comprises:
accessing a size value pointed at by the pointer corresponding to the key; and
determining a location of a first data value based on the size value.
6 . The computer-implemented method of claim 5 , wherein determining the location of the first data value based on the size value comprises:
determining a number of words used to store data values of the vector based on the size value stored in the vector; and
determining the location of the first data value based on the number of words to store data values of the vector and the value stored in the pointer associated with the key.
7 . The computer-implemented method of claim 5 , wherein the database query specifies a join operation of a first database table and a second database table, wherein the first set of records is obtained from the first database table and the second set of records is obtained from the second database table.
8 . A non-transitory computer readable storage medium storing instructions that when processed by a computer processor cause the computer processor to perform steps for executing database queries using a multimap, wherein the multimap is a data structure that stores one or more key-pointer pairs, each of the one or more key-pointer pair includes key associated with a pointer that points to a vector, the vector includes a plurality of elements, at least one of the plurality of elements stores a data value associated with the key, and at least one of the plurality of elements stores a size value representing a number of data values stored in the vector, the steps comprising:
receiving, by a database system, a database query for processing data stored in one or more database tables;
receiving a first set of records based on the one or more database tables;
repeating for each record of the first set of records:
determining a key and a data value corresponding to the record,
identifying a vector mapped to the key,
adding the data value to the vector, wherein the data value is stored in a next available contiguous memory location after a previously stored data value in the vector,
updating a size value representing a number of data values stored in the vector based on the adding of the data value, and
storing in the multimap a pointer to the vector in association with the key, wherein the pointer references a memory location of the vector that stores the size value;
receiving a second set of records based on the one or more database tables;
repeating for each record of the second set of records:
determining a key corresponding to the record,
accessing a pointer corresponding to the key from the multimap,
accessing data values stored in a vector pointed at by the pointer, and
determining one or more result values based on the data values accessed; and
sending a result of processing the database query based on result values determined using the multimap.
9 . The non-transitory computer readable storage medium of claim 8 , wherein the size value representing the number of data values is stored in the vector as an element adjacent to a last data value stored in the vector.
10 . The non-transitory computer readable storage medium of claim 8 , wherein the instructions further cause the computer processor to perform steps comprising:
determining that the vector is full based on the size value;
allocating a new vector that is larger than the vector; and
copying the data values of the vector to the new vector.
11 . The non-transitory computer readable storage medium of claim 10 , wherein each vector stores a number of elements that is a power of two, wherein determining that the vector is full comprises comparing the size value with a power of two.
12 . The non-transitory computer readable storage medium of claim 8 , wherein accessing data values stored in a vector comprises:
accessing a size value pointed at by the pointer corresponding to the key; and
determining a location of a first data value based on the size value.
13 . The non-transitory computer readable storage medium of claim 12 , wherein determining the location of the first data value based on the size value comprises:
determining a number of words used to store data values of the vector based on the size value stored in the vector; and
determining the location of the first data value based on the number of words to store data values of the vector and the value stored in the pointer associated with the key.
14 . The non-transitory computer readable storage medium of claim 12 , wherein the database query specifies a join operation of a first database table and a second database table, wherein the first set of records is obtained from the first database table and the second set of records is obtained from the second database table.
15 . A computer system comprising:
a computer processor; and
a non-transitory computer readable storage medium storing instructions that when processed by a computer processor cause the computer processor to perform steps for executing database queries using a multimap, wherein the multimap is a data structure that stores one or more key-pointer pairs, each of the one or more key-pointer pair includes key associated with a pointer that points to a vector, the vector includes a plurality of elements, at least one of the plurality of elements stores a data value associated with the key, and at least one of the plurality of elements stores a size value representing a number of data values stored in the vector, the steps comprising:
receiving, by a database system, a database query for processing data stored in one or more database tables;
receiving a first set of records based on the one or more database tables;
repeating for each record of the first set of records:
determining a key and a data value corresponding to the record,
identifying a vector mapped to the key,
adding the data value to the vector, wherein the data value is stored in a next available contiguous memory location after a previously stored data value in the vector,
updating a size value representing a number of data values stored in the vector based on the adding of the data value, and
storing in the multimap a pointer to the vector in association with the key,
wherein the pointer references a memory location of the vector that stores the size value;
receiving a second set of records based on the one or more database tables;
repeating for each record of the second set of records:
determining a key corresponding to the record,
accessing a pointer corresponding to the key from the multimap,
accessing data values stored in a vector pointed at by the pointer, and
determining one or more result values based on the data values accessed; and
sending a result of processing the database query based on result values determined using the multimap.
16 . The computer system of claim 15 , wherein the size value representing the number of data values is stored in the vector as an element adjacent to a last data value stored in the vector.
17 . The computer system of claim 15 , wherein the instructions further cause the computer processor to perform steps comprising:
determining that the vector is full based on the size value;
allocating a new vector that is larger than the vector; and
copying the data values of the vector to the new vector.
18 . The computer system of claim 17 , wherein each vector stores a number of elements that is a power of two, wherein determining that the vector is full comprises comparing the size value with a power of two.
19 . The computer system of claim 15 , wherein accessing data values stored in a vector comprises:
accessing a size value pointed at by the pointer corresponding to the key; and
determining a location of a first data value based on the size value.
20 . The computer system of claim 19 , wherein determining the location of the first data value based on the size value comprises:
determining a number of words used to store data values of the vector based on the size value stored in the vector; and
determining the location of the first data value based on the number of words to store data values of the vector and the value stored in the pointer associated with the key.