Systems and methods of generating and using a bitmap index
Systems and methods of generating and using a bitmap index are disclosed. The bitmap index stores bit strings. In a particular implementation, each bit string in the bitmap index corresponds to an attribute. Each location in each bit string corresponds to the same source. The bitmap index supports parallelized and distributed execution of queries with respect to data indexed by the bitmap index. For example, the described bitmap index can be used to index advertising data, healthcare data, financial data, etc.
1. A method comprising:
receiving data at a computing device comprising a processor, wherein the data indicates that a first source has a first attribute, that the first source has a second attribute, and that a second source has the first attribute;
in response to receiving the data, modifying a first bit at a first location of a first bit string of a bitmap index, a second bit at a second location of the first bit string, and a third bit at the first location of a second bit string of the bitmap index, wherein each of the first bit string and the second bit string corresponds to a distinct attribute, and wherein a value stored in a particular location in each of the first bit string and the second bit string indicates whether a corresponding source has a corresponding attribute;
receiving a query directed to the bitmap index; and
outputting a result of the query based on performing one or more set operations with respect to at least two bits selected from the group consisting of the first bit, the second bit, and the third bit.
2. The method of claim 1 , wherein the first location is in a first column of the bitmap index that corresponds to the first attribute and in a first row of the bitmap index that corresponds to the first source.
3. The method of claim 1 , wherein a query execution plan identifies:
the one or more set operations;
that one or more first nodes are to send stored portions of one or more bit strings to a second node;
that the second node is to perform the one or more set operations with respect to:
the portions of the one or more bit strings received from the one or more first nodes; and
stored portions of one or more bit strings stored at the second node; and
that the second node is to concatenate results of performing the one or more set operations to generate a result bit string that indicates a result of the query.
4. The method of claim 1 , wherein the first attribute is assigned to the first source based on a registration event.
5. The method of claim 4 , wherein the registration event indicates demographic information of a first user based on social network data.
6. The method of claim 1 , wherein the first bit string, the second bit string, or a combination thereof corresponds to a demographic attribute, a behavior, a brand affinity, or a combination thereof.
7. The method of claim 1 , further comprising determining a top Z signals for a subset of users based on the bitmap index, wherein Z is a positive integer.
8. The method of claim 7 , wherein determining the top Z signals comprises:
generating a filter string corresponding to the subset of users;
performing intersection operations of the filter string to generate a plurality of result strings; and
identifying the top Z signals for the subset of users based on the result strings.
9. The method of claim 8 , wherein generating the filter string includes performing one or more logical operations.
10. The method of claim 1 , wherein a first portion of the first bit string is stored at a first particular node and a second portion of the first bit string is stored at a second particular node that is different from the first particular node.
11. The method of claim 1 , wherein the first bit string, the second bit string, or a combination thereof is stored as one or more distributed slices.
12. The method of claim 11 , wherein:
each of the one or more distributed slices comprises one or more chunks; and
a particular chunk is stored in a compressed form if each bit of the particular chunk has a particular value.
13. The method of claim 1 , wherein data corresponding to each row of the bitmap index is stored in a data store.
14. The method of claim 1 , wherein data corresponding to at least one signal is stored in a data store but not in the bitmap index.
15. The method of claim 1 , wherein the bitmap index is associated with at least one of advertising data, customer data, inventory data, sensor data, healthcare data, or financial data.
16. A system comprising:
a processor; and
a memory storing instructions executable by the processor to perform operations comprising:
receiving data, wherein the data indicates a first source having a first attribute, the first source having a second attribute, and a second source having the first attribute; and
in response to receiving the data, modifying a first bit at a first location of a first bit string of a bitmap index, a second bit at a second location of the first bit string, and a third bit at the first location of a second bit string of the bitmap index, wherein each of the first bit string and the second bit string corresponds to a distinct attribute, and wherein a value stored in a particular location in each of the first bit string and the second bit string indicates whether a corresponding source has a corresponding attribute.
17. The system of claim 16 , wherein the bitmap index is associated with at least one of advertising data, customer data, inventory data, sensor data, healthcare data, or financial data.
18. The system of claim 16 , wherein the first bit string, the second bit string, or a combination thereof corresponds to a demographic attribute, a behavior, a brand affinity, or a combination thereof.
19. A non-transitory computer readable storage device storing instructions that, when executed, cause a computer to perform operations comprising:
receiving first data at a computing device comprising a processor, wherein the first data indicates a first source having a first attribute, the first source having a second attribute, and a second source having the first attribute; and
in response to receiving the first data, modifying a first bit at a first location of a first bit string of a bitmap index, a second bit at a second location of the first bit string, and a third bit at the first location of a second bit string of the bitmap index, wherein each bit of the first bit string and the second string corresponds to a distinct attribute, and wherein a value stored in a particular location in each of the first bit string and the second bit string indicates whether a corresponding source has a corresponding attribute.
20. The non-transitory computer readable storage device of claim 19 , wherein the bitmap index is associated with at least one of advertising data, customer data, inventory data, sensor data, healthcare data, or financial data.