Scalable fine grained access control within a search engine
A system and method for providing fine-grained access control in a search engine. Access control predicates associated with a search query, including fixed-width and/or variable-width tokens, are received from the search engine, and are formatted in a machine-readable binary format to generate a single byte array. A pre-sorted memory index structure associated with the single byte array is generated, by sorting the access control predicates according to their token width. The pre-sorted memory index structure is merge joined with an uninverted terms index that includes a sorted list of all terms in a field associated with the search query, and a document index mapping each document identifier (ID) to a term ordinal for a specific field.
1. A method of providing access control in a search engine, the method comprising:
receiving one or more access control predicates associated with a search query;
pre-sorting the one or more access control predicates according to at least one or more of fixed-width and variable-width tokens based on an offset value calculated for a single byte array for at least one access control predicate, the offset value being maintained in an integer array separate from the single byte array;
formatting the one or more access control predicates in a machine-readable binary format to generate the single byte array;
generating a pre-sorted memory index structure associated with the single byte array; and
merge joining the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of a plurality of search terms associated with the search query, the uninverted terms index further having a document index mapping one or more document identifiers (ID) to a respective term ordinal for a first field, a merge joining identifying document ID to match against the one or more access control predicates.
2. The method in accordance with claim 1 , wherein the pre-sorting the one or more access control predicates according to an associated fixed-width or variable width tokens further comprises calculating offsets of any variable-width tokens of the machine-readable binary-formatted access control.
3. The method in accordance with claim 2 , wherein the pre-sorting the one or more access control predicates according to the associated fixed-width or variable width tokens further comprises storing the calculated offsets in an integer array associated with the single byte array.
4. The method in accordance with claim 1 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index includes intersecting the pre-sorted memory index structure with the sorted list of one or more terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term.
5. The method in accordance with claim 4 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index further includes iterating through one or more of the document ID's in the document index and retrieving the respective term ordinal.
6. The method in accordance with claim 5 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index further includes testing one or more ordinal against the bitset of a collected ordinal to identify the document ID matches against the one or more access control predicates.
7. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receiving one or more access control predicates associated with a search query;
pre-sorting the one or more access control predicates according to at least one or more of fixed-width and variable-width tokens based on an offset value calculated for a single byte array for at least one access control predicate, the offset value being maintained in an integer array separate from the single byte array;
formatting the one or more access control predicates in a machine-readable binary format to generate the single byte array;
generating a pre-sorted memory index structure associated with the single byte array; and
merge joining the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of a plurality of search terms associated with the search query, the uninverted terms index further having a document index mapping one or more document identifiers (ID) to a respective term ordinal for a first field, a merge joining identifying document ID to match against the one or more access control predicates.
8. The computer program product in accordance with claim 7 , wherein the pre-sorting the one or more access control predicates according to an associated fixed-width or variable width tokens further comprises calculating offsets of any variable-width tokens of the machine-readable binary-formatted access control.
9. The computer program product in accordance with claim 8 , wherein the pre-sorting the one or more access control predicates according to the associated fixed-width or variable width tokens further comprises storing the calculated offsets in an integer array associated with the single byte array.
10. The computer program product in accordance with claim 7 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index includes intersecting the pre-sorted memory index structure with the sorted list of one or more terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term.
11. The computer program product in accordance with claim 10 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index further includes iterating through one or more of the document ID's in the document index and retrieving the respective term ordinal.
12. The computer program product in accordance with claim 10 , wherein merge joining the pre-sorted memory index structure with an uninverted terms index further includes testing one or more ordinal against the bitset of a collected ordinal to identify the document ID matches against the one or more access control predicates.
13. A system for providing fine-grained access control in a search engine, the system comprising:
at least one programmable processor; and
a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising:
receiving one or more access control predicates associated with a search query;
pre-sorting the one or more access control predicates according to at least one or more of fixed-width and variable-width tokens based on an offset value calculated for a single byte array for at least one access control predicate, the offset value being maintained in an integer array separate from the single byte array;
formatting the one or more access control predicates in a machine-readable binary format to generate the single byte array;
generating a pre-sorted memory index structure associated with the single byte array; and
merge joining the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of a plurality of search terms associated with the search query, the uninverted terms index further having a document index mapping one or more document identifiers (ID) to a respective term ordinal for a first field, a merge joining identifying document ID to match against the one or more access control predicates.
14. The system in accordance with claim 13 , wherein the pre-sorting the one or more access control predicates according to an associated fixed-width or variable width tokens further comprises calculating offsets of any variable-width tokens of the machine-readable binary-formatted access control.
15. The system in accordance with claim 14 , wherein the pre-sorting the one or more access control predicates according to the associated fixed-width or variable width tokens further comprises storing the calculated offsets in an integer array associated with the single byte array.
16. The system in accordance with claim 13 , wherein the merge joining the pre-sorted memory index structure with an uninverted terms index includes intersecting the pre-sorted memory index structure with the sorted list of one or more terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term.
17. The system in accordance with claim 16 , wherein merge joining the pre-sorted memory index structure with an uninverted terms index further includes iterating through one or more of the document ID's in the document index and retrieving the respective term ordinal.
18. The system in accordance with claim 17 , wherein merge joining the pre-sorted memory index structure with an uninverted terms index further includes testing one or more ordinal against the bitset of a collected ordinal to identify the document ID matches against the one or more access control predicates.
19. The system of claim 13 , wherein a fixed-width token is stored in the single byte array without padding to reduce memory overhead.
20. The system of claim 13 , wherein a variable-width token is stored in the single byte array with a single byte header describing length of the variable-width token.