Scalable fine grained access control within a search engine
View Patent ↗A system and method for providing fine-grained access control in a search engine. Access control predicates associated with a search query, including fixed-width and/or variable-width tokens, are received from the search engine, and are formatted in a machine-readable binary format to generate a single byte array. A pre-sorted memory index structure associated with the single byte array is generated, by sorting the access control predicates according to their token width. The pre-sorted memory index structure is merge joined with an uninverted terms index that includes a sorted list of all terms in a field associated with the search query, and a document index mapping each document identifier (ID) to a term ordinal for a specific field.
1. A method of providing fine-grained access control in a search engine, the method comprising:
receiving, by a computer processor from the search engine, one or more access control predicates associated with a search query, the one or more access control predicates including fixed-width and/or variable-width tokens;
formatting, by the computer processor, the one or more access control predicates in a machine-readable binary format to generate a single byte array;
generating, by the computer processor, a pre-sorted memory index structure associated with the single byte array, the pre-sorted memory index structure being generated by sorting the one or more access control predicates according to their associated fixed-width and/or variable width tokens; and
merge joining, by the computer processor, the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of all terms in a field associated with the search query, and having a document index mapping each document identifier (ID) to a term ordinal for a specific field, the merge joining identifying document ID matches against the one or more access control predicates, wherein the merge joining of the pre-sorted memory index structure with the uninverted terms index comprises:
intersecting, by the computer processor, the pre-sorted memory index structure with the sorted list of all terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term;
determining, in response to the intersecting, a term ordinal for each term that is in the sorted list of all terms, the determining comprising iterating, by the computer processor, through each of the document ID's in the document index and retrieving the term ordinal; and
testing, by the computer processor, each ordinal against the bitset of each collected ordinal to identify the document ID matches against the one or more access control predicates.
2. The method in accordance with claim 1 , wherein sorting the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprises calculating, by the computer processor, offsets of any variable-width tokens of the machine-readable binary-formatted access control.
3. The method in accordance with claim 2 , wherein sorting the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprises storing, in a memory by the computer processor, the calculated offsets in an integer array associated with the single byte array.
4. A non-transitory computer program product storing instructions that, when executed by at least one programmable processor, cause the at least one programmable processor to perform operations comprising:
receive, from a search engine, one or more access control predicates associated with a search query, the one or more access control predicates including fixed-width and/or variable-width tokens;
format the one or more access control predicates in a machine-readable binary format to generate a single byte array;
sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens;
generate a pre-sorted memory index structure associated with the single byte array based on the sorting; and
merge join the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of all terms in a field associated with the search query, and having a document index mapping each document identifier (ID) to a term ordinal for a specific field, the merge joining identifying document ID matches against the one or more access control predicates, wherein the operations to merge join the pre-sorted memory index structure with an uninverted terms index include operations to:
intersect the pre-sorted memory index structure with the sorted list of all terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term;
determine, in response to the intersecting, a term ordinal for each term that is in the sorted list of all terms, the determining comprising iterating, by the computer processor, through each of the document ID's in the document index and retrieving the term ordinal; and
test each ordinal against the bitset of each collected ordinal to identify the document ID matches against the one or more access control predicates.
5. The computer program product in accordance with claim 4 , wherein the operations to sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprise operations to calculate offsets of any variable-width tokens of the machine-readable binary-formatted access control.
6. The computer program product in accordance with claim 5 , wherein the operations to sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprise operations to store in a memory the calculated offsets in an integer array associated with the single byte array.
7. A system for providing fine-grained access control in a search engine, the system comprising:
at least one programmable processor; and
a machine-readable medium storing instructions that, when executed by the at least one processor, cause the at least one programmable processor to perform operations comprising:
receive, from the search engine, one or more access control predicates associated with a search query, the one or more access control predicates including fixed-width and/or variable-width tokens;
format the one or more access control predicates in a machine-readable binary format to generate a single byte array;
sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens;
generate a pre-sorted memory index structure associated with the single byte array based on the sorting; and
merge join the pre-sorted memory index structure with an uninverted terms index, the uninverted terms index having a sorted list of all terms in a field associated with the search query, and having a document index mapping each document identifier (ID) to a term ordinal for a specific field, the merge joining identifying document ID matches against the one or more access control predicates wherein the operations to merge join the pre-sorted memory index structure with an uninverted terms index include operations to:
intersect the pre-sorted memory index structure with the sorted list of all terms in the field associated with the search query to collect a bitset of each ordinal number of each intersecting term;
determine, in response to the intersecting, a term ordinal for each term that is in the sorted list of all terms, the determining comprising iterating, by the computer processor, through each of the document ID's in the document index and retrieving the term ordinal; and
test each ordinal against the bitset of each collected ordinal to identify the document ID matches against the one or more access control predicates.
8. The system in accordance with claim 7 , wherein the operations to sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprise operations to calculate offsets of any variable-width tokens of the machine-readable binary-formatted access control.
9. The system in accordance with claim 8 , wherein the operations to sort the one or more access control predicates according to their associated fixed-width and/or variable width tokens further comprise operations to store in a memory the calculated offsets in an integer array associated with the single byte array.