Efficient ingest and search of access controlled records
View Patent ↗Large data sets that use ACLS can be efficiently ingested into a search index and subsequently searched. The ingestion may replace the ACLS of records with virtual ACL nodes that are common to a number of individual records, and a virtual ACL node to ACL entities index is provided for use in searching the records with the virtual ACL nodes.
1. A method comprising:
ingesting a plurality of records comprising:
receiving a plurality of records each comprising record data and an associated access control list (ACL) specifying one or more authorized entities that are authorized to access the record data;
slitting the received plurality of records into data portions comprising the record data, and ACL portions comprising the ACLs;
optimizing the ACL portions to generate a plurality of virtual ACL nodes each specifying a plurality of authorized entities from the authorized entities of the received plurality of records, wherein the optimizing comprises:
identifying all identical ACLs of the received ACLs, and
replacing each of the identical ACLs with a single virtual ACL node;
combining the data portions and the optimized ACL portions by replacing respective authorized entities of the ACL portions with corresponding virtual ACL nodes having the same respective authorized entities;
generating an index of the combined data portions and optimized ACL portions based at least on the plurality of virtual ACL nodes; and
querying the plurality of ingested records comprising:
receiving a query including an ACL filter specifying an entity;
determining one or more virtual ACL nodes specifying the entity of the ACL filter from the index;
transforming the received query by replacing the entity specified in the query with the determined one or more virtual ACL nodes;
querying the index using the transformed query; and
returning at least one of the plurality of ingested records matching the index in response to the received query.
2. The method of claim 1 , wherein the generating the index comprises:
indexing a first mapping between the respective authorized entities and the corresponding virtual ACL nodes; and
indexing a second mapping between the corresponding virtual ACL nodes and the respective authorized entities specified on the corresponding virtual ACL nodes.
3. The method of claim 1 , wherein the optimizing the ACL portions comprises replacing authorized entities of the plurality of authorized entities common to records of the plurality of received records with the single virtual ACL node.
4. The method of claim 3 , comprising; determining a processing savings by replacing the authorized entities of the plurality of authorized entities common to the records of the plurality of received records with the single virtual ACL node.
5. The method of claim 4 , wherein the authorized entities of the plurality of authorized entities common to the records of the plurality of received records are replaced with the single virtual ACL node if the processing savings is above a threshold value.
6. The method of claim 3 , further comprising:
hashing the common authorized entities that have been replaced by the single virtual ACL node; and
indexing the hashed common authorized entities and the single virtual ACL node that replaced the common authorized entities.
7. The method of claim 6 , the replacing the respective authorized entities of the ACL portions with the corresponding virtual ACL nodes comprises:
hashing the respective authorized entities of the ACL portions;
using the hashed authorized entities to determine a corresponding indexed virtual ACL node; and
replacing the hashed authorized entities with the corresponding indexed virtual ACL node.
8. A system comprising:
one or more processors; and
one or more computer readable memories coupled to the one or more processors, the one or more computer readable memories having instructions stored thereon, which when executed by the one or more processors configure the system to:
ingest a plurality of records comprising:
receiving a plurality of records each comprising record data and an associated access control list (ACL) specifying one or more authorized entities that are authorized to access the record data;
slitting the received plurality of records into data portions comprising the record data, and ACL portions comprising the ACLs;
optimizing the ACL portions to generate a plurality of virtual ACL nodes each specifying a plurality of authorized entities from the authorized entities of the received plurality of records, wherein the optimizing comprises:
identifying all identical ACLs of the received ACLs, and
replacing each of the identical ACLs with a single virtual ACL node;
combining the data portions and the optimized ACL portions by replacing respective authorized entities of the ACL portions with corresponding virtual ACL nodes having the same respective authorized entities;
generating an index of the combined data portions and optimized ACL portions based at least on the plurality of virtual ACL nodes; and
query the plurality of ingested records comprising:
receiving a query including an ACL filter specifying an entity;
determining one or more virtual ACL nodes specifying the entity of the ACL filter from the index;
transforming the received query by replacing the entity specified in the query with the determined one or more virtual ACL nodes;
querying the index using the transformed query; and
returning at least one of the plurality of ingested records matching the index in response to the received query.
9. The system of claim 8 , wherein the generating the index comprises:
indexing a first mapping between the respective authorized entities and the corresponding virtual ACL nodes; and
indexing a second mapping between the corresponding virtual ACL nodes and the respective authorized entities specified on the corresponding virtual ACL nodes.
10. The system of claim 8 , wherein the optimizing the ACL portions comprises replacing authorized entities of the plurality of authorized entities common to records of the plurality of received records with the single virtual ACL node.
11. The system of claim 10 , comprising: determining a processing savings by replacing the authorized entities of the plurality of authorized entities common to the records of the plurality of received records with the single virtual ACL node.
12. The system of claim 8 , wherein the authorized entities of the plurality of authorized entities common to the records of the plurality of received records are replaced with the single virtual ACL node if the processing savings is above a threshold value.
13. The system of claim 10 , further comprising:
hashing the common authorized entities that have been replaced by the single virtual ACL node; and
indexing the hashed common authorized entities and the single virtual ACL node that replaced the common authorized entities.
14. The system of claim 13 , the replacing the respective authorized entities of the ACL portions with the corresponding virtual ACL nodes comprises:
hashing the respective authorized entities of the ACL portions;
using the hashed authorized entities to determine a corresponding indexed virtual ACL node; and
replacing the hashed authorized entities with the corresponding indexed virtual ACL node.