Granted Patent B2

US 12,724,771 · App. 18/970,661 · Granted Sep 1, 2026

Inventor: Steve Newman (Portola Valley, CA)

Assignee: Sentinel Labs Israel Ltd.

G06F16/24539G06F16/24561

Abstract

Claims (83)

1 . A computer-implemented method for optimizing query processing in a cloud-based storage system, the computer-implemented method comprising:

receiving, by a computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;

transmitting, by the computing system to the cloud-based storage system, the time range;

receiving, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;

distributing, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;

determining, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;

if the corresponding subset of the plurality of files is not stored on the local storage cache:

retrieving, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;

storing, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;

scanning, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;

copying, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;

determining that the scanning is complete;

determining if the copying is complete; and

if the copying is not complete:

abandoning the copying; and

discarding the subset of the plurality of files from the memory;

if the corresponding subset of the plurality of files is stored on the local storage cache:

scanning, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;

transmitting, from each of the plurality of processing nodes to the computing system, the subset of query results;

aggregating, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result; and

transmitting, by the computing system to the query source, the final query result,

wherein the computing system and each of the plurality of processing nodes comprise a processor and computer-readable memory.

2 . The method of claim 1 , wherein the local storage cache is stored in a local storage drive in communication with a corresponding node.

3 . The method of claim 1 , wherein the memory comprises random-access memory (RAM).

4 . The method of claim 1 , wherein the local storage cache is stored on a file server or a network attached storage (NAS) device.

5 . The method of claim 1 , wherein the cloud database storage system comprises Amazon Simple Storage Service (S3).

6 . The method of claim 5 , wherein the local storage drive comprises an SSD device.

7 . The method of claim 1 , wherein the plurality of processing nodes comprise epoch readers.

8 . A computer system for optimizing query processing in a cloud database storage system, the system comprising:

a computing system comprising a processor and a plurality of processing nodes;

one or more non-transitory computer storage media in communication with the processor and/or the plurality of processing nodes and configured to store computer-executable instructions that when executed by the processor and/or the plurality of processing nodes, cause the processor and/or the plurality of processing nodes to:

receive, by the computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;

transmit, by the computing system to the cloud-based storage system, the time range;

receive, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;

distribute, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;

determine, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;

if the corresponding subset of the plurality of files is not stored on the local storage cache:

retrieve, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;

store, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;

scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;

copy, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;

determine that the scanning is complete;

determine if the copying is complete; and

if the copying is not complete:

abandon the copying; and

discard the subset of the plurality of files from the memory;

if the corresponding subset of the plurality of files is stored on the local storage cache:

scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;

transmit, from each of the plurality of processing nodes to the computing system, the subset of query results;

aggregate, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result; and

transmit, by the computing system to the query source, the final query result.

9 . The system of claim 8 , wherein the local storage cache is stored in a local storage drive in communication with a corresponding node.

10 . The system of claim 8 , wherein the local storage cache is stored on a file server or a network attached storage (NAS) device.

11 . The system of claim 8 , wherein the cloud database storage system includes Amazon S3.

12 . The system of claim 11 , wherein the local storage drive comprises an SSD device.

13 . The system of claim 8 , wherein the plurality of processing nodes comprise epoch readers.

14 . The system of claim 8 , wherein the memory comprises random-access memory (RAM).

15 . A non-transitory computer storage media in communication with the processor and/or the plurality of processing nodes and configured to store computer-executable instructions that when executed by the processor and/or the plurality of processing nodes, cause the processor and/or the plurality of processing nodes to:

receive, by the computing system from a query source, a query of a dataset stored in a database storage system, wherein the query comprises one or more query parameters, wherein the one or more query parameters comprises a time range;

transmit, by the computing system to the cloud-based storage system, the time range;

receive, by the computing system, a list comprising a plurality of files, wherein each file of the plurality of files is selected from the time range;

distribute, by the computing system to a plurality of processing nodes, the plurality of files, wherein each processing node of the plurality of processing nodes is assigned a corresponding subset of the plurality of files, and wherein each processing node of the plurality of processing nodes comprises a local storage cache;

determine, by each of the plurality of processing nodes, whether the corresponding subset of the plurality of files is stored on the local storage cache;

if the corresponding subset of the plurality of files is not stored on the local storage cache:

retrieve, by each of the plurality of processing nodes, the corresponding subset of the plurality of files not stored on the local storage cache;

store, by each of the plurality of processing nodes, the retrieved corresponding subset of the plurality of files in a memory corresponding to each of the plurality of processing nodes;

scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored in the memory for data matching one or more query parameters to generate a subset of query results;

copy, concurrently with the scanning and using a separate processing thread from the scanning, the corresponding subset of the plurality of files stored in the memory to the local storage cache;

determine that the scanning is complete;

determine if the copying is complete; and

if the copying is not complete:

abandon the copying; and

discard the subset of the plurality of files from the memory;

if the corresponding subset of the plurality of files is stored on the local storage cache:

scan, by each of the plurality of processing nodes, the corresponding subset of the plurality of files stored on the local storage cache for data matching the one or more query parameters to generate the subset of query results;

transmit, from each of the plurality of processing nodes to the computing system, the subset of query results;

aggregate, by the computing system, each subset of query results from each of the plurality of processing nodes to generate a final query result; and

transmit, by the computing system to the query source, the final query result.

16 . The non-transitory storage medium of claim 15 , wherein the local storage cache is stored in a local storage drive in communication with a corresponding node.

17 . The non-transitory storage medium of claim 15 , wherein the local storage cache is stored on a file server or a network attached storage (NAS) device.

18 . The non-transitory storage medium of claim 15 , wherein the cloud database storage system includes Amazon S3.

19 . The non-transitory storage medium of claim 18 , wherein the local storage drive comprises an SSD device.

20 . The non-transitory storage medium of claim 15 , wherein the plurality of processing nodes comprise epoch readers.

Assignments (1)

Continuity (3)

References Cited (400)