Systems and methods of packet object database management
View Patent ↗The present disclosure generally provides systems and methods of packet object database management. The database management system includes a database server designed specifically to process binary network packet data. The database server is associated with a parser, query engine, retrieval engine, virtual machine, data manger, and file processor. The database management system uses a proprietary query language to support all accesses to the database. The parser identifies whether the query is a data management query or if it is a data retrieval instruction. If the query is a data management query, the data manager manages the query request and attempts to satisfy the query request. Otherwise, query engine could further analyze or parse the query into a particular query structure or sub-structures to attempt to satisfy the query request.
1. A method, comprising:
receiving a plurality of data packages at one or more hardware database servers, each of the plurality of data packages including a plurality of packets:
receiving a query that includes one or more selection criteria;
indexing the query to generate an initial indexing set;
creating, using the one or more hardware database servers, one or more lists of one or more packets for each of the plurality of data packages, the one or more lists configured to index a sequence of packets within each of the plurality of data packages by source address information and destination address information, the one or more lists indicating, for each of the plurality of data packages, an offset of each of: (a) a first packet and a last packet in the sequence that contains given source address information, and (b) a first packet and a last packet in the sequence that contains given destination address information, the one or more lists further configured to allow the one or more hardware database servers to bypass packets within each of the plurality of data packages during execution of the query using one or more of the offsets, each of the offsets provided as a number of bytes;
storing both the query and the initial indexing set;
passing both the query and the initial indexing set to a parser to generate a symbolic program;
compiling and executing the symbolic program to retrieve, by the one or more hardware database servers and based at least in part upon the one or more lists, one or more packets that satisfy the query, wherein retrieving the one or more packets includes executing a plurality of threads to satisfy the query;
creating a table, using the one or more hardware database servers, that includes data from each retrieved packet matching the one or more selection criteria; and
combining results from the plurality of threads to generate a final result of the query.
2. The method of claim 1 , wherein the packets within each of the plurality of data packages are stored in a sequence according to an order of their arrival at a respective network device.
3. A system, comprising: a hardware database server, the hardware database server configured to:
store a plurality of data packages, each of the plurality of data packages including a sequence of packets;
receive a query, the query including selection criteria;
index the query to generate an initial indexing set;
storing both the query and the initial indexing set;
create one or more lists of one or more packets for each of the plurality of data packages, the one or more lists configured to index a sequence of packets within each of the plurality of data packages by source address information and destination address information, the one or more lists indicating, for each of the plurality of data packages, an offset of each of: (a) a first packet and a last packet in the sequence that contains given source address information, and (b) a first packet and a last packet in the sequence that contains given destination address information, the one or more lists further configured to allow the one or more hardware database servers to bypass packets within each of the plurality of data packages during execution of the query using one or more of the offsets, each of the offsets provided as a number of bytes;
pass both the query and the initial indexing set to a parser to generate a symbolic program;
compile and execute the symbolic program to retrieve, by the one or more hardware database servers and based at least in part upon the one or more lists, one or more packets that satisfy the query, wherein retrieving the one or more packets includes executing a plurality of threads to satisfy the query;
create a table, using the one or more hardware database servers, that includes data from each retrieved packet matching the one or more selection criteria; and
combine results from each of the plurality of threads to generate a final result of the query.
4. The system of claim 3 , wherein the packets in each of the plurality of data packages are stored in order of their arrival at a respective network device.
5. The system of claim 4 , the hardware database server further configured to: launch a separate thread for each of the plurality of data packages, each of the separate threads configured to execute the query with respect to each corresponding data package using an indexed list, the indexed list indexing the sequence of packets within each of the plurality of data packages by at least one of: source address or destination address, the indexed list including, for each of the at least one of the source address or destination address, an offset, configured to enable each of the separate threads to bypass as many packets as possible that do not match the selection criteria within a respective data package;
wherein, for each of the plurality of data packages, the indexed list includes the offsets of a first packet in the sequence that contains a given source address and of a last packet in the sequence that contains the given source address.
6. The system of claim 5 , wherein, for each of the plurality of data packages, the indexed list includes the offsets of a first packet in the sequence that contains a given destination address and of a last packet in the sequence that contains the given destination address.
7. The system of claim 5 , wherein each of the plurality of data packages includes packets of disparate data types.
8. The system of claim 7 , wherein the indexed list includes data contained in a given packet as fields in positions that are unassociated with their actual positions in the given packet.
9. The method of claim 1 , wherein combining the query results includes creating a table that excludes fields of packets that are not responsive to the query.
10. The method of claim 1 , wherein the one or more selection criteria include a requested time, the method further comprising:
selecting data packages corresponding to the requested time; and
launching by the plurality of threads for the selected data packages.
11. The method of claim 10 , wherein each of the data packages corresponds to an hour's traffic, the method further comprising, in response to the requested time being greater than one hour, selecting two or more data packages from different network devices.