Method and apparatus for data capture and analysis system
View Patent ↗Content leaving a local network can be captured and indexed so that queries can be performed on the captured data. In one embodiment, the present invention comprises an apparatus that connects to a network. In one embodiment, this apparatus includes a network interface module to connect the apparatus to a network, a packet capture module to intercept packets being transmitted on the network, an object assembly module to reconstruct objects being transmitted on the network from the intercepted packets, an object classification module to determine the content in the reconstructed objects, and an object store module to store the objects. This apparatus can also have a user interface to enable a user to search objects stored in the object store module.
1. A method, comprising:
receiving a flow of packets in a network;
applying a filter to the flow in order to identify a protocol for the flow;
extracting a plurality of objects associated with flow;
determining a content type for each of the objects based on a signature identified within the objects; and
providing a user interface to enable a user to use a query to search for stored objects, wherein the query includes search criteria used to identify certain objects that match the search criteria, wherein a particular search is scheduled for a recurring time interval and includes a particular search query with selected terms, and wherein certain results of the particular search trigger an e-mail message to be sent to an administrator.
2. The method of claim 1 , wherein the determining includes identifying malicious content based on a file extension associated with the flow.
3. The method of claim 1 , further comprising:
determining whether the object is to be stored or discarded based on a plurality of capture rules.
4. The method of claim 3 , further comprising:
providing the user interface for authoring the capture rules that designate which objects is to be stored or discarded.
5. The method of claim 1 , further comprising:
storing at least some of the objects in a content store, which is indexed by a tag database in which each record has an associated tag that indexes a corresponding object.
6. The method of claim 1 , further comprising:
receiving a search query for particular content associated with the flow.
7. The method of claim 1 , further comprising:
providing at least some of the objects in response to the search query.
8. The method of claim 1 , further comprising:
providing at least some of the packets to an object assembly module, which is configured to reconstruct an element originally sought for propagation in the network.
9. The method of claim 1 , wherein the packets are reassembled into unique flows based on a source Internet Protocol (IP) address and a destination IP address.
10. The method of claim 1 , wherein the packets form an e-mail.
11. The method of claim 1 , wherein the packets form a document.
12. The method of claim 1 , wherein a network interface receives the flow of packets.
13. The method of claim 12 , wherein the network interface comprises a plurality of network interface cards (NICs).
14. The method of claim 12 , wherein the network interface comprises a plurality of Ethernet cards.
15. Logic encoded in one or more non-transitory media that includes code for execution and when executed by a processor operable to perform operations comprising:
receiving a flow of packets in a network;
applying a filter to the flow in order to identify a protocol for the flow;
extracting a plurality of objects associated with flow;
determining a content type for each of the objects based on a signature identified within the objects; and
providing a user interface to enable a user to use a query to search for stored objects, wherein the query includes search criteria used to identify certain objects that match the search criteria, wherein a particular search is scheduled for a recurring time interval and includes a particular search query with selected terms, and wherein certain results of the particular search trigger an e-mail message to be sent to an administrator.
16. The logic of claim 15 , wherein the determining includes identifying malicious content based on a file extension associated with the flow, and wherein a determination is made as to whether the object is to be stored or discarded based on a plurality of capture rules, and wherein the user interface is provided for authoring the capture rules that designate which objects is to be stored or discarded.
17. The media of claim 15 , wherein the determining includes identifying malicious content based on a file extension associated with the flow.
18. The media of claim 15 , the operations further comprising:
determining whether the object is to be stored or discarded based on a plurality of capture rules.
19. The media of claim 18 , the operations further comprising:
providing the user interface for authoring the capture rules that designate which objects is to be stored or discarded.
20. The media of claim 15 , the operations further comprising:
storing at least some of the objects in a content store, which is indexed by a tag database in which each record has an associated tag that indexes a corresponding object.
21. The media of claim 15 , the operations further comprising:
receiving a search query for particular content associated with the flow.
22. The media of claim 15 , the operations further comprising:
providing at least some of the objects in response to the search query.
23. The media of claim 15 , the operations further comprising:
providing at least some of the packets to an object assembly module, which is configured to reconstruct an element originally sought for propagation in the network.
24. The media of claim 15 , wherein the packets are reassembled into unique flows based on a source Internet Protocol (IP) address and a destination IP address.
25. The media of claim 15 , wherein a network interface receives the flow of packets, and wherein the network interface comprises a plurality of network interface cards (NICs) or a plurality of Ethernet cards.
26. An apparatus, comprising:
a memory element; and
a processor coupled to the memory element, wherein the apparatus is configured for:
receiving a flow of packets in a network;
applying a filter to the flow in order to identify a protocol for the flow;
extracting a plurality of objects associated with flow;
determining a content type for each of the objects based on a signature identified within the objects; and
providing a user interface to enable a user to use a query to search for stored objects, wherein the query includes search criteria used to identify certain objects that match the search criteria, wherein a particular search is scheduled for a recurring time interval and includes a particular search query with selected terms, and wherein certain results of the particular search trigger an e-mail message to be sent to an administrator.
27. The apparatus of claim 26 , wherein the determining includes identifying malicious content based on a file extension associated with the flow.
28. The apparatus of claim 26 , wherein the apparatus is further configured for:
determining whether the object is to be stored or discarded based on a plurality of capture rules.
29. The apparatus of claim 28 , wherein the apparatus is further configured for:
providing the user interface for authoring the capture rules that designate which objects is to be stored or discarded.
30. The apparatus of claim 26 , wherein the apparatus is further configured for:
storing at least some of the objects in a content store, which is indexed by a tag database in which each record has an associated tag that indexes a corresponding object.
31. The apparatus of claim 26 , wherein the apparatus is further configured for:
receiving a search query for particular content associated with the flow.
32. The apparatus of claim 26 , wherein the apparatus is further configured for:
providing at least some of the objects in response to the search query.
33. The apparatus of claim 26 , wherein the apparatus is further configured for:
providing at least some of the packets to an object assembly module, which is configured to reconstruct an element originally sought for propagation in the network.
34. The apparatus of claim 26 , wherein the packets are reassembled into unique flows based on a source Internet Protocol (IP) address and a destination IP address.
35. The apparatus of claim 26 , wherein a network interface receives the flow of packets, and wherein the network interface comprises a plurality of network interface cards (NICs) or a plurality of Ethernet cards.