Object classification in a capture system
View Patent ↗Objects can be extracted from data flows captured by a capture device. Each captured object can then be classified according to content. In one embodiment, the present invention includes determining whether a captured object is binary or textual in nature, and classifying the captured object as one of a plurality of textual content types based tokens found in the captured object if the captured object is determined to be textual in nature.
1. A method for classifying an object according to content comprising: determining whether the object is binary or textual in nature, wherein the object is captured and the captured object is a plurality of packets that are broken down by a capture system and then reassembled; classifying the object as one of a plurality of textual content types based on tokens found in the object if the object is determined to be textual in nature, wherein each of the tokens found in the object have an associated weight and the weights of the tokens are used to determine the content type of the object, and wherein a confidence level is assigned for classifying the content type of the object; and inserting the content type into a content field of a tag that indexes the object in a storage location and contains a plurality of fields to describe the object, wherein the capture system is configured to allow a document that includes the captured object to be forwarded from the capture system to its intended destination at a network node unless a capture rule prohibits forwarding the document based on the document including the captured object; and further classifying the object as an encrypted document based on a statistical characteristic of the object if the object is determined to be binary in nature, wherein the statistical characteristic of the object comprises a byte distribution of the object.
2. The method of claim 1 , wherein classifying the object comprises generating a list of known tokens found in the object, each token being associated with a textual content type, calculating the probability of the object being each of the plurality of textual content types, and classifying the object as the textual content type with the highest probability.
3. The method of claim 2 , wherein calculating the probability of the object being each of the plurality of textual content types comprises performing Bayesian statistics on the list of known tokens found in the object.
4. The method of claim 1 , wherein the plurality of textual content types include Englishtext, Frenchtext, Germantext, Spanishtext, Japanesetext, Chinesetext, Koreantext, Russiantext, Basic_Source, C++_Source, C_Source, Java_Source, FORTRAN_Source, Verilog_Source, VHDL_Source, Assembly_Source, Pascal_Source, Cobol_Source, Ada_Source, Lisp_Source, Perl_Source, XQuery_Source, Hypertext Markup Language, Cascaded Style Sheets, JavaScript, DXF, Spice, Gerber, Mathematica, Matlab, AllegroPCB, ViewLogic, TangoPCAD, BSDL, C_Shell, K_Shell, Bash_Shell, Bourne_Shell, FTP, Telnet, MSExchange, POP3, RFC822, CVS, CMS, SQL, RTSP, MIME, PDF, PS, and Stockdata.
5. The method of claim 1 , further comprising attempting to classify the object by searching for a known binary signature associated with a content type in the object.
6. The method of claim 1 , wherein the object being textual in nature comprises the object containing ASCII text.
7. A method comprising: intercepting a flow; classifying the flow according to transmission protocol; extracting one or more objects from the classified flow using a protocol handler corresponding with the transmission protocol, wherein the objects are respective pluralities of packets that are captured and broken down by a capture system and then reassembled; classifying the one or more objects based on content type by statistically analyzing the object, wherein tokens found in the object have an associated weight and the weights of the tokens are used to determine the content type of the object, and wherein a confidence level is assigned for classifying the content type of the object; and inserting the content type into a content field of a tag that indexes the object in a storage location and contains a plurality of fields to describe the object, wherein a capture system that receives the captured object, which is part of a document, is configured to allow the document to be forwarded from the capture system to its intended destination at a network node unless a capture rule prohibits forwarding the document based on the document including one or more objects; and further classifying the object as an encrypted document based on a statistical characteristic of the object if the object is determined to be binary in nature, wherein statistically analyzing the object comprises determining a byte distribution.
8. The method of claim 7 , wherein classifying the flow according to protocol comprises not recognizing a transmission protocol carrying the flow and classifying the flow as unknown; and wherein extracting one or more object comprises using a default protocol handler to extract the one or more objects from the unknown flow.
9. The method of claim 7 , wherein the unknown protocol handler classifies the unknown flow as one object.
10. An apparatus comprising: an object statistics module to determining whether an object is binary or textual in nature, wherein the object is captured and the captured object is a plurality of packets that are broken down by a capture system and then reassembled; a token database to store a plurality of tokens, each token being associated with a textual content type, wherein each of the tokens found in the object have an associated weight and the weights of the tokens are used to determine the content type of the object, and wherein a confidence level is assigned for classifying the content type of the object; and a token analyzer to classify the object as one of the plurality of textual content types by accessing the token database if the object is determined to be textual in nature, wherein the content type is inserted into a content field of a tag that indexes the object in a storage location and contains a plurality of fields to describe the object, wherein the capture system is configured to allow a document that includes the captured object to be forwarded from the capture system to its intended destination at a network node unless a capture rule prohibits forwarding the document based on the document including the captured object; and wherein the object statistics module is configured to classify the object as an encrypted document based on a statistical characteristic of the object if the object is binary in nature, wherein the statistical characteristic of the object comprises a byte distribution of the object.
11. The apparatus of claim 10 , the token analyzer is configured to classify the object by generating a list of known tokens found in the object, each token being associated with a textual content type, calculating the probability of the object being each of the plurality of textual content types, and classifying the object as the textual content type with the highest probability.
12. The apparatus of claim 11 , wherein the token analyzer is configured to calculate the probability of the object being each of the plurality of textual content types by performing Bayesian statistics on the list of known tokens found in the object.
13. The apparatus of claim 10 , wherein the plurality of textual content types include Englisthtext, Frenchtext, Germantext, Spanishtext, Japanesetext, Chinesetext, Koreantext, Russiantext, Basic_Source, C++_Source, C_Source, Java_Source, FORTRAN_Source, Verilog_Source, VHDL_Source, Assembly_Source, Pascal_Source, Cobol_Source, Ada_Source, Lisp_Source, Perl_Source, XO.uery_Source, Hypertext Markup Language, Cascaded Style Sheets, JavaScript, DXF, Spice, Gerber, Mathematica, Matlab, AllegroPCB, ViewLogic, TangoPCAD, BSDL, C_Shell, K_Shell, Bash_Shell, Bourne_Shell, FTP, Telnet, MSExchange, POP3, RFC822, CVS, CMS, SO.L, RTSP, MIME, PDF, PS, and Stockdata.
14. The apparatus of claim 10 , further comprising a binary signature module to classify the object by searching for a known binary signature associated with a content type in the object.
15. A non-transitory computer storage medium having stored thereon data representing instructions that, when executed by a processor of a capture system, cause the processor to perform operations comprising: determining whether a captured object is binary or textual in nature, wherein the object is captured and the captured object is a plurality of packets that are broken down by a capture system and then reassembled; classifying the captured object as one of a plurality of textual content types based on tokens found in the captured object if the captured object is determined to be textual in nature, wherein each of the tokens found in the object have an associated weight and the weights of the tokens are used to determine the content type of the object, and wherein a confidence level is assigned for classifying the content type of the object; and inserting the content type into a content field of a tag that indexes the object in a storage location and contains a plurality of fields to describe the object, wherein the capture system is configured to allow a document that includes the captured object to be forwarded from the capture system to its intended destination at a network node unless a capture rule prohibits forwarding the document based on the document including the captured object; and wherein the instructions further cause the processor to classify the captured object as an encrypted document based on a statistical characteristic of the captured object if the captured object is determined to be binary in nature, wherein the statistical characteristic of the captured object comprises a byte distribution of the captured object.
16. The non-transitory computer storage medium of claim 15 , wherein classifying the captured object comprises generating a list of known tokens found in the captured object, each token being associated with a textual content type, calculating the probability of the captured object being each of the plurality of textual content types, and classifying the captured object as the textual content type with the highest probability.
17. The non-transitory computer storage medium of claim 16 , wherein calculating the probability of the captured object being each of the plurality of textual content types comprises performing Bayesian statistics on the list of known tokens found in the captured object.
18. The non-transitory computer storage medium of claim 15 , wherein the plurality of textual content types include Englishtext, Frenchtext, Germantext, Spanishtext, Japanesetext, Chinesetext, Koreantext, Russiantext, Basic_Source, C++_Source, C_Source, Java_Source, FORTRAN_Source, Verilog_Source, VHDL_Source, Assembly_Source, Pascal_Source, Cobol_Source, Ada_Source, Lisp_Source, Perl_Source, XQuery_Source, Hypertext Markup Language, Cascaded Style Sheets, JavaScript, DXF, Spice, Gerber, Mathematica, Matlab, AllegroPCB, ViewLogic, TangoPCAD, BSDL, C_Shell, K_Shell, Bash_Shell, Bourne_Shell, FTP, Telnet, MSExchange, POP3, RFC822, CVS, CMS, SQL, RTSP, MIME, PDF, PS, and Stockdata.
19. The non-transitory computer storage medium of claim 15 , wherein the instruction further cause the processor to attempt to classify the captured object by searching for a known binary signature associated with a content type in the captured object.
20. The non-transitory computer storage medium of claim 15 , wherein the captured object being textual in nature comprises the captured object containing ASCII text.
21. A non-transitory computer storage medium having stored thereon data representing instructions that, when executed by a processor of a capture system, cause the processor to perform operations comprising: intercepting a flow; classifying the flow according to transmission protocol; extracting one or more objects from the classified flow using a protocol handler corresponding with the transmission protocol, wherein the objects are respective pluralities of packets that are captured and broken down by a capture system and then reassembled; classifying the one or more objects based on content type by statistically analyzing the object, wherein tokens found in the object have an associated weight and the weights of the tokens are used to determine the content type of the object, and wherein a confidence level is assigned for classifying the content type of the object; and inserting the content type into a content field of a tag that indexes the object in a storage location and contains a plurality of fields to describe the object, wherein the capture system that receives the captured objects, which are part of a document, is configured to allow the document to be forwarded from the capture system to its intended destination at a network node unless a capture rule prohibits forwarding the document based on the document including the objects; and wherein the instructions further cause the processor to classify the captured object as an encrypted document based on a statistical characteristic of the captured object if the captured object is determined to be binary in nature, wherein statistically analyzing the object comprises determining a byte distribution.
22. The non-transitory computer storage medium of claim 21 , wherein classifying the flow according to protocol comprises not recognizing a transmission protocol carrying the flow and classifying the flow as unknown; and wherein extracting one or more object comprises using a default protocol handler to extract the one or more objects from the unknown flow.
23. The non-transitory computer storage medium of claim 21 , wherein the unknown protocol handler classifies the unknown flow as one object.