Segmentation of web documents based on detected security vulnerabilities
Various embodiments include systems and methods to implement processing of web content for vulnerability assessments. A plurality of documents comprising web content may be obtained from multiple different web sources, and the documents may be parsed to determine a set of discrete document chunks. Parsing the documents includes determining whether a document satisfies a segmentation condition for segmenting the document into multiple discrete document chunks using a named-entity recognition system configured to segment the document based at least in part on a vulnerability identification. The discrete document chunks may be stored in a database, where vulnerability information is indexed such that each respective entry in the database corresponds to a respective vulnerability identification and a respective discrete document chunk.
1 . A system, comprising:
one or more computer devices that implement a vulnerability assessment system for web content, configured to:
obtain a plurality of documents that includes web content;
in response to a determination that a length of a document exceeds a threshold document length:
segment the document into multiple document chunks using a named-entity recognition (NER) system that segments documents based on vulnerability identifications (IDs); and
store the document chunks in a database, wherein the document chunks are indexed in the database so that each entry in the database corresponds to a respective document chunk and a respective vulnerability ID; and
execute a machine learning model to assess the document for one or more security vulnerabilities based on the document chunks stored in the database.
2 . The system of claim 1 , wherein the vulnerability assessment system is configured to:
in response to a determination that a second document does not exceed the threshold document length, store the second document as a discrete document chunk without segmenting the second document.
3 . The system of claim 1 , wherein the vulnerability assessment system is configured to receive documents submitted by a plurality of web crawlers.
4 . The system of claim 1 , wherein the NER system comprises a transformer machine learning model configured to identify one or more predefined entity types in the document associated with vulnerability information.
5 . The system of claim 4 , wherein the one or more predefined entity types includes at least one of: a product name, an exploitation technique, or a common vulnerabilities and exposures identifier (CVE-ID).
6 . The system of claim 1 , wherein the machine learning model comprises a neural network configured to:
infer vulnerability information about the document based on one or more feature vectors of entries associated with the document stored in database.
7 . The system of claim 1 , wherein the machine learning model comprises a neural network configured to:
output an extractive summary of vulnerability information inferred about the document based on entries associated with the document stored in database.
8 . The system of claim 7 , wherein the vulnerability assessment system is configured to:
use the extractive summary to simulate a vulnerability management scan of the document and generate a synthetic scan result.
9 . The system of claim 1 , wherein the NER system is configured to:
tokenize the document into a plurality of tokens;
scan the tokens to identify a triggering token of an entity type;
continue to scan the tokens to identify a stopping token of the entity type; and
generate a document chunk of the document, wherein the document chunk begins at a beginning of the document and ends immediately before the stopping token.
10 . The system of claim 9 , wherein the NER system is configured to:
store the document chunk as a file separate from the document;
remove the document chunk from the document; and
tag the file with a vulnerability ID identified for the document chunk.
11 . A method comprising:
performing, by one or more computer devices, a vulnerability assessment system for web content, including:
obtaining a plurality of documents that includes web content;
determining that a length of a document exceeds a threshold document length, and in response:
segmenting the document into multiple document chunks using a named-entity recognition (NER) system that segments documents based on vulnerability identifications (IDs); and
storing the document chunks in a database, wherein the document chunks are indexed in the database so that each entry in the database corresponds to a respective document chunk and a respective vulnerability ID; and
executing a machine learning model to assess the document for one or more security vulnerabilities based on the document chunks stored in the database.
12 . The method of claim 11 , further comprising:
in response to determining that a second document does not exceed the threshold document length, storing the second document as a discrete document chunk without segmenting the second document.
13 . The method of claim 11 , further comprising:
receiving the plurality of documents from a plurality of web crawlers.
14 . The method of claim 11 , wherein the NER system comprises a transformer machine learning model configured to identify one or more predefined entity types in the document associated with vulnerability information.
15 . The method of claim 14 , wherein the one or more predefined entity types includes at least one of: a product name, an exploitation technique, or a common vulnerabilities and exposures identifier (CVE-ID).
16 . The method of claim 11 , wherein:
the machine learning model comprises a neural network; and
the method comprises inferring, using the machine learning model, vulnerability information about the document based on one or more feature vectors of entries associated with the document stored in database.
17 . The method of claim 11 , wherein:
the machine learning model comprises a neural network; and
the method comprises outputting, using the machine learning model, an extractive summary of vulnerability information inferred about the document based on entries associated with the document stored in database.
18 . The method of claim 17 , further comprising:
using the extractive summary to simulate a vulnerability management scan of the document and generate a synthetic scan result.
19 . The method of claim 11 , further comprising the NER system:
tokenizing the document into a plurality of tokens;
scanning the tokens to identify a triggering token of an entity type;
scanning the tokens to identify a stopping token of the entity type; and
generating a document chunk of the document, wherein the document chunk begins at a beginning of the document and ends immediately before the stopping token.
20 . The method of claim 19 , further comprising the NER system:
storing the document chunk as a file separate from the document;
removing the document chunk from the document; and
tagging the file with a vulnerability ID identified for the document chunk.