Processing of web content for vulnerability assessments

1. A system, comprising:

one or more hardware processors with associated memory that implement processing of web content for vulnerability assessments, wherein the one or more hardware processors are configured to:

obtain, from multiple different web sources, a plurality of documents comprising web content;

parse the plurality of documents to determine a set of discrete document chunks, wherein, to parse the plurality of documents, the one or more hardware processors are configured to:

determine whether a given document of the plurality of documents satisfies at least one segmentation condition;

responsive to a determination that the given document satisfies the at least one segmentation condition, segment the given document into multiple discrete document chunks using a named-entity recognition (NER) system configured to segment the given document based at least in part on a vulnerability identification (ID); and

responsive to a determination that the given document does not satisfy the at least one segmentation condition, identify the given document as a discrete document chunk without segmenting the given document; and

store the set of discrete document chunks in a database, wherein vulnerability information contained within the set of discrete document chunks is indexed such that each respective entry in the database corresponds to a respective vulnerability ID and a respective discrete document chunk of the set of discrete document chunks.

2. The system of claim 1 , wherein, to determine whether the given document satisfies the least one segmentation condition, the one or more hardware processors are configured to:

determine whether a length of the given document satisfies a threshold document length; and

responsive to a determination that the length of the given document satisfies the threshold document length, the one or more hardware processors are configured to determine that the given document satisfies the at least one segmentation condition based on the given document being too long.

3. The system of claim 1 , wherein the NER system comprises a transformer-based NER system configured to identify one or more predefined entity types included in documents containing vulnerability information, and wherein the one or more predefined entity types comprise a vulnerability identification ID.

4. The system of claim 3 , wherein the one or more predefined entity types further comprise at least one of: a product name; or an affected product.

5. The system of claim 4 , wherein, to segment the given document into multiple discrete document chunks, the one or more hardware processors are configured to:

implement a NER segmentation scheme, wherein, to implement the NER segmentation scheme, the one or more hardware processors are configured to:

scan the given document until a triggering named entity is identified, wherein the triggering named entity comprises at least one of the predefined entity types;

responsive to the triggering named entity being identified:

scan the given document until at least one other entity, of the predefined entities, is identified, wherein the least one other entity is of a different entity type than the triggering named entity;

continue to scan the given document until a stopping entity is identified, wherein the stopping entity is of the same entity type as the triggering entity;

separate a portion of the web content from the given document to produce a discrete document chunk, wherein the portion of the web content begins at the beginning of the given document and ends at the character immediately preceding the stopping entity, and wherein, to separate the portion of the web content, the one or more hardware processors are configured to:

place the portion of the web content into a separate file so that it is segmented from the remaining web content of the given document; and

remove the portion of the web content from the given document; and

tag the separate file with an identified vulnerability ID contained within the separate file.

6. The system of claim 5 , wherein, to segment the given document into multiple discreet document chunks, the one or more hardware processors are further configured to:

continue to segment the given document in accordance with the NER segmentation scheme, until the end of the given document is reached, so as to produce one or more additional discrete document chunks.

7. The system of claim 1 , wherein, to obtain the plurality of documents comprising web content, the one or more hardware processors are configured to:

use web crawlers to collect the plurality of documents from the multiple different web sources; and

wherein the plurality of documents comprise HyperText Markup Language (HTML) pages.

8. The system of claim 1 , wherein the one or more hardware processors are further configured to:

input, into a multi-headed neural network, at least a portion of the vulnerability information from the database, wherein:

a first head of the multi-headed neural network is configured to output a feature vector comprising features extracted from the portion of the vulnerability information, wherein the portion of the vulnerability information relates to a security vulnerability; and

a second head of the multi-headed neural network is configured to output an extractive summary of the portion of the vulnerability information.

9. A method comprising:

implementing, using one or more hardware processors, processing of web content for vulnerability assessments, wherein the implementing comprises:

obtaining, from multiple different web sources, a plurality of documents comprising web content;

parsing the plurality of documents to determine a set of discrete document chunks, wherein parsing the plurality of documents comprises:

determining whether a given document of the plurality of documents satisfies at least one segmentation condition;

responsive to a determination that the given document satisfies the at least one segmentation condition, segmenting the given document into multiple discrete document chunks using a named-entity recognition (NER) system configured to segment the given document based at least in part on a vulnerability identification (ID); and

responsive to a determination that the given document does not satisfy the at least one segmentation condition, identifying the given document as a discrete document chunk without segmenting the given document; and

storing the set of discrete document chunks in a database, wherein vulnerability information contained within the set of discrete document chunks is indexed such that each respective entry in the database corresponds to a respective vulnerability ID and a respective discrete document chunk of the set of discrete document chunks.

10. The method of claim 9 , wherein determining whether the given document satisfies the least one segmentation condition comprises:

determining whether a length of the given document satisfies a threshold document length; and

responsive to a determination that the length of the given document satisfies the threshold document length, determining that the given document satisfies the at least one segmentation condition based on the given document being too long.

11. The method of claim 9 , wherein the NER system comprises a transformer-based NER system configured to identify one or more predefined entity types included in documents containing vulnerability information, and wherein the one or more predefined entity types comprise a vulnerability identification ID.

12. The method of claim 11 , wherein the one or more predefined entity types further comprise at least one of: a product name; or an affected product.

13. The method of claim 9 , wherein obtaining the plurality of documents comprising web content comprises:

using web crawlers to collect the plurality of documents from the multiple different web sources; and

wherein the plurality of documents comprise HyperText Markup Language (HTML) pages.

14. The method of claim 9 , wherein the implementing further comprises:

inputting, into a multi-headed neural network, at least a portion of the vulnerability information from the database, wherein:

a first head of the multi-headed neural network is configured to output a feature vector comprising features extracted from the portion of the vulnerability information, wherein the portion of the vulnerability information relates to a security vulnerability; and

a second head of the multi-headed neural network is configured to output an extractive summary of the portion of the vulnerability information.

15. One or more non-transitory computer-accessible storage media storing program instructions that, when executed on or across one or more hardware processors, implement at least a portion of a system that implements processing of web content for vulnerability assessments and cause the system to:

obtain, from multiple different web sources, a plurality of documents comprising web content;

parse the plurality of documents to determine a set of discrete document chunks, wherein, to parse the plurality of documents, the one or more hardware processors are configured to:

determine whether a given document of the plurality of documents satisfies at least one segmentation condition;

responsive to a determination that the given document satisfies the at least one segmentation condition, segment the given document into multiple discrete document chunks using a named-entity recognition (NER) system configured to segment the given document based at least in part on a vulnerability identification (ID); and

responsive to a determination that the given document does not satisfy the at least one segmentation condition, identify the given document as a discrete document chunk without segmenting the given document; and

store the set of discrete document chunks in a database, wherein vulnerability information contained within the set of discrete document chunks is indexed such that each respective entry in the database corresponds to a respective vulnerability ID and a respective discrete document chunk of the set of discrete document chunks.

16. The one or more non-transitory computer-accessible storage media of claim 15 , wherein, to determine whether the given document satisfies the least one segmentation condition, the program instructions when executed on or across the one or more hardware processors cause the system to:

determine whether a length of the given document satisfies a threshold document length; and

responsive to a determination that the length of the given document satisfies the threshold document length, the one or more hardware processors are configured to determine that the given document satisfies the at least one segmentation condition based on the given document being too long.

17. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the NER system comprises a transformer-based NER system configured to identify one or more predefined entity types included in documents containing vulnerability information, and wherein the one or more predefined entity types comprise a vulnerability identification ID.

18. The one or more non-transitory computer-accessible storage media of claim 17 , wherein the one or more predefined entity types further comprise at least one of: a product name; or an affected product.

19. The one or more non-transitory computer-accessible storage media of claim 15 , wherein, to obtain the plurality of documents comprising web content, the program instructions when executed on or across the one or more hardware processors cause the system to:

use web crawlers to collect the plurality of documents from the multiple different web sources; and

wherein the plurality of documents comprise HyperText Markup Language (HTML) pages.

20. The one or more non-transitory computer-accessible storage media of claim 15 , wherein the program instructions when executed on or across the one or more hardware processors further cause the system to:

input, into a multi-headed neural network, at least a portion of the vulnerability information from the database, wherein:

a first head of the multi-headed neural network is configured to output a feature vector comprising features extracted from the portion of the vulnerability information, wherein the portion of the vulnerability information relates to a security vulnerability; and

a second head of the multi-headed neural network is configured to output an extractive summary of the portion of the vulnerability information.