STREAM EXTRACTION ADMINISTRATION SYSTEM

1 . A computer network security threat monitoring method for a processor and a storage device including instructions configured to run on the processor, including:

continuously gathering machine-readable documents from one or more streams of third-party machine-readable documents from network sources selected based on predetermined selection criteria,

evaluating machine-readable documents gathered from the streams according to one or more productivity rejection criteria and rejecting machine-readable documents that meet the productivity rejection criteria, and

processing the gathered machine-readable documents to extract threat information about conditions on the network except if they meet the one or more productivity rejection criteria.

2 . The method of claim 1 further including evaluating machine-readable documents gathered from the streams according to one or more productivity acceptance criteria and wherein the processing the gathered machine-readable documents processes all of the gathered machine-readable documents that satisfy the acceptance criteria to extract threat information.

3 . The method of claim 1 further including queuing at least some of the documents that do not satisfy the rejection criteria for manual review.

4 . The method of claim 3 further including adjusting at least some of the rejection criteria based on results of the manual review.

5 . The method of claim 4 further including confirming at least some of the rejection criteria based on results of the manual review.

6 . The method of claim 1 wherein the evaluating documents according to rejection criteria applies a plurality of heuristics.

7 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies probabilistic heuristics.

8 . The method of claim 6 wherein the application of probabilistic heuristics includes calculating a distance between a feature representation of a stream of machine-readable documents and identified clusters previously learned by a probabilistic model.

9 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies deterministic heuristics.

10 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies a dispensability score to the documents.

11 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies a dispensability score to the documents based on URLs of the documents.

12 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies a dispensability score to the documents based on the presence in the URL of one or more weighted indicator expressions.

13 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies a dispensability score to the documents based on a ratio of valuable entities to total token value.

14 . The method of claim 6 wherein the evaluating documents according to rejection criteria applies a dispensability score to the documents based on a relationship between true content size and compressed size.

15 . The method of claim 6 wherein the application of heuristics includes utilizing one or more foundation models to reprocess items in a stream of machine readable documents.

16 . The method of claim 15 wherein the application of heuristics includes utilizing one or more Large Language Models (LLMs) to reprocess items in a stream of machine readable documents.

17 . The method of claim 15 wherein the application of heuristics includes utilizing one or more foundation models to gain more information about or summarize items in a stream of machine readable documents.

18 . The method of claim 1 wherein the continuously gathering and evaluating machine-readable documents operate on a plurality of formats including textual, image, video, and pdf formatted machine-readable documents.

19 . A computer network security threat monitoring system, including:

a document analyzer to extract and resolve named entities in an unpredictable data stream,

a dispensability assessor responsive to the document analyzer and operative to apply heuristics utilizing the named entities in the data stream and thereby determine the document's dispensability, and

a dispensability service responsive to the dispensability assessor to receive and store the documents that are determined to be dispensable by the dispensability assessor.

20 . The system of claim 19 further including a user interface responsive to the dispensability service and operative to allow a user to inspect the documents that are determined to be dispensable by the dispensability assessor.

21 . A computer network security threat monitoring system, including:

means for continuously gathering machine-readable documents from one or more streams of third-party machine-readable documents from network sources,

means for evaluating gathered machine-readable documents from the streams according to one or more productivity acceptance criteria and accepting machine-readable documents that meet the productivity acceptance criteria,

means for evaluating gathered machine-readable documents from the streams according to one or more productivity rejection criteria and rejecting machine-readable documents that meet the productivity rejection criteria, and

means for processing the gathered machine-readable documents that satisfy the acceptance criteria and do not satisfy the rejection criteria to extract threat information about conditions on the network.