Threat detection system that reformats log records for NLP model-based threat detection packages

1. A method comprising:

performing, by a threat detection system implemented by one or more computing devices:

receiving a batch of log records indicating operations of devices in a network, wherein individual log records in the batch are formatted in a first format that indicates annotations of individual items in the log records;

using a conversion rule set associated with a first natural language processing (NLP) package, converting the batch of log records to a training dataset of records, wherein individual records in the training dataset are formatted in a second format that indicates positions and lengths of individual items in the records;

using the training dataset, training a NLP model to identify respective types of items in records of the second format and relationships between the items in the records;

after the NLP model is trained:

using the conversion rule set and the NLP model to monitor additional log records in the first format; and

based on the monitoring, detecting an anomalous or malicious activity in the network; and

after initiation of the monitoring of the additional log records using the conversion rule set and the NLP model:

deploying another NLP package that requires training records in a third format;

storing another conversion rule set associated with the other NLP package; and

using the other conversion rule set to convert further log records into the third format.

2. The method of claim 1 , further comprising:

generating a user alert in response to the detecting of the anomalous or malicious activity in the network.

3. The method of claim 1 , further comprising:

based on the monitoring, extracting and storing items and relationships identified in the additional log records; and

using the extracted items and relationships to search the additional log records.

4. The method of claim 1 , wherein

the NPL package requires training records in the second format.

5. The method of claim 1 , wherein NLP model is trained to semantically map an individual log record into log values analyzed by the NLP package.

6. The method of claim 1 , wherein the third format indicates one or more punctuation marks or characters used to delineate individual items in training records of the third format.

7. The method of claim 1 , wherein the NLP model is a convolutional neural network.

8. The method of claim 1 , wherein the individual log records in the batch indicate actions observed in the network comprising:

a timestamp of the action,

a device associated with the action, and

a user associated with the action.

9. The method of claim 8 , wherein a particular one of the log records indicates a connection from a source device to a destination device, comprising:

a source address of the source device,

a destination address of the destination device,

a duration of the connection, and

a number of bytes transferred over the connection.

10. The method of claim 1 , the NLP model training system is configured to receive log records from a plurality of different sources and periodically convert the received log records into training data using different conversion rule sets.

11. A system comprising:

a threat detection system implemented by one or more computing devices, configured to:

receive a batch of log records indicating operations of devices in a network, wherein individual log records in the batch are formatted in a first format that indicates annotations of individual items in the log records;

use a conversion rule set associated with a first natural language processing (NLP) package to convert the batch of log records to a training dataset of records, wherein individual records in the training dataset are formatted in a second format that indicates positions and lengths of individual items in the records;

use the training dataset to train a NLP model to identify respective types of items in additional records of the second format and relationships between items in the additional records;

after the NLP model is trained:

use the conversion rule set and the NLP model to monitor additional log records in the first format; and

based on the monitoring, detect an anomalous or malicious activity in the network; and

after initiation of the monitoring of the additional log records using the conversion rule set and the NLP model:

deploy another NLP package that requires training records in a third format;

store another conversion rule set associated with the other NLP package; and

use the other conversion rule set to convert further log records into the third format.

12. The system of claim 11 , wherein the threat detection system is configured to:

generate a user alert in response to the detecting of the anomalous or malicious activity in the network.

13. The system of claim 11 , wherein the threat detection system includes a log searching system implemented by one or more computing devices, configured to:

use the conversion rule set and the NLP model to extract and store items and relationships identified in additional log records; and

use the extracted items and relationships to search the additional log records.

14. The system of claim 11 , wherein

the NPL package requires training records in the second format.

15. The system of claim 11 , wherein NLP model is trained to semantically map an individual log record into log values analyzed by the NLP package.

16. The system of claim 11 , wherein the third format indicates one or more punctuation marks or characters used to delineate individual items in training records of the third format.

17. The system of claim 11 , wherein the NLP model is a convolutional neural network.

18. The system of claim 11 , wherein the NLP model is trained to output a probabilistic assessment of an item type or relationship identified in a particular input record.

19. The system of claim 11 , wherein the NLP model training system is configured to receive log records from a plurality of different sources over a wide area network.

20. One or more non-transitory computer readable media storing program instructions that when executed on one or more processors cause the one or more processors to implement a threat detection system and to:

receive a batch of log records indicating operations of devices in a network, wherein individual log records in the batch are formatted in a first format that indicates annotations of individual items in the log records;

use a conversion rule set associated with a first natural language processing (NLP) package to convert the batch of log records to a training dataset of records, wherein individual records in the training dataset are formatted in a second format that indicates positions and lengths of individual items in the records;

use the training dataset to train a NLP model to identify respective types of items in additional records of the second format and relationships between items in the additional records;

after the NLP model is trained:

use the conversion rule set and the NLP model to monitor additional log records in the first format; and

based on the monitoring, detect an anomalous or malicious activity in the network; and

after initiation of the monitoring of the additional log records using the conversion rule set and the NLP model:

deploy another NLP package that requires training records in a third format;

store another conversion rule set associated with the other NLP package; and

use the other conversion rule set to convert further log records into the third format.