Generating training data for machine learning models

1. A method for training a language processing model, the method comprising:

receiving at an interface a first log record in a first format, wherein the first log record includes annotations describing items in the first log record;

creating a second log record in a second format comprising data from the first log record utilizing the annotations in the first log record and a conversion rule set;

providing the second log record to a processor executing instructions stored on a memory to cause the processors to perform:

training a language processing model using the second log record, resulting in a trained model configured to identify items and respective types of the items in previously unseen log records and relationships therebetween, wherein the conversion rule set is associated with the language processing model;

processing a third log record using the trained model to identify items and respective types of the items in the third log record and relationships therebetween; and

detecting network activity associated with the third log record based on processing the third log record using the trained model.

2. The method of claim 1 wherein creating the second log record includes converting the first log record into a list of tuples required by the language processing model.

3. The method of claim 1 wherein the described items in the first log record include at least one of an IP address, a byte count, a port, and a user name.

4. The method of claim 1 wherein the language processing model is configured to output a probabilistic assessment regarding the respective types of the items identified in the third log record and the relationships therebetween.

5. The method of claim 1 wherein the language processing model is implemented as a convolutional neural network.

6. The method of claim 1 wherein the language processing model is configured to semantically map the third log record into desired log values.

7. The method of claim 1 wherein the trained model executes at least two different natural language processing packages that are each defined by a different conversion rule set.

8. The method of claim 1 further comprising providing the identified items and respective types of the items in the third log record and the relationships therebetween to a threat detection module for analysis to detect malicious activity.

9. The method of claim 1 further comprising providing the identified items and respective types of the items in the third log record and the relationships therebetween to a log searching tool configured to conduct searches on log records.

10. A system for training a language processing model, the system comprising:

an interface for receiving:

a first log record in a first format, wherein the first log record includes annotations describing items in the first log record;

a mapping module configured to execute a conversion rule set to create a second log record in a second format comprising data from the first log record utilizing the annotations in the first log record; and

a processor executing instructions stored on a memory to:

receive the second log record;

train a language processing model using the second log record, resulting in a trained model configured to identify items and respective types of the items in previously unseen log records and relationships therebetween, wherein the conversion rule set is associated with the language processing model;

process a third log record using the trained model to identify items and respective types of the items in the third log record and relationships therebetween; and

detect network activities associated with the third log record based on processing the third log record using the trained model.

11. The system of claim 10 wherein the second format includes a list of tuples required by the language processing model.

12. The system of claim 10 wherein the described items in the first log record include at least one of an IP address, a byte count, a port, and a user name.

13. The system of claim 10 wherein the language processing model is configured to output a probabilistic assessment regarding the respective types of the items identified in the third log record and the relationships therebetween.

14. The system of claim 10 wherein the language processing model is implemented as a convolutional neural network.

15. The system of claim 10 wherein the language processing model is configured to semantically map the third log record into desired log values.

16. The system of claim 10 wherein the trained model is configured to execute at least two different natural language processing packages that are each defined by a different conversion rule set.

17. The system of claim 10 wherein the processor is further configured to provide the identified items and respective types of the items in the third log record and the relationships therebetween to a threat detection module for analysis to detect malicious activity.

18. The system of claim 10 wherein the processor is further configured to provide the identified items and respective types of the items in the third log record to a log searching tool configured to conduct searches on log records.

19. A non-transitory computer readable medium containing computer-executable instructions for performing a method for training a language processing model, the method comprising:

receiving at an interface a first log record in a first format, wherein the first log record includes annotations describing items in the first log record;

creating a second log record in a second format comprising data from the first log record utilizing the annotations in the first log record and a conversion rule set; and

providing the second log record to a processor executing instructions stored on a memory to cause the processor to perform:

training a language processing model using the second log record, resulting in a trained model configured to identify items and respective types of the items in previously unseen a records and relationships therebetween, wherein the conversion rule set is associated with the language processing model;

processing a third log record using the trained model to identify items and respective types of the items in the third log record and relationships therebetween; and

detecting network activity associated with the third log record based on processing the third log record using the trained model.

20. The non-transitory computer readable medium of claim 19 , the method further comprising:

providing the identified items and respective types of the items in the third log record and the relationships therebetween to a threat detection module for analysis to detect malicious activity.