Automated security analytics platform with multi-level representation conversion for space efficiency and incremental persistence
Active memory for managing network telemetry information, or other types of information stored as objects, has objects partially-serialized to allow greater amounts of information to store in a memory of a given size with slightly increased retrieval times. Storing additional information in an active memory provides an overall increase in network security platform responsiveness by allowing a greater amount of information to be accessible from the active memory instead of archive.
1. A method for managing network information for interaction with a network security platform, the method comprising:
storing the network information as plural objects in random access memory in non-serialized form;
accessing at least some of the plural objects from the random access memory with the network security platform;
selectively converting less than all of the plural objects into a serialized form for storage in the random access memory;
maintaining the less than all of the plural objects having the serialized form in the random access memory for a predetermined time; and
accessing within the predetermined time at least one of the less than all of the plural objects that were converted into serialized form with the network security platform from the random access memory to detect a network security threat, the accessing performed at least in part by converting the at least some of the less than all of the plural objects from the serialized form to the non-serialized form.
2. The method of claim 1 wherein selectively converting further comprises:
analyzing one or more predetermined conditions; and
selecting one or more of the plural objects to convert to serialized form based upon the predetermined conditions.
3. The method of claim 2 wherein the predetermined conditions comprise a measure of the complexity of attribute values of the plural objects, and selecting comprises selecting objects to convert to serialized form that have a smaller measure of complexity relative to unselected objects.
4. The method of claim 2 wherein the predetermined conditions comprise the recency of access of the plural objects by the network security platform and selecting comprises selecting objects to convert to serialized form based at least in part on the recency of access by the network security platform relative to unselected objects.
5. The method of claim 2 wherein the predetermined conditions comprise the length of time of storage of the plural objects in the random access memory, and selecting comprises selecting objects to convert to serialized form that have a greater time of storage relative to unselected objects.
6. The method of claim 1 wherein the random access memory is a dynamic random access memory.
7. The method of claim 1 wherein selectively converting less than all of the plural objects into a serialized form in the active memory further comprises:
maintaining at least some of the plural objects with attribute values having fully-realized objects; and
converting at least some of the plural objects from attribute values having fully-realized objects to objects stored in serialized form without per-object overhead.
8. The method of claim 1 wherein selectively converting less than all of the plural objects into a serialized form in the active memory further comprises:
maintaining at least some of the plural objects with attribute values having fully-realized objects; and
converting at least some of the plural objects from attribute values having fully-realized objects to a fully serialized object with shared context to interpret the serialized representation.
9. The method of claim 6 wherein selectively converting less than all of the plural objects into a serialized form in the active memory further comprises:
maintaining at least some of the plural objects with attribute values having fully-realized objects; and
converting at least some of the plural objects from attributes having full-realized objects to plural fully serialized objects compressed into one or more pages.