Detection of network security breaches based on analysis of network record logs

1. A method, implemented in a first network device, of inspecting logs of security records in a computer network, the method comprising:

receiving security log records from a plurality of network security devices, at the first network device;

processing the log records, including deriving keys to a table, wherein individual keys of the table are tagged with a time stamp;

determining data values from information in the log records and adding a data value including a tag field to a list of data values associated with a key if the data value is not in the list of data values associated with the key, wherein the time stamp and the tag field differ and the tag field indicates that the key has been modified by the addition of the data value since a prior evaluation;

retrieving entries of the table not having the tag field;

retrieving entries of the table having the tag field;

evaluating only those entries of the table having the tag field based on predetermined criteria to detect attempted security breaches; and

resetting the tag field upon the evaluating to indicate that the key has been evaluated since a prior modification and updating the time stamp.

2. The method of claim 1 , wherein the table is a hash table.

3. The method of claim 1 , wherein the list of data values is implemented as a linked list.

4. The method of claim 1 , wherein the list of data values is implemented as a hash table.

5. The method of claim 1 , wherein the list of data values is implemented as a tree.

6. The method of claim 1 , wherein evaluating one or more entries of the table includes evaluating all the entries of the table.

7. A method of inspecting logs of security records in a computer network, comprising:

retrieving log records from a plurality of network security devices;

hashing, for each of the log records, one or more fields of the log record to generate a hash key;

evaluating a hash table using the hash key;

if there is no matching hash table entry, adding a new entry to the hash table;

if there is a matching hash table entry, retrieving a data list associated with the hash table entry;

using, for each of the log records, one of more fields of the log record to compute a data value;

comparing the data value with entries in the data list to determine if there are any matching entries;

inserting the data value into the data list if there are no matching entries, wherein the data value includes a tag field and a time stamp that differ, wherein the tag field indicates that the hash key has been modified by the insertion of the data value since a prior evaluation;

retrieving entries of the hash table which do not have the tag field;

retrieving entries of the hash table which do have the tag field;

evaluating the data value based on predetermined criteria to detect attempted security breaches;

resetting the tag field to indicate that the hash key has been evaluated since a prior modification; and

updating the time stamp.

8. The method of claim 7 , wherein adding a new entry to the hash table includes generating an empty data list associated with the new entry to the hash table.

9. The method of claim 7 , wherein inserting a new entry in the data list includes triggering an evaluation of the data list.

10. The method of claim 7 , including receiving a check table operation to trigger the evaluation of the data list.

11. The method of claim 7 , wherein evaluating the data list based on predetermined criteria results in blocking a packet associated with one of the log records.

12. The method of claim 7 , wherein evaluating the data list based on predetermined criteria results in blocking all future packets from a same source as a packet associated with a given log record.

13. The method of claim 7 , wherein evaluating the data list based on predetermined criteria includes reporting an attempted security breach.

14. The method of claim 7 , wherein the data list is a linked list.

15. The method of claim 7 , wherein the data list is a hash table.

16. The method of claim 7 , wherein the data list is a tree.

17. The method of claim 7 , including evaluating the data list after a plurality of log records have been added to the data list.

18. The method of claim 7 , including evaluating the data list after each of the log records is added to the data list.

19. The method of claim 7 , wherein evaluating the hash table using the hash key includes processing a second hash table.

20. The method of claim 19 , wherein processing a second hash table includes:

using the matching hash table entry to retrieve a second hash table;

using the hash key to evaluate the second hash table;

if there is no matching second hash table entry, adding a new entry to the second hash table;

if there is a matching second hash table entry, retrieving a second data list associated with the second hash table entry;

comparing the data value with entries in the second data list to determine if there are any matching entries;

inserting the data value in the second data list if there are no matching entries; and

evaluating the second data list based on predetermined criteria to detect attempted security breaches.

21. A method of inspecting a log of security records in a computer network, comprising:

retrieving log records from a plurality of network security devices;

hashing, for each of the log records, one or more fields of the log record to generate a hash key;

evaluating a hash table using the hash key;

if there is no matching hash table entry, adding a new entry to the hash table;

if there is a matching hash table entry, retrieving a data list associated with the table entry;

using, for each of the tog records, one or more fields of the log record to compute a data value to be inserted into the data list;

evaluating the data list to determine whether the data value is included in the data list; and

inserting the data value in the data list when the data value is not included in the data list and tagging the inserted data value with a time stamp associated with a predetermined time of expiration and with a tag field indicating that the data list has been modified by the insertion of the data value since a prior evaluation.

22. A computer program product, tangibly embodied in a machine-readable storage medium, the computer program product comprising instructions operable to cause a data processing apparatus in a first network device to:

receive log records from a plurality of network security devices, at the first network device;

process the log records, including deriving keys to a table;

determine data values from information in the log records and adding a data value including a time stamp and a tag field to a list of data values associated with a key if the data value is not in the list of data values associated with the key, wherein the time stamp and the tag field differ and the tag field indicates that the key has been modified by the addition of the data value since a prior evaluation;

retrieving entries of the table not having the tag field;

retrieving entries of the table having the tag field;

evaluate only those entries of the table having respective tag fields based on predetermined criteria to detect attempted security breaches; and

reset the tag fields to indicate that the keys have been evaluated since a prior modification; and

update the time stamps of the evaluated entries.

23. The computer program product of claim 22 , wherein the table is a hash table.

24. The computer program product of claim 22 , wherein the list of data values is implemented as a linked list.

25. The computer program product of claim 22 , wherein the list of data values is implemented as a hash table.

26. The computer program product of claim 22 , wherein the list of data values is implemented as a tree.

27. The computer program product of claim 22 , wherein instructions to evaluate one or more entries of the table include instructions to evaluate all the entries of the table.

28. A computer program product, tangibly embodied in a machine-readable storage medium, the computer program product comprising instructions operable to cause a data processing apparatus to:

retrieve log records from a plurality of network devices;

hash, for each of the log records, one or more fields of the log record to generate a hash key;

evaluate a hash table using the hash key;

if there is no matching hash table entry, add a new entry to the hash table;

if there is a matching hash table entry, retrieve a data list associated with the hash table entry;

use, for each of the log records, one or more fields of the log record to compute a data value;

compare the data value with entries in the data list to determine if there are any matching entries;

insert the data value and an associated tag field and a time stamp into the data list if there are no matching entries, wherein the time stamp and the tag field differ and the tag field indicates that the hash key has been modified by the insertion of the data value since a prior evaluation;

retrieve entries of the table that do not have the tag field;

retrieve entries of the table that have the tag field;

evaluate only those data values of the data list having the associated tag field based on predetermined criteria to detect attempted security breaches; and

reset the respective tag field to indicate that the hash key has been evaluated since a prior modification; and

update the time stamp of the evaluated data values.

29. The computer program product of claim 28 , wherein instructions for adding a new entry to the hash table cause the data processing apparatus to generate an empty data list associated with the new entry to the hash table.

30. The computer program product of claim 28 , wherein instructions for inserting a new entry in the data list cause the data processing apparatus to trigger the evaluation of the data list.

31. The computer program product of claim 28 , wherein instructions for issuing a check table operation cause the data processing apparatus to trigger the evaluation of the data list.

32. The computer program product of claim 28 , wherein instructions for evaluating the data list based on predetermined criteria cause the data processing apparatus to block a packet associated with one of the log records.

33. The computer program product of claim 28 , wherein instructions for evaluating the data list based on predetermined criteria cause the data processing apparatus to block all future packets from a same source as a packet associated with a given log record.

34. The computer program product of claim 28 , wherein instructions for evaluating the data list based on predetermined criteria cause the data processing apparatus to report an attempted security breach.

35. The computer program product of claim 28 , wherein the data list is a linked list.

36. The computer program product of claim 28 , wherein the data list is a hash table.

37. The computer program product of claim 28 , wherein the data list is a tree.

38. The computer program product of claim 28 , wherein instructions for evaluating the data list cause the data processing apparatus to evaluate the data list after a plurality of log records have been added to the data list.

39. The computer program product of claim 28 , wherein instructions for evaluating the data list cause the data processing apparatus to evaluate the data list after each log record is added to the data list.

40. The computer program product of claim 28 , wherein instructions for evaluating the hash table using the hash key cause the data processing apparatus to process a second hash table.

41. The computer program product of claim 40 , wherein instructions for processing a second hash table cause the data processing apparatus to:

use the matching hash table entry to retrieve a second hash table;

use the hash key to evaluate the second hash table;

if there is no matching second hash table entry, add a new entry to the second hash table;

if there is a matching second hash table entry, retrieve a second data list associated with the second hash table entry;

compare the data value with entries in the second data list to determine if there are any matching entries;

insert the data value in the second data list if there are no matching entries; and

evaluate the second data list based on predetermined criteria to detect attempted security breaches.

42. A computerprogram product, tangibly embodied in a machine-readable storage medium, for inspecting a log of security records in a computer network, the computer program product comprising instructions operable to cause a data processing apparatus to:

retrieve log records from a plurality of network security devices;

hash, for each of the log records, one or more fields of the log record to generate a hash key;

evaluate a hash table using the hash key;

if there is no matching hash table entry, add a new entry to the hash table;

if there is a matching hash table entry, retrieve a data list associated with the hash table entry;

use, for each of the log records, one or more fields of the log record to compute a data value to be inserted into the data list; and

evaluate the data list to determine whether the data value is included in the data list; and

insert the data value in the data list when the data value is not included in the data list, wherein the data value is tagged with a time stamp associated with a predetermined time of expiration and with a tap field to indicate that the data list has been modified by the insertion of the data value since a prior evaluation.