High speed logging system
View Patent ↗A method of storing log entries of events from a plurality of network elements in a communication network, comprising the steps of: a) receiving log entries at a control processor of events from a plurality of different elements positioned, the log entries grouped into threads based on a common purpose; b) converting each log entry into a compact log record in a logging module, and c) storing the compact log records in a first memory buffer in random access memory (RAM) forming a first log file.
1. A method comprising:
receiving, by a device, log entries of events from a plurality of network elements,
the log entries being grouped by thread based on a particular purpose;
converting, by the device, each log entry into a log record of log records,
each log record comprising at least one of:
information associated with a record length of the log record,
information associated with a time stamp of data collection at a network element of the plurality of network elements,
information associated with a thread ID of a thread that logged each event of the events,
information associated with a category of each event,
information associated with a message ID code corresponding to a log message, or
information associated with a network element ID for identifying a respective network element associated with each event;
storing, by the device, a first plurality of the log records in a first memory buffer to form a log file;
allocating, by the device and based on an amount of data stored in the first memory buffer, a second memory buffer for a second plurality of the log records;
queuing, by the device and based on the amount of data stored in the first memory buffer, the first memory buffer to a background thread to write the first plurality of the log records to a disk file,
the background thread writing the first plurality of the log records to the disk file when one or more processing cycles are not needed by another thread that has a higher priority than the background thread;
associating, by the device and after the background thread writes the first plurality of the log records to the disk file, the first memory buffer with a free pool; and
storing, by the device and based on associating the first memory buffer with the free pool, a third plurality of the log records in the first memory buffer.
2. The method according to claim 1 , further comprising:
storing a mapping file of text message strings corresponding to a plurality of different message ID codes.
3. The method according to claim 2 ,
where the first plurality of the log records are compact log records, and
where the method includes:
converting, using the mapping file, the compact log records to full log entries,
the full log entries including message ID codes and log messages; and
providing, for display, the full log entries on a graphic user interface (GUI).
4. The method according to claim 3 , where the full log entries are provided, for display, on the GUI by time vs element ID for each thread.
5. The method according to claim 4 , further comprising:
providing a separate line on the GUI for errors.
6. The method according to claim 1 ,
where the first plurality of the log records are compact log records, and
where the method includes:
converting the compact log records to full log entries utilizing a remote computer processor and a mapping file,
the mapping file including message ID codes and log messages; and
providing, for display, the full log entries on a graphic user interface (GUI).
7. The method according to claim 6 , where the full log entries are provided, for display, on the GUI by time vs element ID for each thread.
8. The method according to claim 7 , further comprising:
providing a separate line on the GUI for errors.
9. The method according to claim 1 , where each network element, of the plurality of network elements, includes a test probe.
10. A device comprising:
a processor, implemented at least partially in hardware, to:
receive log entries of events from a plurality of network elements,
the log entries being grouped by thread based on a particular purpose;
convert each log entry into a log record of log records,
each log record comprising at least one of:
information associated with a record length of the log record,
information associated with a time stamp of data collection at a network element of the plurality of network elements,
information associated with a thread ID of a thread that logged each event of the events,
information associated with a category of each event,
information associated with a message ID code corresponding to a log message, or
information associated with a network element ID for identifying a respective network element associated with each event;
store a first plurality of the log records in a first memory buffer to form a log file;
allocate, based on an amount of data stored in the first memory buffer, a second memory buffer for a second plurality of the log records;
queue, based on the amount of data stored in the first memory buffer, the first memory buffer to a background thread to write the first plurality of the log records to a disk file,
the background thread writing the first plurality of the log records to the disk file when one or more processing cycles are not needed by another thread that has a higher priority than the background thread;
associate, after the background thread writes the first plurality of the log records to the disk file, the first memory buffer with a free pool; and
store, based on associating the first memory buffer with the free pool, a third plurality of the log records in the first memory buffer.
11. The device of claim 10 , where the processor is further to:
store a mapping file of text message strings corresponding to a plurality of different message ID codes.
12. The device of claim 11 ,
where the first plurality of the log records are compact log records, and
where the processor is further to:
convert, using the mapping file, the compact log records to full log entries,
the full log entries including message ID codes and log messages; and
provide, for display, the full log entries on a graphic user interface (GUI).
13. The device of claim 12 ,
where the full log entries are provided, for display, on the GUI by time vs element ID for each thread, and
where the processor is further to:
provide a line on the GUI for errors.
14. The device of claim 10 ,
where the first plurality of the log records are compact log records, and
where the processor is further to:
convert the compact log records to full log entries utilizing a remote computer processor and a mapping file,
the mapping file including message ID codes and log messages; and
provide, for display, the full log entries on a graphic user interface (GUI).
15. The device of claim 14 ,
where the full log entries are provided, for display, on the GUI by time vs element ID for each thread, and
where the processor is further to:
provide a line on the GUI for errors.
16. A system comprising:
a device to:
receive log entries of events from a plurality of network elements,
the log entries being grouped by thread based on a particular purpose;
convert each log entry into a log record of log records,
each log record comprising at least one of:
information associated with a record length of the log record,
information associated with a time stamp of data collection at a network element of the plurality of network elements,
information associated with a thread ID of a thread that logged each event of the events,
information associated with a category of each event,
information associated with a message ID code corresponding to a log message, or
information associated with a network element ID for identifying a respective network element associated with each event;
store a first plurality of the log records in a first memory buffer to form a log file;
determine, based on an amount of data stored in the first memory buffer, that a second plurality of the log records should be stored in a second memory buffer;
queue, based on the amount of data stored in the first memory buffer, the first memory buffer to a background thread to write the first plurality of the log records to a disk file,
the background thread writing the first plurality of the log records to the disk file when one or more processing cycles are not needed by another thread that has a higher priority than the background thread;
associate, after the background thread writes the first plurality of the log records to the disk file, the first memory buffer with a free pool; and
store, based on associating the first memory buffer with the free pool, a third plurality of the log records in the first memory buffer.
17. The system of claim 16 , where the device is further to:
store a mapping file of text message strings corresponding to a plurality of different message ID codes.
18. The system of claim 17 ,
where the first plurality of the log records are compact log records, and
where the device is further to:
convert, using the mapping file, the compact log records to full log entries,
the full log entries including message ID codes and log messages; and
provide, for display, the full log entries on a graphic user interface (GUI).
19. The system of claim 18 ,
where the full log entries are provided, for display, on the GUI by time vs element ID for each thread, and
where the device is further to:
provide a line on the GUI for errors.
20. The system of claim 16 ,
where the first plurality of the log records are compact log records, and
where the device is further to:
convert the compact log records to full log entries utilizing a remote computer processor and a mapping file,
the mapping file including message ID codes and log messages; and
provide, for display, the full log entries on a graphic user interface (GUI).