Dynamic monitoring of ability to reassemble streaming data across multiple channels based on history
View Patent ↗Mechanisms are provided for processing streaming data at high sustained data rates. These mechanisms receive a plurality of data elements over a plurality of non-sequential communication channels and write the plurality of data elements directly to the file system of the data processing system in an unassembled manner. The mechanisms determining whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner. The mechanisms perform a data scrubbing operation, in response to a determination to perform data scrubbing, to identify any missing data elements in the plurality of data elements written to the file system and assemble the plurality of data elements into a plurality of data streams in response to results of the data scrubbing indicating that there are no missing data elements.
1. A computer program product comprising a computer recordable storage medium having a computer readable program recorded thereon, wherein the computer readable program, when executed on a computing device, causes the computing device to:
receive a plurality of data elements, in the computing device, over a plurality of non-sequential communication channels;
write the plurality of data elements directly to a file system of the computing device in an unassembled manner;
determine whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner;
perform a data scrubbing operation, in response to a determination to perform data scrubbing, to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system; and
assemble the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements.
2. The computer program product of claim 1 , wherein the data scrubbing operation is performed in response to a determination that the data elements in the plurality of data elements are not being received in a substantially sequential manner.
3. The computer program product of claim 2 , wherein the computer readable program further causes the computing device to:
inhibit performance of the data scrubbing operation in response to a determination that the data elements in the plurality of data elements are being received in a substantially sequential manner.
4. The computer program product of claim 3 , wherein the computer readable program further causes the computing device to:
set a flag to a first value indicating that data scrubbing is to be performed in response to a determination that the data elements in the plurality of data elements are not being received in a substantially sequential manner; and
set the flag to a second value indicating that data scrubbing is not to be performed in response to a determination that the data elements in the plurality of data elements are being received in a substantially sequential manner, wherein if the flag is set, data scrubbing is performed and if the flag is not set, data scrubbing is not performed.
5. The computer program product of claim 1 , wherein the history information comprises data indicating an identifier of a last received data element.
6. The computer program product of claim 2 , wherein the computer readable program further causes the computing device to determine whether to perform data scrubbing or not by determining if the plurality of data elements are being received in a substantially sequential manner based on whether a current data element is within a predetermined range of data elements from a last received data element identified by the history information.
7. An apparatus, comprising:
a processor; and
a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to:
receive a plurality of data elements, in the apparatus, over a plurality of non-sequential communication channels;
write the plurality of data elements directly to a file system of the apparatus in an unassembled manner;
determine whether to perform a data scrubbing operation or not based on history information indicative of whether data elements in the plurality of data elements are being received in a substantially sequential manner;
perform a data scrubbing operation, in response to a determination to perform data scrubbing, to determine if there are any missing data elements that are not present in the plurality of data elements written to the file system; and
assemble the plurality of data elements into a plurality of data streams associated with the plurality of non-sequential communication channels in response to results of the data scrubbing indicating that there are no missing data elements.