Method and system for detecting gaps in a data stream
View Patent ↗Systems and methods are described which allow the detection of gaps in a set of data. These systems and methods may include defining streams of data from a network topology, associating incoming data with one or more of these streams, and processing these streams. A gap may be detected by comparing the times of events in the stream. If a gap is detected remedial action may be taken, and processing of the streams temporarily halted. Processing of the streams may continue when data for a certain stream is received, or after the lapse of a certain period of time.
1. A method for detecting gaps in data, comprising:
defining at a computer at least a first stream associated with a particular user's activities and a second stream associated with that user's activities from a network topology for presenting a logical website, wherein each of the streams is a logical data source associated with one or more servers, wherein each server has hosts, data locations, or a combination thereof associated with the server, and wherein each server is responsible for running a different portion of the logical website, wherein the computer comprises a processor and a computer readable medium storing a computer program executable to detect gaps in data;
receiving incoming data from one or more sources at the computer and associating the incoming data with one of the streams based on a source of the incoming data, wherein the source is one of the one or more servers or one of the hosts or data locations associated therewith and the incoming data comprises data regarding previous activities at one of the one or more servers, hosts, or data locations;
calculating a data loss for each stream, wherein the data loss is calculated between a next event (nextEvent.time) and a last event (lastTime) in the stream, wherein the computer calculates the data loss and wherein the data loss is a time difference between the occurrence of the next event and the last event;
stopping the processing of every stream if the first or second stream's calculated data loss is greater than a first user defined threshold;
resuming processing according to a second user defined threshold;
determining whether each stream has a gap based upon the calculated data loss and a user defined threshold (GAP_TIME), wherein a gap is detected if (nextEvent.time>=(lastTime+GAP_TIME)), wherein the computer determines whether each stream has a gap.
2. The method of claim 1 , further comprising stopping the processing of every stream if the first or second stream's calculated time difference is greater than a first time period.
3. The method of claim 2 , further comprising resuming processing of the first or second stream upon reception of more data associated with the first or second stream.
4. The method of claim 3 , further comprising resuming the processing of each stream in which the calculated time difference is not greater than the first time period.
5. A tangible non-transitory computer readable medium containing computer program instructions translatable for:
defining at least a first stream associated with a particular user's activities and a second stream associated with that user's activities from a network topology for presenting a logical website, wherein each of the streams is a logical data source associated with one or more servers, wherein each server has hosts, data locations, or a combination thereof associated with the server, and wherein each server is responsible for running a different portion of the logical website;
associating incoming data with one of the streams based on a source of the incoming data, wherein the source is one of the one or more servers or one of the hosts or data locations associated therewith and the incoming data comprises data regarding previous activities at one of the one or more servers, hosts, or data locations;
calculating a data loss for each stream, wherein the data loss is calculated between a next event (nextEvent.time) and a last event (lastTime) in the stream, wherein the data loss is a time difference between the occurrence of the next event and the last event;
stopping the processing of every stream if the first or second stream's calculated data loss is greater than a first user defined threshold;
resuming processing according to a second user defined threshold;
determining whether each stream has a gap based upon the calculated data loss and a user defined threshold (GAP_TIME), wherein a gap is detected if (nextEvent.time>=(lastTime+GAP_TIME)).
6. The computer readable medium of claim 5 , wherein the instructions are further translatable for stopping the processing of every stream if the first or second stream's calculated time difference is greater than a first time period.
7. The computer readable medium of claim 6 , wherein the instructions are further translatable for resuming processing of the first or second stream upon reception of more data associated with the first or second stream.
8. The computer readable medium of claim 6 , wherein the instructions are further translatable for resuming the processing of each stream in which the calculated time difference is not greater than the first time period.
9. The computer readable medium of claim 6 , wherein the instructions are further translatable for resuming processing after a second period of time.
10. A system for processing a real-time data feed, compromising:
a computer configured with definitions of at least a first stream associated with a particular user's activities and a second stream associated with that user's activities from a network topology for presenting a logical website, wherein each of the streams is a logical data source associated with one or more servers, wherein each server has hosts, data locations, or a combination thereof associated with the server, and wherein each server is responsible for running a different portion of the logical website, the computer comprising a processor and a tangible storage medium storing instructions executable by the processor to implement a method at the computer comprising:
receiving incoming data at the computer;
associating incoming data received over the network with one of the streams based on a source of the incoming data, wherein the source is one of the one or more servers or one of the hosts or data locations associated therewith and the incoming data comprises data regarding previous activities at one of the one or more servers, hosts, or data locations, wherein the associating is performed at the computer;
calculating a data loss for each stream, wherein the data loss is calculated between a next event (nextEvent.time) and a last event (lastTime) in the stream, wherein the calculating is performed at the computer, wherein the data loss is a time difference between the occurrence of the next event and the last event;
stopping the processing of every stream if the first or second stream's calculated data loss is greater than a first user defined threshold;
resuming the processing according to a second user defined threshold;
determining whether each stream has a gap based upon the calculated data loss and a user defined threshold (GAP_TIME), wherein a gap is detected if (nextEvent.time>=(lastTime GAP_TIME)), wherein the determining is performed at the computer.
11. The system of claim 10 , further comprising stopping the processing of every stream if the first or second stream's calculated time difference is greater than a first time period.
12. The system of claim 10 , further comprising resuming processing of the first or second stream upon reception of more data associated with the first or second stream.
13. The system of claim 10 , further comprising resuming the processing of each stream in which the calculated time difference is not greater than the first time period.
14. The system of claim 10 , wherein the processing resumes after a second period of time.