Using persistent memory to enable consistent data for batch processing and streaming processing

1. A system comprising:

one or more devices;

a processor coupled to the one or more devices; and

a non-volatile memory coupled to the processor and the one or more devices, wherein the non-volatile memory stores instructions that are configured to cause the processor to perform operations comprising:

receiving data from the one or more devices;

validating the data to yield validated data;

storing the validated data in a database on the non-volatile memory, the validated data being used for streaming processing and batch processing;

sending the validated data to a streaming processor that uses a streaming function to present a result of implementing the streaming function in real-time; and

sending the validated data to a remote disk to cause a batch processor to process the validated data and other validated data in the database obtained over a period of time.

2. The system of claim 1 , wherein the streaming processor is communicatively coupled to the processor, the streaming processor executing computer-readable instructions to perform operations comprising:

querying the database of the non-volatile memory for the validated data stored therein;

implementing the streaming function on the validated data received from the non-volatile memory; and

presenting the result of implementing the streaming function on the validated data.

3. The system of claim 1 , wherein the non-volatile memory is further configured to send the validated data to the remote disk to be stored therein upon triggering of an event.

4. The system of claim 3 , wherein the triggering event is a passage of the period of time or when a target amount of validated data is stored in the database of the non-volatile memory.

5. The system of claim 1 , wherein the one or more devices are configured to monitor and collect data in real time.

6. The system of claim 1 , wherein the non-volatile memory is configured to validate the data by performing a data failure check on the data received from the one or more devices.

7. The system of claim 1 , wherein the non-volatile memory is configured to validate the data by performing a data redundancy check on the data received from the one or more devices.

8. The system of claim 1 , wherein the non-volatile memory is in the same geographical location as the one or more devices.

9. A device comprising:

a memory having computer-readable instructions stored therein; and

one or more processors coupled to the memory and configured to execute the computer-readable instructions to perform functions including:

receiving data from one or more sensors;

validating the data to yield validated data;

storing the validated data in on a non-volatile memory associated with the device, the validated data being used for streaming processing and batch processing;

sending the validated data to a streaming processor that uses a streaming function to present a result of implementing the streaming function in real-time; and

sending the validated data to a remote disk to cause a batch processor to process the validated data and other validated data in the database obtained over a period of time.

10. The device of claim 9 , wherein the device is communicatively coupled to the one or more sensors.

11. The device of claim 9 , wherein

the validated data is sent to the streaming process in response to receiving a query from the streaming processor for the validated data stored on the non-volatile memory.

12. The device of claim 9 , wherein the device is further configured to send the validated data to the remote disk to be stored therein upon triggering of an event.

13. The device of claim 12 , wherein the triggering event is a passage of the period of time or when a target amount of validated data is stored in the database of the device.

14. The device of claim 9 , wherein the one or more processors are configured execute the computer-readable instructions to validate the data by performing a data failure check on the data received from the one or more sensors.

15. The device of claim 9 , wherein the one or more processors are configured to execute the computer-readable instructions to validate the data by performing a data redundancy check on the data received from the one or more sensors.

16. The device of claim 9 , wherein the device is in the same geographical location as the one or more sensors.

17. A method comprising:

receiving data from one or more sensors;

validating the data to yield validated data;

storing the validated data in a database of a non-volatile memory, the validated data being used for streaming processing and batch processing; and

sending the validated data to a streaming processor that uses a streaming function to present a result of implementing the streaming function in real-time; and

sending the validated data to a remote disk to cause a batch processor to process the validated data and other validated data in the database obtained over a period of time.

18. The method of claim 17 ,

wherein the validated data is sent to the streaming processor in response to receiving a query from the streaming processor for the validated data stored in the database.

19. The method of claim 17 , further comprising sending the validated data to the remote disk to be stored therein upon triggering of an event, the triggering event being a passage of the period of time or when a target amount of validated data is stored in the database of the non-volatile memory.

20. The method of claim 17 , wherein validating the data is based on performing one of a data failure check or a data redundancy check on the data received from the one or more sensors.