Techniques and systems for storage and processing of operational data
A system stores data, such as sensor data or other operational data, on a plurality of storage volumes in a sequence so as to allow for interpolations or other approximations of the data using a subset of the storage volumes in response to a request for information regarding that data. For example, a plurality of devices connect to the system to provide operational data, which is then stored in a specified sequence on a specified set of volumes. In response to a request for operational information regarding some or all of the devices, the system reads at least one of the volumes, and approximates the values of the data over a specified period of time. In some embodiments, the data may be buffered prior to storage, and a jitter analyzer determines whether the incoming data is anomalous relative to a baseline, which may be determined using related data sets.
1. A computer-implemented method, comprising:
receiving, from a plurality of remote devices connected to the one or more computer systems, sensor data related to operation of the plurality of remote devices;
processing the received sensor data so as to apportion each subset of a plurality of subsets of the sensor data to a respective volume of a sequence of volumes that includes a first volume, a second volume, and a third volume, based at least in part on a time at which each of the subsets of the plurality is received by the one or more computer systems; and
in response to a request for the sensor data over a period of time:
retrieving one or more subsets of the sensor data from the sequence of volumes over the period of time by using information associated with times that each of the subsets of the plurality is received without retrieving the entirety of the sensor data over the period of time;
interpolating the retrieved one or more subsets of sensor data so as to generate interpolated sensor data over the period of time; and
providing, to a requestor associated with the request, the interpolated sensor data as an approximation of the sensor data to satisfy the request.
2. The computer-implemented method of claim 1 , further comprising, in response to the request for the sensor data:
retrieving a third subset of the sensor data from the second volume;
refining the interpolated sensor data by further interpolating the interpolated sensor data using both the retrieved data from the first volume and the second volume; and
providing, to the requestor, the refined interpolated sensor data.
3. The computer-implemented method of claim 1 , wherein the sensor data is stored in the sequence of volumes in a repeated fashion.
4. The computer-implemented method of claim 1 , wherein the plurality of devices are connected to the one or more computer systems via a network protocol selected from the group consisting of: Z-wave, Zigbee, Thread, Bluetooth, Insteon, 6LoWPAN, InternetProtocol, Highway Addressable Remote Transducer Protocol (HART), supervisory control and data acquisition (SCADA), fieldbus, and sneakernet.
5. A system, comprising:
memory storing instructions that, as a result of being executed by one or more processors of the system, cause the system to:
receive data from a device connected to the at least one computing device;
process the data to apportion, based on times at which subsets of the data are received, the subsets of the data to a respective volume of a sequenced plurality of volumes, such that at least one of the subsets of data is apportioned to at least one volume of the plurality of volumes to which a different subset of the data is already apportioned;
store the processed data on the sequenced plurality of volumes as apportioned;
in response to a request for the data over a time range, interpolate an approximation of the data over the time range from the data stored on a subset of volumes of the sequenced plurality of volumes using information from times at which subsets of the data are received, wherein the data from the subset of volumes are less than the entirety of the data stored on the sequenced plurality of volumes over the time range; and
provide the interpolated approximation of data over the time range to satisfy the request.
6. The system of claim 5 , wherein the device is a sensor.
7. The system of claim 5 , wherein the data includes operational data periodically captured by the device.
8. The system of claim 5 , wherein the device connects to the at least one computing device over the Internet.
9. The system of claim 5 , wherein the instructions, as a result of being executed by the one or more processors of the system, further cause the system to process the data by applying a redundancy code to the data in connection with storing the data.
10. The system of claim 5 , wherein the processed data is stored on volumes associated with identity shards.
11. The system of claim 5 , wherein the instructions, as a result of being executed by the one or more processors of the system, further cause the system to receive the data in response to a periodic request by the device to store the data via the one or more services.
12. The system of claim 5 , wherein the request for the data originates from an entity connected to the device.
13. A non-transitory computer-readable storage medium having stored thereon executable instructions that, as a result of being executed by one or more processors of a computer system, cause the computer system to at least:
cause storage of operational data associated with devices connected to the computer system by at least storing subsets of the operational data on respective volumes in a sequence of the volumes based at least in part on timing parameters associated with the operational data;
generate, in response to a request for information related to operation of the devices over a period of time, interpolated data from the operational data stored on a subset of the volumes using information from the timing parameters, the subset of the volumes is less than that of the sequenced volumes; and
provide the interpolated data as an approximation of the operational data over the period of time in response to the request.
14. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to cause encoding of the operational data using a redundancy code in connection with causing storage of the operational data.
15. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to cause storage of the operational data in response to receiving requests from the devices to store the operational data.
16. The non-transitory computer-readable storage medium of claim 13 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to refine the interpolated data by using the operational data stored on an additional subset of the volumes.
17. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to provide the refined interpolated data in connection with the request.
18. The non-transitory computer-readable storage medium of claim 16 , wherein the instructions further comprise instructions that, as a result of being executed by the one or more processors, cause the computer system to refine the interpolated data in connection with a second request.
19. The non-transitory computer-readable storage medium of claim 13 , wherein the devices are connected to the computer system over a network that uses a protocol selected from the group consisting of: Z-wave, Zigbee, Thread, Bluetooth, Insteon, 6LoWPAN, InternetProtocol, Highway Addressable Remote Transducer Protocol (HART), supervisory control and data acquisition (SCADA), fieldbus, and sneakernet.
20. The non-transitory computer-readable storage medium of claim 13 , wherein the operational data includes sensor data associated with the devices.