Storage architecture for heterogeneous multimedia data
Various embodiments for implementing a system and method for managing and querying semi-structured, heterogeneously-typed sensor data in a distributed environment are disclosed. Example embodiments include: enabling data communications between a plurality of compute nodes and a plurality of physical data storage devices via an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer; and receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request including an identification of a topic of a dataset, the topic including a metadata file, a data file, and an index file, the index file including at least one pointer into the data file.
1. A system comprising:
a plurality of compute nodes being in data communication with a data network;
a plurality of physical data storage devices being in communication with the data network;
sensor logic to capture sensor input from an autonomous vehicle; and
a data storage access system enabling communication of data between the plurality of compute nodes and the plurality of physical data storage devices via an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer, the data storage access system receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request including an identification of a topic of a dataset, the topic including a metadata file, a data file, and an index file, the index file including at least one pointer into the data file, the dataset being a heterogeneous multimedia time-series dataset including a plurality of dataset topics, each dataset topic corresponding to a data stream created by a sensor and captured by the sensor logic of the autonomous vehicle, at least two dataset topics of the plurality of dataset topics corresponding to different data streams having different data types.
2. The system of claim 1 further including a set of computer software-implemented routines configured to query and join data produced by different sensors.
3. The system of claim 1 further including a set of computer software-implemented routines configured to import data into the data storage access system.
4. The system of claim 1 further including a set of computer software-implemented routines configured to transfer datasets between geographically disparate regions.
5. The system of claim 1 wherein the server layer is configured to authenticate an originating compute node of the data request.
6. The system of claim 1 wherein the server layer returns data to an originating compute node of the data request via Hypertext Transfer Protocol (HTTP).
7. The system of claim 1 wherein the storage layer is configured to obtain data corresponding to the topic from at least one of the plurality of physical data storage devices.
8. The system of claim 1 wherein the plurality of physical data storage devices is of a type from the group consisting of a plurality of disk drives and a plurality of solid-state drives (SSDs).
9. The system of claim 1 wherein the dataset includes data obtained from a plurality of sensors of an autonomous vehicle.
10. The system of claim 1 wherein the dataset includes semi-structured, heterogeneously-typed sensor data.
11. A method comprising:
enabling data communications between a plurality of compute nodes and a plurality of physical data storage devices via an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer;
using sensor logic to capture sensor input from an autonomous vehicle; and
receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request including an identification of a topic of a dataset, the topic including a metadata file, a data file, and an index file, the index file including at least one pointer into the data file, the dataset being a heterogeneous multimedia time-series dataset including a plurality of dataset topics, each dataset topic corresponding to a data stream created by a sensor and captured by the sensor logic of the autonomous vehicle, at least two dataset topics of the plurality of dataset topics corresponding to different data streams having different data types.
12. The method of claim 11 further including querying and joining data produced by different sensors.
13. The method of claim 11 further including importing data into the data storage access system.
14. The method of claim 11 further including transferring datasets between geographically disparate regions.
15. The method of claim 11 wherein the server layer is configured to authenticate an originating compute node of the data request.
16. The method of claim 11 wherein the server layer returns data to an originating compute node of the data request via Hypertext Transfer Protocol (HTTP).
17. The method of claim 11 wherein the storage layer is configured to obtain data corresponding to the topic from at least one of the plurality of physical data storage devices.
18. The method of claim 11 wherein the plurality of physical data storage devices is of a type from the group consisting of a plurality of disk drives and a plurality of solid-state drives (SSDs).
19. The method of claim 11 wherein the dataset includes data obtained from a plurality of sensors of an autonomous vehicle.
20. The method of claim 11 wherein the dataset includes semi-structured, heterogeneously-typed sensor data.