Storage architecture for heterogeneous multimedia data

1. A system comprising:

a plurality of compute nodes being in data communication with a data network;

a plurality of physical data storage devices being in communication with the data network;

a sensor logic to capture sensor data from an autonomous vehicle; and

a data storage access system enabling communication of data between the plurality of compute nodes and the plurality of physical data storage devices via the data network, the data storage access system receiving a data request from at least one of the plurality of compute nodes, the data request comprising an identification of a topic of a dataset, the dataset being a heterogeneous multimedia time-series dataset comprising a plurality of dataset topics, each dataset topic corresponding to a data stream of the sensor data, at least two dataset topics of the plurality of dataset topics corresponding to different data streams having different data types.

2. The system of claim 1 wherein the data network comprises an application programming interface (API) layer, a cache management layer, a server layer, and a storage layer.

3. The system of claim 2 wherein data transfers between the API layer and the storage layer goes through the server layer, wherein the data transfer between the API layer and the server layer goes through the cache management layer.

4. The system of claim 2 wherein the storage layer, the server layer, and the cache management layer are transparent to users.

5. The system of claim 1 wherein the data request uses a Hypertext Transfer Protocol (HTTP) and a server layer of the data network returns data to an originating compute node of the data request via HTTP.

6. The system of claim 1 wherein the different data streams are aligned to form a time series via interpolation.

7. The system of claim 1 wherein cache data at a cache management layer of the data network is transparently split into portions.

8. The system of claim 1 wherein a server layer of the data network is configured to authenticate an originating compute node of the data request by validating credentials of a requesting user or the originating compute node.

9. The system of claim 1 wherein a storage layer of the data network stores data on an array of the plurality of physical data storage devices connected by a networked file system (NFS).

10. A method comprising:

enabling data communications between a plurality of compute nodes and a plurality of physical data storage devices via a data network;

using a sensor logic to capture sensor data from an autonomous vehicle; and

receiving a data request from at least one of the plurality of compute nodes at the API layer, the data request comprising an identification of a topic of a dataset, the dataset being a heterogeneous multimedia time-series dataset comprising a plurality of dataset topics, each dataset topic corresponding to a data stream of the sensor data, at least two dataset topics of the plurality of dataset topics corresponding to different data streams having different data types.

11. The method of claim 10 wherein the at least one of the plurality of compute nodes is at an application programming interface (API) of the data network.

12. The method of claim 10 wherein the topic comprises a metadata file, a data file, and an index file, wherein the index file comprises at least one pointer into the data file.

13. The method of claim 12 wherein an application programming interface (API) of the data network determines a data type for the dataset topic by using the metadata file.

14. The method of claim 12 wherein the metadata file is formatted as a JSON file.

15. The method of claim 12 wherein an application programming interface (API) layer of the data network uses the index file for the dataset topic to look up an offset and a length of a first message in the data file.

16. A non-transitory machine-useable storage medium embodying instructions which, when executed by a machine, cause the machine to:

enable data communications between a plurality of compute nodes and a plurality of physical data storage devices via a data network;

use a sensor logic to capture sensor data from an autonomous vehicle; and

receive a data request from at least one of the plurality of compute nodes at the API layer, the data request comprising an identification of a topic of a dataset, the dataset being a heterogeneous multimedia time-series dataset comprising a plurality of dataset topics, each dataset topic corresponding to a data stream of the sensor data, at least two dataset topics of the plurality of dataset topics corresponding to different data streams having different data types.

17. The non-transitory machine-useable storage medium of claim 16 wherein the different data streams are created by different sensors respectively, wherein the different data streams are captured by the sensor logic of the autonomous vehicle.

18. The non-transitory machine-useable storage medium of claim 16 wherein the sensor data is selected from the group consisting of global positioning system (GPS) data, video streams, inertial measurement unit (IMU) data, radar data, laser range finder, LIDAR, and internal vehicle subsystem data.

19. The non-transitory machine-useable storage medium of claim 16 wherein the dataset comprises semi-structured, heterogeneously-typed sensor data which is not tabular.

20. The non-transitory machine-useable storage medium of claim 16 wherein an application programming interface (API) automates selection of training and test subsets of data in the data network.