Action-conditioned vehicle control

1. A method, comprising:

determining a high-level command based on a location of a vehicle with respect to a route including a start location and a finish location, wherein the high-level command comprises a command to execute a maneuver to be performed by the vehicle;

acquiring an image of the vehicle external environment;

generating a high-level command cue at a Context Sub-Network of a Deep Neural Network based on the high-level command, wherein the high-level command cue comprises high-level commands and distances;

generating an image cue at a Convolutional Neural Network of the Deep Neural Network based on the image, wherein the image cue comprises an array of data values extracted by the Convolutional Neural Network from the image;

determining steering, braking and powertrain commands at a Neural Sub-Network of the Deep Neural Network based on the high-level command cue and the image cue; and

operating the vehicle by actuating vehicle components based on the steering, braking and powertrain commands.

2. The method of claim 1 , further comprising determining a plurality of high-level commands based on locations of the vehicle with respect to the route and acquiring a plurality of corresponding images of the vehicle external environment, wherein the plurality of locations are distributed from the start location to the finish location.

3. The method of claim 2 , further comprising determining a plurality of steering, braking and powertrain commands based on inputting the plurality of high-level commands and the plurality of corresponding images into the Deep Neural Network; and thereby operating the vehicle to travel from the start location to the finish location along the route by actuating vehicle components based on the plurality of steering, braking and powertrain commands.

4. The method of claim 1 , wherein the steering, braking and powertrain commands include torque with which steering, braking and powertrain vehicle components apply force to vehicle wheels to change vehicle location, speed and direction.

5. The method of claim 1 , wherein the Convolutional Neural Network includes a plurality of pooling layers that process an input image by convolving and down-sampling and one or more fully-connected layers that output the image cues based on matrix multiplication and bias offset.

6. The method of claim 5 , wherein the Context Sub-Network includes a plurality of fully-connected neural network layers that process an input high-level command based on matrix multiplication and bias offset to output the high-level command cues.

7. The method of claim 6 , wherein the Neural Sub-Network includes a plurality of full-connected neural network layers that process the input image cues and the high-level command cues and output the steering, braking and powertrain commands.

8. The method of claim 7 , wherein the Convolutional Neural Network, the Context Sub-Network and the Neural Sub-Network are trained by inputting recorded high-level commands based on recorded locations with respect to the route into the Context Sub-Network and recorded corresponding images into the Convolutional Neural Network and back propagating errors based on comparing Neural Sub-Network output steering, braking and powertrain commands with recorded steering, braking and powertrain commands.

9. The method of claim 8 , wherein back propagating errors includes determining the image cues and the high-level command cues.

10. A system, comprising a processor; and

a memory, the memory including instructions to be executed by the processor to:

determine a high-level command based on a location of a vehicle with respect to a route including a start location and a finish location, wherein the high-level command comprises a command to execute a maneuver to be performed by the vehicle;

acquire an image of a vehicle external environment;

generating a high-level command cue at a Context Sub-Network of a Deep Neural Network based on the high-level command, wherein the high-level command cue comprises high-level commands and distances;

generating an image cue at a Convolutional Neural Network of the Deep Neural Network based on the image, wherein the image cue comprises an array of data values extracted by the Convolutional Neural Network from the image;

determine steering, braking and powertrain commands at a Neural Sub-Network of the Deep Neural Network based on the high-level command cue and the image cue; and

operate the vehicle by actuating vehicle components based on the steering, braking and powertrain commands.

11. The processor of claim 10 , further comprising determining a plurality of high-level commands based on locations of the vehicle with respect to the route and acquiring a plurality of corresponding images of the vehicle external environment, wherein the plurality of locations are distributed from the start location to the finish location.

12. The processor of claim 11 , further comprising determining a plurality of steering, braking and powertrain commands based on the plurality of high-level commands and the plurality of corresponding images into the Deep Neural Network; and thereby operating the vehicle to travel from the start location to the finish location along the route by actuating vehicle components based on the plurality of steering, braking and powertrain commands.

13. The processor of claim 12 , wherein the steering, braking and powertrain commands include the torque with which steering, braking and powertrain vehicle components apply force to vehicle wheels to change vehicle location, speed and direction.

14. The processor of claim 10 , wherein the Convolutional Neural Network includes a plurality of pooling layers that process an input image by convolving and down-sampling and one or more fully-connected layers that output the image cues based on matrix multiplication and bias offset.

15. The processor of claim 14 , wherein the Context Sub-Network includes a plurality of fully-connected neural network layers that process an input high-level command based on matrix multiplication and bias offset to output the high-level command cues.

16. The processor of claim 15 , wherein the Neural Sub-Network includes a plurality of full-connected neural network layers that process the input image cues and the high-level command cues and output the steering, braking and powertrain commands.

17. The processor of claim 16 , wherein the Convolutional Neural Network, the Context Sub-Network and the Neural Sub-Network are trained by inputting recorded high-level commands based on recorded locations with respect to the route and recorded corresponding images into the Convolutional Neural Network and back propagating error based on comparing Neural Sub-Network output steering, braking and powertrain commands to with recorded steering, braking and powertrain commands.

18. The processor of claim 17 , wherein back propagating errors includes determining the image cues and the high-level command cues.

19. A system, comprising:

a video sensor operative to acquire an image of a vehicle external environment;

a Deep Neural Network;

vehicle components operative to operate a vehicle;

a processor; and

a memory, the memory including instructions to be executed by the processor to:

determine a high-level command based on a location of the vehicle with respect to a route including a start location and a finish location, wherein the high-level command comprises a command to execute a maneuver to be performed by the vehicle;

acquire an image of a vehicle external environment;

generating a high-level command cue at a Context Sub-Network of the Deep Neural Network based on the high-level command, wherein the high-level command cue comprises high-level commands and distances;

generating an image cue at a Convolutional Neural Network of the Deep Neural Network based on the image, wherein the image cue comprises an array of data values extracted by the Convolutional Neural Network from the image;

determine steering, braking and powertrain commands at a Neural Sub-Network of the Deep Neural Network based on the high-level command cue and the image cue; and

operate the vehicle by actuating vehicle components based on the steering, braking and powertrain commands.

20. The processor of claim 19 , further comprising determining a plurality of high-level commands based on and acquiring a plurality of corresponding images of the vehicle external environment, wherein the plurality of locations are distributed from the start location to the finish location.