Reducing latency in head-mounted display for the remote operation of machinery

1 . One or more non-transitory computer-readable media storing computer-executable instructions that, when executed by at least one processor, perform a method for reducing latency for imagery captured in a remote location and displayed in a head-mounted display, the method comprising:

receiving, from a remote capture device controlled by a robot assembly located in the remote location, an image of the remote location;

receiving a head-mounted display pose of the head-mounted display;

transmitting, by the remote capture device, the image to a graphics processing unit associated with the head-mounted display; and

causing the graphics processing unit to execute instructions for:

determining, by the graphics processing unit, a rotation difference based on an image pose of the image and the head-mounted display pose;

spherically rendering, by the graphics processing unit, the image on the head-mounted display based on the rotation difference;

responsive to a first movement of the head-mounted display, determining, by the graphics processing unit, a new head-mounted display pose,

wherein the first movement of the head-mounted display occurs while the remote capture device is locked from moving;

determining, by the graphics processing unit, a new rotation difference;

rendering, by the graphics processing unit, a new image based on the new rotation difference;

extrapolating, by the graphics processing unit, at least one future movement of the head-mounted display; and

instructing the robot assembly associated with the remote capture device to initiate a new movement corresponding to the at least one future movement.

2 . The one or more non-transitory computer-readable media of claim 1 , wherein the image pose is determined using at least one of non-linear extrapolation or non-linear interpolation of two or more image poses.

3 . The one or more non-transitory computer-readable media of claim 1 , wherein the method further comprises:

responsive to a second movement of the head-mounted display, causing a corresponding movement of the remote capture device by the robot assembly.

4 . The one or more non-transitory computer-readable media of claim 1 , wherein the method further comprises:

communicating over a dielectric gap, via a fiber optic cable disposed between the remote capture device and the head-mounted display, additional sensor data captured by at least one of:

a gyroscope, an accelerometer, a thermometer, a barometer, a light emitter, a voltage meter, or a current meter.

5 . The one or more non-transitory computer-readable media of claim 1 , further comprising causing the graphics processing unit to execute further instructions for:

demosaicing, by the graphics processing unit, the image; and

converting, by the graphics processing unit, the image from a fisheye image to an equirectangular image,

wherein the image is spherically rendered from the equirectangular image.

6 . The one or more non-transitory computer-readable media of claim 1 , wherein the remote capture device is coupled to an aerial device located on a utility vehicle.

7 . The one or more non-transitory computer-readable media of claim 1 ,

wherein the image and the head-mounted display pose are received asynchronously.

8 . A system for viewing a remote location, comprising:

a remote capture device configured to capture video of the remote location;

a robot assembly coupled to the remote capture device;

a head-mounted display configured to control movement of the remote capture device; and

a graphics processing unit associated with the head-mounted display and configured to execute computer-executable instructions comprising:

receiving, by the graphics processing unit, a frame of the video from the remote capture device;

determining, by the graphics processing unit, an image pose by extrapolating from two or more robot poses associated with the robot assembly;

determining, by the graphics processing unit, a rotation difference using the image pose and a head-mounted display pose; spherically rendering, by the graphics processing unit, the frame of the video on the head-mounted display based on the rotation difference; and

responsive to detecting a vibration at the head-mounted display, mitigating the vibration using a filter.

9 . The system of claim 8 , wherein the head-mounted display comprises:

a plurality of visual displays configured to display a stereoscopic virtual representation of the remote location.

10 . The system of claim 8 , wherein the graphics processing unit is configured to execute additional computer-executable instructions comprising:

responsive to receiving the frame from the remote capture device, demosaicing, by the graphics processing unit, the frame; and

converting, by the graphics processing unit, the frame from a fisheye image frame to an equirectangular image frame,

wherein the frame is spherically rendered from the equirectangular image frame.

11 . The system of claim 8 , wherein the computer-executable instructions further comprise:

responsive to a new movement of the head-mounted display, determining, by the graphics processing unit, a new head-mounted display pose,

wherein the new movement of the head-mounted display occurs while the robot assembly is locked from moving;

determining, by the graphics processing unit, a new rotation difference; and

rendering, by the graphics processing unit, a new image based on the new rotation difference.

12 . The system of claim 8 , wherein the remote capture device further comprises a microphone configure to capture audio at the remote location.

13 . The system of claim 8 ,

wherein the filter used to mitigate the vibration is a low-pass filter.

14 . The system of claim 8 ,

wherein the computer-executable instructions further comprise:

responsive to the vibration at the head-mounted display ceasing to be detected, disabling the filter used to mitigate the vibration.

15 . A method for rendering imagery of a remote location, the method comprising:

receiving an image from a remote capture device coupled to a robot assembly disposed in the remote location;

determining an image pose by interpolating from two or more robot poses associated with the robot assembly;

wherein the image pose is determined using interpolation based on a first robot pose and a second robot pose from the two or more robot poses, the first robot pose preceding a time at which the image was captured and the second robot pose succeeding the time at which the image was captured;

determining, by a graphics processing unit and based on the image pose and a head-mounted display pose of a head-mounted display, a rotation difference;

determining, by the graphics processing unit, the first robot pose based on a plurality of prior robot poses; and

rendering, by the graphics processing unit and based on the rotation difference, the image in the head-mounted display.

16 . The method of claim 15 , further comprising:

receiving, by the graphics processing unit, the image between a first time associated with the first robot pose and a second time associated with the second robot pose; and

buffering, by the graphics processing unit, the first robot pose and the second robot pose.

17 . The method of claim 15 , further comprising:

responsive to a movement of the head-mounted display, determining, by the graphics processing unit, a new head-mounted display pose,

wherein the movement of the head-mounted display occurs while the robot assembly is locked from moving;

determining, by the graphics processing unit, a new rotation difference; and

rendering, by the graphics processing unit, a new image based on the new rotation difference.

18 . The method of claim 15 , further comprising:

responsive to receiving the image, demosaicing, by the graphics processing unit, the image; and

converting the image from a fisheye image to an equirectangular image, wherein the image is rendered from the equirectangular image.

19 . The method of claim 15 ,

wherein the rotation difference is represented by a rotation matrix.

20 . The method of claim 15 , further comprising:

responsive to receiving the image, overlaying the image with a color filter array such that a color image is obtained that matches a color captured by the remote capture device.