IP Library › Granted Patent US 12688550

Granted Patent B1

US 12688550 · App. 18/525,036 · Granted Jul 21, 2026

Motion compensation for time-of-flight devices

Inventors: Yongzhe Chen (Santa Clara, CA); Robert Nicholas Moor (San Mateo, CA); Jonathan Solheim (Snoqualmie, WA); Shaminda Subasingha (San Ramon, CA)

Assignee: Zoox, Inc.

G06T3/4038 G06T5/50 G06T7/254 G06T2207/10028 G06T2207/20221 G06T2207/30252

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12688550
App. No.: 18/525,036
Granted: Jul 21, 2026
Kind: B1

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Techniques for generating high resolution images based on ToF sensor data are discussed herein. A vehicle may generate a composite image by combining multiple images, such as a first image and a second image. Since the images may be captured at different times and/or positions in the environment, to accurately combine the images, the ToF device may compensate for the motion of the vehicle. Thus, the ToF device may unproject the pixels of the first image into a three-dimensional point cloud. Further, the ToF device may update the position of the points in the point cloud. Based on the updated point cloud, the ToF device can determine a motion vector which may be applied to the first image to generate an updated image. The ToF device can determine a composite image by combining the updated image with the second image. The vehicle can be controlled based on the composite image.

Claims (76)

1 . A system comprising:

one or more processors; and

one or more non-transitory computer-readable media storing computer-executable instructions that, when executed, cause the system to perform operations comprising:

receiving first sensor data associated with a first image generated by a time-of-flight device at a first time, the first sensor data comprising a set of pixels;

determining, based at least in part on unprojecting the set of pixels to a three-dimensional domain, a set of points;

determining, based at least in part on velocity data of a vehicle and a third time associated with the first image, a displacement vector;

determining, based at least in part on the displacement vector, a modified set of points associated with the set of points;

determining, based at least in part on the modified set of points and the time-of-flight device, a motion vector, wherein determining the motion vector further comprises:

determining a first index of a pixel associated with the set of pixels, the first index associated with a first angle from the time-of-flight device to a point in the set of points;

determining, based at least in part on a second angle, a second index of an updated pixel, the updated pixel being associated with the pixel and the point; and

determining, based at least in part on the first index and the second index, the motion vector;

determining, based at least in part on the motion vector and the first image, an updated image;

receiving second sensor data associated with a second image generated by the time-of-flight device at a second time that is after the first time;

generating, based at least in part on the updated image and the second sensor data, a composite image that compensates for motion of the time-of-flight device over time; and

controlling the vehicle based at least in part on the composite image.

2 . The system of claim 1 , wherein determining the motion vector comprises:

determining, as the motion vector, a difference between the first index and the second index.

3 . The system of claim 1 , the operations further comprising:

identifying a second pixel at third index in the first image and a third pixel at the third index in the second image; and

determining, based at least in part on comparing a first depth associated with the second pixel with a second depth associated with the third pixel, that the second pixel is a motion artifact,

wherein generating the composite image is further based at least in part on the second pixel being a motion artifact.

4 . The system of claim 1 , wherein the velocity data is determined based at least in part on:

receiving, from a component of the vehicle external to the time-of-flight device, the velocity data.

5 . One or more non transitory computer readable media storing instructions executable by one or more processors, wherein the instructions, when executed, cause a system to perform operations comprising:

receiving first sensor data associated with a first image generated by a sensor device;

determining, based at least in part a portion of the first image, a set of three-dimensional points in an environment;

determining, based at least in part on a displacement vector, a modified set of points associated with the set of three-dimensional points;

determining, based at least in part on the modified set of points and the sensor device, a motion vector, wherein determining the motion vector further comprises:

determining a first index of a pixel associated with a set of pixels associated with the first image, the first index associated with a first angle from the sensor device to a point in the set of three-dimensional points;

determining, based at least in part on a second angle, a second index of an updated pixel, the updated pixel being associated with the pixel and the point; and

determining, based at least in part on the first index and the second index, the motion vector;

determining, based at least in part on the motion vector and the first image, an updated image;

receiving second sensor data associated with a second image generated by the sensor device; and

generating, based at least in part on the updated image and the second sensor data, a composite image that compensates for motion of the sensor device over time.

6 . The one or more non transitory computer readable media of claim 5 , wherein determining the motion vector is further based at least in part on:

determining, as the motion vector, a difference between the first index and the second index.

7 . The one or more non transitory computer readable media of claim 5 , wherein unprojecting the set of pixels is based at least in part on:

identifying a second pixel at third index in the first image and a third pixel at the third index in the second image;

determining, based at least in part on comparing a first depth associated with the second pixel with a second depth associated with the third pixel, a difference; and

determining, based at least in part on the difference meeting or exceeding a threshold, that the second pixel is a motion artifact.

8 . The one or more non transitory computer readable media of claim 5 , wherein the sensor device is a time-of-flight device.

9 . The one or more non transitory computer readable media of claim 5 , wherein determining the modified set of points is based at least in part on:

determining, based at least in part on velocity data of a vehicle and a time associated with the first image, the displacement vector; and

determining, based at least in part on the displacement vector, the modified set of points.

10 . The one or more non transitory computer readable media of claim 9 , wherein the velocity data is determined based at least in part on:

receiving, from a component of the vehicle external to the sensor device, the velocity data.

11 . The one or more non transitory computer readable media of claim 5 , the operations further comprising:

controlling a vehicle based at least in part on the composite image.

12 . A method comprising: