IP Library Granted Patent US 11,635,507
Granted Patent B2
US 11,635,507 · App. 17/190,711 · Granted Apr 25, 2023

Systems for estimating three-dimensional trajectories of physical objects

Inventors: Jennifer Anne Healey (San Jose, CA); Haoliang Wang (San Jose, CA); Ding Zhang (Fairfax Station, VA)
Assignee: Adobe Inc.
G01S13/589G01S7/20G01S7/352G01S7/356G01S13/583
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 11,635,507
App. No.
17/190,711
Granted
Apr 25, 2023
Kind
B2
Abstract

In implementations of systems for estimating three-dimensional trajectories of physical objects, a computing device implements a three-dimensional trajectory system to receive radar data describing millimeter wavelength radio waves directed within a physical environment using beamforming and reflected from physical objects in the physical environment. The three-dimensional trajectory system generates a cloud of three-dimensional points based on the radar, each of the three-dimensional points corresponds to a reflected millimeter wavelength radio wave within a sliding temporal window. The three-dimensional points are grouped into at least one group based on Euclidean distances between the three-dimensional points within the cloud. The three-dimensional trajectory system generates an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group using a Kalman filter to track a position and a velocity a centroid of the at least one group in three-dimensions.

Claims (43)

1. A method comprising:

transmitting, by a processing device, millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment;

receiving, by the processing device, radar data describing the millimeter wavelength radio waves;

generating, by the processing device, a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;

determining, by the processing device, a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;

removing, by the processing device, additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;

grouping, by the processing device, the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud; and

generating, by the processing device for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a position and a velocity of a centroid of the at least one group in three-dimensions.

2. The method as described in claim 1 , wherein the Kalman filter is a six state Kalman filter.

3. The method as described in claim 1 , wherein the physical object is a person.

4. The method as described in claim 3 , further comprising generating an indication for display in the user interface that the person is within a threshold distance of an additional person within the physical environment.

5. The method as described in claim 3 , wherein the indication of the three-dimensional trajectory is generated without identifying the person or capturing data usable to identify the person.

6. The method as described in claim 1 , wherein the indication of the three-dimensional trajectory is generated without using a digital camera.

7. The method as described in claim 1 , wherein the sliding temporal window corresponds to period of time that is less than 300 milliseconds.

8. The method as described in claim 1 , wherein the indication of the three-dimensional trajectory is displayed in the user interface as a virtual object relative to the physical object in digital video frames depicting the physical environment.

9. The method as described in claim 1 , wherein the grouping includes performing density-based spatial clustering of applications with noise.

10. A system comprising:

a memory component; and

a processing device coupled to the memory component, the processing device to perform operations comprising:

transmitting millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment

receiving radar data describing the millimeter wavelength radio waves; and

generating a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;

determining a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;

removing additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;

grouping the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud; and

generating, for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a position and a velocity of a centroid of the at least one group in three-dimensions.

11. The system as described in claim 10 , wherein the physical object is a person.

12. The system as described in claim 11 , wherein the indication of the three-dimensional trajectory is generated without identifying the person or capturing data usable to identify the person.

13. The system as described in claim 11 , wherein operations further comprise generating an indication for display in the user interface that the person is within a threshold distance of an additional person within the physical environment.

14. The system as described in claim 10 , wherein the indication of the three-dimensional trajectory is displayed in the user interface as a virtual object relative to the physical object in digital video frames depicting the physical environment.

15. The system as described in claim 10 , wherein the operations further comprise accessing path data describing path objects corresponding to detections of the physical objects in the physical environment and associate the physical object with a particular path object based on a distance between the centroid and a center of the particular path object.

16. A non-transitory computer-readable storage medium storing executable instructions, which when executed by a processing device, cause the processing device to perform operations comprising:

transmitting millimeter wavelength radio waves directed within a physical environment using beamforming that reflect from physical objects in the physical environment

receiving radar data describing the millimeter wavelength radio waves;

generating a cloud of three-dimensional points based on the radar data, each of the three-dimensional points corresponding to a reflected millimeter wavelength radio wave within a sliding temporal window and removing three-dimensional points corresponding to reflected millimeter wavelength radio waves that have arrival times greater than a threshold arrival time;

determining a position and a velocity for each of the three-dimensional points in the cloud of three-dimensional points;

removing additional three-dimensional points having velocities below a threshold velocity from the three-dimensional points within the cloud;

grouping the three-dimensional points into at least one group based on Euclidean distances between the three-dimensional points within the cloud, the Euclidean distances computed in an x-plane, a y-plane, and a z-plane; and

generating, for display in a user interface of a display device, an indication of a three-dimensional trajectory of a physical object corresponding to the at least one group, displayed relative to a current position of the physical object, using a Kalman filter to track a velocity and a position of a centroid of the at least one group in the x-plane, the y-plane, and the z-plane.

17. The non-transitory computer-readable storage medium as described in claim 16 , wherein the operations further comprise associating the physical object with a particular path object of a plurality of path objects based on a distance between the centroid and a center of the particular path object.

18. The non-transitory computer-readable storage medium as described in claim 16 , wherein the physical object is a person.

19. The non-transitory computer-readable storage medium as described in claim 18 , wherein the indication of the three-dimensional trajectory is generated without identifying the person or capturing data usable to identify the person.

20. The non-transitory computer-readable storage medium as described in claim 18 , wherein the indication of the three-dimensional trajectory is generated without using a digital camera.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Mar 3, 2021
From: HEALEY, JENNIFER ANNE; WANG, HAOLIANG; ZHANG, DING
To: ADOBE INC.
Reel/Frame 055476/0827 →
Continuity (1)
Related Publication 20220283289A1 · Sep 8, 2022
Cited By (2)
US 12,248,056 US 12,566,250