IP Library Granted Patent US 10,609,355
Granted Patent B2
US 10,609,355 · App. 15/796,425 · Granted Mar 31, 2020

Dynamically adjusting sampling of a real-time depth map

Inventors: Yin Hu Chen (Deerfield, IL); Valeriy Marchevsky (Glenview, IL); Susan Yanqing Xu (Westmont, IL)
Assignee: Motorola Mobility LLC
H04N13/139G06T7/50H04N5/232H04N13/296H04N2013/0081H04N2013/0085H04N2013/0096
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 10,609,355
App. No.
15/796,425
Granted
Mar 31, 2020
Kind
B2
Abstract

A method, a system, and a computer program product for dynamically adjusting sampling of a depth map based on detected motion in a current scene. The method includes capturing real-time scan data at a first resolution by a first camera and a second camera of an image capturing device. The method further includes synchronizing a first plurality of frames of the real-time scan data to create a plurality of synchronized frames at a first frame rate. The method further includes analyzing the synchronized frames to determine whether motion exists. The method further includes, in response to determining motion exists: determining, based on the plurality of synchronized frames, a rate of motion within the current scene; and dynamically calculating a target resolution and a target frame rate for a real-time depth map. The method further includes generating a real-time depth map at the target resolution and target frame rate.

Claims (110)

1. A method comprising:

capturing real-time scan data by a first camera and a second camera of an image capturing device;

synchronizing, by a processor, a first plurality of frames of the real-time scan data captured by the first camera with a second plurality of frames captured by the second camera to create a plurality of synchronized frames at a first frame rate;

analyzing the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene:

determining, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculating a target resolution and a target frame rate based on the rate of motion; and

generating, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to the analysis of the plurality of synchronized frames indicating that the current scene is a static scene, combining, by the processor, the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and a full resolution of at least one of the first camera and the second camera.

2. The method of claim 1 , wherein calculating the target resolution and the target frame rate further comprises:

selecting, based on the determined rate of motion, a second resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution; and

selecting, based on the determined rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

3. The method of claim 2 , wherein generating the real-time depth map further comprises:

down-sampling the real-time scan data to the second resolution based on the determined rate of motion; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

4. The method of claim 1 , further comprising:

in response to determining motion exists within the current scene:

determining whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range:

selecting, from a database, a high-motion resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution;

selecting, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-sampling the real-time scan data to the high-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

5. The method of claim 4 , further comprising:

in response to determining motion exists within the current scene:

determining whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range:

selecting, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selecting, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-sampling the real-time scan data to the low-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

6. The method of claim 1 , wherein dynamically calculating the target resolution and the target frame rate further comprises:

calculating, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculating a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishing the minimum frame rate as the target frame rate; and

establishing the maximum resolution as the target resolution.

7. An image capturing device comprising:

a first camera that captures a first scan data of a current scene at a full resolution;

a second camera that captures a second scan data of the current scene at the full resolution; and

at least one processor that:

synchronizes the first scan data and the second scan data to create a plurality of synchronized frames at a first frame rate;

analyzes the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene:

determines, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculates a target resolution and a target frame rate based on the rate of motion; and

generates, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to determining that the analysis of the plurality of synchronized frames indicates that the current scene is a static scene, combines the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and the full resolution.

8. The image capturing device of claim 7 , wherein in calculating the target resolution and the target frame rate, the at least one processor:

selects, based on the rate of motion, a second resolution that is lower than the full resolution as the target resolution; and

selects, based on the rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

9. The image capturing device of claim 8 , wherein in generating the real-time depth map, the at least one processor:

down-samples the real-time scan data to the second resolution based on the determined rate of motion; and

combines the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

10. The image capturing device of claim 7 , wherein in response to determining motion exists within the current scene, the at least one processor:

determines whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range:

selects, from a database, a high-motion resolution that is lower than the full resolution as the target resolution;

selects, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-samples the real-time scan data to the high-motion resolution; and

combines the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

11. The image capturing device of claim 10 , wherein in response to determining motion exists within the current scene, the at least one processor:

determines whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range:

selects, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selects, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-samples the real-time scan data to the low-motion resolution; and

combines, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

12. The image capturing device of claim 7 , wherein in calculating the target resolution and the target frame rate, the processor:

calculates, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculates a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishes the minimum frame rate as the target frame rate; and

establishes the maximum resolution as the target resolution.

13. A computer program product comprising:

a computer readable storage device; and

program code on the computer readable storage device that, when executed by a processor associated with an image capturing device, configures the image capturing device to provide the functionality of:

capturing real-time scan data at a full resolution by a first camera and a second camera of an image capturing device;

synchronizing, by a processor, a first plurality of frames of the real-time scan data captured by the first camera with a second plurality of frames captured by the second camera to create a plurality of synchronized frames at a first frame rate;

analyzing the plurality of synchronized frames to determine whether motion exists within the current scene;

in response to determining motion exists within the current scene:

determining, based on the plurality of synchronized frames, a rate of motion within the current scene;

dynamically calculating a target resolution and a target frame rate based on the rate of motion; and

generating, from the real-time scan data, a real-time depth map at the target resolution and target frame rate; and

in response to the analysis of the plurality of synchronized frames indicating that the current scene is a static scene, combining the real-time scan data captured by the first camera and the second camera at the first frame rate to generate a full-resolution real-time depth map of the current scene at the first frame rate and the full resolution.

14. The computer program product of claim 13 , the program code for calculating the target resolution and the target frame rate further comprising code for:

selecting, based on the determined rate of motion, a second resolution that is lower than the full resolution as the target resolution; and

selecting, based on the determined rate of motion, a second frame rate that is higher than the first frame rate as the target frame rate.

15. The computer program product of claim 14 , the program code for generating the real-time depth map further comprising code for:

down-sampling the real-time scan data to the second resolution based on the determined rate of motion; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the second frame rate to generate the real-time depth map at the second frame rate and the second resolution.

16. The computer program product of claim 13 , the program code further comprising code for:

in response to determining motion exists within the current scene:

determining whether the rate of motion corresponds to a high-speed motion range; and

in response to determining the rate of motion corresponds to a high-speed motion range:

selecting, from a database, a high-motion resolution that is lower than a full resolution of at least one of the first camera and the second camera as the target resolution;

selecting, from the database, a high-motion frame rate that is higher than the first frame rate as the target frame rate;

down-sampling the real-time scan data to the high-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the high-motion frame rate to generate the real-time depth map at the high-motion frame rate and the high-motion resolution.

17. The computer program product of claim 16 , the program code further comprising code for:

in response to determining motion exists within the current scene:

determining whether the rate of motion corresponds to a low-speed motion range; and

in response to determining the rate of motion corresponds to a low-speed motion range:

selecting, from the database, a low-motion resolution that is lower than the full resolution and higher than the high-motion resolution as the target resolution;

selecting, from the database, a low-motion frame rate that is higher than the first frame rate and lower than the high-motion frame rate as the target frame rate;

down-sampling the real-time scan data to the low-motion resolution; and

combining, by the processor, the down-sampled real-time scan data captured by the first camera and the second camera at the low-motion frame rate to generate the real-time depth map at the low-motion frame rate and the low-motion resolution.

18. The computer program product of claim 13 , the program code for dynamically calculating the target resolution and the target frame rate further comprising code for:

calculating, based on the rate of motion, a minimum frame rate to capture the motion within the current scene;

calculating a maximum resolution of the real-time depth map based on the minimum frame rate and a maximum calculation time of each frame of a real-time depth map;

establishing the minimum frame rate as the target frame rate; and

establishing the maximum resolution as the target resolution.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 1, 2017
From: CHEN, YIN HU; MARCHEVSKY, VALERIY; XU, SUSAN YANQING
To: MOTOROLA MOBILITY LLC
Reel/Frame 044000/0543 →
Continuity (1)
Related Publication 20190132570A1 · May 2, 2019
Cited By (1)
US 12,340,016