IP Library Granted Patent US 9,384,556
Granted Patent B2
US 9,384,556 · App. 14/661,524 · Granted Jul 5, 2016

Image processor configured for efficient estimation and elimination of foreground information in images

Inventors: Ivan Leonidovich Mazurenko (Khimki, RU); Denis Vladimirovich Parkhomenko (Mytyschy, RU); Pavel Aleksandrovich Aliseitchik (Moscow, RU); Barrett J. Brickner (Savage, MN); Dmitry Nicolaevich Babin (Moscow, RU)
Assignee: Avago Technologies General IP (Singapore) Pte. Ltd.
G06T7/0081G06K9/00355G06K9/00375G06K9/469
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,384,556
App. No.
14/661,524
Granted
Jul 5, 2016
Kind
B2
Abstract

An image processing system comprises an image processor having image processing circuitry and an associated memory. The image processor is configured to implement a foreground processing module utilizing the image processing circuitry and the memory. The foreground processing module is configured to obtain one or more images, to estimate a foreground region of interest from the one or more images, to determine a plurality of segments of the foreground region of interest, to calculate amplitude statistics for respective ones of the plurality of segments, to classify respective segments as being respective portions of static foreground objects or as being respective portions of dynamic foreground objects based at least in part on the calculated amplitude statistics and one or more defined patterns for known static and dynamic objects, and to remove one or more segments classified as static foreground objects from the foreground region of interest.

Claims (54)

1. A method comprising the steps of:

obtaining one or more images;

estimating a foreground region of interest from the one or more images;

determining a plurality of segments of the foreground region of interest;

calculating amplitude statistics for respective ones of the plurality of segments;

classifying respective segments as being respective portions of static foreground objects or as being respective portions of dynamic foreground objects based at least in part on the calculated amplitude statistics and one or more defined patterns for known static and dynamic objects; and

removing one or more segments classified as static foreground objects from the foreground region of interest;

wherein the steps are implemented in an image processor comprising a processor coupled to a memory.

2. The method of claim 1 further comprising calculating depth statistics for respective ones of the plurality of segments, wherein classifying respective segments as being respective portions of static foreground objects or as being respective portions of dynamic foreground objects is based at least in part on the calculated amplitude statistics, the calculated depth statistics and the one or more defined patterns for known static and dynamic objects.

3. The method of claim 1 wherein estimating the foreground region of interest comprises generating a binary foreground region of interest mask in which pixels within the foreground region of interest all have a first binary value and pixels outside the foreground region of interest all have a second binary value complementary to the first binary value.

4. The method of claim 3 wherein:

the one or more images comprise a amplitude image and a corresponding depth image;

respective pixels in the amplitude image having amplitude values above a defined threshold are assigned the first binary value and respective pixels having amplitude values below the defined threshold are assigned the second binary value; and

respective pixels in the depth image having a depth within a defined threshold are assigned the first binary value and respective pixels having a depth outside the defined threshold are assigned the second binary value.

5. The method of claim 1 wherein determining the plurality of segments of the foreground region of interest comprises one of:

segmenting the foreground region of interest into a predefined constant number of segments;

segmenting the foreground region of interest into segments comprising respective groups of pixels such that respective numbers of pixels in each group of pixels is approximately the same; and

segmenting the foreground region of interest into areas defined by a rectangular grid.

6. The method of claim 1 wherein obtaining the one or more images comprises obtaining a amplitude image, estimating per-pixel amplitude statistics for each pixel of the amplitude image, and storing the per-pixel amplitude statistics in a amplitude history buffer.

7. The method of claim 6 wherein calculating amplitude statistics for respective ones of the plurality of segments is based on an analysis of the per-pixel amplitude statistics of pixels within the respective segments.

8. The method of claim 7 wherein the amplitude statistics of a given segment comprise an arithmetic mean value of per-pixel amplitude values of pixels within the given segment and a standard deviation of the arithmetic mean value.

9. The method of claim 1 further comprising classifying respective segments as being skin material or as being non-skin material based at least in part on the calculated amplitude statistics and one or more defined patterns for skin and non-skin materials.

10. The method of claim 9 further comprising removing one or more segments classified as non-skin material from the foreground region of interest.

11. The method of claim 9 further comprising classifying one or more of the plurality of segments of the foreground region of interest as a hand region of interest responsive to classifying the one or more segments as dynamic objects and classifying the one or more objects as being skin material.

12. The method of claim 11 further comprising removing from the foreground region of interest segments not classified as the hand region of interest.

13. The method of claim 9 further comprising:

building a connectivity graph for the plurality of segments;

assigning a skin flag to each segment, wherein the skin flag identifies whether respective segments are classified as skin material or non-skin material;

propagating the skin flag based on the connectivity graph to define one or more skin areas of the foreground region of interest;

assigning a dynamic flag to each segment, wherein the dynamic flag identifies whether respective segments are classified as being respective portions of static foreground objects or as being respective portions of dynamic foreground objects;

propagating the dynamic flag based on the connectivity graph to define one or more dynamic areas of the foreground region of interest; and

defining dynamic skin areas of the foreground region of interest as intersections of the skin and dynamic areas.

14. The method of claim 13 further comprising:

classifying at least a portion of the dynamic skin area as a hand region of interest; and

removing from the foreground region of interest segments not classified as the hand region of interest.

15. An apparatus comprising:

an image processor comprising image processing circuitry and an associated memory;

wherein the image processor is configured to implement a foreground processing module utilizing the image processing circuitry and the memory;

wherein the foreground processing module is configured:

to obtain one or more images;

to estimate a foreground region of interest from the one or more images;

to determine a plurality of segments of the foreground region of interest;

to calculate amplitude statistics for respective ones of the plurality of segments;

to classify respective segments as being respective portions of static foreground objects or as being respective portions of dynamic foreground objects based at least in part on the calculated amplitude statistics and one or more defined patterns for known static and dynamic objects; and

to remove one or more segments classified as static foreground objects from the foreground region of interest.

16. The apparatus of claim 15 wherein the foreground processing module is further configured:

to classify respective segments as being skin material or as being non-skin material based at least in part on the calculated amplitude statistics and one or more defined patterns for skin and non-skin materials; and

to remove one or more segments classified as non-skin material from the foreground region of interest.

17. The apparatus of claim 16 wherein the foreground processing module is further configured:

to classify one or more of the plurality of segments of the foreground region of interest as a hand region of interest responsive to classifying the one or more segments as dynamic objects and classifying the one or more objects as being skin material; and

to remove from the foreground region of interest segments not classified as the hand region of interest.

18. The apparatus of claim 15 wherein the one or more images comprise a amplitude image and a corresponding depth image.

19. The apparatus of claim 18 wherein respective pixels in the amplitude image having amplitude values above a defined threshold are assigned a first binary value and respective pixels having amplitude values below the defined threshold are assigned the second binary value.

20. The apparatus of claim 19 wherein respective pixels in the depth image having a depth within a defined threshold are assigned the first binary value and respective pixels having a depth outside the defined threshold are assigned a second binary value.

Assignments (6)
CORRECTIVE ASSIGNMENT TO CORRECT THE PATENT NUMBER 9,385,856 TO 9,385,756 PREVIOUSLY RECORDED AT REEL: 47349 FRAME: 001. ASSIGNOR(S) HEREBY CONFIRMS THE MERGER. Recorded Mar 22, 2019
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 051144/0648 →
CORRECTIVE ASSIGNMENT TO CORRECT THE EFFECTIVE DATE PREVIOUSLY RECORDED ON REEL 047229 FRAME 0408. ASSIGNOR(S) HEREBY CONFIRMS THE THE EFFECTIVE DATE IS 09/05/2018. Recorded Oct 29, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047349/0001 →
MERGER Recorded Oct 4, 2018
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: AVAGO TECHNOLOGIES INTERNATIONAL SALES PTE. LIMITED
Reel/Frame 047229/0408 →
TERMINATION AND RELEASE OF SECURITY INTEREST IN PATENTS Recorded Feb 3, 2017
From: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
To: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Reel/Frame 041710/0001 →
PATENT SECURITY AGREEMENT Recorded Feb 11, 2016
From: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
To: BANK OF AMERICA, N.A., AS COLLATERAL AGENT
Reel/Frame 037808/0001 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded May 20, 2015
From: MAZURENKO, IVAN LEONIDOVICH; PARKHOMENKO, DENIS VLADIMIROVICH; ALISEITCHIK, PAVEL ALEKSANDROVICH; BRICKNER, BARRETT J.; BABIN, DMITRY NICOLAEVICH
To: AVAGO TECHNOLOGIES GENERAL IP (SINGAPORE) PTE. LTD.
Reel/Frame 035683/0983 →
Priority Claims (1)
RU 2014110361 · Mar 18, 2014 · national
Continuity (1)
Related Publication 20150269740A1 · Sep 24, 2015