Video manipulation of red, green, blue, distance (RGB-Z) data including segmentation, up-sampling, and background substitution techniques
View Patent ↗RGB-Z imaging systems acquire RGB data typically with a high X-Y resolution RGB pixel array, and acquire Z-depth data with an array of physically larger Z pixels having additive signal properties. In each acquired frame, RGB pixels are mapped to a corresponding Z pixel. Z image resolution is enhanced by identifying Z discontinuities and identifying corresponding RGB pixels where the Z discontinuities occur. Thus segmented data enables RGB background substitution, which preferably blends foreground pixel color and substitute background color. The segmented data also enables up-sampling in which a higher XY resolution Z image with accurate Z values is obtained. Up-sampling uses an equation set enabling assignment of accurate Z values to RGB pixels. Fixed acquisition frame rates are enabled by carefully culling bad Z data. Segmenting and up-sampling enhanced video effects and enable low cost, low Z resolution arrays to function comparably to higher quality, higher resolution Z arrays.
1. A method to increase resolution of a Z image for an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, the method including the following steps:
(a) acquiring said RGB data from said RGB array and acquiring said Z data from said Z array such that at least one RGB pixel in said RGB array corresponds to and is associated with at least one Z pixel in said Z array;
(b) determining from Z data acquired at step (a) where Z discontinuities occur in said Z array, and for determined said Z discontinuities in said Z array, identifying from RGB data acquired at step (a) corresponding RGB pixels at which said Z discontinuities occur; and
(c) applying determinations and identifications from step (b) to compute a solution to a set of constraints and to minimize a set of costs to determine a correct Z value for said at least one said RGB pixel corresponding to said Z pixel; wherein a resultant Z-image exhibits enhanced resolution compared to native Z resolution of a Z image produced from said Z array.
2. The method of claim 1 , wherein at step (c), said set of constraints utilizes said additive signal properties of said Z pixels.
3. The method of claim 1 , wherein at step (c), said additive signal properties of said Z pixels are satisfied by providing said Z pixel array with Z pixels for which a Z pixel response is the sum of responses of associated RGB pixels.
4. The method of claim 1 , wherein applying said determinations and identifications from step (b) to minimize said set of costs includes minimizing said set of costs to render Z data at each RGB pixel so as to smooth at least a portion of a surface of an object within said scene imaged by said RGB-Z system and displayed on said resultant image.
5. The method of claim 1 , wherein step (b) locates said Z discontinuities using at least one of (i) Z-pixel acquired active brightness discontinuity data, (ii) Z-gradient data, and (iii) RGB pixel data.
6. The method of claim 1 , wherein:
said RGB-Z system is disposed in a motor vehicle, said scene is external to said motor vehicle, which scene may include an object hazardous to said motor vehicle.
7. The method of claim 1 , wherein determining Z discontinuities at step (a) includes using a priori technique by which alpha values are assigned to regions of said scene.
8. The method of claim 1 , wherein at step (c), said set of constraints includes a linear additive constraint and a smoothness cost constraint.
9. The method of claim 8 , said smoothness cost constraint is satisfied by minimizing object surface discontinuities.
10. A method to identify discontinuities in Z in an RGB image for an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, the method including the steps of:
(a) determining where Z discontinuities occur in said Z array;
(b) acquiring said RGB data and said Z data such that at least one RGB pixel corresponds to and is associated with at least one Z pixel in said Z array at which said Z discontinuities occur;
(c) using said RGB data to determine where Z discontinuities occur in RGB pixels neighboring said at least one RGB pixel;
wherein discontinuities identifiable in a Z image are now identifiable in an RGB image.
11. The method of claim 10 , further including using a confidence map for Z depth values to reduce errors in determination of Z discontinuities at step (c).
12. The method of claim 11 , wherein data from Z pixels associated with a low confidence is not used.
13. A method to identify discontinuities in Z in an RGB image for an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, the method including the steps of:
(a) determining where Z discontinuities occur in said Z array;
(b) acquiring said RGB data and said Z data such that at least one RGB pixel corresponds to and is associated with at least one Z pixel in said Z array at which said Z discontinuities occur;
(c) using said RGB data to determine where Z discontinuities occur in RGB pixels neighboring said at least one RGB pixel, wherein at step (c) discontinuities in Z are identified using vector-disparity between RGB values;
wherein discontinuities identifiable in a Z image are now identifiable in an RGB image.
14. The method of claim 13 , wherein at step (c), vector-disparity between RGB values is augmented to include a fourth dimension Z depth estimate for said RGB pixels.
15. A method to replace RGB data in an RGB image formed by an RGB-Z imaging system useable to image a scene, which RGB-Z imaging system has an RGB array of RGB pixels to acquire RGB data useable to form an RGB image, and has a Z array of Z-pixels having additive signal properties to acquire Z data useable to form a Z-image, the method including the steps of:
(a) determining at least one location whereat Z discontinuity occurs in said RGB array of RGB pixels such that at least a first RGB pixel neighboring said Z discontinuity is on one side of said Z discontinuity, and at least a second RGB pixel neighboring said Z discontinuity is on a second said of said Z discontinuity;
(b) replacing a value of said at least said first RGB pixel;
(c) leaving unchanged a value of said at least second RGB pixel;
wherein said Z discontinuity delineates foreground from background within an RGB image formed by said RGB-Z system, wherein one of said foreground and background is substituted at step (b).
16. The method of claim 15 , wherein value replacement for said first RGB pixel at step (b) depends at least in part upon a previous value of said first RGB pixel.