Granted Patent B2

US 12,475,649 · App. 18/099,186 · Granted Nov 18, 2025

Inventors: Matthew A. Shreve (Campbell, CA); Robert R. Price (Palo Alto, CA)

Assignee: Xerox Corporation

G06T17/205G06T15/04H04N19/17

Abstract

Claims (98)

1 . A computer-implemented method, comprising:

capturing, by a recording device, a scene with a plurality of physical objects, wherein the scene is displayed as a three-dimensional (3D) mesh;

marking 3D annotations for a physical object in the scene;

identifying a background mask in the scene displayed as a 3D mesh, wherein the background mask indicates pixels corresponding to a region behind or partially obstructed by the physical object and each pixel in the identified background mask is associated with a value;

capturing a plurality of images of the scene with varying features, wherein a respective image includes:

two-dimensional (2D) projections corresponding to the marked 3D annotations for the physical object; and

the background mask based on the associated value for each pixel in the background mask;

updating, in the respective image, the value of each pixel in the identified background mask with a new value; and

training a machine model using the respective image as generated labeled data, thereby obtaining the generated labeled data in an automated manner based on a minimal amount of marked annotations.

2 . The method of claim 1 , wherein identifying the background mask further comprises at least one of:

inserting, by a user associated with the recording device, the background mask in the scene using tools associated with the recording device or another computing device; and

detecting, automatically by the recording device, predetermined categories of 2D surfaces or 3D surfaces or shapes.

3 . The method of claim 1 ,

wherein the background mask comprises a virtual green screen, and

wherein the value associated with each background pixel comprises a chroma key value corresponding to a shade of green.

4 . The method of claim 1 , wherein the background mask corresponds to at least one of:

a 2D surface within the 3D mesh scene that is behind or underneath the physical object relative to the recording device; and

a 3D surface or shape within the 3D mesh scene that is behind or underneath the physical object relative to the recording device.

5 . The method of claim 1 , wherein the varying features of the captured plurality of images of the scene include or are based on at least one of:

a location, pose, or angle of the recording device relative to the physical object;

a lighting condition associated with the scene; and

an occlusion factor of the physical object in the scene.

6 . The method of claim 1 , wherein the value associated with each background pixel comprises at least one of:

a chroma key value;

a red green blue (RGB) value;

a hue saturation value (HSV) value;

a hue saturation brightness (HSB) value;

a monochrome value;

a random value;

a noisy value; and

a value or flag indicating that a respective background pixel of the background mask is to be subsequently replaced by a pixel with a different value.

7 . The method of claim 1 ,

wherein a respective background pixel is of a same or a different value than a remainder of the background pixels.

8 . The method of claim 1 , wherein updating the value of each background pixel with the new value comprises at least one of:

replacing the background pixels indicated by the background mask with a natural image, wherein the natural image comprises a differing texture from the region behind the physical object; and

replacing the background pixels indicated by the background mask with pixels of a same value or a different value as each other.

9 . The method of claim 1 , further comprising:

storing an image of the scene, including the marked 3D annotations and the identified background mask, captured by the recording device;

storing the respective image, including the 2D projections and the background mask, captured by the recording device; and

storing the respective image with the updated background pixels.

10 . A computer system, comprising:

a processor; and

a storage device storing instructions that when executed by the processor cause the processor to perform a method, the method comprising:

capturing, by a recording device, a scene with a plurality of physical objects, wherein the scene is displayed as a three-dimensional (3D) mesh;

marking 3D annotations for a physical object in the scene;

identifying a background mask in the scene displayed as a 3D mesh, wherein the background mask indicates pixels corresponding to a region behind or partially obscured by the physical object and each pixel in the identified background mask is associated with a value;

capturing a plurality of images of the scene with varying features, wherein a respective image includes:

two-dimensional (2D) projections corresponding to the marked 3D annotations for the physical object; and

the background mask based on the associated value for each background pixel;

updating, in the respective image, the value of each pixel in the identified background mask with a new value; and

training a machine model using the respective image as generated labeled data, thereby obtaining the generated labeled data in an automated manner based on a minimal amount of marked annotations.

11 . The computer system of claim 10 , wherein identifying the background mask further comprises at least one of:

inserting, by a user associated with the recording device, the background mask in the scene using tools associated with the recording device or another computing device; and

detecting, automatically by the recording device, predetermined categories of 2D surfaces or 3D surfaces or shapes.

12 . The computer system of claim 10 ,

wherein the background mask comprises a virtual green screen, and

wherein the value associated with each background pixel comprises a chroma key value corresponding to a shade of green.

13 . The computer system of claim 10 , wherein the background mask corresponds to at least one of:

a 2D surface within the 3D mesh scene that is behind or underneath the physical object relative to the recording device; and

a 3D surface or shape within the 3D mesh scene that is behind or underneath the physical object relative to the recording device.

14 . The computer system of claim 10 , wherein the varying features of the captured plurality of images of the scene include or are based on at least one of:

a location, pose, or angle of the recording device relative to the physical object;

a lighting condition associated with the scene; and

an occlusion factor of the physical object in the scene.

15 . The computer system of claim 10 , wherein the value associated with each background pixel comprises at least one of:

a chroma key value;

a red green blue (RGB) value;

a hue saturation value (HSV) value;

a hue saturation brightness (HSB) value;

a monochrome value;

a random value;

a noisy value; and

a value or flag indicating that a respective background pixel of the background mask is to be subsequently replaced by a pixel with a different value.

16 . The computer system of claim 10 , wherein updating the value of each background pixel with the new value comprises at least one of:

replacing the background pixels indicated by the background mask with a natural image, wherein the natural image comprises a differing texture from the region behind the physical object; and

replacing the background pixels indicated by the background mask with pixels of a same value or a different value as each other.

17 . The computer system of claim 10 , wherein the method further comprises:

storing an image of the scene, including the marked 3D annotations and the identified background mask, captured by the recording device;

storing the respective image, including the 2D projections and the background mask, captured by the recording device; and

storing the respective image with the updated background pixels.

18 . A non-transitory computer-readable storage medium storing instructions that when executed by a computer cause the computer to perform a method, the method comprising:

capturing, by a recording device, a scene with a plurality of physical objects, wherein the scene is displayed as a three-dimensional (3D) mesh;

marking 3D annotations for a physical object in the scene;

identifying a background mask in the scene displayed as a 3D mesh, wherein the background mask indicates pixels corresponding to a region behind or partially obstructed by the physical object and each pixel in the identified background mask is associated with a value;

capturing a plurality of images of the scene with varying features, wherein a respective image includes:

two-dimensional (2D) projections corresponding to the marked 3D annotations for the physical object; and

the background mask based on the associated value for each background pixel;

updating, in the respective image, the value of each pixel in the identified background mask with a new value; and

training a machine model using the respective image as generated labeled data, thereby obtaining the generated labeled data in an automated manner based on a minimal amount of marked annotations.

19 . The non-transitory computer-readable storage medium of claim 18 , wherein identifying the background mask further comprises at least one of:

inserting, by a user associated with the recording device, the background mask in the scene using tools associated with the recording device or another computing device; and

detecting, automatically by the recording device, predetermined categories of 2D surfaces or 3D surfaces or shapes.

20 . The non-transitory computer-readable storage medium of claim 18 , wherein the background mask corresponds to at least one of:

a 2D surface within the 3D mesh scene that is behind or underneath the physical object relative to the recording device; and

a 3D surface or shape within the 3D mesh scene that is behind or underneath the physical object relative to the recording device, and

wherein updating the value of each background pixel with the new value comprises at least one of:

replacing the background pixels indicated by the background mask with a natural image, wherein the natural image comprises a differing texture from the region behind the physical object; and

replacing the background pixels indicated by the background mask with pixels of a same value or a different value as each other.

Assignments (8)

Continuity (1)

References Cited (55)