Granted Patent B2

US 12,651,416 · App. 18/263,861 · Granted Jun 9, 2026

Inventors: Mikio Ishimaru (Tokyo, JP); Sayuri Wakabayashi (Kanagawa, JP); Go Matsumori (Kanagawa, JP)

Assignees: SONY SEMICONDUCTOR SOLUTIONS CORPORATION; SONY GROUP CORPORATION

G06T19/006

Abstract

Claims (102)

1 . An image processing apparatus, comprising:

a central processing unit (CPU) configured to:

acquire first sensing data that corresponds to first image sensing data;

acquire second sensing data different from the first sensing data;

extract, from the second sensing data, first distance information associated with a real object;

augment a reality space based on each of the first sensing data and the second sensing data, wherein the reality space includes the real object;

superimpose a virtual object on the real object based on each of the second sensing data and the extracted first distance information;

control a display process to display the augmented reality space, wherein the displayed augmented reality space includes the superimposed virtual object on the real object;

determine, based on at least one of the first sensing data or the second sensing data, a specific portion of a user in the displayed augmented reality space;

switch, based on the determination of the specific portion of the user in the displayed augmented reality space, between a first mode and a second mode;

accept one of a first operation or a second operation based on the switch between the first mode and the second mode, wherein

the first mode is associated with

the accepted first operation for the virtual object, and

the determination of the specific portion of the user in the displayed augmented reality space, and

the second mode is associated with the accepted second operation for the virtual object; and

perform an occlusion process associated with the virtual object, wherein the occlusion process is based on

the accepted first operation for the virtual object, and

second distance information associated with the determined specific portion.

2 . The image processing apparatus according to claim 1 , wherein

the first operation is based on movement of the determined specific portion in the augmented reality space,

the second operation corresponds to a touch operation on a display region of the virtual object,

the touch operation is based on a user input, and

the display region is on a display screen.

3 . The image processing apparatus according to claim 2 ,

wherein the CPU is further configured to:

designate, in the first mode, a first display mode of a display object; and

designate, in the second mode, a second display mode of the display object,

wherein the first display mode of the display object is different from the second display mode of the display object.

4 . The image processing apparatus according to claim 3 ,

wherein the CPU is further configured to control the display process to

designate a first size of the virtual object in the first mode; and

designate a second size of the virtual object in the second mode, wherein the first size of the virtual object is different from the second size of the virtual object.

5 . The image processing apparatus according to claim 1 , wherein

the CPU is further configured to determine the virtual object based on the first sensing data and the determined specific portion of the user,

the virtual object corresponds to a target of a specific operation, and

the specific operation is associated with the determined specific portion.

6 . The image processing apparatus according to claim 1 , wherein the CPU is further configured to:

acquire, from a first sensor, second image sensing data, wherein

the first sensor captures the second image sensing data from a first direction,

a second sensor captures the first image sensing data from a second direction, and

the first direction is opposite to the second direction;

determine, based on the second image sensing data, the specific portion in the augmented reality space; and

switch to the second mode based on the determination of the specific portion in the augmented reality space.

7 . The image processing apparatus according to claim 1 , wherein the CPU is further configured to:

determine, based on the extracted first distance information, a specific step associated with the real object; and

control the display process to issue an alert associated with the determined specific step.

8 . The image processing apparatus according to claim 1 , wherein the CPU is further configured to determine a size of the virtual object based on the extracted first distance information.

9 . The image processing apparatus according to claim 1 , wherein the CPU is further configured to:

determine, based on the second sensing data, a material of the real object; and

control, after the determination of the material of the real object, the display process to superimpose the virtual object on the real object.

10 . The image processing apparatus according to claim 9 , wherein

the CPU is further configured to:

determine a self-position of the image processing apparatus; and

correct, based on material information, the self-position of the image processing apparatus, and

the material information is associated with the determined material of the real object.

11 . The image processing apparatus according to claim 9 , wherein

the CPU is further configured to control the display process to display the virtual object in one of a third mode or a fourth mode different from the third mode,

in the third mode, the determined material of the real object corresponds to a transparent material, and

in the fourth mode, the determined material of the real object corresponds to a specific material different from the transparent material.

12 . The image processing apparatus according to claim 9 , wherein

the CPU is further configured to determine, based on the second sensing data, a polarized state of light, and

the light is reflected from the real object.

13 . The image processing apparatus according to claim 9 , wherein the CPU is further configured to:

determine the real object based on the second sensing data, wherein the real object corresponds to a part of a human body; and

control, based on the determined real object, the display process to superimpose the virtual object on the real object.

14 . The image processing apparatus according to claim 1 , wherein the CPU is further configured to generate, based on signals generated from a sensor, the first sensing data and the second sensing data.

15 . An image processing method, comprising:

in an image processing apparatus:

acquiring first sensing data that corresponds to image sensing data;

acquiring second sensing data different from the first sensing data;

extracting, from the second sensing data, first distance information associated with a real object;

augmenting a reality space based on each of the first sensing data and the second sensing data, wherein the reality space includes the real object;

superimposing a virtual object on the real object based on each of the second sensing data and the extracted first distance information;

controlling a display process for displaying the augmented reality space, wherein the displayed augmented reality space includes the superimposed virtual object on the real object;

determining, based on at least one of the first sensing data or the second sensing data, a specific portion of a user in the displayed augmented reality space;

switching, based on the determination of the specific portion of the user in the displayed augmented reality space, between a first mode and a second mode;

accepting one of a first operation or a second operation based on the switch between the first mode and the second mode, wherein

the first mode is associated with

the accepted first operation for the virtual object, and

the determination of the specific portion of the user in the displayed augmented reality space, and

the mode is associated with the accepted second operation for the virtual object; and

performing an occlusion process associated with the virtual object, wherein the occlusion process is based on

the accepted first operation for the virtual object, and

second distance information associated with the determined specific portion.

16 . An image processing system, comprising:

a first sensor configured to acquire first sensing data that corresponds to image sensing data;

a second sensor configured to acquire second sensing data different from the first sensing data; and

a central processing unit (CPU) configured to:

extract, from the second sensing data, first distance information associated with a real object;

augment a reality space based on each of the first sensing data and the second sensing data, wherein the reality space includes the real object;

superimpose a virtual object on the real object based on each of the second sensing data and the extracted first distance information;

control a display process to display the augmented reality space, wherein the displayed augmented reality space includes the superimposed virtual object on the real object;

determine, based on at least one of the first sensing data or the second sensing data, a specific portion of a user in the displayed augmented reality space;

switch, based on the determination of the specific portion of the user in the displayed augmented reality space, between a first mode and a second mode;

accept one of a first operation or a second operation based on the switch between the first mode and the second mode, wherein

the first mode is associated with

the accepted first operation for the virtual object, and

the determination of the specific portion of the user in the displayed augmented reality space, and

the second mode is associated with the accepted second operation for the virtual object; and

perform an occlusion process associated with the virtual object, wherein the occlusion process is based on

the accepted first operation for the virtual object, and

second distance information associated with the determined specific portion.

Assignments (1)

Priority Claims (1)

Continuity (1)

References Cited (14)