IP Library › Granted Patent US 12,283,006
Granted Patent B2
US 12,283,006 · App. 17/911,316 · Granted Apr 22, 2025

Information processing device and information processing method

Inventors: Tomoya Ishikawa (Tokyo, JP); Gaku Narita (Tokyo, JP); Tomu Tahara (Tokyo, JP); Takashi Seno (Tokyo, JP)
Assignee: SONY GROUP CORPORATION
G06T19/006G06F3/0484
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,283,006
App. No.
17/911,316
Granted
Apr 22, 2025
Kind
B2
Abstract

A content display device that functions as an information processing device includes an acquisition unit, a determination unit, and a display control unit. The acquisition unit acquires template data defining relative relationships between a first real object, a second real object, and a virtual object. The determination unit determines the relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by the sensor. The display control unit controls the display device to arrange the virtual object on or near the first real object so as to face the second real object, based on the relative relationships and the template data.

Claims (34)

1. An information processing device, comprising:

a processor configured to:

acquire template data defining relative relationships between a first real object, a second real object, and a virtual object, wherein the relative relationships include relative positional relationships between the first real object, the second real object, and the virtual object, and a first interactive action of the virtual object with respect to the first real object;

determine a relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by a sensor;

control a display device to arrange the virtual object on or near the first real object and facing the second real object, based on the relative relationships and the template data; and

control the display device not to arrange the virtual object based on a determination that, in the relative relationship, a distance between the first real object and the second real object is at least a specific distance.

2. The information processing device according to claim 1 , wherein,

after arranging the virtual object, the processor is further configured to execute a second interactive action of the virtual object with respect to the second real object.

3. The information processing device according to claim 2 , wherein the second interactive action includes sitting on the first real object, holding the first real object, and talking to the second real object.

4. The information processing device according to claim 1 ,

wherein based on a determination that the three-dimensional data includes a plurality of first real objects, the processor is further configured to arrange the virtual object on the first real object on which a third real object is not arranged,

each of the first real object and the third real object are included in the plurality of first real objects, and

the arrangement is based on the template data.

5. The information processing device according to claim 1 ,

wherein based on a determination that the three-dimensional data includes a plurality of first real objects, the processor is further configured to arrange the virtual object on one of the plurality of first real objects which is closer to the second real object.

6. The information processing device according to claim 1 , wherein the processor is further configured to:

determine a movement of the first real object; and

rearrange the virtual object on the first real object.

7. The information processing device according to claim 1 ,

wherein the processor is further configured to rearrange the virtual object on the first real object based on a determination that at least a certain time has elapsed since the first interactive action of the virtual object.

8. The information processing device according to claim 1 ,

wherein the processor is further configured to rearrange the virtual object on the first real object based on a determination that the virtual object performs the first interactive action with respect to the first real object within a certain time.

9. The information processing device according to claim 1 ,

wherein the processor is further configured to rearrange the virtual object on the first real object based on a determination that the virtual object has a positional relationship enabling a specific interactive action which is executed with respect to the second real object by switching a video content scene that includes the first real object, the second real object, and the virtual object.

10. The information processing device according to claim 1 , wherein the template data is scene graph data.

11. The information processing device according to claim 1 ,

wherein the processor is further configured to determine, from the three-dimensional data, the relative relationship between the first real object and the second real object based on scene graph data that represents the real space in an abstract manner.

12. The information processing device according to claim 1 ,

wherein the first real object is an object for seating or an object that can be held, the second real object is a person, and the virtual object is a person character that changes based on the relative relationship.

13. An information processing method, comprising:

acquiring template data defining relative relationships between a first real object, a second real object, and a virtual object, wherein the relative relationships include relative positional relationships between the first real object, the second real object, and the virtual object, and an interactive action of the virtual object with respect to the first real object or the second real object;

determining a relative relationship between the first real object and the second real object from three-dimensional data of a real space detected by a sensor;

controlling a display device to arrange the virtual object on or near the first real object and facing the second real object, based on the relative relationships and the template data; and

controlling the display device not to arrange the virtual object based on a determination that, in the relative relationship, a distance between the first real object and the second real object is at least a specific distance.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 13, 2022
From: ISHIKAWA, TOMOYA; NARITA, GAKU; TAHARA, TOMU; SENO, TAKASHI
To: SONY GROUP CORPORATION
Reel/Frame 061080/0033 →
Priority Claims (1)
JP 2020-051779 · Mar 23, 2020 · national
Continuity (1)
Related Publication 20230112368A1 · Apr 13, 2023
References Cited (15)
US 7768534B2 · Pentenrieder · 2010 [cited by examiner]
US 20140347391A1 · Keane · 2014 [cited by examiner]
US 20150302665A1 · Miller · 2015 [cited by examiner]
US 20190197785A1 · Tate-Gans · 2019 [cited by examiner]
US 20210042992A1 · Newman · 2021 [cited by examiner]
JP 2001273520A · 2001 [cited by applicant]
JP 2006072668A · 2006 [cited by applicant]
JP 2016516241A · 2016 [cited by applicant]
International Search Report and Written Opinion of PCT Application No. PCT/JP2021/004935, issued on May 11, 2021, 09 pages of ISRWO. [cited by applicant]
Takeshima, et al., “Integration and Visualization Method of Virtual Space by Scene Graph with Semantic Information”, Proceedings of the 14th Data Engineering Workshop, May 16, 2003. [cited by applicant]
Narita, et al., “PanopticFusion: Online Volumetric Semantic Mapping at the Level of Stuff and Things”, IEEE/RSJ, International Conference on Intelligent Robots and Systems, Sep. 9, 2019, 8 pages. [cited by applicant]
Grinvald, et al., “Volumetric Instance-Aware Semantic Mapping and 3D Object Discovery”, IEEE, Robotics and Automation Letters, Jul. 10, 2019, 8 pages. [cited by applicant]
Wang, et al., “Normalized Object Coordinate Space for Category-Level 6D Object Pose and Size Estimation”, IEEE, Conference on Computer Vision and Pattern Recognition, Jun. 23, 2019, 12 pages. [cited by applicant]
Xu, et al., “Scene Graph Generation by Iterative Message Passing”, IEEE, Conference on Computer Vision and Pattern Recognition, Apr. 12, 2017, 10 pages. [cited by applicant]
Savva, et al., “PiGraphs: Learning Interaction Snapshots from Observations”, ACM Transactions on Graphics, Jul. 24-28, 2016, 12 pages. [cited by applicant]