IP Library Granted Patent US 12,423,907
Granted Patent B2
US 12,423,907 · App. 17/850,077 · Granted Sep 23, 2025

Displaying a rendered volumetric representation according to different display modes

Inventors: David Lui (San Jose, CA); Xiao Jin Yu (Sunnyvale, CA); Tyler L. Casella (San Mateo, CA); Hon-Ming Chen (San Jose, CA); Shuai Song (Fremont, CA)
Assignee: APPLE INC.
G06T15/08G06T19/006
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,423,907
App. No.
17/850,077
Granted
Sep 23, 2025
Kind
B2
Abstract

A method is performed at an electronic device including one or more processors, a non-transitory memory, and a display. The method includes rendering a first volumetric object in order to generate first object data. The method includes displaying, on the display, the first object data according to a first display mode. The first display mode includes displaying the first object data within a two-dimensional (2D) content region. The method includes detecting a request to change from the first display mode to a second display mode. The method includes, in response to detecting the request, displaying, on the display, the first object data according to the second display mode. The second display mode includes displaying the first object data within a representation of a physical environment.

Claims (27)

1. A method comprising: at an electronic device including one or more processors, a non-transitory memory, and a display: rendering a three-dimensional (3D) object model of a first object in order to generate two-dimensional (2D) image data of the first object and a 3D object model of a second object in order to generate 2D image data of the second object; displaying, on the display, the 2D image data of the first object and the 2D image data of the second object according to a first display mode, wherein the first display mode includes displaying the 2D image data of the first object and the 2D image data of the second object within a 2D content region, and wherein in the first display mode the display does not include a representation of a physical environment; while the 2D image data of the first object is displayed according to the first display mode: receiving a manipulation input via one or more input devices; and rendering the 3D object model of the first object based on the manipulation input to generate updated 2D image data of the first object; displaying, on the display, the updated 2D image data of the first object according to the first display mode; and in response to detecting a request to change from the first display mode to a second display mode, displaying, on the display, the updated 2D image data of the first object according to the second display mode, wherein the second display mode includes displaying the updated 2D image data of the first object within the representation of the physical environment and ceasing to display the 2D content region.

2. The method of claim 1 , wherein displaying the 2D image data of the first object according to the first display mode includes:

obtaining 2D content data that indicates the 2D content region;

combining the 2D content data with the 2D image data of the first object in order to generate first display data, wherein the first display data indicates the first object within the 2D content region; and

sending the first display data to the display.

3. The method of claim 1 , wherein the electronic device includes an image sensor, and wherein displaying the updated 2D image data of the first object according to the second display mode includes:

capturing, via the image sensor, image data characterizing the physical environment, wherein the image data indicates the representation of the physical environment;

compositing the image data with the updated 2D image data of the first object in order to generate second display data, wherein the second display data indicates the first object within the representation of the physical environment; and

sending the second display data to the display.

4. The method of claim 2 , wherein the 2D content data further indicates 2D content that is within the 2D content region.

5. The method of claim 4 , wherein displaying the 2D image data of the first object within the 2D content region includes displaying the 2D image data of the first object in-line with respect to the 2D content.

6. The method of claim 1 , further comprising:

in further response to detecting the request, processing the updated 2D image data of the first object; and

detecting completion of the processing before displaying the updated 2D image data of the first object according to the second display mode.

7. The method of claim 1 , wherein the one or more input devices includes a positional sensor that receives the manipulation input, and wherein the positional sensor detects a positional change of the electronic device based on the manipulation input.

8. The method of claim 1 , wherein the manipulation input is spatially directed to the 2D image data of the first object.

9. The method of claim 1 , wherein detecting the request includes receiving an input directed to the 2D image data of the first object displayed within the 2D content region.

10. The method of claim 1 , wherein the second display mode corresponds to an augmented reality (AR) environment or mixed reality (MR) environment.

11. A system comprising: a display; non-transitory memory; and one or more processors to: render a three-dimensional (3D) object model of a first object in order to generate two-dimensional (2D) image data of the first object and a 3D object model of a second object in order to generate 2D image data of the second object; display, on the display, the 2D image data of the first object and the 2D image data of the second object according to a first display mode, wherein the first display mode includes displaying the 2D image data of the first object and the 2D image data of the second object within a 2D content region, and wherein in the first display mode the display does not include a representation of a physical environment; while the 2D image data of the first object is displayed according to the first display, mode: receive a manipulation input via one or more input devices; and render the 3D object model of the first object based on the manipulation input to generate updated 2D image data of the first object display, on the display, the updated 2D image data of the first object according to the first display mode; and in response to detecting a request to change from the first display mode to the second display mode, display, on the display, the updated 2D image of the first object according to the second display mode, wherein the second display mode includes displaying the updated 2D image of the first object within the representation of the physical environment and ceasing to display the 2D content region.

12. The system of claim 11 , wherein the one or more processors are to display the 2D image data of the first object according to the first display mode by combining the 2D image data of the first object with 2D content data, and wherein the one or more processors are to display the updated 2D image data of the first object according to the second display mode by compositing the updated 2D image data of the first object with image data of the physical environment.

13. The system of claim 12 , wherein the system comprises an image sensor that captures the image data of the physical environment.

14. The system of claim 11 , wherein the one or more processors are further to process the updated 2D image data of the first object based on detection of the request.

15. A non-transitory computer readable storage medium storing one or more programs, the one or more programs comprising instructions, which, when executed by an electronic device including a display, cause the electronic device to: render a three-dimensional (3D) object model of a first object in order to generate two-dimensional (2D) image data of the first object and a 3D object model of a second object in order to generate 2D image data of the second object; display, on the display, the 2D image data of the first object and the 2D image data of the second object according to a first display mode, wherein the first display mode includes displaying the 2D image data of the first object and the 2D image data of the second object within a 2D content region, and wherein in the first display mode the display does not include a representation of a physical environment; while the 2D image data of the first object is displayed according to the first display mode: receive a manipulation input via one or more input devices; render the 3D object model of the first object to generate updated 2D image data of the first object before detecting a request to change from the first display mode to a second display mode; display, on the display, the updated 2D image data of the first object according to the first display mode: and in response to detecting a request to change from the first display mode to the second display mode, display, on the display, the updated 2D image data of the first object according to the second display mode, wherein the second display mode includes displaying the updated 2D image data of the first object within the representation of the physical environment and ceasing to display the 2D content region.

16. The method of claim 1 , further comprising, in response to detecting the request to change from the first display mode to the second display mode, activating an image sensor that captures image data corresponding to the representation of the physical environment, wherein the image sensor is inactive in the first display mode.

17. The method of claim 1 , wherein the manipulation input corresponds to a rotation of the first object.

18. The system of claim 11 , wherein the one or more input devices includes a positional sensor that receives the manipulation input, and wherein the positional sensor detects a positional change of the system based on the manipulation input.

19. The system of claim 11 , wherein the manipulation input is spatially directed to the 2D image of the first object.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jun 28, 2022
From: LUI, DAVID; YU, XIAO JIN; CASELLA, TYLER L.; CHEN, HON-MING; SONG, SHUAI
To: APPLE INC.
Reel/Frame 060336/0941 →
Continuity (2)
Provisional Application 63239332 · Aug 31, 2021
Related Publication 20230065077A1 · Mar 2, 2023
References Cited (7)
US 10163261B2 · Bell et al. · 2018 [cited by applicant]
US 10643387B2 · Sakamoto et al. · 2020 [cited by applicant]
US 10777017B1 · Saiger · 2020 [cited by examiner]
US 20030231191A1 · Glen · 2003 [cited by examiner]
US 20180203951A1 · Jovanovic · 2018 [cited by examiner]
US 20210081579A1 · Jovanovic · 2021 [cited by applicant]
US 20210183154A1 · Canada · 2021 [cited by examiner]