Granted Patent B2

US 12,254,797 · App. 18/906,950 · Granted Mar 18, 2025

Inventors: Kamran Qaderi (San Gabriel, CA); Jonathan Seamus Blackley (South Pasadena, CA); Watson Brent Boyett (Los Angeles, CA); Robin James Green (Duvall, WA); Stephen John Hart (San Juan Capistrano, CA); Robert Alan Hess (Mesa, AZ); Mark Anthony Loya (Temple City, CA); Benjamin Francis Neil (South Pasadena, CA); Jesus Manuel Caridad Ramirez (Altadena, CA); William Luke Snitzer (Los Angeles, CA)

Assignee: Pacific Light & Hologram, Inc.

G09G3/003G02F1/134309G02F1/13439G09G3/3413G09G3/36

Abstract

Claims (137)

1. A system comprising:

a hologram generation system configured to generate one or more digital holograms corresponding to a live scene that comprises one or more three-dimensional (3D) objects; and

a holographic display system configured to reconstruct the live scene in a 3D space based on the one or more digital holograms,

wherein the hologram generation system comprises:

one or more scene acquisition devices configured to capture visual data of the live scene from one or more views, the visual data comprising at least one of one or more images or one or more videos; and

a computing system configured to:

obtain primitive data associated with the live scene based on the captured visual data of the live scene; and

generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display of the holographic display system, and

wherein the computing system comprises a computing device coupled to the one or more scene acquisition devices and configured to:

generate a 3D representation of the live scene based on the captured visual data of the live scene; and

obtain primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene comprises the primitive data of the 3D representation of the live scene.

2. The system of claim 1 , wherein the computing device is configured to generate the 3D representation of the live scene based on the captured visual data of the live scene using a 3D rendering algorithm.

3. The system of claim 2 , wherein the one or more scene acquisition devices are configured to transmit sequential visual data of the live scene to the computing device in a time period, the sequential visual data comprising first visual data and second visual data sequential to the first visual data, and

wherein the computing device is configured to:

generate a first 3D representation of the live scene based on the first visual data of the live scene using the 3D rendering algorithm, and

generate a second 3D representation of the live scene by updating the first 3D representation of the live scene based on a difference between the first visual data and the second visual data using the 3D rendering algorithm.

4. The system of claim 1 , wherein the computing device is configured to:

load the 3D representation of the live scene into a 3D simulation application, and

obtain the primitive data of the 3D representation of the live scene based on an output of the 3D simulation application that is associated with the 3D representation of the live scene.

5. The system of claim 4 , wherein the primitive data of the 3D representation of the live scene comprises:

data of a plurality of primitives corresponding to the 3D representation of the live scene, the data comprising primitive data of each primitive of the plurality of primitives, a primitive comprising at least one vertex, primitive data of the primitive comprising data of the at least one vertex.

6. The system of claim 5 , wherein the primitive data of the primitive comprises at least one of:

a primitive identifier of the primitive,

at least one vertex identifier of the at least one vertex,

coordinate information of the primitive in a 3D coordinate system,

color information of the primitive,

texture coordinate information of the primitive,

shading information for the primitive,

viewpoint dependent shading information associated with the primitive, or

occlusion information of the primitive.

7. The system of claim 5 , wherein the computing system comprises a processing device coupled to the computing device and configured to:

for each primitive of the plurality of primitives, determining an electromagnetic (EM) field contribution to each of a plurality of display elements of the display based on primitive data of the primitive; and

for each of the plurality of display elements of the display, generating a sum of the EM field contributions of the plurality of primitives to the display element,

wherein the digital hologram comprises the sums of the EM field contributions for the plurality of display elements of the display.

8. The system of claim 7 , wherein the computing device is configured to: generate the primitive data of the 3D representation of the live scene based on the output of 3D simulation application using an application programming interface (API), and

wherein the computing device is configured to execute the API to:

for each of a plurality of vertices of the plurality of primitives, associate a respective vertex identifier of the vertex with respective vertex data of the vertex, and store the association between the respective vertex identifier and the respective vertex data of the vertex in a memory of the computing device; and

for each of the plurality of primitives, associate a respective primitive identifier of the primitive with one or more respective vertex identifiers of one or more vertices of the primitive in the memory, and store an association between the respective primitive identifier and the one or more respective vertex identifiers for the primitive in the memory.

9. The system of claim 8 , wherein the computing device is configured to execute the API to:

determine primitive identifiers of multiple primitives associated with a command instruction,

determine vertex identifiers associated with the primitive identifiers, and

transmit a command including the command instruction, the vertex identifiers associated with the primitive identifiers, and the primitive identifiers of the multiple primitives, to the processing device, without the primitive data of the plurality of primitives,

wherein the command indicates drawing the multiple primitives according to the command instruction and based on at least one of the primitive identifiers of the multiple primitives or the vertex identifiers associated with the primitive identifiers.

10. The system of claim 9 , wherein the processing device comprises:

a command processor configured to:

receive the command from the computing device, and

process the command to obtain primitive data of the multiple primitives from the computing device based on the command;

a plurality of computing units configured to calculate an electromagnetic (EM) field contribution of each of the multiple primitives to each of the plurality of display elements based on the primitive data of the multiple primitives; and

an accumulator configured to:

accumulate EM field contributions of the multiple primitives to each of the plurality of display elements.

11. The system of claim 10 , wherein the command processor, the plurality of computing units, and the accumulator are connected in series, and the plurality of computing units are connected in parallel between the command processor and the accumulator.

12. The system of claim 7 , wherein the digital hologram is a complex-valued hologram, a phase hologram, or an amplitude hologram.

13. The system of claim 1 , wherein the holographic display system comprises a driving device coupled to the display and configured to:

generate respective modulation control signals for a plurality of display elements of the display based on the digital hologram corresponding to the live scene.

14. The system of claim 13 , wherein the digital hologram is a complex-valued hologram, and the driving device is configured to:

convert the complex-valued hologram to a phase-only hologram, and

generate the respective modulation control signals for the plurality of display elements based on the phase-only hologram.

15. The system of claim 13 , wherein the holographic display system further comprises an illuminator,

wherein the driving device is configured to: transmit an illumination control signal to the illuminator to activate the illuminator to illuminate light on the display such that the light is caused by modulated display elements of the display to form a volumetric light field corresponding to the live scene, and

wherein the driving device is configured to:

output the respective modulation control signal to each display element of the plurality of display elements, in coordination with transmitting the illumination control signal to the illuminator.

16. The system of claim 15 , wherein the driving device is configured to:

sequentially output a first modulation control signal to modulate the display with a first digital hologram associated with a first color during a first time period, and a second modulation control signal to modulate the display with a second digital hologram associated with a second color during a second time period sequential to the first time period; and

sequentially output a first illumination control signal to activate the illuminator to turn on a first coherent light element to emit light with a first color during the first time period, and a second illumination control signal to activate the illuminator to turn on a second coherent light element to emit light with the second color during the second time period.

17. The system of claim 1 , wherein the hologram generation system is configured to generate sequential digital holograms corresponding to the live scene, and

wherein the holographic display system is configured to continuously reconstruct the live scene in the 3D space based on the sequential digital holograms.

18. A method comprising:

capturing visual data of a live scene from one or more views, the live scene comprising one or more three-dimensional (3D) objects, the visual data comprising at least one of one or more images or one or more videos;

obtaining primitive data associated with the live scene based on the captured visual data of the live scene;

generating one or more digital holograms corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display; and

reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms,

wherein obtaining the primitive data associated with the live scene based on the captured visual data of the live scene comprises:

generating a 3D representation of the live scene based on the captured visual data of the live scene, and

obtaining primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene comprises the primitive data of the 3D representation of the live scene.

19. The method of claim 18 , wherein generating the 3D representation of the live scene based on the captured visual data of the live scene comprises:

processing the captured visual data of the live scene using a 3D rendering algorithm to generate the 3D representation of the live scene, and

wherein the method comprises:

generating sequential visual data of the live scene in a time period, the sequential visual data comprising first visual data and second visual data sequential to the first visual data;

generating a first 3D representation of the live scene based on the first visual data of the live scene using the 3D rendering algorithm; and

generating a second 3D representation of the live scene by updating the first 3D representation of the live scene based on a difference between the first visual data and the second visual data using the 3D rendering algorithm.

20. The method of claim 18 , wherein obtaining the primitive data of the 3D representation of the live scene based on the 3D representation of the live scene comprises:

loading the 3D representation of the live scene into a 3D simulation application, and

obtaining the primitive data of the 3D representation of the live scene based on an output of the 3D simulation application that is associated with the 3D representation of the live scene.

21. The method of claim 18 , wherein the primitive data of the 3D representation of the live scene comprises:

data of a plurality of primitives corresponding to the 3D representation of the live scene, the data comprising primitive data of each primitive of the plurality of primitives, a primitive comprising at least one vertex, primitive data of the primitive comprising data of the at least one vertex, and

wherein generating the one or more digital holograms corresponding to the live scene comprises:

for each primitive of the plurality of primitives, determine an electromagnetic (EM) field contribution to each of a plurality of display elements of the display based on primitive data of the primitive; and

for each of the plurality of display elements of the display, generate a sum of the EM field contributions of the plurality of primitives to the display element,

wherein a digital hologram comprises the sums of the EM field contributions for the plurality of display elements of the display.

22. The method of claim 21 , comprising:

for each of a plurality of vertices of the plurality of primitives, associating a respective vertex identifier of the vertex with respective vertex data of the vertex, and storing the association between the respective vertex identifier and the respective vertex data of the vertex in a memory;

for each of the plurality of primitives, associating a respective primitive identifier of the primitive with one or more respective vertex identifiers of one or more vertices of the primitive in the memory, and storing an association between the respective primitive identifier and the one or more respective vertex identifiers for the primitive in the memory;

determining primitive identifiers of multiple primitives associated with a command instruction;

determining vertex identifiers associated with the primitive identifiers; and

generating a command including the command instruction, the vertex identifiers associated with the primitive identifiers, and the primitive identifiers of the multiple primitives,

wherein the command indicates drawing the multiple primitives according to the command instruction and based on at least one of the primitive identifiers of the multiple primitives or the vertex identifiers associated with the primitive identifiers.

23. The method of claim 22 , comprising:

processing the command to obtain primitive data of the multiple primitives based on the command;

calculating an electromagnetic (EM) field contribution of each of the multiple primitives to each of the plurality of display elements based on the primitive data of the multiple primitives; and

accumulating EM field contributions of the multiple primitives to each of the plurality of display elements.

24. The method of claim 18 , wherein reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms comprises:

generating respective modulation control signals for a plurality of display elements of the display based on a digital hologram corresponding to the live scene.

25. The method of claim 24 , wherein the digital hologram is a complex-valued hologram, and the method comprises:

converting the complex-valued hologram to a phase-only hologram, and

generating the respective modulation control signals for the plurality of display elements based on the phase-only hologram.

26. The method of claim 24 , wherein reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms comprises:

transmitting an illumination control signal to an illuminator to activate the illuminator to illuminate light on the display such that the light is caused by modulated display elements of the display to form a volumetric light field corresponding to the live scene; and

outputting the respective modulation control signal to each display element of the plurality of display elements, in coordination with transmitting the illumination control signal to the illuminator.

27. The method of claim 18 , comprising:

generating sequential digital holograms corresponding to the live scene based on captured sequential visual data of the live scene; and

continuously reconstructing the live scene in the 3D space based on the sequential digital holograms.

28. A system for generating digital holograms corresponding to live scenes, the system comprising:

one or more scene acquisition devices configured to capture visual data of a live scene from one or more views, the live scene comprising one or more three-dimensional (3D) objects, the visual data comprising at least one of one or more images or one or more videos; and

a computing system configured to:

obtain primitive data associated with the live scene based on the captured visual data of the live scene; and

generate a digital hologram corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display,

wherein the computing system comprises a computing device coupled to the one or more scene acquisition devices and configured to:

generate a 3D representation of the live scene based on the captured visual data of the live scene; and

obtain primitive data of the 3D representation of the live scene based on the 3D representation of the live scene, wherein the primitive data associated with the live scene comprises the primitive data of the 3D representation of the live scene.

29. A method comprising:

capturing visual data of a live scene from one or more views, the live scene comprising one or more three-dimensional (3D) objects, the visual data comprising at least one of one or more images or one or more videos;

obtaining primitive data associated with the live scene based on the captured visual data of the live scene;

generating one or more digital holograms corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display; and

reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms,

wherein reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms comprises: generating respective modulation control signals for a plurality of display elements of the display based on a digital hologram corresponding to the live scene, and

wherein the digital hologram is a complex-valued hologram, and the method comprises:

converting the complex-valued hologram to a phase-only hologram, and

generating the respective modulation control signals for the plurality of display elements based on the phase-only hologram.

30. A method comprising:

capturing visual data of a live scene from one or more views, the live scene comprising one or more three-dimensional (3D) objects, the visual data comprising at least one of one or more images or one or more videos;

obtaining primitive data associated with the live scene based on the captured visual data of the live scene;

generating one or more digital holograms corresponding to the live scene based on the primitive data associated with the live scene and display element information of a display; and

reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms,

wherein reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms comprises: generating respective modulation control signals for a plurality of display elements of the display based on a digital hologram corresponding to the live scene, and

wherein reconstructing the live scene in a 3D space by modulating the display with the one or more digital holograms comprises:

transmitting an illumination control signal to an illuminator to activate the illuminator to illuminate light on the display such that the light is caused by modulated display elements of the display to form a volumetric light field corresponding to the live scene; and

outputting the respective modulation control signal to each display element of the plurality of display elements, in coordination with transmitting the illumination control signal to the illuminator.

Assignments (1)

Continuity (6)

References Cited (272)

Cited By (3)

US 12,417,722 US 12,561,907 US 12,579,919