System and method for immersive and interactive multimedia generation

1. A method, implemented by an immersive and interactive multimedia generation system, comprising:

projecting, by a first component of the system, a plurality of patterns to a physical environment in which a second component of the system is located;

acquiring, by a first detector of the second component of the system, a first image and a second image of at least a part of the physical environment, the first and second images including, respectively, first and second pixel data corresponding to at least some of the plurality of projected patterns; and

determining, by the second component of the system, a change in at least one of an orientation or a position of the first detector relative to the plurality of projected patterns based on a relationship between the first and the second pixel data, the first component and the plurality of projected patterns being static relative to the physical environment and the first detector being movable relative to the physical environment;

acquiring, by a second detector of the second component of the system, a third image and a fourth image of the at least a part of the physical environment; and

aligning, by the second component of the system, the first image with the third image, and the second image with the fourth image to generate a rendering of the at least a part of the physical environment,

wherein aligning the first image with the third image, and the second image with the fourth image comprises reflecting, at a location, the plurality of patterns before they are acquired by the first detector, and letting a visual light pass through, at the same location, before the visual light is acquired by the second detector.

2. The method of claim 1 , wherein the physical environment is an indoor environment.

3. The method of claim 1 , wherein:

the first component is disposed at a fixed position; and

the second component is attached to a user moving in the physical environment.

4. The method of claim 1 , wherein the patterns are infrared patterns and the second component has no prior knowledge of the patterns.

5. The method of claim 1 , further comprising:

detecting a hand gesture from the first and second images;

determining a user instruction associated with the hand gesture; and

determining a rendering of the physical environment based on the determined user instruction.

6. The method of claim 1 , wherein the patterns comprise a turning angle.

7. The method of claim 6 , wherein the patterns are “L” shaped.

8. The method of claim 1 , further comprising:

monitoring, in real-time, positions of a user of the system relative to objects in the physical environment;

determining, based on the monitored positions, if the user will collide with one of the objects in the physical environment; and

providing instructions to display a warning overlaying a rendering of the physical environment to the user.

9. The method of claim 1 , further comprising:

monitoring, in real-time, objects in the physical environment;

generating a 3D model of the physical environment, the 3D model including a position of a real world object; and

providing instructions to display a virtual object at the position of the real world object in a rendering of the physical environment.

10. An immersive and interactive multimedia generation system, comprising:

a first apparatus configured to project a plurality of patterns to a physical environment in which a second component of the system is located; and

a second apparatus configured to:

acquire a first image and a second image of at least a part of the physical environment by a first detector of the second apparatus, the first and second images including, respectively, first and second pixel data corresponding to at least some of the plurality of projected patterns, and

determine a change in at least one of an orientation or a position of the second component relative to the plurality of projected patterns based on a relationship between the first and the second pixel data, the first component and the plurality of projected patterns being static relative to the physical environment and the first detector being movable relative to the physical environment;

acquire a third image and a fourth image of the at least a part of the physical environment by a second detector of the second apparatus; and

align the first image with the third image, and the second image with the fourth image to generate a rendering of the at least a part of the physical environment,

wherein the second apparatus is configured to align the first image with the third image, and the second image with the fourth image by reflecting, at a location, the plurality of patterns before they are acquired by the first detector of the second apparatus, and letting a visual light pass through, at the same location, before the visual light is acquired by the second detector of the second apparatus.

11. The system of claim 10 , wherein the physical environment is an indoor environment.

12. The system of claim 10 , wherein:

the first apparatus is disposed at a fixed position; and

the second apparatus is attached to a user moving in the physical environment.

13. The system of claim 10 , wherein the patterns are infrared patterns and the second apparatus has no prior knowledge of the patterns.

14. The system of claim 10 , wherein the second apparatus is further configured to:

detect a hand gesture from the first and second images;

determine a user instruction associated with the hand gesture; and

determine a rendering of the physical environment based on the determined user instruction.

15. The system of claim 10 , wherein the patterns comprise a turning angle.

16. The system of claim 15 , wherein the patterns are “L” shaped.

17. The system of claim 10 , wherein the second apparatus is further configured to:

monitor, in real-time, positions of a user of the system relative to objects in the physical environment;

determine, based on the monitored positions, if the user will collide with one of the objects in the physical environment; and

provide instructions to display a warning overlaying a rendering of the physical environment to the user.

18. The system of claim 10 , wherein the second apparatus is further configured to:

monitor, in real-time, objects in the physical environment;

generate a 3D model of the physical environment, the 3D model including a position of a real world object; and

provide instructions to display a virtual object at the position of the real world object in a rendering of the physical environment.