Lidar-Based Immersive 3D Reality Capture Systems, And Related Methods And Apparatus

1 . A reality capture system comprising:

a set of LiDAR sensors disposed around an environment and configured to capture one or more events occurring within the environment;

a corresponding set of cameras disposed around the environment, wherein each camera is mounted on a same gimbal with a corresponding LiDAR sensor and has a same optical axis as the corresponding LiDAR sensor;

a base station viewpoint generator coupled to the set of LiDAR sensors and the cameras to generate a video feed based on data received from the LiDAR sensors and the cameras; and

a virtual reality device coupled to the base station viewpoint generator to receive and display the video feed generated by the base station viewpoint generator.

2 . The reality capture system of claim 1 , further comprising:

at least one additional LiDAR sensor and one additional camera disposed within the environment, to capture the one or more events occurring within the environment.

3 . The reality capture system of claim 1 , wherein the LiDAR sensors and the cameras are disposed at fixed locations that remain unchanged during the one or more events.

4 . The reality capture system of claim 1 , wherein each LiDAR sensor comprises a respective CMOS sensor-based receiver.

5 . The reality capture system of claim 1 , further comprising an edge device coupled to the virtual reality device and configured to capture and track head movements of a user of the virtual reality device.

6 . A reality capture method comprising:

receiving a set of point cloud data captured by a corresponding set of LiDAR sensors based on scans of a physical environment;

processing the set of point cloud data to generate a first representation of the physical environment as a point cloud image or video; and

integrating the first representation of the physical environment into a virtual environment for presenting to a user.

7 . The reality capture method of claim 6 , further comprising:

receiving a set of imaging data representing images of the physical environment captured by a corresponding set of cameras disposed at the same respective locations as the set of LiDAR sensors;

processing the set of imaging data to generate a second representation of the physical environment as a photorealistic image or video; and

integrating the second representation of the physical environment into the virtual environment for presenting to the user.

8 . The reality capture method of claim 7 , further comprising:

before integrating the first representation or the second representation of the physical environment into the virtual environment, combining the first and second representations of the physical environment to generate a third representation of the physical environment; and

integrating the third representation of the physical environment into the virtual environment for presenting to the user.

9 . The reality capture method of claim 6 , further comprising:

performing a fusion and/or alignment of the set of point cloud data captured from different LiDAR components when generating the first representation of the physical environment.

10 . The reality capture method of claim 9 , wherein performing the fusion and/or alignment comprises:

detecting one or more identifiable features from each of the set of point cloud data; and

fusing the set of point cloud data based on an alignment of the one or more identifiable features from each of the set of point cloud data.

11 . The reality capture method of claim 6 , further comprising:

performing a temporal calibration for the set of point cloud data captured from different LiDAR sensors when generating the first representation of the physical environment.

12 . The reality capture method of claim 11 , wherein the temporal calibration is performed following a precision time protocol or generalized precision time protocol.

13 . The reality capture method of claim 12 , wherein the temporal calibration is performed by enabling each of the LiDAR sensors to listen to a shared listening station or listen to at least one other of the LiDAR sensors.

14 . The reality capture method of claim 6 , further comprising:

identifying a first member from the first representation of the physical environment; and

replacing the first member with a first avatar when presenting the first representation to the user.

15 . The reality capture method of claim 14 , further comprising:

identifying a second member from the first representation of the physical environment; and

replacing the second member with a second avatar when presenting the first representation to the user.

16 . The reality capture method of claim 15 , wherein the first avatar and the second avatar are different avatars.

17 . The reality capture method of claim 9 , further comprising:

adding a user profile to the identified first member when presenting the first representation to the user.

18 . The reality capture method of claim 9 , further comprising:

adding a sound effect based on a user activity of the first member occurring in the physical environment when presenting the first representation to the user.

19 . A reality capture system for generating an immersive virtual reality experience, the system comprising:

a set of reality capture units disposed at a plurality of vantage points around a physical environment, each reality capture unit comprising: a light detection and ranging (lidar) sensor configured to generate point cloud data of the environment, and a camera coaxially mounted on a common gimbal with the lidar sensor and configured to generate image data of the environment;

a base station viewpoint generator communicatively coupled to the set of reality capture units, the generator configured to:

receive the point cloud data and the image data from the set of reality capture units;

perform a calibration process to spatially and temporally align the point cloud data and the image data from the plurality of vantage points;

generate a fused, three-dimensional (3D) point cloud video based on the aligned data; and

transmit the fused 3D point cloud video; and

a virtual reality (VR) device configured to receive and display the fused 3D point cloud video, thereby enabling a user to view the physical environment from a virtual perspective.

20 . The reality capture system of claim 19 , wherein the base station viewpoint generator is further configured to:

identify a person within the physical environment based on the aligned data; and

replace a representation of the identified person with a corresponding virtual avatar in the fused 3D point cloud video.