Dynamic multi-dimensional media content projection

1 . A method, executed by a processor in an electronic device, comprising:

acquiring image data that includes two-dimensional (2D) images depicting a view of a real-world location;

detecting one or more 3D polygons corresponding to one or more objects on a ground region of the 2D images;

acquiring a three-dimensional (3D) surface model of the real-world location;

segmenting the 3D surface model into a plurality of 3D segments comprising a ground segment and one or more non-ground segments; and

extracting the ground segment of the 3D surface model by removing the one or more non-ground segments from the plurality of 3D segments;

determining, by applying a ray casting operation on the 2D images and the ground segment of the 3D surface model, correspondence information between 2D pixel coordinates in the 2D images and 3D surface coordinates of the ground segment; and

projecting, based on the correspondence information, the one or more 3D polygons corresponding to the one or more objects onto the 3D surface model;

wherein the correspondence information includes a table that includes the 2D pixel coordinates and the 3D surface coordinates corresponding to the 2D pixel coordinates.

2 . The method according to claim 1 , further comprising:

estimating initial camera pose parameters corresponding to an initial view of the 3D surface model in a 3D space; and

optimizing the initial camera pose parameters to align a view of the 3D surface model to a view of the 2D images.

3 . The method according to claim 1 , further comprising:

generating vertex information that includes a mapping of the 3D surface coordinates with vertices of the one or more 3D polygons; and

generating edge information that includes a mapping of the vertices to edges of the one or more 3D polygons;

wherein the projection of the one or more 3D polygons is performed further based on the vertex information and the edge information.

4 . The method according to claim 3 , wherein the vertex information includes a primary table that includes a vertex ID, and a polygon ID, a vertex type, and the 3D surface coordinates, and

the edge information includes a secondary table that includes the polygon ID, an edge ID, and IDs of edge vertices.

5 . The method according to claim 3 , wherein the vertex type includes one of a ground or an object.

6 . The method according to claim 1 , wherein each 3D polygon of the one or more 3D polygons encloses a corresponding object of the plurality of objects in a 3D space of the 3D surface model.

7 . The method according to claim 1 , further comprising:

retrieving, from a database, historical data associated with a plurality of 2D polygons corresponding to a plurality of objects in a set of images,

wherein each 3D polygon of the plurality of 3D polygon encloses a corresponding object of the plurality of objects;

training a polygon estimation model based on the historical data;

estimating one or more 3D polygons corresponding to the detected one or more objects in the 2D images, based on the trained polygon estimation model; and

projecting the detected one or more 3D polygons onto the ground region of the 2D images based on a determination that a difference between the estimated one or more 3D polygons and the detected one or more 3D polygons is less than a predefined threshold.

8 . The method according to claim 7 , further comprising projecting the estimated one or more 3D polygons onto the ground region of the 2D images based on a determination that a difference between the estimated one or more 3D polygons and the detected one or more 3D polygons is greater than the predefined threshold.

9 . The method according to claim 1 , wherein the determining of the correspondence information includes:

extracting an image size of the 2D images;

selecting a 2D coordinate from 2D pixel coordinates;

applying the ray casting operation on the selected 2D coordinate to estimate a corresponding 3D surface coordinate of the 3D surface coordinates on the ground segment in a 3D space; and

determine the correspondence information based on the 2D pixel coordinate and the 3D surface coordinate.

10 . The method according to claim 1 , wherein the acquiring of the 3D surface model further includes:

capturing 3D data via an image capturing device;

generating a 3D point cloud based on the 3D data; and

converting the point cloud data into the 3D surface model.

11 . One or more non-transitory computer-readable storage media configured to store instructions that, in response to being executed, cause an electronic device to perform operations, the operations comprising:

acquiring image data that includes two-dimensional (2D) images depicting a view of a real-world location;

detecting one or more 3D polygons corresponding to one or more objects on a ground region of the 2D images;

acquiring a three-dimensional (3D) surface model of the real-world location;

segmenting the 3D surface model into a plurality of 3D segments comprising a ground segment and one or more non-ground segments; and

extracting the ground segment of the 3D surface model by removing the one or more non-ground segments from the plurality of 3D segments;

determining, by applying a ray casting operation on the 2D images and the ground segment of the 3D surface model, correspondence information between 2D pixel coordinates in the 2D images and 3D surface coordinates of the ground segment; and

projecting, based on the correspondence information, the one or more 3D polygons corresponding to the one or more objects onto the 3D surface model;

wherein the correspondence information includes a table that includes the 2D pixel coordinates and the 3D surface coordinates corresponding to the 2D pixel coordinates.

12 . The one or more non-transitory computer-readable storage media according to claim 11 , wherein the operations further comprises:

estimating initial camera pose parameters corresponding to an initial view of the 3D surface model in a 3D space; and

optimizing the initial camera pose parameters to align a view of the 3D surface model to a view of the 2D images.

13 . The one or more non-transitory computer-readable storage media according to claim 11 , wherein the operations further comprises:

generating vertex information that includes a mapping of the 3D surface coordinates with vertices of the one or more 3D polygons;

generating edge information that includes a mapping of the vertices to edges of the one or more 3D polygons, wherein the projection is performed further based on the vertex information and the edge information.

14 . The one or more non-transitory computer-readable storage media according to claim 13 , wherein the vertex information includes a primary table that includes a vertex ID, and a polygon ID, a vertex type, and the 3D surface coordinates, and

the edge information includes a secondary table that includes the polygon ID, an edge ID, and IDs of edge vertices.

15 . The one or more non-transitory computer-readable storage media according to claim 11 , further comprising:

retrieving, from a database, historical data associated with a plurality of 3D polygons corresponding to a plurality of objects in a set of images,

wherein each 3D polygon of the plurality of 3D polygon encloses a corresponding object of the plurality of objects;

training a polygon estimation model based on the historical data;

estimating one or more 3D polygons corresponding to the detected one or more objects in the 2D images, based on the trained polygon estimation model; and

projecting the detected one or more 3D polygons onto the ground region of the 2D images based on a determination that a difference between the estimated one or more 3D polygons and the detected one or more 3D polygons is less than a predefined threshold.

16 . The one or more non-transitory computer-readable storage media according to claim 11 , wherein the operations further comprises:

extracting an image size of the 2D images;

selecting a 2D coordinate from 2D pixel coordinates;

applying the ray casting operation on the selected 2D coordinate to estimate a corresponding 3D surface coordinate of the 3D surface coordinates on the ground segment in a 3D space; and

determining the correspondence information based on the 2D pixel coordinate and the 3D surface coordinate.

17 . The one or more non-transitory computer-readable storage media according to claim 11 , wherein the operations further comprises:

capturing 3D data via an image capturing device;

generating a 3D point cloud based on the 3D data; and

converting the point cloud data into the 3D surface model.

18 . An electronic device, comprising:

a memory configured to store instructions; and

a processor, coupled to the memory, configured to execute the instructions to perform a process comprising:

acquiring image data that includes two-dimensional (2D) images depicting a view of a real-world location;

detecting one or more 3D polygons corresponding to one or more objects on a ground region of the 2D images;

acquiring a three-dimensional (3D) surface model of the real-world location;

segmenting the 3D surface model into a plurality of 3D segments comprising a ground segment and one or more non-ground segments;

extracting the ground segment of the 3D surface model by removing the one or more non-ground segments from the plurality of 3D segments;

determining, by applying a ray casting operation on the 2D images and the ground segment of the 3D surface model, correspondence information between 2D pixel coordinates in the 2D images and 3D surface coordinates of the ground segment; and

projecting, based on the correspondence information, the one or more 3D polygons corresponding to the one or more objects onto the 3D surface model;

wherein the correspondence information includes a table that includes the 2D pixel coordinates and the 3D surface coordinates corresponding to the 2D pixel coordinates.