3D video reconstruction system
A server receives video data and location data from mobile devices. Each mobile device records a video of a target. The location data identifies a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device to the target. The location data is associated with a corresponding video frame from the video data. The server identifies video frames from the video data captured from the mobile devices. The server scales parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target. The server extracts the scaled parts of the identified video frames and generates a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices.
1. A method comprising:
receiving, at a server, video data and location data from a plurality of mobile devices, each mobile device configured to record a video of a target, the location data identifying a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device to the target, the location data associated with a corresponding video frame from the video data;
identifying video frames from the video data captured from the plurality of mobile devices;
scaling parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target;
extracting the scaled parts of the identified video frames;
generating a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices;
identifying a first video frame from the video data captured from a first mobile device and a second video frame from the video data captured from a second mobile device, both the first and second video frames captured at a same time;
extracting a first region in the first video frame and a second region in the second video frame;
scaling the first region of the first video frame based on the position and distance of the first mobile device, and the second region of the second video frame based on the position and distance of the second mobile device;
determining whether a portion of the first scaled region and the second region overlap and contain a substantially similar pixel pattern; and
including the first scaled region and the second scaled region in a portion of the three-dimensional model of the target in response to determining that the first portion of the first scaled region and the second region overlap and contain the substantially similar pixel pattern.
2. The method of claim 1 , further comprising:
forming an augmented reality content comprising the three-dimensional model of the target;
associating the augmented reality content with an identifier of a physical object; and
communicating the augmented reality content to an augmented reality viewing device, the augmented reality viewing device configured to display the three-dimensional model in response to detecting the identifier of the physical object in a field of view of the augmented reality viewing device.
3. The method of claim 2 , further comprising:
identifying a missing portion on the three-dimensional model of the target;
identifying a best positioned mobile device with a first head mounted display from among the plurality of mobile devices based on its respective location and orientation having the closest proximity to the location of the missing portion on the three-dimensional model compared to the other respective locations and orientation of the other plurality of mobile devices;
generating a first augmented reality content for the first head mounted display, the first augmented reality content including a suggested virtual path to guide the first head mounted display to a second position and a second orientation of the first head mounted display, the first head mounted display configured to generate video data corresponding to the missing portion on the three-dimensional model while at the second position and the second orientation; and
communicating the first augmented reality content to the first head mounted display.
4. The method of claim 3 , wherein the first mobile device includes a transparent display configured to display the suggested virtual path in the transparent display relative to a position of the target within a field of view of the transparent display.
5. The method of claim 2 , further comprising:
receiving a request from the augmented reality viewing device, the request identifying a side view of the target;
identifying the first mobile device based on its location and position and the requested side view of the target;
generating and communicating instructions to the first mobile device, the instructions identifying a direction in which the mobile device is to move to generate the requested side view of the target.
6. The method of claim 1 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content identifying the target;
a camera configured to capture video frames of the target;
an inertial navigation sensor configured to determine a position and an orientation of the mobile device;
a location sensor configured to determine a geographic location of the mobile device and detect a frame of reference at a geographic location of the target; and
a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target and the frame of reference, the location data including the location and position of the mobile device relative to the target and the frame of reference.
7. The method of claim 1 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content identifying the target;
a camera configured to capture video frames of the target;
an inertial navigation sensor configured to determine a position and an orientation of the mobile device;
a depth sensor configured to determine a depth of the target relative to the mobile device;
a location sensor configured to identify a geographic location of the mobile device and detect a frame of reference at a geographic location of the target; and
a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target, the location data including the position and orientation of the mobile device relative to the target, and the depth of the target relative to the mobile device.
8. The method of claim 1 , further comprising:
generating an augmented reality content corresponding to the target, the augmented reality content identifying a physical object as the target in the plurality of mobile devices; and
communicating the augmented reality content to the plurality of mobile devices, each mobile device configured to display the augmented reality content as a layer on top of the physical object in a corresponding field of view of the mobile device.
9. The method of claim 1 , wherein the plurality of mobile devices includes:
the first mobile device having a static location and forming a frame of reference with the target; and
the second mobile device moving along an undetermined path, a location of the second mobile device determined relative to the frame of reference.
10. A server comprising:
a processor; and
a memory storing instructions that, when executed by the processor, configure the server to:
receive video data and location data from a plurality of mobile devices, each mobile device configured to record a video of a target, the location data identifying a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device and the target, the location data associated with a corresponding video frame from the video data;
identify video frames from the video data captured from the plurality of mobile devices;
scale parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target;
extract the scaled parts of the identified video frames;
generate a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices;
identify a first video frame from the video data captured from a first mobile device and a second video frame from the video data captured from a second mobile device, both first and second video frames captured at a same time;
extract a first region in the first video frame and a second region in the second video frame;
scale the first region of the first video frame based on the position and distance of the first mobile device, and the second region of the second video frame based on the position and distance of the second mobile device;
determine whether a portion of the first scaled region and the second region overlap and contain a substantially similar pixel pattern; and
include the first scaled region and the second scaled region in a portion of the three-dimensional model of the target in response to determining that the first portion of the first scaled region and the second region overlap and contain the substantially similar pixel pattern.
11. The server of claim 10 , wherein the instructions further configure the server to:
form an augmented reality content comprising the three-dimensional model of the target;
associate the augmented reality content with an identifier of a physical object; and
communicate the augmented reality content to an augmented reality viewing device, the augmented reality viewing device configured to display the three-dimensional model in response to detecting the identifier of the physical object in a field of view of the augmented reality viewing device.
12. The server of claim 11 , wherein the instructions further configure the server to:
identify a missing portion on the three-dimensional model of the target;
identify a best positioned mobile device with a first head mounted display from among the plurality of mobile devices based on its respective location and orientation having the closest proximity to the location of the missing portion on the three-dimensional model compared to the other respective locations and orientation of the other plurality of mobile devices;
generate a first augmented reality content for the first head mounted display, the first augmented reality content including a suggested virtual path to guide the first head mounted display to a second position and a second orientation of the first head mounted display, the first head mounted display configured to generate video data corresponding to the missing portion on the three-dimensional model while at the second position and the second orientation; and
communicate the first augmented reality content to the first head mounted display.
13. The server of claim 12 , wherein the first mobile device includes a transparent display configured to display the suggested virtual path in the transparent display relative to a position of the target within a field of view of the transparent display.
14. The server of claim 11 , wherein the instructions further configure the server to:
receive a request from the augmented reality viewing device, the request identifying a side view of the target;
identify a first mobile device based on its location and position and the requested side view of the target;
generate and communicate instructions to the first mobile device, the instructions identifying a direction in which the mobile device is to move to generate the requested side view of the target.
15. The server of claim 10 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content that identifies the target;
a camera configured to capture video frames of the target;
an inertial navigation sensor configured to determine a position and an orientation of the mobile device;
a location sensor configured to determine a geographic location of the mobile device and detect a frame of reference at a geographic location of the target; and
a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target and the frame of reference, the location data including the location and position of the mobile device relative to the target and the frame of reference.
16. The server of claim 10 , wherein each mobile device comprises:
a transparent display configured to display an augmented reality content that identifies the target;
a camera configured to capture video frames of the target;
an inertial navigation sensor configured to determine a position and an orientation of the mobile device;
a depth sensor configured to determine a depth of the target relative to the mobile device;
a location sensor configured to determine a geographic location of the mobile device and detect a frame of reference at geographic location of the target; and
a processor configured to generate video data and associate a video frame of the video data with a corresponding location and position of the mobile device relative to the target, the location data including the position and orientation of the mobile device relative to the target, and the depth of the target relative to the mobile device.
17. The server of claim 10 , wherein the instructions further configure the server to:
generate an augmented reality content corresponding to the target, the augmented reality content identifying a physical object as the target in the plurality of mobile devices; and
communicate the augmented reality content to the plurality of mobile devices, each mobile device configured to display the augmented reality content as a layer on top of the physical object in a corresponding field of view of the mobile device.
18. A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:
receive, at a server, video data and location data from a plurality of mobile devices, each mobile device configured to record a video of a target, the location data identifying a position of the corresponding mobile device relative to the target and a distance between the corresponding mobile device and the target, the location data associated with a corresponding video frame from the video data;
identify video frames from the video data captured from the plurality of mobile devices;
scale parts of the identified video frames based on the position and distance of the corresponding mobile devices to the target;
extract the scaled parts of the identified video frames;
generate a three-dimensional model of the target based on the extracted scaled parts of the identified video frames from the plurality of mobile devices;
identify a first video frame from the video data captured from a first mobile device and a second video frame from the video data captured from a second mobile device, both first and second video frames captured at a same time;
extract a first region in the first video frame and a second region in the second video frame;
scale the first region of the first video frame based on the position and distance of the first mobile device, and the second region of the second video frame based on the position and distance of the second mobile device;
determine whether a portion of the first scaled region and the second region overlap and contain a substantially similar pixel pattern; and
include the first scaled region and the second scaled region in a portion of the three-dimensional model of the target in response to determining that the first portion of the first scaled region and the second region overlap and contain the substantially similar pixel pattern.