System and method for capturing and sharing a location based experience
A system and method for capturing a location based experience at an event including a plurality of mobile devices having a camera employed near a point of interest to capture random, crowdsourced images and associated metadata near said point of interest. In a preferred form, the images include depth camera information from prepositioned devices around the point of interest during the event. A network communicates images, depth information, and metadata to build a 3D model of the region, preferably with the location of contributors known. Users connect to this experience platform to view the 3D model from a user selected location and orientation and to participate in experiences with, for example, a social network.
1. A method to build or update a 3D model, comprising:
operating one or more cameras to collect at least a first and a second camera images of a target near a point of interest, the at least first and second camera images including associated metadata having position and orientation of the respective first and second camera when capturing an image of a common target near a point of interest;
transmitting at least the first and second camera images and associated metadata to an image processing server; and
operating said image processing server to build or update a 3D model near the point of interest,
comparing first and second camera images to detect a common target in the first and second images using at least in part said metadata,
determining a first location of said detected common target using associated
metadata including position and orientation metadata.
2. The method of claim 1 , wherein the step of operating one or more cameras, further comprises operating two or more conventional cameras wherein the two or more conventional cameras are offset from each other and operable for capturing an image from each of the two or more cameras simultaneously for use in a stereo triangulation.
3. The method of claim 2 , each camera comprising a time of flight camera, a structured light sensor, a stereo triangulation or a light field camera.
4. The method of claim 1 , wherein said processing step further comprises using an existing 3D model proximate to the common target and updating said existing 3D model using said first location of said common target.
5. The method of claim 1 , further comprising inserting a message into said 3D model at a geo-referenced position relative to the common target first location.
6. The method of claim 1 , further comprising processing the at least first and second camera images to identify said detected common target in an image using a feature point matching process.
7. The method of claim 6 , wherein the feature point matching process comprises a simultaneous localization and map-building (SLAM) process.
8. The method of claim 6 , wherein the feature point matching process comprises a parallel Tracking and Mapping (PTAM) process.
9. The method of claim 1 , wherein the step of operating one or more cameras, further comprises using one or more depth cameras to capture depth information including range between a depth camera and a target.
10. A system for developing a 3D model for use in artificial reality comprising:
one or more cameras to capture images and associated metadata near a point of interest, wherein the metadata includes camera position and orientation when capturing an image,
a communication network for receiving and transmitting images and metadata from said one or more cameras,
an image processing server connected to the communication network for receiving said images and position metadata, wherein the server processes the images and metadata to develop a 3D model of one or more targets proximate to the point of interest based at least in part on said images and metadata,
the image processing server being operable to select a first image and associated metadata proximate to the point of interest,
the image processing server being operable to select a second image and associated metadata proximate to the point of interest, and
the image processing server determining the location of a common target near said point of interest where common target location in said 3D model is based at least in part on a feature point matching process searching for and connecting features in each image.
11. The system of claim 10 , wherein the image processing server uses photogrammetry to determine the location of the common target.
12. The system of claim 10 , further comprising an experience platform having said 3D model near the point of interest and a user connected to the experience platform can view targets associated with a user selected location and orientation.
13. The system of claim 10 , wherein the image processing server is operable to stitch a number of images together to form a panorama.
14. The system of claim 10 , wherein at least one of the one or more cameras is provided by a mobile device with approximate position metadata comprising GPS, Wi-Fi, Cell ID, repeater, or RSS information.
15. The system of claim 10 , wherein said 3D model comprises a mesh having a number of feature points and one or more of said common targets serving as feature points in said mesh.
16. The system of claim 10 , wherein the feature point matching process connects common features in each image using Simultaneous Localization and Map-building (SLAM)/Parallel Tracking And Mapping (PTAM) algorithm.
17. The system of claim 10 , further comprises acquiring at least the first and second images from mobile devices at different locations.
18. The system of claim 10 , wherein the metadata includes time of image acquisition and
said first and second camera images are selected based at least in part on time of image acquisition.
19. The system of claim 10 , wherein the feature point matching process connects common features in each image using scale-invariant feature transform (SIFT) process.
20. The system of claim 10 , wherein the feature point matching process comprises a simultaneous localization and map-building (SLAM) process.
21. The system of claim 10 , wherein the feature point matching process comprises a parallel Tracking and Mapping (PTAM) process.
22. The system of claim 10 , wherein the one or more cameras further comprise one or more depth cameras to capture depth information including range between a depth camera and a target.
23. A computer-implemented method to build or update a 3D model comprising:
collecting at least a first and a second camera image of a common target near a point of interest, the at least first and second camera images including associated metadata having position and orientation of the respective first and second camera images;
comparing first and second camera images to detect a common target in the first and second camera images using at least in part said metadata,
determining a first location of said detected common target using at least in part associated metadata from said first and second camera images.
24. The method of claim 23 , further comprising operating one or more cameras to collect said first and a second camera images of a target near a point of interest.
25. The method of claim 24 , wherein the step of operating one or more cameras, further comprises using one or more conventional cameras wherein the two or more conventional cameras are offset from each other and operable for capturing an image from each of the two or more cameras simultaneously for use in a stereo triangulation.
26. The method of claim 23 , wherein the step of operating one or more cameras, further comprises using one or more depth cameras to capture depth information including range between a depth camera and a target.
27. The method of claim 23 , further comprising processing the at least first and second camera images to identify said detected common target in an image using a feature point matching process.
28. The method of claim 27 , wherein the feature point matching process comprises a simultaneous localization and map-building (SLAM) process.
29. The method of claim 27 , wherein the feature point matching process comprises a parallel Tracking and Mapping (PTAM) process.
30. The method of claim 27 , wherein the feature point matching process connects common features in each image using scale-invariant feature transform (SIFT) process.