Methods and apparatus for making environmental measurements and/or using such measurements
Methods and apparatus for making environmental measurements are described. In some embodiments different devices are used to capture environmental information at different times, rates and/or resolutions. Environmental information, e.g., depth information, from multiples sources captured using a variety of devices is processed and combined. Some environmental information is captured during an event. Such information is combined, in some embodiments, with environmental information that was captured prior to the event. Environmental depth model is generated in some embodiments by combining, e.g., reconciling, depth information from at least two different sources including: i) depth information obtained from a static map, ii) depth information obtained from images captured by light field cameras, and iii) depth information obtained from images captured by stereoscopic camera pairs. The reconciliation process may involve a variety of information weighting operations taking into consideration the advantages of different depth information sources and the availability of such information.
1. A method of operating an imaging system, comprising:
acquiring depth measurements that were made using LIDAR;
determining depths using at least one non-LIDAR depth measurement technique;
determining a scaling factor from a LIDAR measured depth corresponding to a location and a non-LIDAR measured depth corresponding to said location,
said non-LIDAR measured depth corresponding to said location differing from said LIDAR measured depth corresponding to said location due to a systematic non-LIDAR measurement error,
said scaling factor being a factor which scales the non-LIDAR measured depth corresponding to said location by an amount required to match the LIDAR measured depth corresponding to said location;
scaling one or more non-LIDAR measured depths by said determined scaling factor to compensate for systematic non-LIDAR measurement errors so that LIDAR and non-LIDAR depth measurements can be combined in a reliable manner,
said scaling producing one or more scaled non-LIDAR depths;
generating an environmental depth map of an environment of interest from LIDAR measured depths and said scaled non-LIDAR measured depths; and
outputting said generated environmental depth map.
2. The method of claim 1 , wherein said non-LIDAR measured depth is a depth generated from images of the environment of interest.
3. The method of claim 1 , further comprising: updating the environmental depth map based on depth information obtained from the processing of images captured by said one or more cameras while an event is ongoing.
4. The method of claim 3 , wherein generating the environmental depth map of the environment of interest includes
performing a weighting operation giving greater weight to depth information from a first source of depth information than a second source of depth information;
and wherein the method further comprises: updating the environmental depth map based on depth information generated from stereoscopic images captured by a pair of cameras used to capture a stereoscopic image pair while the event is ongoing.
5. The method of claim 4 , wherein said stereoscopic image pair is processed to generate environmental depth information which provides more depth measurements of the environment for a point in time than is provided by the environmental depth information generated from images captured by one or more light field cameras.
6. The method of claim 1 , wherein said LIDAR measured depth used to
generate the scaling factor is a depth obtained from a static depth model based on LIDAR measurements that were made when the environment of interest was not occupied by spectators; and
wherein depth information obtained from stereoscopic image pairs provides depth information which includes the effect of persons, props, stage sets, or signs present while the event is ongoing but not at the time of LIDAR measurements used to generate said static depth map.
7. The method of claim 1 , wherein said step of generating the environmental depth map of the environment of interest is performed by a processor which accesses a static depth map which provides LIDAR measured depths corresponding to the environment of interest.
8. The method of claim 1 , wherein generating the environmental depth map of the environment of interest includes: detecting a change in a depth measurement; determining a duration of the change in the depth measurement; and considering an amount of time for which the change in the depth measurement is detected as part of deciding whether to include the detected change in the depth measurement in said environmental depth map.
9. The method of claim 1 , wherein said non-LIDAR depth measurement was obtained from stereoscopic images which provide depth information which is of higher resolution in terms of the number of spatial locations to which measurements correspond than depth information obtained from images captured by one or more light field cameras and which is also used to generate said environmental depth map of the environment of interest.
10. The method of claim 9 , wherein said generating of the environmental depth map of the environment of interest includes: using depth information obtained from stereoscopic images to determine depths for environmental points which are located between environmental points for which light field camera provided depth information is available.
11. The method of claim 1 , wherein the environmental depth map is in the form of a 3D model of the environment in which images are captured.
12. The method of claim 1 , further comprising: displaying an image captured in the environment of interest to which said environmental depth map corresponds by wrapping the image onto the 3D model of the environment produced from depth measurements.
13. An imaging system, comprising:
a processor configured to:
acquire depth measurements that were made using LIDAR;
determine depths using at least one non-LIDAR depth measurement technique;
determine a scaling factor from a LIDAR measured depth corresponding to a location and a non-LIDAR measured depth corresponding to said location,
said non-LIDAR measured depth corresponding to said location differing from said LIDAR measured depth corresponding to said location due to a systematic non-LIDAR measurement error,
said scaling factor being a factor which scales the non-LIDAR measured depth corresponding to said location by an amount required to match the LIDAR measured depth corresponding to said location;
scale one or more non-LIDAR measured depths by said determined scaling factor to compensate for systematic non-LIDAR measurement errors so that LIDAR and non-LIDAR depth measurements can be combined in a reliable manner,
said scaling producing one or more scaled non-LIDAR depths;
generate an environmental depth map of an environment of interest from LIDAR measured depths and said scaled non-LIDAR measured depths; and
output said generated environmental depth map.
14. The imaging system of claim 13 , further comprising: an interface configured to receive a static map of the environment; a light field camera for capturing images of the environment; and a stereoscopic pair of cameras for capturing pairs of images of the environment.
15. The imaging system of claim 14 , wherein said processor is configured, as part of being configured to generate the environmental depth map, to: combine depth information from said at least two different sources of depth information, said depth information including at least two items from the list including
i) depth information obtained from said static map,
ii) depth information obtained from images captured by said light field camera,
iii) depth information obtained from images captured by said stereoscopic pair of cameras.
16. The method of claim 1 , wherein said generating an environmental depth map includes: using depth information obtained from stereoscopic images to determine depths for environmental points which are located between environmental points for which light field camera provided depth information is available.
17. The method of claim 4 , wherein said generating an environmental depth map includes: using depth information obtained from stereoscopic images to determine depths for environmental points which are located between environmental points for which light field camera provided depth information is available.
18. The method of claim 1 , wherein said generating an environmental depth map includes: combining light field depth information with stereo depth information, said combining including treating the light field depth information as being more reliable than the stereo depth information with the light field depth information being weighted more heavily than stereoscopic determined depth information when both sources of information are available for the same location.
19. The imaging system of claim 13 , wherein said non-LIDAR measured depth is a depth generated from images of the environment of interest.
20. The imaging system of claim 13 , wherein said processor is further configured to: update the environmental depth map based on depth information obtained from the processing of images captured by one or more cameras while an event is ongoing.
21. The imaging system of claim 20 , wherein said processor is configured, as part of being configured to generate the environmental depth map of the environment of interest, to perform a weighting operation giving greater weight to depth information from a first source of depth information than a second source of depth information; and wherein said processor is further configured to: update the environmental depth map based on depth information generated from stereoscopic images captured by a pair of cameras used to capture a stereoscopic image pair while the event is ongoing.
22. The imaging system of claim 21 , wherein said stereoscopic image pair is processed to generate environmental depth information which provides more depth measurements of the environment for a point in time than is provided by environmental depth information generated from images captured by one or more light field cameras.
23. The imaging system of claim 13 , wherein said LIDAR measured depth used to generate the scaling factor is a depth obtained from a static depth model based on LIDAR measurements that were made when the environment of interest was not occupied by spectators; and wherein depth information obtained from stereoscopic image pairs provides depth information which includes the effect of persons, props, stage sets, or signs present while the event is ongoing but not at the time of LIDAR measurements used to generate said static depth map.