Integrated internal and external camera system in vehicles
Devices, systems and processes for an integrated internal and external camera system that enhances the passenger experience in vehicles are described. One example method for enhancing a passenger experiences includes capturing a first set of images of an area around the vehicle using an external camera system, capturing a second set of images of one or more passengers inside the vehicle using an internal camera system, recognizing at least one gesture made by the one or more passengers based on the second set of images, identifying an object or a location external to the vehicle based on the first set of images and the at least one gesture, and displaying information related to the object or the location to the one or more passengers.
1 . A camera system for use with a vehicle, comprising:
a processor;
a first camera system coupled to the processor, wherein the first camera system is configured to capture a first set of images of an area around the vehicle;
a second camera system coupled to the processor, wherein the second camera system is configured to capture a second set of images of one or more passengers inside the vehicle; and
a display system coupled to the processor,
wherein the processor is configured to:
recognize, based on the second set of images, at least one gesture made by at least one passenger of the one or more passengers,
modify the first set of images to generate a modified first set of images that corresponds to an actual field of view of the one or more passengers, wherein the modified first set of images captures a portion of the area around the vehicle that the at least one passenger was viewing at a time that the at least one passenger made the at least one gesture,
generate, based on the actual field of view, a cropped image by cropping at least one of the modified first set of images,
identify, based on the modified first set of images and the at least one gesture, an object or location located outside the vehicle that the at least one passenger gestured at, wherein the cropped image comprises the object or location,
receive, from the at least one passenger, an indication of a search performed on the cropped image by the at least one passenger,
retrieve, based on the indication, information related to the object or location from a point-of-interest dataset using a web application programming interface (API), and
display, using the display system, the information related to the object or location to the one or more passengers.
2 . The camera system of claim 1 ,
wherein the processor is further configured to determine a location of the vehicle, and
wherein the processor is configured to identify the object or location located outside the vehicle further based on the location of the vehicle.
3 . The camera system of claim 1 ,
wherein the processor is further configured to determine a field of view of the at least one passenger of the one or more passengers, and
wherein the processor is configured to identify the object or location located outside the vehicle further based on the field of view.
4 . The camera system of claim 1 , wherein the point-of-interest dataset is generated based on a location of the vehicle.
5 . The camera system of claim 4 , wherein the point-of-interest dataset is generated further based on a direction of travel of the vehicle.
6 . The camera system of claim 1 , wherein the at least one gesture includes a pointing gesture or a framing gesture.
7 . The camera system of claim 1 , wherein the display system displays the information related to the object or location on a window surface of the vehicle or in an augmented reality format.
8 . A method for use with a vehicle, comprising:
capturing a first set of images of an area around the vehicle;
capturing a second set of images of one or more passengers inside the vehicle;
recognizing, based on the second set of images, at least one gesture made by at least one passenger of the one or more passengers;
modifying the first set of images to generate a modified first set of images that corresponds to an actual field of view of the one or more passengers, wherein the modified first set of images captures a portion of the area around the vehicle that the at least one passenger was viewing at a time that the at least one passenger made the at least one gesture;
generating, based on the actual field of view, a cropped image by cropping at least one of the modified first set of images;
identifying, based on the modified first set of images and the at least one gesture, an object or location located outside the vehicle that the at least one passenger gestured at, wherein the cropped image comprises the object or location;
receiving, from the at least one passenger, an indication of a search performed on the cropped image by the at least one passenger;
retrieving, based on the indication, information related to the object or location from a point-of-interest dataset using a web application programming interface (API); and
displaying the information related to the object or location to the one or more passengers.
9 . The method of claim 8 , further comprising:
determining a location of the vehicle,
wherein the identifying the object or location located outside the vehicle is further based on the location of the vehicle.
10 . The method of claim 8 , further comprising:
determining a field of view of the one or more passengers, and
wherein the identifying the object or location located outside the vehicle is further based on the field of view.
11 . The method of claim 8 , wherein the point-of-interest dataset is generated based on a location of the vehicle.
12 . The method of claim 11 , wherein the point-of-interest dataset is generated further based on a direction of travel of the vehicle.
13 . The method of claim 8 , wherein the at least one gesture includes a pointing gesture.
14 . The method of claim 8 , wherein the at least one gesture includes a framing gesture.
15 . The method of claim 8 , wherein the displaying the information related to the object or location is performed so that the information related to the object or location is displayed on a window surface of the vehicle.
16 . The method of claim 15 , wherein the displaying the information related to the object or location is performed so that the information related to the object or location is displayed in an augmented reality format.
17 . The method of claim 8 , wherein the modified first set of images is generated based on overlaying adjacent images of the first set of images.
18 . The method of claim 10 , further comprising:
determining, based on the second set of images, a pose of the at least one passenger,
wherein the field of view is determined based on the second set of images and the pose of the at least one passenger.