Recreating peripheral vision on a wearable device
A head-wearable apparatus includes a frame having a front piece configured to hold left and right lenses. A left temple is coupled to the front piece and a right temple coupled to the front piece. A camera system includes one or more cameras coupled to the front piece, one or more left peripheral cameras coupled to an outside surface of the frame, and one or more right peripheral cameras coupled to an outside surface of the frame. A left peripheral display is coupled to an inside surface of the frame. The left peripheral display is configured to receive and display input from the one or more left peripheral cameras. A right peripheral display is coupled to an inside surface of the frame. The right peripheral display is configured to receive and display input from the one or more right peripheral cameras.
1 . A head-wearable apparatus comprising:
a frame comprising a front piece configured to hold left and right lenses, a left temple coupled to the front piece, and a right temple coupled to the front piece;
a camera system, comprising one or more cameras coupled to the front piece, one or more left peripheral cameras coupled to an outside surface of the frame, and one or more right peripheral cameras coupled to an outside surface of the frame;
a left peripheral display coupled to an inside surface of the frame, the left peripheral display configured to receive and display input from the one or more left peripheral cameras;
a right peripheral display coupled to an inside surface of the frame the right peripheral display configured to receive and display input from the one or more right peripheral cameras, and
a controller configured to:
(1) input an image or video captured by the one or more left peripheral cameras and the one or more right peripheral cameras into a machine learning model, the image or video includes a blocked user's field of view that is blocked by at least a portion of the frame, the machine learning model trained to;
identify a portion of the image or video that corresponds to the blocked field of view; and
generate an adjusted image or video that replaces the blocked field of view of the user with field of view that aligns with the user's view of a real world absent the frame of the head-wearable apparatus, the adjusted image or video being based on a position of an eye of the user;
(2) receiving the adjusted image or video that was generated by the machine learning model, and
(3) generate display data that enables display of the adjusted image or video onto an interior portion of the frame to display to the user a view of the real world absent the frame of the head-wearable apparatus.
2 . The head-wearable apparatus of claim 1 , further comprising a right lens display coupled to an inside surface of the right lens and a left lens display coupled to an inside surface of the left lens.
3 . The head-wearable apparatus of claim 1 , wherein one or both of the left peripheral display and the right peripheral display is a projector.
4 . The head-wearable apparatus of claim 1 , wherein the controller is configured to adjust the input that is displayed by the left peripheral display and the right peripheral display based on an actual vision of a user of the head-wearable apparatus.
5 . The head-wearable apparatus of claim 1 , wherein the controller is configured to adjust the input based on a distance between at least one of the left peripheral camera and the right peripheral camera and a respective eye of a user of the head-wearable apparatus.
6 . The head-wearable apparatus of claim 1 , further comprising an eye position sensor configured to detect a position of one or both eyes of a user of the head-wearable apparatus and wherein the controller is configured to activate one or both of the peripheral displays based on the detected position.
7 . The head-wearable apparatus of claim 1 , wherein the controller is configured to adjust the input that is displayed based on at least one of a position of the one or more left peripheral cameras, the one or more right peripheral cameras, or eye measurements of a user of the head-wearable apparatus.
8 . The head-wearable apparatus of claim 1 , wherein the controller is configured to control at least one of a resolution or a refresh rate of the left and right peripheral cameras.
9 . The head-wearable apparatus of claim 1 , wherein the controller is configured to cause the left peripheral display and the right peripheral display to display at least one of a mixed reality or an augmented reality input to a user of the head-wearable apparatus.
10 . To: A method, comprising: receiving input including an image or video from one or more left peripheral cameras and one or more right peripheral cameras peripheral cameras of a head-wearable apparatus, the input including a user's field of view that is blocked by at least a portion of a frame of the head-wearable apparatus; adjusting the input generating an adjusted image or video based on at least one of a position of the one or more peripheral cameras or eye measurements of a user of the head-wearable apparatus by inputting the input into using a machine learning model, the adjusted input adjusting for a vision of the user, the machine learning model trained to receive as input the frame that includes the blocked portion and identify the blocked field of view in images or videos and generate an adjusted image or video that replaces the blocked field of view in the images or videos with a field of view that aligns with the user's view of a real world absent a frame of the head-wearable apparatus based on a position of an eye of the user; and displaying the adjusted input to a peripheral vision of the user of the head-wearable apparatus to display to the user a view of a real world absent the frame of the head-wearable apparatus.
11 . The method of claim 10 , wherein adjusting the input comprises adjusting the input based on an actual field of vision of the user of the head-wearable apparatus.
12 . The method of claim 10 , wherein displaying the adjusted input comprises displaying at least one of mixed reality or an augmented reality input to the user of the head-wearable apparatus.
13 . The method of claim 10 , further comprising controlling at least one of a resolution or a refresh rate of the received input.
14 . The method of claim 10 , wherein adjusting the input based on a distance between the one or more peripheral cameras and a respective eye of a user of the head-wearable apparatus.
15 . The method of claim 10 , further comprising:
detecting a position of one or both eyes of a user of the head-wearable apparatus; and
activating one or both of the peripheral displays based on the detected position.
16 . To: A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a controller of a head-wearable apparatus, cause the controller to perform operations comprising: receiving input including an image or video from one or more left peripheral cameras and one or more right peripheral cameras peripheral cameras of a head-wearable apparatus, the input including a user's field of view that is blocked by at least a portion of a frame of the head-wearable apparatus; adjusting the input generating an adjusted image or video based on at least one of a position of the one or more peripheral cameras or eye measurements of a user of the head-wearable apparatus by inputting the input into using a machine learning model, the machine learning model trained to receive as input the frame that includes the blocked portion and generate the adjusted input to enable display of the adjusted input onto an interior portion of the head-wearable apparatus that blocks the user's field of view, the adjusted input adjusting for a vision of the user, the machine learning model trained to identify the blocked field of view in images or videos and replace the blocked field of view in the images or videos with a field of view that aligns with the user's view of a real world absent a frame of the head-wearable apparatus based on a position of an eye of the user; and displaying the adjusted input to a peripheral vision of the user of the head-wearable apparatus to display to the user a view of a real world absent the frame of the head-wearable apparatus.
17 . The non-transitory computer-readable storage medium of claim 16 , wherein adjusting the input comprises adjusting the input based on an actual field of vision of the user of the head-wearable apparatus.
18 . The non-transitory computer-readable storage medium of claim 16 , wherein displaying the adjusted input comprises displaying at least one of mixed reality or an augmented reality input to the user of the head-wearable apparatus.
19 . The non-transitory computer-readable storage medium of claim 16 , wherein adjusting the input based on a distance between the one or more peripheral cameras and a respective eye of a user of the head-wearable apparatus.
20 . The non-transitory computer-readable storage medium of claim 16 , further comprising:
detecting a position of one or both eyes of a user of the head-wearable apparatus; and
activating one or both of the peripheral displays based on the detected position.