System and method for video image registration and/or providing supplemental data in a heads up display
Video sources and inertial sensors are attached to a weapon and to goggles. A computer receives video images from the weapon- and goggles-mounted sources and inertial data from the sensors. The computer calculates a location for an image from the weapon-mounted source within an image from the goggles-mounted source using the inertial sensor data. The sensor-based location is checked (and possibly adjusted) based on a comparison of the images. A database contains information about real-world objects in a field of view of the goggles-mounted source, and is used to generate icons or other graphics concerning such objects.
1. A method comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
determining, based on comparing data associated with the portion of the fourth image to data associated with a portion of the third image, that the determined position is correct, wherein the portion of the third image comprises the determined position;
determining, based on the motion data adjustment, based on the third motion data, and based on the fourth motion data, a rotation associated with the portion of the fourth image;
determining, based on comparing the data associated with the portion of the fourth image to the data associated with the portion of the third image, that the determined rotation is within a predetermined limit; and
generating a modified version of the third image comprising the portion of the fourth image in the determined position.
2. The method of claim 1 , wherein the first motion data and the third motion data comprise data from an inertial measurement unit (IMU) sensor coupled to the first video source, and wherein the second motion data and the fourth motion data comprise data from an IMU sensor coupled to the second video source.
3. The method of claim 1 , wherein the motion data adjustment comprises an amount of rotation.
4. The method of claim 1 , further comprising:
receiving, while a first field of view (FOV) of a Heads-Up Display (HUD) associated with the first video source does not overlap with a second FOV associated with the second video source, video data from the first video source comprising a fifth image and video data from the second video source comprising a sixth image; and
generating a modified version of the fifth image comprising a portion of the sixth image.
5. The method of claim 1 , further comprising:
determining a position and an orientation of the first video source in an external environment; and
identifying, based on the determined position and orientation of the first video source, objects in the external environment,
wherein the generating the modified version of the third image comprises generating graphic indicia associated with the identified objects.
6. The method of claim 5 , wherein the graphic indicia comprise icons scaled based on distances between the first video source and the identified objects.
7. The method of claim 1 , further comprising wirelessly receiving data from the second video source.
8. The method of claim 1 , wherein the first video source is mounted to a helmet and the second video source is mounted to a weapon.
9. A system comprising:
a processor; and
a memory having instructions stored thereon, that, when executed by the processor, cause the processor to perform operations, the operations comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
determining, based on comparing data associated with the portion of the fourth image to data associated with a portion of the third image, that the determined position is correct, wherein the portion of the third image comprises the determined position;
determining, based on the motion data adjustment, based on the third motion data, and based on the fourth motion data, a rotation associated with the portion of the fourth image;
determining, based on comparing the data associated with the portion of the fourth image to the data associated with the portion of the third image, that the determined rotation is within a predetermined limit; and
generating a modified version of the third image comprising the portion of the fourth image in the determined position.
10. The system of claim 9 , wherein the first motion data and the third motion data comprise data from an inertial measurement unit (IMU) sensor coupled to the first video source, and wherein the second motion data and the fourth motion data comprise data from an IMU sensor coupled to the second video source.
11. The system of claim 9 , wherein the motion data adjustment comprises an amount of rotation.
12. The system of claim 9 , wherein the instructions, when executed by the processor, cause the processor to perform operations comprising:
receiving, while a first field of view (FOV) of a Heads-Up Display (HUD) associated with the first video source does not overlap with a second FOV associated with the second video source, video data from the first video source comprising a fifth image and video data from the second video source comprising a sixth image; and
generating a modified version of the fifth image comprising a portion of the sixth image.
13. The system of claim 9 , wherein the instructions, when executed by the processor, cause the processor to perform operations comprising:
determining a position and an orientation of the first video source in an external environment; and
identifying, based on the determined position and orientation of the first video source, objects in the external environment,
wherein the generating the modified version of the third image comprises generating graphic indicia associated with the identified objects.
14. The system of claim 13 , wherein the graphic indicia comprise icons scaled based on distances between the first video source and the identified objects.
15. The system of claim 9 , wherein the instructions, when executed by the processor, cause the processor to perform operations comprising:
wirelessly receiving data from the second video source.
16. The system of claim 9 , wherein the first video source is mounted to a helmet and the second video source is mounted to a weapon.
17. A non-transitory machine-readable medium having machine executable instructions for performing a method comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
generating a modified version of the third image comprising the portion of the fourth image in the determined position;
receiving, while a first field of view (FOV) of a Heads-Up Display (HUD) associated with the first video source does not overlap with a second FOV associated with the second video source, video data from the first video source comprising a fifth image and video data from the second video source comprising a sixth image; and
generating a modified version of the fifth image comprising a portion of the sixth image.
18. The method of claim 1 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.
19. The system of claim 9 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.
20. The non-transitory machine-readable medium of claim 17 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.
21. A method comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
generating a modified version of the third image comprising the portion of the fourth image in the determined position;
receiving, while a first field of view (FOV) of a Heads-Up Display (HUD) associated with the first video source does not overlap with a second FOV associated with the second video source, video data from the first video source comprising a fifth image and video data from the second video source comprising a sixth image; and
generating a modified version of the fifth image comprising a portion of the sixth image.
22. The method of claim 21 , wherein the first motion data and the third motion data comprise data from an inertial measurement unit (IMU) sensor coupled to the first video source, and wherein the second motion data and the fourth motion data comprise data from an IMU sensor coupled to the second video source.
23. The method of claim 21 , wherein the motion data adjustment comprises an amount of rotation.
24. The method of claim 21 , wherein the first video source is mounted to a helmet and the second video source is mounted to a weapon.
25. The method of claim 21 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.
26. A system comprising:
a processor; and
a memory having instructions stored thereon, that, when executed by the processor, cause the processor to perform operations, the operations comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
generating a modified version of the third image comprising the portion of the fourth image in the determined position;
receiving, while a first field of view (FOV) of a Heads-Up Display (HUD) associated with the first video source does not overlap with a second FOV associated with the second video source, video data from the first video source comprising a fifth image and video data from the second video source comprising a sixth image; and
generating a modified version of the fifth image comprising a portion of the sixth image.
27. The system of claim 26 , wherein the first motion data and the third motion data comprise data from an inertial measurement unit (IMU) sensor coupled to the first video source, and wherein the second motion data and the fourth motion data comprise data from an IMU sensor coupled to the second video source.
28. The system of claim 26 , wherein the motion data adjustment comprises an amount of rotation.
29. The system of claim 26 , wherein the first video source is mounted to a helmet and the second video source is mounted to a weapon.
30. The system of claim 26 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.
31. A non-transitory machine-readable medium having machine executable instructions for performing a method comprising:
predicting, based on first motion data for a first video source and based on second motion data for a second video source, a position in a first image, from the first video source, for a portion of a second image from the second video source;
determining, based on comparing data associated with a portion of the first image to data associated with the portion of the second image, a motion data adjustment;
determining, based on the motion data adjustment, based on third motion data for the first video source, and based on fourth motion data for the second video source, a position in a third image, from the first video source, for a portion of a fourth image from the second video source;
determining, based on comparing data associated with the portion of the fourth image to data associated with a portion of the third image, that the determined position is correct, wherein the portion of the third image comprises the determined position;
determining, based on the motion data adjustment, based on the third motion data, and based on the fourth motion data, a rotation associated with the portion of the fourth image;
determining, based on comparing the data associated with the portion of the fourth image to the data associated with the portion of the third image, that the determined rotation is within a predetermined limit; and
generating a modified version of the third image comprising the portion of the fourth image in the determined position.
32. The non-transitory machine-readable medium of claim 31 , wherein the first motion data and the third motion data comprise data from an inertial measurement unit (IMU) sensor coupled to the first video source, and wherein the second motion data and the fourth motion data comprise data from an IMU sensor coupled to the second video source.
33. The non-transitory machine-readable medium of claim 31 , wherein the motion data adjustment comprises an amount of rotation.
34. The non-transitory machine-readable medium of claim 31 , wherein the first video source is mounted to a helmet and the second video source is mounted to a weapon.
35. The non-transitory machine-readable medium of claim 31 , wherein the data associated with the portion of the first image is based on pixels for a first region of the first image, and wherein the data associated with the portion of the second image is based on pixels for a second region of the second image.