Monitoring icon status in a display from an external device
Systems and methods for monitoring of icon in an external display device are disclosed. Images of an icon displayed in a display device may be continually captured as video frames by a video camera of an icon monitoring system. While operating in a first mode, video frames may be continually analyzed to determine if the captured image matches an active template icon known to match the captured image of the icon. While the captured image matches the active template icon, operating in the first mode continues. Upon detecting a failed match to the active template icon, the system starts operating in a second to search among known template icons for a new match. Upon finding a new match, the active template icon may be updated to the new match, and operation switches back to the first mode. Times of transitions between the first and second modes may be recorded.
1. A method carried out by a system including a computing device and a video camera communicatively connected with the computing device, the method comprising:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the computing device and video camera, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.
2. The method of claim 1 , wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises maintaining a threshold-limited fixed orientation of a field-of-view (FOV) direction of the video camera with respect to the display region of the display device during a time in which the video images are continually captured,
wherein the threshold-limited fixed orientation comprises an orientation with respect to the display region of the display device that remains constant to within a threshold amount of misalignment with respect to a fixed direction.
3. The method of claim 1 , wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises capturing a video image of the icon in the corresponding sub-region of the image plane during each respective video frame.
4. The method of claim 1 , wherein computationally comparing the template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame comprises computationally comparing the template icon with video image data captured only within the corresponding sub-region of the image plane.
5. The method of claim 4 , wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises applying normalized cross correlation template matching to the video image data captured only within the corresponding sub-region of the image plane.
6. The method of claim 4 , wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises inputting the video image data captured only within the corresponding sub-region of the image plane to an artificial neural network application configured for image recognition and trained with an image of the template icon.
7. The method of claim 1 , wherein each respective video frame has a respective time stamp marking a time during which the image of the icon was captured in image plane,
wherein recording the information associated with the respective video frame comprises recording the time stamp and information identifying the template icon,
and wherein recording the information associated with the given one of the respective video frames comprises recording the time stamp and information indicating the failure to match the template icon to within the matching threshold.
8. The method of claim 1 , wherein determining the new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold comprises:
computationally comparing the captured image of the icon during the given one of the respective video frames with each of a plurality of candidate template icons until either: a match to within the matching threshold is found for a given candidate template icon, or no match to within the matching threshold is found for any of the plurality of candidate template icons;
if a match is found, setting the new icon to the given candidate template icon; and
if no match is found: (i) in an area of the image plane in which an area of the display region of the display device extending beyond the sub-region is captured, scanning for a further icon that matches one of the plurality of candidate template icons to within the matching threshold, (ii) setting the new icon to the one of the plurality of candidate template icons that matches the further icon, and (iii) setting the sub-region of the display region to a new sub-region in which the further icon is displayed.
9. The method of claim 1 , further comprising:
prior to operation in either of the first or second modes, operating in an initialization mode, wherein operation in the initialization mode comprises:
in an area of the image plane in which an area of the entire display region of the display device is captured, scanning for an initial icon that matches an initial template icon to within the matching threshold;
setting the icon to the initial template icon;
setting the sub-region of the display region to an initial sub-region in which the initial icon is displayed; and
switching to operation in the first mode.
10. A system comprising:
one or more processors;
a video camera; and
memory storing instructions that, when executed by the one or more processors, cause the system to carry out operations including:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the system, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.
11. The system of claim 10 , wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises maintaining a threshold-limited fixed orientation of a field-of-view (FOV) direction of the video camera with respect to the display region of the display device during a time in which the video images are continually captured,
wherein the threshold-limited fixed orientation comprises an orientation with respect to the display region of the display device that remains constant to within a threshold amount of misalignment with respect to a fixed direction.
12. The system of claim 10 , wherein continually capturing the video images in the image plane of the video camera of the icon displayed in the sub-region of the display region of the display device comprises capturing a video image of the icon in the corresponding sub-region of the image plane during each respective video frame.
13. The system of claim 10 , wherein computationally comparing the template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame comprises computationally comparing the template icon with video image data captured only within the corresponding sub-region of the image plane.
14. The system of claim 13 , wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises applying normalized cross correlation template matching to the video image data captured only within the corresponding sub-region of the image plane.
15. The system of claim 13 , wherein computationally comparing the template icon with the video image data captured only within the corresponding sub-region of the image plane comprises inputting the video image data captured only within the corresponding sub-region of the image plane to an artificial neural network application configured for image recognition and trained with an image of the template icon.
16. The system of claim 10 , wherein each respective video frame has a respective time stamp marking a time during which the image of the icon was captured in image plane,
wherein recording the information associated with the respective video frame comprises recording the time stamp and information identifying the template icon,
and wherein recording the information associated with the given one of the respective video frames comprises recording the time stamp and information indicating the failure to match the template icon to within the matching threshold.
17. The system of claim 10 , wherein determining the new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold comprises:
computationally comparing the captured image of the icon during the given one of the respective video frames with each of a plurality of candidate template icons until either: a match to within the matching threshold is found for a given candidate template icon, or no match to within the matching threshold is found for any of the plurality of candidate template icons;
if a match is found, setting the new icon to the given candidate template icon; and
if no match is found: (i) in an area of the image plane in which an area of the display region of the display device extending beyond the sub-region is captured, scanning for a further icon that matches one of the plurality of candidate template icons to within the matching threshold, (ii) setting the new icon to the one of the plurality of candidate template icons that matches the further icon, and (iii) setting the sub-region of the display region to a new sub-region in which the further icon is displayed.
18. The system of claim 10 , wherein the operations further include:
prior to operation in either of the first or second modes, operating in an initialization mode, wherein operation in the initialization mode comprises:
in an area of the image plane in which an area of the entire display region of the display device is captured, scanning for an initial icon that matches an initial template icon to within the matching threshold;
setting the icon to the initial template icon;
setting the sub-region of the display region to an initial sub-region in which the initial icon is displayed; and
switching to operation in the first mode.
19. The system of claim 10 , further comprising a further video camera,
wherein operating in the first mode further comprises:
with the further video camera, continually capturing further video images in an image plane of the further video camera of a further icon displayed in a sub-region of a display region of a further display device, wherein the further display device is external to the system, wherein the sub-region of the display region of the further display device is fractionally larger than the further icon, and maps to a corresponding sub-region of the image plane of the further video camera, and wherein the further video images captured in the image plane of the further video camera comprise further video frames;
for each respective further video frame of at least a subset of the further video frames, computationally comparing a further template icon with the further video image of the further icon captured in the corresponding sub-region of the image plane of the further video camera during the respective further video frame;
for each computational comparison for which the captured image of the further icon during the respective further video frame matches the further template icon to within a further matching threshold: (i) recording information associated with the respective further video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the further icon during a given one of the respective further video frames fails to match the further template icon to within the further matching threshold: (i) recording information associated with the given one of the respective further video frames, and (ii) switching to operation in a second mode;
and wherein operating in the second mode further comprises:
determining a new further template icon that matches the captured image of the further icon during the given one of the respective further video frames to within the further matching threshold;
setting the further template icon to the new further template icon; and
switching to operation in the first mode.
20. A non-transitory computer-readable storage medium, having stored thereon program instructions that, upon execution by one or more processors of a system comprising a video camera, cause the system to carry out operations including:
operating in a first mode, wherein operating in the first mode comprises:
with the video camera, continually capturing video images in an image plane of the video camera of an icon displayed in a sub-region of a display region of a display device, wherein the display device is external to the system, wherein the sub-region of the display region is fractionally larger than the icon and maps to a corresponding sub-region of the image plane, and wherein the video images captured in the image plane comprise video frames;
for each respective video frame of at least a subset of the video frames, computationally comparing a template icon with the video image of the icon captured in the corresponding sub-region of the image plane during the respective video frame;
for each computational comparison for which the captured image of the icon during the respective video frame matches the template icon to within a matching threshold: (i) recording information associated with the respective video frame, and (ii) continuing operation in the first mode; and
upon occurrence of at least one computational comparison for which the captured image of the icon during a given one of the respective video frames fails to match the template icon to within the matching threshold: (i) recording information associated with the given one of the respective video frames, and (ii) switching to operation in a second mode;
and
operating in the second mode upon switching to operation in the second mode, wherein operating in the second mode comprises:
determining a new template icon that matches the captured image of the icon during the given one of the respective video frames to within the matching threshold;
setting the template icon to the new template icon; and
switching to operation in the first mode.