IP Library › Granted Patent US 12664756
Granted Patent B2
US 12664756 · App. 18/256,983 · Granted Jun 23, 2026

Fusion-based sensing intelligence and reporting

Inventors: Karen A. Panetta (Rockport, MA); Sos S. Agaian (New York, NY); Aruna Ramesh (Lincoln, MA); Shishir Paramathma Rao (Burlington, MA); Shreyas Kamath Kalasa Mohandas (Burlington, MA); Srijith Rajeev (Burlington, MA); Rahul Rajendran (Belleville, MI)
Assignees: Trustees of Tufts College; Research Foundation of the City University of New York
G06V10/764G06T7/70G06V40/18G06T2207/10048G06T2207/30204
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12664756
App. No.
18/256,983
Granted
Jun 23, 2026
Kind
B2
Abstract

A method comprising: obtaining a plurality of data sets, each of the plurality of data sets being generated, at least in part, by using by a different one of a plurality of sensors, each of a plurality of sensors including a wearable sensor that is worn by a user or a sensor that is positioned at a same location as the user; combining the plurality of data sets to produce a fused data set; processing the fused data set to identify at least one of a performance of the user in completing a task, a cognitive load of the user, and/or one or more objects that are positioned at the same location as the user; and outputting an indication of a state of the user based on an outcome of the processing.

Claims (76)

1 . A method comprising:

obtaining a plurality of data sets, each of the plurality of data sets being generated, at least in part, by using a different one of a plurality of sensors, each of a plurality of sensors including a wearable sensor that is worn by a user or a sensor that is positioned at a same location as the user;

combining the plurality of data sets to produce a fused data set;

processing the fused data set to identify at least one of a performance of the user in completing a task, a cognitive load of the user, and/or one or more objects that are positioned at the same location as the user; and

outputting an indication of a state of the user based on an outcome of the processing,

wherein the plurality of data sets includes an image of a field of view of the user at a given time instant, and an indication of a gaze point of the user at the given time instant;

processing the fused data set includes identifying an object on which a gaze of the user is focused; and

outputting the indication of the state of the user includes outputting a marker that denotes the object on which the gaze of the user is focused, the marker being super-imposed on an image that shows a position of the user relative to the object.

2 . The method of claim 1 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and an image that is being observed by the user during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in detecting a condition that is evident from the image; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

3 . The method of claim 1 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and plurality of images of a field of view of the user that are captured during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in fulfilling an objective; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

4 . The method of claim 1 , wherein:

the plurality of data sets includes a data set that is generated by an Infrared (IR) imager and a visible range image of a field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a gas leak; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the gas leak.

5 . The method of claim 1 , wherein:

the plurality of data sets includes a thermal image of a field of view of the user and a visible range image of the field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a hot object; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the hot object.

6 . The method of claim 1 , wherein generating the fused data set includes bringing the plurality of data sets into a temporal alignment.

7 . A system, comprising:

a memory; and

a processing circuitry that is operatively coupled to the memory, the processing circuitry being configured to perform the operations of:

obtaining a plurality of data sets, each of the plurality of data sets being generated, at least in part, by using a different one of a plurality of sensors, each of a plurality of sensors including a wearable sensor that is worn by a user or a sensor that is positioned at a same location as the user;

combining the plurality of data sets to produce a fused data set;

processing the fused data set to identify at least one of a performance of the user in completing a task, a cognitive load of the user, and/or one or more objects that are positioned at the same location as the user; and

outputting an indication of a state of the user based on an outcome of the processing,

wherein the plurality of data sets includes an image of a field of view of the user at a given time instant, and an indication of a gaze point of the user at the given time instant;

processing the fused data set includes identifying an object on which a gaze of the user is focused; and

outputting the indication of the state of the user includes outputting a marker that denotes the object on which the gaze of the user is focused, the marker being super-imposed on an image that shows a position of the user relative to the object.

8 . The system of claim 7 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and an image that is being observed by the user during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in detecting a condition that is evident from the image; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

9 . The system of claim 7 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and plurality of images of a field of view of the user that are captured during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in fulfilling an objective; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

10 . The system of claim 7 , wherein:

the plurality of data sets includes a data set that is generated by an Infrared (IR) imager and a visible range image of a field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a gas leak; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the gas leak.

11 . The system of claim 7 , wherein:

the plurality of data sets includes a thermal image of a field of view of the user and a visible range image of the field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a hot object; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the hot object.

12 . The system of claim 7 , wherein generating the fused data set includes bringing the plurality of data sets into a temporal alignment.

13 . A non-transitory computer-readable medium storing one or more processor-executable instructions, which when executed by at least one processor, cause the at least one processor to perform the operations of:

obtaining a plurality of data sets, each of the plurality of data sets being generated, at least in part, by using a different one of a plurality of sensors, each of a plurality of sensors including a wearable sensor that is worn by a user or a sensor that is positioned at a same location as the user;

combining the plurality of data sets to produce a fused data set;

processing the fused data set to identify at least one of a performance of the user in completing a task, a cognitive load of the user, and/or one or more objects that are positioned at the same location as the user; and

outputting an indication of a state of the user based on an outcome of the processing,

wherein the plurality of data sets includes an image of a field of view of the user at a given time instant, and an indication of a gaze point of the user at the given time instant;

processing the fused data set includes identifying an object on which a gaze of the user is focused; and

outputting the indication of the state of the user includes outputting a marker that denotes the object on which the gaze of the user is focused, the marker being super-imposed on an image that shows a position of the user relative to the object.

14 . The non-transitory computer-readable medium of claim 13 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and an image that is being observed by the user during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in detecting a condition that is evident from the image; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

15 . The non-transitory computer-readable medium of claim 13 , wherein:

the plurality of data sets includes a gaze tracking data set for the user and plurality of images of a field of view of the user that are captured during a period in which the gaze tracking data set is generated;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set to identify a level of performance of the user in fulfilling an objective; and

outputting the indication of the state of the user includes outputting an indication of the level of performance of the user.

16 . The non-transitory computer-readable medium of claim 13 , wherein:

the plurality of data sets includes a data set that is generated by an Infrared (IR) imager and a visible range image of a field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a gas leak; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the gas leak.

17 . The non-transitory computer-readable medium of claim 13 , wherein:

the plurality of data sets includes a thermal image of a field of view of the user and a visible range image of the field of view of the user;

processing the fused data set includes classifying, with a neural network, at least a portion of the fused data set as being associated with a hot object; and

outputting the indication of the state of the user includes outputting an indication of a position of the user relative to the hot object.