IP Library › Granted Patent US 12705792

Granted Patent B2

US 12705792 · App. 17/962,238 · Granted Aug 11, 2026

Apparatus and methods for improving driver monitoring systems

Inventors: Min Guo (San Diego, CA); Srinivasa Deevi (San Diego, CA); Ning Bi (San Diego, CA)

Assignee: QUALCOMM Incorporated

G06T7/74 G06T2207/20221 G06T2207/30196

View Patent ↗

Loading inventors, assignments & file history…

Monitor This Case

Get email alerts when status or documents change.

Order Certified Copies

Most orders are placed with the USPTO same day — all within 24 business hours.

Order via The Patent Place →

Pre-filled with this patent's details

Quick Facts

Patent No.: US 12705792
App. No.: 17/962,238
Granted: Aug 11, 2026
Kind: B2

External Links

USPTO Patent Center↗ Full Patent PDF↗

Abstract

Methods, systems, and apparatuses to fuse a first dataset with a second dataset, and determine head pose estimation(s) based on the fused first dataset and second dataset. The first dataset may be associated with sensor data generated by a set of sensors of a first device, while the second dataset may be associated with sensor data generated by a first sensor of an apparatus. For example, an apparatus may obtain the first dataset and the second dataset. Additionally, the apparatus may generate a fused dataset based on the first dataset and the second dataset, and determine a head pose estimation of a head of the user based on the fused dataset. Further, the apparatus may output the third head pose estimation.

Claims (57)

1 . An apparatus of a vehicle, the apparatus comprising:

a non-transitory, machine-readable storage medium storing instructions; and

at least one processor coupled to the non-transitory, machine-readable storage medium, the at least one processor being configured to execute the instructions to:

obtain, from a first device, a first dataset indicating a first head pose estimation of a head of a user in each of a first set of images generated by the first device, the first dataset being based on sensor data of a set of sensors of the first device;

obtain a second dataset indicating a second head pose estimation of the head of the user in each of a set of images generated by a first sensor of the vehicle;

generate a fused dataset based on the first dataset and the second dataset, the fused dataset comprising time-aligned data derived from the first dataset and the second dataset, wherein the time-aligned data comprises one or more first elements of the first dataset and one or more second elements of the second dataset, and wherein a difference between a first timestamp associated with the one or more first elements and a second timestamp associated with the one or more second elements is within a predetermined time threshold;

determine a third head pose estimation of the head of the user based on the fused dataset; and

output the third head pose estimation.

2 . The apparatus of claim 1 , wherein the first dataset is generated based on an application of a first trained machine learning process to the sensor data of the set of sensors of the first device.

3 . The apparatus of claim 2 , wherein the first trained machine learning process is a simultaneous location and mapping (SLAM) algorithm.

4 . The apparatus of claim 1 , wherein the first device is a head mounted display (HMD) device including the set of sensors, and wherein the at least one processor is further configured to:

obtain, over one or more networks and from the first device, the first dataset.

5 . The apparatus of claim 1 , wherein the at least one processor is further configured to:

obtain, over one or more networks and from the first device, the sensor data of the set of sensors of the first device;

apply a first trained machine learning process to the sensor data of the set of sensors; and

generate the first dataset based on an application of the first trained machine learning process to the sensor data.

6 . The apparatus of claim 1 , wherein the first device is a HMD device including the set of sensors, and wherein the at least one processor is further configured to:

receive, over one or more networks and from a cloud computing system, the first dataset.

7 . The apparatus of claim 6 , wherein the first dataset is generated based on an application of a first trained machine learning process to the sensor data of the set of sensors.

8 . The apparatus of claim 1 , wherein the second dataset is generated based on an application of a second trained machine learning process to the set of images generated by the first sensor of the vehicle.

9 . The apparatus of claim 8 , wherein the second trained machine learning process is a three-dimensional morphable model (3DMM).

10 . The apparatus of claim 1 , wherein the set of sensors includes at least one of an optical sensor, a gyroscopic sensor and an inertial measurement unit.

11 . The apparatus of claim 1 , wherein the first sensor is an optical sensor.

12 . The apparatus of claim 1 , wherein generating the fused dataset includes applying an extended Kalman filter (EKF) to the first dataset and the second dataset.

13 . The apparatus of claim 1 , wherein outputting the third head pose estimation comprises providing the third head pose estimation to a driver monitoring application.

14 . The apparatus of claim 1 , wherein outputting the third head pose estimation comprises determining content for a display in response to the determination of the third head pose estimation of the head of the user.

15 . The apparatus of claim 14 , wherein determining content for the display includes determining content of a subsystem of the apparatus that is associated with the third head pose estimation of the head of the user and presenting the determined content on the display.

16 . The apparatus of claim 15 , wherein the first device includes the display, and wherein presenting the determined content on the display includes communicating with the first device to cause the first device to present the determined content on the display of the first device.

17 . The apparatus of claim 15 , wherein the apparatus further comprises the display, and wherein presenting the determined content on the display includes communicating with the first device to cause the first device to present the determined content on the display of the first device.

18 . The apparatus of claim 17 , wherein the display is a heads up display (HUD).

19 . The apparatus of claim 1 , wherein the apparatus comprises the first sensor.

20 . The apparatus of claim 1 , wherein the first timestamp indicates a first time when the set of sensors of the first device generates or captures the one or more first elements of the first dataset, and wherein the second timestamp indicates a second time when the first sensor of the vehicle generates or captures the one or more second elements of the second dataset.

21 . A non-transitory, machine-readable storage medium storing instructions that, when executed by at least one processor of a vehicle, causes the at least one processor to perform operations that include:

obtaining, from a first device, a first dataset indicating a first head pose estimation of a head of a user in each of a first set of images generated by the first device, the first dataset being based on sensor data of a set of sensors of the first device;

obtaining a second dataset indicating a second head pose estimation of the head of the user in each of a set of images generated by a first sensor of the vehicle;

generating a fused dataset based on the first dataset and the second dataset, the fused dataset comprising time-aligned data derived from the first dataset and the second dataset, wherein the time-aligned data comprises one or more first elements of the first dataset and one or more second elements of the second dataset, and wherein a difference between a first timestamp associated with the one or more first elements and a second timestamp associated with the one or more second elements is within a predetermined time threshold;

determining a third head pose estimation of the head of the user based on the fused dataset; and

outputting the third head pose estimation.

22 . A computer-implemented method performed by an application program executed on at least one processor of a vehicle, the computer-implemented method comprising:

obtaining, from a first device by the application program executed on the at least one processor, a first dataset indicating a first head pose estimation of a head of a user in each of a first set of images generated by the first device, the first dataset being based on sensor data of a set of sensors of the first device;

obtaining, by the application program executed on the at least one processor, a second dataset indicating a second head pose estimation of the head of the user in each of a set of images generated by a first sensor of the vehicle;

generating, by the application program executed on the at least one processor, a fused dataset based on the first dataset and the second dataset, the fused dataset comprising time-aligned data derived from the first dataset and the second dataset, wherein the time-aligned data comprises one or more first elements of the first dataset and one or more second elements of the second dataset, and wherein a difference between a first timestamp associated with the one or more first elements and a second timestamp associated with the one or more second elements is within a predetermined time threshold;

determining, by the application program executed on the at least one processor, a third head pose estimation of the head of the user based on the fused dataset; and

outputting the third head pose estimation.

23 . The computer-implemented method of claim 22 , wherein the first dataset is generated based on an application of a first trained machine learning process to the sensor data of the set of sensors of the first device.

24 . The computer-implemented method of claim 23 , wherein the first trained machine learning process is a simultaneous location and mapping (SLAM) algorithm.

25 . The computer-implemented method of claim 22 , wherein the first device is a HMD device including the set of sensors, and wherein the computer-implemented method further comprises: obtaining, over one or more networks and from the first device, the first dataset.

26 . The computer-implemented method of claim 22 , wherein the computer-implemented method further comprises:

obtaining, over one or more networks and from the first device, the sensor data of the set of sensors of the first device;

applying a first trained machine learning process to the sensor data of the set of sensors, and

generating the first dataset based on an application of the first trained machine learning process to the sensor data.

27 . The computer-implemented method of claim 22 , wherein the first device is a HMD device including the set of sensors, and wherein the computer-implemented method further comprises: obtaining, over one or more networks and from a cloud computing system, the first dataset.

28 . The computer-implemented method of claim 22 , wherein generating the fused dataset includes applying an extended Kalman filter (EKF) to the first dataset and the second dataset.

29 . The computer-implemented method of claim 22 , wherein outputting the third head pose estimation comprises:

determining, by the application program executed on the at least one processor, content for a display in response to determining the third head pose estimation of the head of the user.

30 . The computer-implemented method of claim 22 , wherein outputting the third head pose estimation comprises providing the third head pose estimation to a driver monitoring application.

31 . The computer-implemented method of claim 22 , wherein the second dataset is generated based on an application of a second trained machine learning process to the set of images generated by the first sensor of the vehicle, and wherein the second trained machine learning process is a three-dimensional morphable model (3DMM).