IP Library Granted Patent US 12,195,022
Granted Patent B2
US 12,195,022 · App. 17/824,675 · Granted Jan 14, 2025

Apparatuses, systems and methods for classifying digital images

Inventors: Aaron Scott Chan (San Jose, CA); Kenneth Jason Sanchez (San Francisco, CA)
Assignee: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
B60W50/08B60R21/01542B60R21/01552B60W40/08B60W40/09B60W60/0053B60W60/0059G01S19/42G06F16/51G06F18/22G06F18/24G06F18/2413G06T7/337G06T7/70G06T7/73G06T15/005G06V20/59G06V20/593G06V20/597G06V20/64G06V40/10G06V40/11H04N19/17H04N19/423H04N23/45B60R2300/30B60R2300/8006B60W2040/0872B60W2050/0095B60W50/14B60W2420/403B60W2420/54B60W2540/22B60W2540/223B60W2540/225B60W2540/229B60W2540/26B60Y2400/30B60Y2400/3015B60Y2400/3017G06T2207/30196G06T2207/30268
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,195,022
App. No.
17/824,675
Granted
Jan 14, 2025
Kind
B2
Abstract

The present disclosure is directed to apparatuses, systems and methods for automatically classifying digital images of occupants inside a vehicle. More particularly, the present disclosure is directed to apparatuses, systems and methods for automatically classifying digital images of occupants inside a vehicle by comparing current image data to previously classified image data.

Claims (29)

1. A vehicle in-cabin imaging device, the vehicle in-cabin imaging device comprising:

a processor and a memory, wherein previously classified image data is stored on the memory wherein the previously classified image data is representative of known images of at least one vehicle interior, wherein the previously classified image data is normalized for a range of different drivers, and wherein the previously classified image data is representative of at least one of: a vehicle occupant elbow orientation, or a seat belt location;

at least one sensor for generating current image data, wherein the current image data is representative of current images of a vehicle interior, and wherein the current image data is representative of a current vehicle occupant elbow orientation; and

a current image classification module stored on the memory that, when executed by the processor, causes the processor to classify current images of the vehicle interior based on a comparison of the current image data with the previously classified image data, wherein at least one current image is classified as representative of the current vehicle occupant elbow orientation.

2. The vehicle in-cabin imaging device as in claim 1 , wherein the at least one sensor is selected from: at least one digital image sensor, at least one ultra-sonic sensor, at least one radar-sensor, at least one infrared light sensor, or at least one laser light sensor.

3. The vehicle in-cabin imaging device as in claim 1 , wherein the current image data is representative of a three-dimensional location of at least one vehicle occupant within the vehicle interior.

4. The vehicle in-cabin imaging device as in claim 1 , wherein the current image data is representative of an orientation of at least a portion of at least one vehicle occupant within the vehicle interior.

5. The vehicle in-cabin imaging device as in claim 1 , wherein the current image data is representative of a three-dimensional location of at least a portion of at least one vehicle occupant within the vehicle interior and an orientation of the portion of the at least one vehicle occupant.

6. The vehicle in-cabin imaging device as in claim 1 , wherein the current image data is representative of at least one of: a vehicle occupant head location/orientation, a vehicle occupant hand location/orientation, a vehicle occupant arm location/orientation, a vehicle occupant torso location/orientation, a cellular telephone location, a vehicle occupant eye location/orientation, or a vehicle seat location/orientation.

7. The vehicle in-cabin imaging device as in claim 1 , wherein the previously classified image data is representative of both the vehicle occupant elbow orientation, and the seat belt location.

8. A computer-implemented method for automatically classifying images of an interior of a vehicle, the method comprising:

receiving previously classified image data at a processor, from a remote computing device, in response to the processor executing a previously classified image data receiving module, wherein the previously classified image data is representative of known images of at least one vehicle interior, wherein the previously classified image data is normalized for a range of different drivers, and wherein the previously classified image data is representative of at least one of: a vehicle occupant elbow orientation, or a seat belt location;

receiving current image data at the processor, from at least one sensor, wherein the current image data is representative of current images of a vehicle interior, and wherein the current image data is representative of a current vehicle occupant elbow orientation; and

classifying current images, using the processor, based on a comparison of the current image data with the previously classified image data, wherein at least one current image is classified as representative of the current vehicle occupant elbow orientation.

9. The computer-implemented method of claim 8 , wherein the current image data is representative of at least one of: a vehicle occupant head location/orientation, a vehicle occupant hand location/orientation, a vehicle occupant arm location/orientation, a vehicle occupant torso location/orientation, a cellular telephone location, a vehicle occupant eye location/orientation, or a vehicle seat location/orientation.

10. The computer-implemented method of claim 8 , wherein the previously classified image data is representative of both the vehicle occupant elbow orientation, and the seat belt location.

11. The computer-implemented method of claim 8 , wherein the current image data is representative of a three-dimensional location of at least one vehicle occupant within the vehicle interior.

12. The computer-implemented method of claim 8 , wherein the current image data is representative of an orientation of at least a portion of at least one vehicle occupant within the vehicle interior.

13. The computer-implemented method of claim 8 , wherein the current image data is representative of a three-dimensional location of at least a portion of at least one vehicle occupant within the vehicle interior and an orientation of the portion of the at least one vehicle occupant.

14. A non-transitory computer-readable medium storing computer-readable instructions that, when executed by a processor, cause the processor to automatically classify images of an interior of a vehicle, the non-transitory computer-readable medium comprising:

a previously classified image data receiving module that, when executed by the processor, causes the processor to receive previously classified image data from a remote computing device, wherein the previously classified image data is representative of known images of at least one vehicle interior, wherein the previously classified image data is normalized for a range of different drivers, and wherein the previously classified image data is representative of at least one of: a vehicle occupant elbow orientation, or a seat belt location;

a current image data receiving module that, when executed by the processor, causes the processor to receive current image data from at least one sensor, wherein the current image data is representative of current images of a vehicle interior, and wherein the current image data is representative of a current vehicle occupant elbow orientation; and

a current image classification module that, when executed by the processor, causes the processor to classify current images based on a comparison of the current image data with the previously classified image data, wherein at least one current image is classified as representative of the current vehicle occupant elbow orientation.

15. The non-transitory computer-readable medium of claim 14 , wherein the current image data is representative of a three-dimensional location of at least a portion of at least one vehicle occupant within the vehicle interior and an orientation of the portion of the at least one vehicle occupant.

16. The non-transitory computer-readable medium of claim 14 , wherein the current image data is representative of an orientation of at least a portion of at least one vehicle occupant within the vehicle interior.

17. The non-transitory computer-readable medium of claim 14 , wherein the current image data is representative of at least one of: a vehicle occupant head location/orientation, a vehicle occupant hand location/orientation, a vehicle occupant arm location/orientation, a vehicle occupant torso location/orientation, a cellular telephone location, a vehicle occupant eye location/orientation, or a vehicle seat location/orientation.

18. The non-transitory computer-readable medium of claim 14 , wherein the previously classified image data is representative of both the vehicle occupant elbow orientation, and the seat belt location.

19. The non-transitory computer-readable medium of claim 14 , wherein the at least one sensor is selected from: at least one ultra-sonic sensor, at least one radar-sensor, at least one infrared light sensor, or at least one laser light sensor.

20. The vehicle in-cabin imaging device as in claim 1 , wherein the current image data is representative of a current seat belt location.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 15, 2024
From: CHAN, AARON SCOTT; SANCHEZ, KENNETH J.
To: STATE FARM MUTUAL AUTOMOBILE INSURANCE COMPANY
Reel/Frame 066119/0274 →
Continuity (5)
Continuation 17039916 · Sep 30, 2020
Continuation 16797009 · Feb 21, 2020
Continuation 14994305 · Jan 13, 2016
Provisional Application 62102672 · Jan 13, 2015
Related Publication 20220292851A1 · Sep 15, 2022
References Cited (125)
US 5627765A · Robotham et al. · 1997 [cited by applicant]
US 5818535A · Asnis et al. · 1998 [cited by applicant]
US 7659827B2 · Gunderson · 2010 [cited by examiner]
US 7792328B2 · Albertson · 2010 [cited by examiner]
US 8441548B1 · Nechyba · 2013 [cited by examiner]
US 8527146B1 · Jackson · 2013 [cited by examiner]
US 8606492B1 · Botnen · 2013 [cited by examiner]
US 8861804B1 · Johnson et al. · 2014 [cited by applicant]
US 8930072B1 · Lambert · 2015 [cited by examiner]
US 8957779B2 · Wu et al. · 2015 [cited by applicant]
US 9020482B2 · Jones · 2015 [cited by applicant]
US 9081999B2 · Baele et al. · 2015 [cited by applicant]
US 9104535B1 · Brinkmann · 2015 [cited by examiner]
US 9117246B2 · McClellan · 2015 [cited by examiner]
US 9428052B1 · Raz · 2016 [cited by examiner]
US 9676392B1 · Brinkmann · 2017 [cited by examiner]
US 9690292B1 · Chan et al. · 2017 [cited by applicant]
US 9710717B1 · Sanchez et al. · 2017 [cited by applicant]
US 9886637B1 · Chan et al. · 2018 [cited by applicant]
US 9928434B1 · Chan · 2018 [cited by examiner]
US 9944296B1 · Sanchez et al. · 2018 [cited by applicant]
US 9990553B1 · Chan · 2018 [cited by examiner]
US 9990554B1 · Sanchez et al. · 2018 [cited by applicant]
US 10013620B1 · Sanchez et al. · 2018 [cited by applicant]
US 10089542B1 · Chan et al. · 2018 [cited by applicant]
US 10140533B1 · Chan et al. · 2018 [cited by applicant]
US 10147007B1 · Chan et al. · 2018 [cited by applicant]
US 10147008B1 · Chan et al. · 2018 [cited by applicant]
US 10163018B1 · Chan · 2018 [cited by examiner]
US 10189480B1 · Sanchez et al. · 2019 [cited by applicant]
US 10229333B1 · Sanchez et al. · 2019 [cited by applicant]
US 10241512B1 · Chan et al. · 2019 [cited by applicant]
US 10282624B1 · Chan · 2019 [cited by examiner]
US 10311749B1 · Kypri et al. · 2019 [cited by applicant]
US 10325167B1 · Chan et al. · 2019 [cited by applicant]
US 10562536B1 · Sanchez et al. · 2020 [cited by applicant]
US 10565460B1 · Chan · 2020 [cited by examiner]
US 10607095B1 · Chan et al. · 2020 [cited by applicant]
US 10943136B1 · Chan · 2021 [cited by examiner]
US 10958987B1 · Hegar et al. · 2021 [cited by applicant]
US 11367293B1 · Chan · 2022 [cited by examiner]
US 11417121B1 · Chan · 2022 [cited by examiner]
US 11423671B1 · Chan · 2022 [cited by examiner]
US 20030154190A1 · Misawa et al. · 2003 [cited by applicant]
US 20040197014A1 · Oohashi · 2004 [cited by applicant]
US 20040247192A1 · Kajiki et al. · 2004 [cited by applicant]
US 20050046584A1 · Breed · 2005 [cited by examiner]
US 20050073136A1 · Larsson · 2005 [cited by examiner]
US 20060013495A1 · Duan et al. · 2006 [cited by applicant]
US 20060255241A1 · Shibao · 2006 [cited by applicant]
US 20060267317A1 · Ida · 2006 [cited by examiner]
US 20070120697A1 · Ayoub · 2007 [cited by examiner]
US 20070135982A1 · Breed · 2007 [cited by examiner]
US 20080130998A1 · Maidment · 2008 [cited by examiner]
US 20080158357A1 · Connell et al. · 2008 [cited by applicant]
US 20080169914A1 · Albertson · 2008 [cited by examiner]
US 20080212850A1 · Adachi et al. · 2008 [cited by applicant]
US 20090180697A1 · Erol et al. · 2009 [cited by applicant]
US 20100157059A1 · Ishikawa · 2010 [cited by examiner]
US 20100205012A1 · McClellan · 2010 [cited by examiner]
US 20100306373A1 · Wormley · 2010 [cited by applicant]
US 20110254956A1 · Ishikawa · 2011 [cited by examiner]
US 20120028682A1 · Danne · 2012 [cited by applicant]
US 20120252350A1 · Steinmetz et al. · 2012 [cited by applicant]
US 20130030811A1 · Olleon et al. · 2013 [cited by applicant]
US 20130166098A1 · Lavie · 2013 [cited by examiner]
US 20130314429A1 · Croxford et al. · 2013 [cited by applicant]
US 20140139451A1 · Levesque et al. · 2014 [cited by applicant]
US 20140172467A1 · He · 2014 [cited by examiner]
US 20140210625A1 · Nemat-Nasser · 2014 [cited by examiner]
US 20140240132A1 · Bychkov · 2014 [cited by examiner]
US 20140256303A1 · Jones · 2014 [cited by applicant]
US 20140257659A1 · Dariush · 2014 [cited by applicant]
US 20140274023A1 · Rajeevalochana et al. · 2014 [cited by applicant]
US 20140322676A1 · Raman · 2014 [cited by examiner]
US 20140375808A1 · Kao et al. · 2014 [cited by applicant]
US 20150077237A1 · Chou · 2015 [cited by examiner]
US 20150092056A1 · Rau · 2015 [cited by examiner]
US 20150095804A1 · Grossman et al. · 2015 [cited by applicant]
US 20150161913A1 · Dominguez et al. · 2015 [cited by applicant]
US 20150193885A1 · Akiva · 2015 [cited by examiner]
US 20150314682A1 · Enriquez Ortiz · 2015 [cited by applicant]
US 20160042543A1 · Hashimoto · 2016 [cited by examiner]
US 20160070958A1 · Whelan et al. · 2016 [cited by applicant]
US 20160090041A1 · Nagasawa et al. · 2016 [cited by applicant]
US 20160093212A1 · Barfield, Jr. · 2016 [cited by examiner]
US 20160174132A1 · Hynes · 2016 [cited by applicant]
US 20160241817A1 · Sun et al. · 2016 [cited by applicant]
US 20170140293A1 · Vij · 2017 [cited by examiner]
US 20170227841A1 · Niemela et al. · 2017 [cited by applicant]
US 20190156150A1 · Krishnan · 2019 [cited by applicant]
US 20210264179A1 · Sanchez et al. · 2021 [cited by applicant]
US 20210284066A1 · Pedersen · 2021 [cited by applicant]
CA 2849725A1 · 2013 [cited by examiner]
Raymond Brodeur et al; “The Initial Position and Postural Attitudes of Driver Occupants, Posture”; MSU, Ergonomics research laboratory, Jun. 21, 1996 (Year: 1996). [cited by examiner]
Erik Murphy et al; “Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness”; IEEE June (Year: 2010). [cited by examiner]
Raymond Brodeur et al.; “The Initial Position and Postural Attitudes of Driver Occupants, Posture”; MSU, Ergonomics research laboratory, Jun. 21 (Year: 1996). [cited by examiner]
Erik Murphy et al; “Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness”; IEEE Jun. 2010 (Year: 2010). [cited by examiner]
Yang et al ; “Objective Evaluation Method of Steering Comfort Based on Movement Quality Evaluation of Driver Steering Maneuver”; Chinese journal of mechanical engineering, 2014 (Year: 2014). [cited by examiner]
Caird et al., A meta-analysis of the effects of texting on driving, Accident Analysis & Prevention, 71:311-8 (2014). [cited by applicant]
Erik Murphy et al., “Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness”, IEEE 2010. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/994,308 dated Feb. 26, 2018. [cited by applicant]
Final Office Action for U.S. Appl. No. 14/994,308 dated May 15, 2019. [cited by applicant]
Murphy-Chutorian et al., Head Pose Estimation and Augmented Reality Tracking: An Integrated System and Evaluation for Monitoring Driver Awareness, IEEE Transactions on Intelligent Transportation Systems, vol. 11, No. 2,… [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/994,308 dated Oct. 18, 2018. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 14/994,308 dated Sep. 7, 2017. [cited by applicant]
Non-Final Office Action for U.S. Appl. No. 15/964,209 dated Feb. 25, 2020. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 14/994,308 dated Oct. 18, 2019. [cited by applicant]
Simons-Morton et al., Keep Your Eyes on the Road: Young Driver Crash Risk Increases According to Duration of Distraction, Journal of Adolescent Health, 54:561-7 (2014). [cited by applicant]
Tran et al., Vision for Driver Assistance: Looking at People in a Vehicle. In: Moeslund et al. (eds), Visual Analysis of Humans. Springer, London, 2011. [cited by applicant]
U.S. Appl. No. 15/964,209, Final Office Action, dated Jun. 15, 2020. [cited by applicant]
U.S. Appl. No. 15/964,209, Final Office Action, dated Mar. 30, 2021. [cited by applicant]
U.S. Appl. No. 15/964,209, Nonfinal Office Action, dated Dec. 15, 2020. [cited by applicant]
U.S. Appl. No. 16/735,198, Final Office Action, dated Sep. 10, 2021. [cited by applicant]
U.S. Appl. No. 16/735,198, Nonfinal Office Action, dated Dec. 20, 2021. [cited by applicant]
U.S. Appl. No. 16/735,198, Nonfinal Office Action, dated Jun. 3, 2021. [cited by applicant]
U.S. Appl. No. 16/797,009, Final Office Action, dated Nov. 18, 2021. [cited by applicant]
U.S. Appl. No. 16/797,009, Nonfinal Office Action, dated Jul. 12, 2021. [cited by applicant]
U.S. Appl. No. 17/039,916, Nonfinal Office Action, dated Oct. 26, 2021. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 16/797,009 dated Mar. 23, 2022. [cited by applicant]
Notice of Allowance for U.S. Appl. No. 17/039,916 dated Feb. 23, 2022. [cited by applicant]
U.S. Appl. No. 17/859,866, Nonfinal Office Action, dated Dec. 28, 2022. [cited by applicant]
U.S. Appl. No. 16/735,198, Notice of Allowance, dated Apr. 13, 2022. [cited by applicant]
U.S. Appl. No. 17/859,866, Notice of Allowance, dated Feb. 13, 2023. [cited by applicant]
U.S. Appl. No. 18/144,701, Notice of Allowance, dated Nov. 7, 2024. [cited by applicant]
Cited By (1)
US 12,664,797