IP Library › Granted Patent US 12,466,422
Granted Patent B2
US 12,466,422 · App. 18/161,290 · Granted Nov 11, 2025

Large animal detection and intervention in a vehicle

Inventors: Alireza Rahimpour (Palo Alto, CA); Navid Fallahinia (Watertown, MA); Devesh Upadhyay (Canton, MI); Justin Miller (Berkley, MI)
Assignee: Ford Global Technologies, LLC
B60W50/14B60W30/0956B60W50/0097G06V10/25G06V10/751G06V10/82G06V20/58G06V40/10H04N23/23B60W2050/146B60W2420/403B60W2554/4045B60W2556/10
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,466,422
App. No.
18/161,290
Granted
Nov 11, 2025
Kind
B2
Abstract

A far-infrared camera mounted in a vehicle generates an image frame. When an image of a large animal is identified in the image frame, a pixel intensity of the large animal image is determined. An estimated distance to the large animal from the far-infrared camera based on the pixel intensity is determined. When the animal is classified as a tracked animal, and future trajectories of the tracked animal and the vehicle intersect, a component in the vehicle is actuated.

Claims (44)

1 . A computing device, comprising a processor and a memory, wherein the memory stores instructions executable by the processor such that the computing device is programmed to:

upon identifying an image of a large animal in an image frame obtained with a far-infrared camera mounted to a vehicle, wherein the identifying includes identifying a type of the large animal selected from a group including one or more of deer, bear, elk, horse, or cow:

determine a pixel intensity of the large animal image;

determine an estimated distance to the large animal from the far-infrared camera based on the pixel intensity;

classify the large animal as one of a tracked animal and a non-tracked animal based on the large animal image, including a pose based on the type of the large animal and whether the pose indicates that the large animal is likely to occupy a road; and

upon classifying the large animal as the tracked animal and upon determining that a future trajectory of the tracked animal and a future trajectory of the vehicle intersect, actuate a component in the vehicle.

2 . The computing device of claim 1 , wherein the computing device is further programmed to:

generate a sequence of image frames; and

predict the future trajectory of the tracked animal based on a past trajectory of the tracked animal, a scene context, and a pixel motion.

3 . The computing device of claim 1 , wherein the computing device is further programmed to generate a bounding box that bounds the large animal image.

4 . The computing device of claim 3 , wherein the computing device is further programmed to, prior to classifying the large animal, determine the estimated distance based on both the pixel intensity and on a size of the bounding box.

5 . The computing device of claim 3 , wherein the computing device is further programmed to:

generate a sequence of image frames; and

determine a past trajectory of the tracked animal based on a movement of the bounding box of the large animal image in the sequence of the image frames.

6 . The computing device of claim 1 , wherein the computing device is further programmed to:

generate a sequence of image frames; and

determine the future trajectory of the tracked animal based on a motion of pixels in the sequence of the image frames.

7 . The computing device of claim 1 , wherein the computing device is further programmed to:

generate a sequence of image frames; and

predict the future trajectory of the tracked animal by modeling the future trajectory with a convolutional long short term memory attention-based multi-stream encoder-decoder model that includes an attention module that assigns weights to vectors derived from a past trajectory of the tracked animal, a scene context, and a pixel motion.

8 . The computing device of claim 1 , wherein the component actuated comprises a heads-up display that displays a visual notification of the tracked animal to a vehicle operator.

9 . The computing device of claim 1 , wherein the computing device is further programmed to, prior to classifying the large animal, identify key points of the large animal image and their respective locations in the image frames to determine a pose of the large animal and include the pose as a factor in classifying the large animal.

10 . A method of predicting a future trajectory of a large animal and actuating a component, the method comprising the steps of:

upon identifying an image of a large animal in an image frame obtained with a far-infrared camera mounted to a vehicle, wherein the identifying includes identifying a type of the large animal selected from a group including one or more of deer, bear, elk, horse, or cow:

determining a pixel intensity of the large animal image;

determining an estimated distance to the large animal from the far-infrared camera based on the pixel intensity;

classifying the large animal is classified as one of a tracked animal and a non-tracked animal based on the large animal image, including a pose based on the type of the large animal and whether the pose indicates that the large animal is likely to occupy a road; and

upon classifying the large animal as a tracked animal and upon determining that the future trajectory of the tracked animal and the future trajectory of the vehicle intersect, actuating a component in the vehicle.

11 . The method of claim 10 , further comprising the steps of:

generating a sequence of image frames; and

predicting the future trajectory of the tracked animal based on a past trajectory of the tracked animal, a scene context, and a pixel motion.

12 . The method of claim 10 , further comprising the step of generating a bounding box that bounds the large animal image.

13 . The method of claim 12 , further comprising the step of, prior to classifying the large animal, determining the estimated distance based on both the pixel intensity and on a size of the bounding box.

14 . The method of claim 12 , further comprising the steps of:

generating a sequence of image frames; and

determining a past trajectory of the tracked animal based on a movement of the bounding box of the large animal image in the sequence of the image frames.

15 . The method of claim 10 , further comprising the steps of:

generating a sequence of image frames; and

determining the future trajectory of the tracked animal based on a motion of pixels in the sequence of the image frames.

16 . The method of claim 10 , further comprising the steps of:

generating a sequence of image frames; and

predicting the future trajectory of the tracked animal by modeling the future trajectory with a convolutional long short term memory attention-based multi-stream encoder-decoder model that includes an attention module and assigning weights to vectors derived from a past trajectory of the tracked animal, a scene context, and a pixel motion.

17 . The method of claim 10 , wherein the component being actuated comprises a heads-up display that displays a visual notification of the tracked animal to a vehicle operator.

18 . The method of claim 10 , further comprising the step of, prior to classifying the large animal, identifying key points of the large animal image and their respective locations in the image frames to determine a pose of the large animal and including the pose as a factor in classifying the large animal.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Jan 30, 2023
From: RAHIMPOUR, ALIREZA; FALLAHINIA, NAVID; UPADHYAY, DEVESH; MILLER, JUSTIN
To: FORD GLOBAL TECHNOLOGIES, LLC
Reel/Frame 062530/0769 →
Continuity (1)
Related Publication 20240253652A1 · Aug 1, 2024
References Cited (53)
US 12128887B1 · Kiiski · 2024 [cited by examiner]
US 12159481B2 · Quinn · 2024 [cited by examiner]
US 20170327112A1 · Yokoyama · 2017 [cited by examiner]
US 20200294266A1 · Botonjic · 2020 [cited by examiner]
US 20210201052A1 · Ranga et al. · 2021 [cited by applicant]
US 20210295171A1 · Kamenev et al. · 2021 [cited by applicant]
US 20220057806A1 · Guo · 2022 [cited by examiner]
US 20220101635A1 · Koivisto et al. · 2022 [cited by applicant]
US 20230237873A1 · Quinn · 2023 [cited by examiner]
US 20240073545A1 · Wilton · 2024 [cited by examiner]
WO 2018132608A2 · 2018 [cited by applicant]
Huang, Y. et al., “Head Pose based Intention Prediction Using Discrete Dynamic Bayesian Network,” Seventh International Conference on Distributed Smart Cameras, 2013, 6 pages. [cited by applicant]
Vidal, M. et al., “Perspectives on Individual Animal Identification from Biology and Computer Vision,” Integrated and Comparative Biology, vol. 61, No. 3, Jan. 2021, pp. 900-916. [cited by applicant]
Ukwuoma, C. et al., “Animal species detection and classification framework based on modified multi-scale attention mechanism and feature pyramid network,” Scientific African, 2022, 18 pages. [cited by applicant]
Vecvanags, A. et al., “Ungulate Detection and Species Classification from Camera Trap Images Using RetinaNet and Faster R-CNN,” entropy, 2022, 13 pages. [cited by applicant]
Adams, E., “Volvo's cars now spot moose and hit the Brakes for You,” www.wired.com, Jan. 27, 2017, 7 pages. [cited by applicant]
Ahmed. S. et al., “Pedestrian and Cyclist Detection and Intent Estimation for Autonomous Vehicles: A Survey,” Applied Sciences, 2019, 38 pages. [cited by applicant]
Fang, Z. et al., “On-Board Detection of Pedestrian Intentions,” Sensors, 2017, 15 pages. [cited by applicant]
Gupta, A. et al., “Social GAN: Socially Acceptable Trajectories with Generative Adversarial Networks,” arXiv:1803.10892v1 [cs.CV], Mar. 29, 2018, 10 pages. [cited by applicant]
Keller, C. et al., “Will the Pedestrian Cross? A Study on Pedestrian Path Prediction,” IEEE Transactions on Intelligent Transportation Systems, Apr. 2014, 14 pages. [cited by applicant]
Kooij, J. et al., “Context-based Pedestrian Path Prediction,” ECCV, 2014, 3 pages. [cited by applicant]
Malla, S. et al., “Titan: Future Forecast using Action Priors,” Computer Vision Foundation, 2020, 11 pages. [cited by applicant]
Rasouli, R. et al., “PIE: A Large-Scale Dataset and Models for Pedestrian Intention Estimation and Trajectory Prediction,” Computer Vision Foundation, 2019, 10 pages. [cited by applicant]
Rasouli, A. et al., “Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior,” Computer Vision Foundation, 2017, 8 pages. [cited by applicant]
Ridel, D. et al., “A Literature Review on the Prediction of Pedestrian Behavior in Urban Scenarios,” 21st International Conference on Intelligent Transportation Systems, Nov. 2018, 8 pages. [cited by applicant]
Schulz, A. et al., “Pedestrian Intention Recognition using Latent-dynamic Conditional Random Fields,” IEEE Intelligent Vehicles Symposium, 2015, 6 pages. [cited by applicant]
Hore, P. et al., “A Comprehensive Guide to Attention Mechanism in Deep Learning for Everyone,” www.analyticsvidhya.com, Nov. 20, 2019, 18 pages. [cited by applicant]
Ren, S. et al., “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” arXiv:1506.01497v3 [cs.CV], Jan. 6, 2016, 14 pages. [cited by applicant]
Mur-Artal. R. et al., “ORB-SLAM: A Versatile and Accurate Monocular SLAM System,” IEEE Transactions on Robotics, vol. 31, Issue 5, Oct. 2015, 33 pages. [cited by applicant]
Vaswani, A. et al., “Attention Is All You Need,” 31st Conference on Neural Information Processing Systems, 2017, 11 pages. [cited by applicant]
Park, J. et al., “Deep Learning Based Detection of Missing Tooth Regions for Dental Implant Planning in Panoramic Radiographic Images,” Applied Sciences, 2022, 10 pages. [cited by applicant]
Gupta, P. et al., “Skin Lesion Detection using VGG-16 and ResNet-50 based Hybrid CNN Model,” International Journal of Creative Research Thoughts, vol. 10, Issue 6, Jun. 6, 2022, 8 pages. [cited by applicant]
Tan, M. et al., “Animal Detection and Classification from Camera Trap Images Using Different Mainstream Object Detection Architectures,” animals, 2022, 16 pages. [cited by applicant]
Li, W. et al., “Multiple attention-based encoder-decoder networks for gasmeter character recognition,” Scientific Reports, Jun. 20, 2022, 35 pages. [cited by applicant]
“Deep Learning,” Wikipedia, 2022, 41 pages. [cited by applicant]
Keldenich, T. et al., “What is Attention Mechanism in Deep Learning?—Quickly Understand,” Inside Machine Learning, Oct. 20, 2021, 11 pages. [cited by applicant]
Clark, D. et al., “Using Machine Learning Methods to Predict the Movement Trajectories of the Louisiana Black Bear,” SMU Data Science Review, vol. 5, No. 1, 2021, 28 pages. [cited by applicant]
“How is RELU used on convolutional layer,” Cross Validated, 2019, 2 pages. [cited by applicant]
Brownlee, J., “How Does Attention Work in Encoder-Decoder Recurrent Neural Networks,” Machine Learning Mastery, Oct. 13, 2017, 29 pages. [cited by applicant]
Cristina, S., “A Tour of Attention-Based Architectures,” Machine Learning Mastery, Aug. 30, 2022, 17 pages. [cited by applicant]
Rizzoli, A., “Annotating With Bounding Boxes: Quality Best Practices,” www.v7labs.com/blog, Oct. 3, 2022, 14 pages. [cited by applicant]
Mansouri, I., “Computer Vision Part 8: Pose Estimation, stick figures using AI,” Jul. 28, 2020, 21 pages. [cited by applicant]
“How to Classify Animal Images via a Convolutional Neural Network,” https://hackernoon.com, Jul. 29, 2020, 17 pages. [cited by applicant]
Paul. S., et al., “Keypoint Detection with Transfer Learning,” Keras, May 2, 2021, 11 pages. [cited by applicant]
Yang, Y. et al., “APT-36K: A Large-scale Benchmark for Animal Pose Estimation and Tracking,” arXiv:2206.05683v2 [cs.CV], Oct. 3, 2022, 19 pages. [cited by applicant]
“Convolutional neural network,” Wikipedia, 2022, 35 pages. [cited by applicant]
“What is a convolutional neural network?” www.wgu.edu, Aug. 3, 2020, 8 pages. [cited by applicant]
“Zero to Hero: Guide to Object Detection using Deep Learning: Faster R-CNN, YOLO, SSD,” CV-Tricks.com, 2022, 12 pages. [cited by applicant]
Ulhaq, A. et al., “Automated Detection of Animals in Low-Resolution Airborne Thermal Imagery,” remote sensing, 2021, 14 pages. [cited by applicant]
Vidal, M. et al., “Perspectives on Individual Animal Identification from Biology and Computer Vision,” Integrative and Comparative Biology, vol. 61, No. 3, Jan. 2021, pp. 900-916. [cited by applicant]
Yu, H. et al., “AP-10K: A Benchmark for Animal Pose Estimation in the Wild,” 35th Conference on Neural Information Processing Systems, 2021, 12 pages. [cited by applicant]
Cao, J. et al., “Cross-Domain Adaptation for Animal Pose Estimation,” Computer Vision Foundation, 2019, 10 pages. [cited by applicant]
Mowen, D. et al., “Improving Road Safety during Nocturnal Hours by Characterizing Animal Poses Utilizing CNN-based Analysis of Thermal Images,” sustainability, 2022, 15 pages. [cited by applicant]