IP Library › Granted Patent US 12,347,194
Granted Patent B2
US 12,347,194 · App. 17/818,283 · Granted Jul 1, 2025

Automated generation of haptic effects based on haptics data

Inventors: David S. Resnick (Los Angeles, CA); Bryce Horwood (Shoreline, WA); Keith J. Comito (Seaford, NY); Steve Sang-Hyuck Lee (Ann Arbor, MI); Nicholas M. Arciero (Los Angeles, CA); Joseph A. Inzerillo (Brooklyn, NY); Frederik S. Boll (Los Angeles, CA); Gad Rouache (Los Angeles, CA); Samuel J. Reisner (Farmington, CT)
Assignees: Disney Enterprises, Inc.; Hulu, LLC
G06V20/44G06F3/016G06V10/7715G08B6/00H04N21/4104
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,347,194
App. No.
17/818,283
Granted
Jul 1, 2025
Kind
B2
Abstract

Embodiments provide systems and techniques for automated generation of haptic effects based on haptics data. An example technique includes obtaining a set of haptic data associated with media content being streamed to a computing device. The media content is displayed on the first computing device. One or more haptic effects are generated using one or more haptic devices while displaying the media content, based on the set of haptic data. One or more parameters of the one or more haptic effects is adjusted when a predetermined condition(s) is satisfied.

Claims (43)

1. A computer-implemented method comprising:

obtaining a first set of haptic data associated with media content being streamed to a first computing device;

displaying the media content on the first computing device;

generating, based on the first set of haptic data, one or more haptic effects using one or more haptic devices associated with the first computing device, while displaying the media content;

determining, by the first computing device, resources available at a first point in time on the first computing device while displaying the media content; and

upon detecting one or more conditions, dynamically adjusting (i) the media content and (ii) one or more parameters of the one or more haptic effects generated using the one or more haptic devices, based on the resources available at the first point in time on the first computing device, the resources comprising a quality of connection between the first computing device and a second computing device, wherein dynamically adjusting the one or more parameters comprises:

requesting, by the first computing device, (i) a second set of haptic data, different from the first set of haptic data and associated with the media content and (ii) a third set of haptic data, different from the first set of haptic data and the second set of haptic data and associated with the media content, based on the resources available at the first point in time on the first computing device; and

generating the one or more haptic effects according to the second set of haptic data and the third set of haptic data.

2. The computer-implemented method of claim 1 , wherein the one or more parameters comprise an amount of the one or more haptic effects generated using the one or more haptic devices.

3. The computer-implemented method of claim 1 , wherein the one or more parameters comprise an intensity of the one or more haptic effects generated using the one or more haptic devices.

4. The computer-implemented method of claim 1 , wherein the one or more parameters comprise a frequency of the one or more haptic effects generated using the one or more haptic devices.

5. The computer-implemented method of claim 1 , wherein obtaining the first set of haptic data comprises requesting the first set of haptic data based on the resources available at a second point in time.

6. The computer-implemented method of claim 1 , wherein the one or more conditions comprises an amount of the resources available at the first point in time being less than a threshold.

7. The computer-implemented method of claim 1 , wherein the one or more conditions comprises an amount of the resources available at the first point in time exceeding a threshold.

8. The computer-implemented method of claim 1 , wherein the one or more conditions comprises the quality of connection between the first computing device and the second computing device being less than a threshold.

9. The computer-implemented method of claim 1 , wherein the one or more haptic effects is generated on a second computing device while the media content is played on the first computing device.

10. The computer-implemented method of claim 1 , further comprising controlling the one or more parameters of the one or more haptic effects, based on input received via a user interface of the first computing device.

11. The computer-implemented method of claim 1 , wherein the one or more conditions is based on at least one of (i) one or more environmental conditions or (ii) one or more accessibility factors associated with a user of the first computing device.

12. A computing device comprising:

one or more memories collectively storing instructions; and

one or more processors communicatively coupled to the one or more memories, the one or more processors being collectively configured to execute the instructions to cause the computing device to perform an operation comprising:

obtaining a first set of haptic data associated with media content being streamed to the computing device;

displaying the media content on the computing device;

generating, based on the first set of haptic data, one or more haptic effects using one or more haptic devices associated with the computing device while displaying the media content;

determining resources available at a first point in time on the computing device while displaying the media content; and

upon detecting one or more conditions, dynamically adjusting (i) the media content and (ii) one or more parameters of the one or more haptic effects generated using the one or more haptic devices, based on the resources available at the first point in time on the computing device, the resources comprising a quality of connection between the computing device and another computing device, wherein dynamically adjusting the one or more parameters comprises:

requesting (i) a second set of haptic data, different from the first set of haptic data and associated with the media content and (ii) a third set of haptic data, different from the first set of haptic data and the second set of haptic data and associated with the media content, based on the resources available at the first point in time on the computing device; and

generating the one or more haptic effects according to the second set of haptic data and the third set of haptic data.

13. The computing device of claim 12 , wherein the one or more parameters comprise at least one of (i) an amount of the one or more haptic effects generated using the one or more haptic devices, (ii) an intensity of the one or more haptic effects generated using the one or more haptic devices, or (iii) a frequency of the one or more haptic effects generated using the one or more haptic devices.

14. The computing device of claim 12 , wherein obtaining the first set of haptic data comprises requesting the first set of haptic data based on the resources available at a second point in time.

15. The computing device of claim 12 , wherein the one or more conditions comprises (i) an amount of the resources available at the first point in time being less than a threshold, (ii) an amount of the resources available at the first point in time exceeding a threshold, or (iii) the quality of connection between the computing device and the other computing device being less than a threshold.

16. The computing device of claim 12 , wherein the one or more conditions is based on at least one of one or more environmental conditions or one or more accessibility factors associated with a user of the computing device.

17. A non-transitory computer-readable medium containing computer program code that, when collectively executed by operation of one or more computer processors, performs an operation comprising:

obtaining a first set of haptic data associated with media content being streamed to a first computing device;

displaying the media content on the first computing device;

generating, based on the first set of haptic data, one or more haptic effects using one or more haptic devices associated with the first computing device, while displaying the media content;

determining, by the first computing device, resources available at a first point in time on the first computing device while displaying the media content; and

upon detecting one or more conditions, dynamically adjusting (i) the media content and (ii) one or more parameters of the one or more haptic effects generated using the one or more haptic devices, based on the resources available at the first point in time on the first computing device, the resources comprising a quality of connection between the first computing device and a second computing device, wherein adjusting the one or more parameters comprises:

requesting, by the first computing device, (i) a second set of haptic data, different from the first set of haptic data and associated with the media content and (ii) a third set of haptic data, different from the first set of haptic data and the second set of haptic data and associated with the media content, based on the resources available at the first point in time on the first computing device; and

generating the one or more haptic effects according to the second set of haptic data and the third set of haptic data.

18. The non-transitory computer-readable medium of claim 17 , wherein the one or more parameters comprise an amount of the one or more haptic effects generated using the one or more haptic devices.

19. The non-transitory computer-readable medium of claim 17 , wherein the one or more parameters comprise an intensity of the one or more haptic effects generated using the one or more haptic devices.

20. The non-transitory computer-readable medium of claim 17 , wherein the one or more parameters comprise a frequency of the one or more haptic effects generated using the one or more haptic devices.

Assignments (2)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 29, 2023
From: RESNICK, DAVID S.; HORWOOD, BRYCE; LEE, STEVE SANG-HYUCK
To: HULU, LLC
Reel/Frame 065703/0364 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 29, 2023
From: ARCIERO, NICHOLAS M.; COMITO, KEITH J.; INZERILLO, JOSEPH A.; BOLL, FREDERIK S.; ROUACHE, GAD; REISNER, SAMUEL J.
To: DISNEY ENTERPRISES, INC.
Reel/Frame 065703/0515 →
Continuity (2)
Provisional Application 63231142 · Aug 9, 2021
Related Publication 20230039530A1 · Feb 9, 2023
References Cited (66)
US 10013857B2 · Sridhara · 2018 [cited by examiner]
US 20030058216A1 · Lacroix · 2003 [cited by examiner]
US 20080062143A1 · Shahoian · 2008 [cited by examiner]
US 20090096632A1 · Ullrich · 2009 [cited by examiner]
US 20140347176A1 · Modarres · 2014 [cited by examiner]
US 20150242608A1 · Kim · 2015 [cited by examiner]
US 20170232338A1 · Goslin et al. · 2017 [cited by applicant]
US 20170242483A1 · Goslin et al. · 2017 [cited by applicant]
US 20170301140A1 · Smith et al. · 2017 [cited by applicant]
US 20170345195A1 · Eatedali et al. · 2017 [cited by applicant]
US 20180124370A1 · Bejot · 2018 [cited by applicant]
US 20180151036A1 · Cha et al. · 2018 [cited by applicant]
US 20180165925A1 · Israr et al. · 2018 [cited by applicant]
US 20180196523A1 · Arana et al. · 2018 [cited by applicant]
US 20180280793A1 · Panec et al. · 2018 [cited by applicant]
US 20190056780A1 · Bombard et al. · 2019 [cited by applicant]
US 20190066359A1 · Chen et al. · 2019 [cited by applicant]
US 20190066387A1 · Grossmann et al. · 2019 [cited by applicant]
US 20190114064A1 · Monibi · 2019 [cited by applicant]
US 20190139281A1 · Chapman et al. · 2019 [cited by applicant]
US 20190171302A1 · Su · 2019 [cited by examiner]
US 20190379878A1 · Chapman et al. · 2019 [cited by applicant]
US 20200019370A1 · Doggett et al. · 2020 [cited by applicant]
US 20200064922A1 · Nocon et al. · 2020 [cited by applicant]
US 20200142484A1 · Maalouf · 2020 [cited by examiner]
US 20200298113A1 · Smithers · 2020 [cited by examiner]
US 20210397260A1 · Birnbaum · 2021 [cited by examiner]
US 20220113801A1 · Bharitkar · 2022 [cited by examiner]
US 20240033624A1 · Phillips · 2024 [cited by examiner]
Camille Moussette, “Simple Haptics: Sketching Perspectives for the design of Haptic Interactions,” Thesis for the Degree of Doctor of Philosophy in Industrial Design, Dated: Oct. 2012, pp. 1-284. [cited by applicant]
Ali Israr, Siyan Zhao, Kaitlyn Schwalje, Roberta Klatzky, and Jill Lehman. 2014. Feel effects: Enriching storytelling with haptic feedback. ACM Trans. Appl. Percept. 11, 3, Article 11 (Sep. 2014), 17 pages. DOI: http://… [cited by applicant]
Siyan Zhao et al., “FeelCraft: Crafting Tactile Experiences for Media using a Feel Effect Library,” UIST '14, Dated Oct. 5-8, 2014, pp. 1-2 http://dx.doi.org/10.1145/2658779.2659109. [cited by applicant]
D. Reusser et al., “Feeling Fireworks,” UIST'17 Adjunct, Dated Oct. 22-25, 20107, pp. 1-3. [cited by applicant]
Nesra Yannier et al., “FeelSleeve: Haptic Feedback to Enhance Early Reading,” ResearchGate, Dated: Apr. 2015, pp. 1-11 DOI: 10.1145/2702123.2702396. [cited by applicant]
Paul Bach-y-Rita et al., “Form perception with a 49-point electrotactile stimulus array on the tongue: A technical note,” Journal of Rehabilitation Research and Development vol. 35 No. 4, Oct. 1998 pp. 427-430. [cited by applicant]
Youngjun Cho, “Haptic Cushion: Automatic Generation of Vibro-tactile Feedback Based on Audio Signal for Immersive Interaction with Multimedia,” Creative Innovation Center, Dated: Jun. 2014, pp. 1-5. [cited by applicant]
Nooshin Jafari et al., “Haptics to improve task performance in people with disabilities: A review of previous studies and a guide to future research with children with disabilities,” Journal of Rehabilitation and Assist… [cited by applicant]
“Haptics in Touch Screen Hand-Held Devices,” Immersion, Dated Aug. 2017, pp. 1-10. [cited by applicant]
“Haptics White Paper: Improving the Mobile User Experience through Touch,” Immersion, Year: 2010, pp. 1-12. [cited by applicant]
M. McLinden, “Mediating haptic exploratory strategies in children who have visual impairment and intellectual disabilities,” Journal of Intellectual Disability Research, vol. 56 Part 2, Dated: Feb. 2012, pp. 129-139. [cited by applicant]
David Navarre et al., “A Formal Description of Multimodal Interaction Techniques for Immersive Virtual Reality Applications,” IFIP International Federation for Information Processing, Year: 2005, pp. 170-183. [cited by applicant]
Geraldine Dawson et al., “Interventions to Facilitate Auditory, Visual, and Motor Integration in Autism: A Review of the Evidence,” Journal of Autism and Developmental Disorders, vol. 30, No. 5, 2000, pp. 1-7. [cited by applicant]
Kathryn Linn Geurts, “On Rocks, Walks, and Talks in West Africa: Cultural Categories and an Anthropology of the Senses,” American Anthropological Association, Year: 2003, pp. 178-198. [cited by applicant]
Priscialla Ramsamy et al., “Using Haptics to Improve Immersion in Virtual Environments,” Centre for Advanced Computing and Emerging Technologies, Year: 2006, pp. 603-609. [cited by applicant]
Grace T. Baranek, “Efficacy of Sensory and Motor Interventions for Children with Autism,” Journal of Autism and Developmental Disorders, vol. 32, No. 5, Oct. 2002, pp. 397-422. [cited by applicant]
Grace Iarocci et al., “Sensory Integration and the Perceptual Experience of Persons with Austim,” Journal of Autism and Developmental Disorders, vol. 36, No. 1, Jan. 2006, pp. 77-90. [cited by applicant]
Crétien van Campen, “Synesthesia and Artistic Experimentation,” Psyche, 3(6), Nov. 1997, pp. 1-8. [cited by applicant]
S. Zhao et al., Using Haptic Inputs to Enrich Story Listening for Young Children, IDC'Jun. 21-25, 15, 2015, pp. 1-4. [cited by applicant]
“Vision Substitution by Tactile Image Projection,” Nature, vol. 221, Mar. 8, 1969, pp. 963-964. [cited by applicant]
“A Theater Experience in the Palm of your Hands,” Immersion, Year: 2017, pp. 1-7. [cited by applicant]
Sanya Attari et al., “Advancing Haptic Design: Focusing on Human Interpretations of Tactile Effects,” Immersion, Dated Accessed: Nov. 4, 2022, pp. 1-10. [cited by applicant]
“Framework: Core Haptics: Compose and Play Haptic Patterns to Customize your IOS app's Haptic Feedback,” Apple Developer, Year: 2022, pp. 1-7. [cited by applicant]
Chris Morris, “Wild new haptic feedback VR vest looks like something out of Ready Plater One,” Digitaltrends, Dated: Jan. 11, 2021, pp. 1-16. [cited by applicant]
“The Next Generation of Hearing Science,” Neosensory, Date Accessed: Nov. 4, 2022, pp. 1-6 <https://neosensory.com/>. [cited by applicant]
Oliver Schneider et al., “Tactile Animation by Direct Manipulation of Grid Displays,” UIST'15, Nov. 8-11, 2015, pp. 1-10, https://dx.doi.org/10.1145/2807442.2807470. [cited by applicant]
J. M. Weisenberger et al., “Evaluation of two multichannel tactile aids for the hearing impaired,” J Acoust Soc Am . Nov. 1989;86(5):1764-75. doi: 10.1121/1.398608. [Abstract Only]. [cited by applicant]
Peter Shull et al., “Haptic Wearables as Sensory Replacement, Sensory Augmentation and Trainer—A review,” Journal of NeuroEngineering and Rehabilitation (2015) 12:59 DOI 10.1186/s12984-015-0055-z. [cited by applicant]
Matthew Hutson, “Here's What the Future of Haptic Technology Looks (or rather, feels} Like, ” Smithsonian Magazine, Dated: Dec. 28, 2018, pp. 1-10. [cited by applicant]
Chris Ullrich, “The Haptic Stack—Software Layer,” Immersion, Dated: May 13, 2020, pp. 1-6. [cited by applicant]
Chris Ullrich, “The Haptic Stack,” Immersion, Dated: Feb. 12, 2020, pp. 1-6. [cited by applicant]
“Human Augmentation Technologies & Sensory Enhancements,” CyborgNest, Date Accessed: Nov. 4, 2022, pp. 1-5 https://www.cyborgnest.net/. [cited by applicant]
Gwydion ap Dafydd, “Pushing Innovation in Haptic DSP with Predictive Analysis,” Lofelt, Dated: May 6, 2020, pp. 1-3. [cited by applicant]
Eric Gunther, “Skinscape: A Tool for Composition in the Tactile Modality,” Department of Electrical Engineering and Computer Science, Dated: May 23, 2001, pp. 1-118. [cited by applicant]
“Emoti-Chair lets your feel the sound,” The London Free Press, Date Accessed: Nov. 4, 2022, p. 1. [cited by applicant]
“Upgrade your sleep. Downgrade your stress,” Apollo, Date Access: Nov. 4, 2022, pp. 1-14 <https://apolloneuro.com/>. [cited by applicant]
“Inspiring Innovation,” Sensel, Year: 2022, pp. 1-18, <https://sensel.com/>. [cited by applicant]