IP Library › Granted Patent US 12,379,770
Granted Patent B2
US 12,379,770 · App. 18/371,990 · Granted Aug 5, 2025

Integrated sensor framework for multi-device communication and interoperability

Inventors: Richard T. Vaughan (Saratoga, CA); Jamil Dhanani (San Francisco, CA); Juan C. Garcia (San Francisco, CA); Seyedmehdi Mohaimenianpour (Cupertino, CA); Geoffrey Nagy (Cupertino, CA); Timothy S. Paek (Mercer Island, WA); Naga Rama Abhishek Pratapa (Seattle, WA); Muhammad Amir Shafiq (Redmond, WA)
Assignee: Apple Inc.
G06F3/011G06T7/20G06T7/70G06T17/00G06V20/50G06V40/10G06T2207/30241G06T2207/30244
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,379,770
App. No.
18/371,990
Granted
Aug 5, 2025
Kind
B2
Abstract

Systems and processes for an integrated sensor framework are provided. For example, a first electronic device receives at least one input including sensor data from a second device. A representation of a physical environment associated with the first electronic device is obtained based on sensor data from the first electronic device and the sensor data from the second device. Movement information corresponding to movement of an object within the physical environment is identified. Event information is determined corresponding to activity within the physical environment, wherein the event information is determined based on the identified movement information and the representation of the physical environment. Accordingly, an output is provided to the user based on the event information.

Claims (115)

1. A first electronic device, comprising:

one or more processors;

a memory; and

one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for:

receiving at least one input including sensor data from a second electronic device;

obtaining, based on sensor data from the first electronic device and the sensor data from the second electronic device, a representation of a physical environment associated with the first electronic device;

identifying movement information corresponding to movement of an object within the physical environment;

determining, based on the identified movement information and the representation of the physical environment, event information corresponding to activity within the physical environment; and

providing an output based on the event information.

2. The electronic device of claim 1 , wherein the sensor data from the second electronic device includes at least one of a distance between the second electronic device and the object, a movement trajectory of the object, a gaze direction of a person within the physical environment, audio content, and a speech direction.

3. The electronic device of claim 1 , wherein obtaining the representation of the physical environment comprises:

generating, based on a plurality of sets of sensor data from a plurality of devices, a multi-dimensional representation of the physical environment, wherein the multi-dimensional representation of the physical environment includes a plurality of locations corresponding to a plurality of objects identified based on the plurality of sets of sensor data.

4. The electronic device of claim 3 , wherein the representation of the physical environment includes at least one approximation of a wall boundary within the physical environment.

5. The electronic device of claim 1 , the one or more programs including instructions for:

identifying, from the at least one input, sensor data including identification of a plurality of respective objects within the physical environment, wherein

the plurality of respective objects are not electronic devices, and

the representation includes a plurality of identifiers corresponding to the plurality of respective objects.

6. The electronic device of claim 1 , the one or more programs including instructions for:

identifying, from the at least one input, sensor data including a device location corresponding to a respective device within the physical environment; and

determining, based on the device location, an approximation of a second object adjacent to the respective device, wherein the representation of the physical environment includes the approximation.

7. The electronic device of claim 1 , the one or more programs including instructions for:

receiving a user input corresponding to a respective object, wherein the user input includes an indication that the respective object is lost;

identifying movement information corresponding to movement of the respective object; and

providing, as the output, an indication that the respective object is found and a location of the respective object.

8. The electronic device of claim 1 , the one or more programs including instructions for:

receiving, as sensor data from a respective device, authentication information corresponding to a user of the respective device; and

while identifying movement information corresponding to movement of the respective device, maintaining an authentication corresponding to the user in accordance with a determination that a predetermined criteria is satisfied.

9. The electronic device of claim 1 , wherein the object corresponds to one of a person, the first electronic device, or a different electronic device.

10. The electronic device of claim 1 , wherein identifying movement information corresponding to movement of an object within the physical environment comprises:

during a period of time, tracking movement of an object around the physical environment based on sensor data from a plurality of devices.

11. The electronic device of claim 1 , wherein providing an output based on the event information comprises:

sending information to the second electronic device; and

causing the output to be provided at the second electronic device based on the sent information.

12. The electronic device of claim 1 , the one or more programs including instructions for:

identifying, based on the at least one input, an unknown object, wherein the event information corresponding to activity within the physical environment includes an identification that the unknown object corresponds to an unknown person; and

providing, as the output, an indication of the unknown person.

13. The electronic device of claim 1 , the one or more programs including instructions for:

in accordance with a determination that the event information satisfies a predetermined criteria, causing a respective device to provide a respective output.

14. The electronic device of claim 1 , the one or more programs including instructions for:

in accordance with a determination that the event information includes identification of a specific person, providing a content recommendation based on the specific person.

15. A non-transitory computer-readable storage medium storing one or more programs, the one or more programs configured to be executed by one or more processors of a first electronic device, the one or more programs including instructions for:

receiving at least one input including sensor data from a second electronic device;

obtaining, based on sensor data from the first electronic device and the sensor data from the second electronic device, a representation of a physical environment associated with the first electronic device;

identifying movement information corresponding to movement of an object within the physical environment;

determining based on the identified movement information and the representation of the physical environment, event information corresponding to activity within the physical environment; and

providing an output based on the event information.

16. A computer-implemented method, comprising:

at a first electronic device with one or more processors and memory:

receiving at least one input including sensor data from a second electronic device;

obtaining, based on sensor data from the first electronic device and the sensor data from the second electronic device, a representation of a physical environment associated with the first electronic device;

identifying movement information corresponding to movement of an object within the physical environment;

determining, based on the identified movement information and the representation of the physical environment, event information corresponding to activity within the physical environment; and

providing an output based on the event information.

17. The computer-readable storage medium of claim 15 , wherein the sensor data from the second electronic device includes at least one of a distance between the second electronic device and the object, a movement trajectory of the object, a gaze direction of a person within the physical environment, audio content, and a speech direction.

18. The computer-readable storage medium of claim 15 , wherein obtaining the representation of the physical environment comprises:

generating, based on a plurality of sets of sensor data from a plurality of devices, a multi- dimensional representation of the physical environment, wherein the multi-dimensional representation of the physical environment includes a plurality of locations corresponding to a plurality of objects identified based on the plurality of sets of sensor data.

19. The computer-readable storage medium of claim 18 , wherein the representation of the physical environment includes at least one approximation of a wall boundary within the physical environment.

20. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

identifying, from the at least one input, sensor data including identification of a plurality of respective objects within the physical environment, wherein

the plurality of respective objects are not electronic devices, and

the representation includes a plurality of identifiers corresponding to the plurality of respective objects.

21. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

identifying, from the at least one input, sensor data including a device location corresponding to a respective device within the physical environment; and

determining, based on the device location, an approximation of a second object adjacent to the respective device, wherein the representation of the physical environment includes the approximation.

22. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

receiving a user input corresponding to a respective object, wherein the user input includes an indication that the respective object is lost;

identifying movement information corresponding to movement of the respective object; and

providing, as the output, an indication that the respective object is found and a location of the respective object.

23. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

receiving, as sensor data from a respective device, authentication information corresponding to a user of the respective device; and

while identifying movement information corresponding to movement of the respective device, maintaining an authentication corresponding to the user in accordance with a determination that a predetermined criteria is satisfied.

24. The computer-readable storage medium of claim 15 , wherein the object corresponds to one of a person, the first electronic device, or a different electronic device.

25. The computer-readable storage medium of claim 15 , wherein identifying movement information corresponding to movement of an object within the physical environment comprises:

during a period of time, tracking movement of an object around the physical environment based on sensor data from a plurality of devices.

26. The computer-readable storage medium of claim 15 , wherein providing an output based on the event information comprises:

sending information to the second electronic device; and

causing the output to be provided at the second electronic device based on the sent information.

27. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

identifying, based on the at least one input, an unknown object, wherein the event information corresponding to activity within the physical environment includes an identification that the unknown object corresponds to an unknown person; and

providing, as the output, an indication of the unknown person.

28. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

in accordance with a determination that the event information satisfies a predetermined criteria, causing a respective device to provide a respective output.

29. The computer-readable storage medium of claim 15 , the one or more programs including instructions for:

in accordance with a determination that the event information includes identification of a specific person, providing a content recommendation based on the specific person.

30. The method of claim 16 , wherein the sensor data from the second electronic device includes at least one of a distance between the second electronic device and the object, a movement trajectory of the object, a gaze direction of a person within the physical environment, audio content, and a speech direction.

31. The method of claim 16 , wherein obtaining the representation of the physical environment comprises:

generating, based on a plurality of sets of sensor data from a plurality of devices, a multi- dimensional representation of the physical environment, wherein the multi-dimensional representation of the physical environment includes a plurality of locations corresponding to a plurality of objects identified based on the plurality of sets of sensor data.

32. The method of claim 31 , wherein the representation of the physical environment includes at least one approximation of a wall boundary within the physical environment.

33. The method of claim 16 , comprising:

identifying, from the at least one input, sensor data including identification of a plurality of respective objects within the physical environment, wherein

the plurality of respective objects are not electronic devices, and

the representation includes a plurality of identifiers corresponding to the plurality of respective objects.

34. The method of claim 16 , comprising:

identifying, from the at least one input, sensor data including a device location corresponding to a respective device within the physical environment; and

determining, based on the device location, an approximation of a second object adjacent to the respective device, wherein the representation of the physical environment includes the approximation.

35. The method of claim 16 , comprising:

receiving a user input corresponding to a respective object, wherein the user input includes an indication that the respective object is lost;

identifying movement information corresponding to movement of the respective object; and

providing, as the output, an indication that the respective object is found and a location of the respective object.

36. The method of claim 16 , comprising:

receiving, as sensor data from a respective device, authentication information corresponding to a user of the respective device; and

while identifying movement information corresponding to movement of the respective device, maintaining an authentication corresponding to the user in accordance with a determination that a predetermined criteria is satisfied.

37. The method of claim 16 , wherein the object corresponds to one of a person, the first electronic device, or a different electronic device.

38. The method of claim 16 , wherein identifying movement information corresponding to movement of an object within the physical environment comprises:

during a period of time, tracking movement of an object around the physical environment based on sensor data from a plurality of devices.

39. The method of claim 16 , wherein providing an output based on the event information comprises:

sending information to the second electronic device; and

causing the output to be provided at the second electronic device based on the sent information.

40. The method of claim 16 , comprising:

identifying, based on the at least one input, an unknown object, wherein the event information corresponding to activity within the physical environment includes an identification that the unknown object corresponds to an unknown person; and

providing, as the output, an indication of the unknown person.

41. The method of claim 16 , comprising:

in accordance with a determination that the event information satisfies a predetermined criteria, causing a respective device to provide a respective output.

42. The method of claim 16 , comprising:

in accordance with a determination that the event information includes identification of a specific person, providing a content recommendation based on the specific person.

Assignments (1)
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 12, 2024
From: VAUGHAN, RICHARD T.; PAEK, TIMOTHY S.; GARCIA, JUAN C.; SHAFIQ, MUHAMMAD AMIR; MOHAIMENIANPOUR, SEYEDMEHDI; NAGY, GEOFFREY; DHANANI, JAMIL; PRATAPA, NAGA RAMA ABHISHEK
To: APPLE INC.
Reel/Frame 069229/0611 →
Continuity (2)
Provisional Application 63408985 · Sep 22, 2022
Related Publication 20240118744A1 · Apr 11, 2024
References Cited (109)
US 7599918B2 · Shen et al. · 2009 [cited by applicant]
US 7603684B1 · Ellis · 2009 [cited by applicant]
US 8654936B1 · Eslambolchi et al. · 2014 [cited by applicant]
US 8762156B2 · Chen · 2014 [cited by applicant]
US 9628844B2 · Conrad et al. · 2017 [cited by applicant]
US 9646313B2 · Kim et al. · 2017 [cited by applicant]
US 9747083B1 · Roman et al. · 2017 [cited by applicant]
US 9760566B2 · Heck et al. · 2017 [cited by applicant]
US 9842168B2 · Heck et al. · 2017 [cited by applicant]
US 9978180B2 · Margolis et al. · 2018 [cited by applicant]
US 10147421B2 · Liddell et al. · 2018 [cited by applicant]
US 10180572B2 · Osterhout et al. · 2019 [cited by applicant]
US 10339721B1 · Dascola et al. · 2019 [cited by applicant]
US 10437928B2 · Bhaya et al. · 2019 [cited by applicant]
US 10489651B2 · Luccin et al. · 2019 [cited by applicant]
US 10496364B2 · Yao · 2019 [cited by applicant]
US 10642934B2 · Heck et al. · 2020 [cited by applicant]
US 10691473B2 · Karashchuk et al. · 2020 [cited by applicant]
US 11043086B1 · Daoura et al. · 2021 [cited by applicant]
US 11223699B1 · Niewczas · 2022 [cited by applicant]
US 20100127854A1 · Helvick et al. · 2010 [cited by applicant]
US 20100318357A1 · Istvan et al. · 2010 [cited by applicant]
US 20100332235A1 · David · 2010 [cited by applicant]
US 20120232886A1 · Capuozzo et al. · 2012 [cited by applicant]
US 20130024910A1 · Verma et al. · 2013 [cited by applicant]
US 20130035086A1 · Chardon et al. · 2013 [cited by applicant]
US 20130080177A1 · Chen · 2013 [cited by applicant]
US 20130183944A1 · Mozer et al. · 2013 [cited by applicant]
US 20140006483A1 · Garmark et al. · 2014 [cited by applicant]
US 20140006947A1 · Garmark et al. · 2014 [cited by applicant]
US 20140045547A1 · Singamsetty et al. · 2014 [cited by applicant]
US 20140057610A1 · Olincy et al. · 2014 [cited by applicant]
US 20140067702A1 · Rathod · 2014 [cited by applicant]
US 20140098247A1 · Rao et al. · 2014 [cited by applicant]
US 20140129006A1 · Chen et al. · 2014 [cited by applicant]
US 20140145914A1 · Latta et al. · 2014 [cited by applicant]
US 20140163751A1 · Davis et al. · 2014 [cited by applicant]
US 20140330951A1 · Sukoff et al. · 2014 [cited by applicant]
US 20150045007A1 · Cash · 2015 [cited by applicant]
US 20150067811A1 · Agnew · 2015 [cited by examiner]
US 20150088518A1 · Kim et al. · 2015 [cited by applicant]
US 20150140990A1 · Kim et al. · 2015 [cited by applicant]
US 20150162006A1 · Kummer · 2015 [cited by applicant]
US 20150170073A1 · Baker · 2015 [cited by applicant]
US 20150193005A1 · Di et al. · 2015 [cited by applicant]
US 20150193018A1 · Venable et al. · 2015 [cited by applicant]
US 20150309691A1 · Seo et al. · 2015 [cited by applicant]
US 20150317831A1 · Ebstyne et al. · 2015 [cited by applicant]
US 20150317832A1 · Ebstyne et al. · 2015 [cited by applicant]
US 20160080165A1 · Ehsani et al. · 2016 [cited by applicant]
US 20160234206A1 · Tunnell et al. · 2016 [cited by applicant]
US 20160283455A1 · Mardanbegi et al. · 2016 [cited by applicant]
US 20170264451A1 · Yu et al. · 2017 [cited by applicant]
US 20180054505A1 · Hart et al. · 2018 [cited by applicant]
US 20180102914A1 · Kawachi et al. · 2018 [cited by applicant]
US 20180221783A1 · Gamero · 2018 [cited by applicant]
US 20190065027A1 · Hauenstein · 2019 [cited by examiner]
US 20190138268A1 · Andersen et al. · 2019 [cited by applicant]
US 20190188895A1 · Miller et al. · 2019 [cited by applicant]
US 20190228581A1 · Dascola et al. · 2019 [cited by applicant]
US 20190287522A1 · Lambourne et al. · 2019 [cited by applicant]
US 20190324533A1 · Itkowitz et al. · 2019 [cited by applicant]
US 20200112454A1 · Brown et al. · 2020 [cited by applicant]
US 20200298394A1 · Han et al. · 2020 [cited by applicant]
US 20200312318A1 · Olson · 2020 [cited by examiner]
US 20220277505A1 · Baszucki et al. · 2022 [cited by applicant]
US 20230409174A1 · Liang et al. · 2023 [cited by applicant]
US 20230409179A1 · Liang et al. · 2023 [cited by applicant]
CN 101833286A · 2010 [cited by applicant]
CN 102882752A · 2013 [cited by applicant]
CN 103035240A · 2013 [cited by applicant]
CN 103365279A · 2013 [cited by applicant]
CN 103475551A · 2013 [cited by applicant]
CN 103477592A · 2013 [cited by applicant]
CN 104820488A · 2015 [cited by applicant]
CN 107003738A · 2017 [cited by applicant]
EP 2938022A1 · 2015 [cited by applicant]
EP 3036594A2 · 2016 [cited by applicant]
IN 2011MU01369A · 2011 [cited by applicant]
JP 2009110300A · 2009 [cited by applicant]
JP 201373240A · 2013 [cited by applicant]
JP 201460600A · 2014 [cited by applicant]
KR 101506510B1 · 2015 [cited by applicant]
KR 1020150041974A · 2015 [cited by applicant]
TW M348993U · 2009 [cited by applicant]
WO 2015053485A1 · 2015 [cited by applicant]
WO 2016027933A1 · 2016 [cited by applicant]
WO 2016089638A1 · 2016 [cited by applicant]
WO 2018009397A1 · 2018 [cited by applicant]
WO 2018213401A1 · 2018 [cited by applicant]
Accessibility on iOS, Apple Inc., online available at: https://developer.apple.com/accessibility/ios/, Retrieved on Jul. 26, 2021, 2 pages. [cited by applicant]
Apple, “Apple previews innovative accessibility features combining the power of hardware, software, and machine learning”, Available online at: https://www.apple.com/newsroom/2022/05/apple-previews-innovative-accessibil… [cited by applicant]
Ashbrook Daniell., “Enabling Mobile Micro interactions”, Retrieved from the Internet: URL: http://danielashbrook.com/wp-content/uploads/2012/06/2009-Ashbrook-Thesis.pdf, May 2010, 186 pages. [cited by applicant]
Ask Alexa—Things That Are Smart Wiki, Available online at: <http://thingsthataresmart.wiki/index.php?title=Ask_Alexa&oldid=4283>, Jun. 8, 2016, pp. 1-31. [cited by applicant]
Bell Jason, “Machine Learning Hands-On for Developers and Technical Professionals”, Wiley, Nov. 3, 2014, 82 pages. [cited by applicant]
DIRECTV™ Voice, Now Part of the DIRECTTV Mobile App for Phones, Sep. 18, 2013, 5 pages. [cited by applicant]
Gomes et al., “Mining Recurring Concepts in a Dynamic Feature Space”, IEEE Transactions on Neural Networks and Learning Systems, vol. 25, No. 1, Jul. 31, 2013, pp. 95-110. [cited by applicant]
Guo et al., “Viz Lens: A Robust and Interactive Screen Reader for Interfaces in the Real World”, In Proceedings of the 29th Annual Symposium on User Interface Software and Technology (UIST '16), Tokyo, Japan, online ava… [cited by applicant]
Gupta Naresh, “Inside Bluetooth Low Energy”, Artech House, Mar. 1, 2013, 274 pages. [cited by applicant]
Iclarified, “How to Use Voice Control on Your iPhone 3GS”, Online Available at: <https://www.iclarified.com/4296/how-to-use-voice-control-on-your-iphone-3g-s>, Jun. 19, 2009, 16 pages. [cited by applicant]
Interactive Voice, available at <http://www.helloivee.com/company/>, retrieved on Feb. 10, 2014, 2 pages. [cited by applicant]
Kickstarter, “Ivee Sleek: Wi-Fi Voice-Activated Assistant”, available at <https://www.kickstarter.com/projects/ivee/ivee-sleek-wi-fi-voice-activated-assistant>, retrieved on Feb. 10, 2014, pp. 1-13. [cited by applicant]
Kumar Shiu, “Ubiquitous Smart Home System Using Android Application”, International Journal of Computer Networks & Communications (IJCNC) vol. 6, No. 1, Available online at: https://arxiv.org/ftp/arxiv/papers/1402/1402.… [cited by applicant]
Loukides et al., “What Is the Internet of Things?”, O'Reilly Media, Inc., Online Available at: https://www.oreilly.com/library/view/what-is-the/9781491975633/, Jan. 13, 2015, 31 pages. [cited by applicant]
Nuance Dragon Naturally Speaking, Version 13 End-User Workbook, Nuance Communications, Inc. Online Available at: https://www.nuance.com/content/dam/nuance/en_us/collateral/dragon/guide/gd-dragon-naturally-speaking-13-wo… [cited by applicant]
Rowland et al., “Designing Connected Products: UX for the Consumer Internet of Things”, O'Reilly, May 31, 2015, 452 pages. [cited by applicant]
Wikipedia, “Home Automation”, Available At: <https://en.wikipedia.org/w/index.php?title=Home_automation&oldid=686569068>, Oct. 19, 2015, 9 Pages. [cited by applicant]
Wikipedia, “Speech Recognition”, available at <http://en.wikipedia.org/wiki/Speech_recognition>, retrieved on Sep. 14, 2011, 12 pages. [cited by applicant]
Young et al., “POMDP-Based Statistical Spoken Dialog Systems: A Review”, Proceedings of the IEEE, vol. 101, No. 5, 2013, 18 pages. [cited by applicant]