IP Library › Granted Patent US 12,298,879
Granted Patent B2
US 12,298,879 · App. 18/511,769 · Granted May 13, 2025

Electronic device and method for controlling same

Inventors: Yehoon Kim (Suwon-si, KR); Chanwon Seo (Suwon-si, KR); Sojung Yun (Suwon-si, KR); Junik Jang (Suwon-si, KR)
Assignee: SAMSUNG ELECTRONICS CO., LTD.
G06F11/3438G06N3/08
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12,298,879
App. No.
18/511,769
Filed
Nov 16, 2023
Granted
May 13, 2025
Kind
B2
Art Unit
2444
USPC
709/224
Abstract

An example method for controlling an electronic device includes detecting at least one user and acquiring user information of the detected at least one user; determining a user mode based on the acquired user information; determining a service to be provided to the detected at least one user, by inputting the user information and the determined user mode as input data to a model learned by an artificial intelligence algorithm; and providing the determined service corresponding to the user mode. A method for providing the service by the electronic device may at least partially use an artificial intelligence model learned according to at least one of machine learning, neural network, and deep learning algorithms.

Claims (77)

1. A method for controlling an electronic device, the method comprising:

obtaining, by the electronic device, an image regarding at least one person around the electronic device;

analyzing, by the electronic device, the image to determine a number of persons in the image;

based on determining the number of persons in the image to be one person, determining, by the electronic device, a first service to be provided to the one person based on a context of the one person, and providing, by the electronic device, the first service; and

based on determining the number of persons in the image to be a plurality of persons, determining, by the electronic device, a second service to be provided to the plurality of persons based on a relationship between the plurality of persons, and providing, by the electronic device, the second service.

2. The method of claim 1 , further comprising:

obtaining the context using a first neural network model, and

obtaining the relationship between the plurality of persons using a second neural network model.

3. The method of claim 2 ,

wherein the first service is determined by inputting, as input data to the first neural network model, at least one of information of the one person, information of objects around the one person, or information of a situation around the one person.

4. The method of claim 3 , further comprising:

based on determining the number of persons in the image to be at least one user being one person, predicting next behavior of the one person based on at least one of information of the one person, information of objects around the one person, or information of a situation around the one person,

wherein the first service is determined based on the predicted next behavior of the one person.

5. The method of claim 2 , further comprising:

updating the first neural network model using, as learning data, information regarding the first service; and

updating the second neural network model using, as learning data, information regarding the second service.

6. The method of claim 1 ,

wherein the relationship between the plurality of persons comprises a level of closeness between the plurality of persons, and

wherein the second service is determined based on the level of closeness.

7. The method of claim 1 ,

further comprising:

determining a reaction to the first service or the second service;

based on the reaction being positive, providing the first service or the second service; and

based on the reaction being negative, determining another service to be provided.

8. An electronic device comprising:

a camera;

a display;

memory; and

at least one processor,

wherein the memory stores instructions which, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

obtaining, through the camera, an image regarding at least one person around the electronic device,

analyzing the image to determine a number of persons in the image,

based on determining the number of persons in the image to be one person, determining a first service to be provided to the one person based on a context of the one person, and providing the first service, and

based on determining the number of persons in the image to be a plurality of persons, determining a second service to be provided to the plurality of persons based on a relationship between the plurality of persons, and providing the second service.

9. The electronic device of claim 8 ,

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

obtaining the context using a first neural network model, and

obtaining the relationship between the plurality of persons using a second neural network model.

10. The electronic device of claim 9 ,

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

determining the first service by inputting, as input data to the first neural network model, at least one of information of the one person, information of objects around the one person, or information of a situation around the one person.

11. The electronic device of claim 10 ,

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

based on determining the number of persons in the image to be one person, predicting next behavior of the one person based on at least one of information of the one person, information of objects around the one person, or information of a situation around the one person, and

determining the first service based on the predicted next behavior of the one person.

12. The electronic device of claim 9 ,

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

updating the first neural network model using, as learning data, information regarding the first service, and

updating the second neural network model using, as learning data, information regarding the second service.

13. The electronic device of claim 8 ,

wherein the relationship between the plurality of persons comprises a level of closeness between the plurality of persons, and

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising determining the second service based on the level of closeness.

14. The electronic device of claim 8 ,

wherein the instructions, when executed by the at least one processor individually and/or collectively, cause the electronic device to perform operations comprising:

determining a reaction to the first service or the second service,

based on the reaction being positive, providing the first service or the second service, and

based on the reaction being negative, determining another service to be provided.

15. A non-transitory computer-readable storage medium storing instructions which, when executed by at least one processor of an electronic device, configure the at least one processor to perform operations comprising:

analyzing, by the electronic device, an image regarding at least one person around the electronic device;

based on determining a number of persons in the image to be one person, determining, by the electronic device, a first service to be provided to the one person based on a context of the one person, and providing the first service; and

based on determining the number of persons in the image to be a plurality of persons, determining a second service to be provided to the plurality of persons based on a relationship between the plurality of persons, and providing the second service.

16. The non-transitory computer-readable storage medium of claim 15 , wherein the operations further comprise:

obtaining the context using a first neural network model, and

obtaining the relationship between the plurality of persons using a second neural network model.

17. The non-transitory computer-readable storage medium of claim 16 ,

wherein the first service is determined by inputting, as input data to the first neural network model, at least one of information of the one person, information of objects around the one person, or information of a situation around the one person.

18. The non-transitory computer-readable storage medium of claim 17 , wherein the operations further comprise:

based on determining the number of persons in the image to be one person, predicting next behavior of the one person based on at least one of information of the one person, information of objects around the one person, or information of a situation around the one person,

wherein the first service is determined based on the predicted next behavior of the one person.

19. The non-transitory computer-readable storage medium of claim 15 ,

wherein the relationship between the plurality of persons comprises a level of closeness between the plurality of persons, and

wherein the second service is determined based on the level of closeness.

20. The non-transitory computer-readable storage medium of claim 15 ,

wherein the operations further comprise:

determining a reaction to the first service or the second service;

based on the reaction being positive, providing the first service or the second service; and

based on the reaction being negative, determining another service to be provided.

Priority Claims (1)
KR 10-2018-0001692 · Jan 5, 2018 · national
Continuity (2)
Continuation 16768452
Related Publication 20240095143A1 · Mar 21, 2024
References Cited (61)
US 8909370B2 · Stiehl et al. · 2014 [cited by applicant]
US 10534900B2 · Cheong et al. · 2020 [cited by applicant]
US 11139065B1 · Gottman · 2021 [cited by examiner]
US 11361336B1 · Almosilhi · 2022 [cited by examiner]
US 11869150B1 · Mason · 2024 [cited by examiner]
US 20100185328A1 · Kim et al. · 2010 [cited by applicant]
US 20130073972A1 · Yung · 2013 [cited by applicant]
US 20150066158A1 · Kim · 2015 [cited by applicant]
US 20150326454A1 · Geis · 2015 [cited by applicant]
US 20160088352A1 · Agarwal · 2016 [cited by examiner]
US 20160105308A1 · Dutt · 2016 [cited by applicant]
US 20160199977A1 · Breazeal · 2016 [cited by applicant]
US 20170010677A1 · Roh · 2017 [cited by applicant]
US 20170140285A1 · Dotan-Cohen · 2017 [cited by applicant]
US 20170358201A1 · Govers · 2017 [cited by applicant]
US 20190087887A1 · Westphal · 2019 [cited by applicant]
US 20190095775A1 · Lembersky · 2019 [cited by applicant]
US 20190176333A1 · Hager, IV · 2019 [cited by examiner]
US 20200226416A1 · Bapat · 2020 [cited by examiner]
US 20200364588A1 · Knox · 2020 [cited by examiner]
US 20200371895A1 · Kim · 2020 [cited by applicant]
US 20210170585A1 · Kim · 2021 [cited by applicant]
CN 105431820 · 2016 [cited by applicant]
CN 105677333 · 2016 [cited by applicant]
CN 105975622 · 2016 [cited by applicant]
CN 106897404 · 2017 [cited by applicant]
JP 2002355779 · 2002 [cited by applicant]
JP 4565196 · 2010 [cited by applicant]
JP 201648417 · 2016 [cited by applicant]
KR 100905523 · 2009 [cited by applicant]
KR 1020100086262 · 2010 [cited by applicant]
KR 101126524 · 2012 [cited by applicant]
KR 1020130071755 · 2013 [cited by applicant]
KR 20130082701A · 2013 [cited by applicant]
KR 1020140058860 · 2014 [cited by applicant]
KR 101413558 · 2014 [cited by applicant]
KR 1020150099430 · 2015 [cited by applicant]
KR 1020150137307 · 2015 [cited by applicant]
KR 101772583 · 2017 [cited by applicant]
WO 0161505 · 2001 [cited by applicant]
WO 2015034295 · 2015 [cited by applicant]
Notice of Allowance dated Feb. 29, 2024 for Chinese Patent Application No. 202210152977.X and English-language translation. [cited by applicant]
Scott W. Ambler, “Mapping objects to relational databases”, IBM, developer Works, Jul. 1, 2000, 10 pages. [cited by applicant]
Sean Andrist, et al., “Conversational Gaze Aversion for Humanlike Robots”, ResearchGate, HRI '14 Proceedings of the 2014 ACM/IEEE international conference on Human-robot interaction, Mar. 2014, 9 pages. [cited by applicant]
Sean Andrist, et al., “Look Like Me: Matching Robot Personality via Gaze to Increase Motivation”, CHI '15, Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems, Apr. 2015, 10 pages. [cited by applicant]
Jong-Hwan Kim, et al., “Two-Layered Confabulation Architecture for an Artificial Creature's Behavior Selection”, IEEE Transactions on Systems, Man and Cybernetics—Part C: Applications and Reviews, vol. 38, No. 6, Nov. 2… [cited by applicant]
Maximilian Nickel, et al., “A Review of Relational Machine Learning for Knowledge Graphs”, Proceedings of the IEEE, Sep. 28, 2015, 23 pages. [cited by applicant]
Adrià Recasens, et al., “Where are they looking?”, Advances in Neural Information Processing Systems 28, Curran Associates, Inc., 2015, 9 pages. [cited by applicant]
Adam Santoro, et al., “A simple neural network module for relational reasoning”, 31st Conference on Neural Information Processing Systems, NIPS, 2017, 10 pages. [cited by applicant]
Aaron P. Shon, et al., “Probabilistic Gaze Imitation and Saliency Learning in a Robotic Head”, May 2005, 6 pages. [cited by applicant]
Zeynep Yücel, et al., “Joint Attention by Gaze Interpolation and Saliency”, IEEE Transactions on Cybernetics, vol. 43, No. 3, pp. 829-842. [cited by applicant]
International Search Report for PCT/KR2018/016623 dated Apr. 9, 2019, 5 pages with English Translation. [cited by applicant]
Written Opinion of the ISA for PCT/KR2018/016623 dated Apr. 9, 2019, 13 pages with English Translation. [cited by applicant]
Office Action dated Feb. 5, 2021 in counterpart Chinese Patent Application No. 201880085453.5 and English-language translation. [cited by applicant]
Notice of Allowance dated Dec. 3, 2021 in counterpart Chinese Patent Application No. 201880085453.5 and English-language translation. [cited by applicant]
Notice of Preliminary Rejection dated Dec. 23, 2021 in counterpart Korean Patent Application No. 10-2018-0001692 and English-language translation. [cited by applicant]
Notice of Decision of Rejection dated Jun. 27, 2022 in counterpart Korean Patent Application No. 10-2018-0001692 and English-language translation. [cited by applicant]
Office Action dated Nov. 8, 2022 in KR Application No. 10-2018-0001692 and English-language machine translation. [cited by applicant]
Office Action dated Sep. 20, 2023 in CN Application No. 2022101529777.X and English-language translation. [cited by applicant]
Office Action dated Oct. 31, 2023 in counterpart Korean Patent Application No. 10-2023-0017619 and English-language translation. [cited by applicant]
Kim et al., U.S. Appl. No. 16/768,452, filed May 29, 2020. [cited by applicant]