Electronic device and controlling method of electronic device

1 . An electronic device comprising:

at least one sensor;

a first memory storing a plurality of neural network models and one or more instructions; and

at least one processor comprising a second memory, the at least one processor operatively coupled to the first memory,

wherein the one or more instructions, when executed by the at least one processor, cause the electronic device to:

in response to reception of a user input for selection of a cooking menu, obtain recipe information corresponding to the cooking menu,

identify, from among a plurality of cooking steps included in the recipe information, a first cooking step and at least one cooking step to be performed after the first cooking step,

load from the first memory into the second memory, based on the recipe information and a resource of the electronic device, at least one first neural network model among the plurality of neural network models, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step,

obtain sensing data from the at least one sensor, the sensing data corresponding to at least the first cooking step,

obtain context information on a cooking situation indicated in the sensing data by inputting the sensing data to the at least one first neural network model,

control an operation of the electronic device based on the recipe information and the context,

detect a neural network model change event based on the context information, and

in response to detection of the neural network model change event, load, from the first memory into the second memory, at least one second neural network model among the plurality of neural network models based on the context information and resource information of the electronic device,

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to, in response to a determination that a plurality of first neural network models exist:

identify a resource required to execute each of the plurality of first neural network models, and

determine a number of neural network models among the plurality of first neural network models to be simultaneously loaded, based on the resource of the electronic device and the resource required to execute each of the plurality of first neural network models,

wherein each first neural network model of the plurality of first neural network models corresponds to the first cooking step and the at least one cooking step to be performed after the first cooking step, and

wherein the plurality of first neural network models are included in the plurality of neural network models.

2 . The device as claimed in claim 1 , wherein the neural network model change event corresponds to an event in which performance of a second cooking step different from the first cooking step is detected, and

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

identify, based on the context information, the second cooking step and the at least one cooking step to be performed after the second cooking step among the plurality of cooking steps included in the recipe information, and

load from the first memory into the second memory, based on the recipe information and the resource of the electronic device, the at least one second neural network model among the plurality of neural network models, the at least one second neural network model corresponding to the second cooking step and the at least one cooking step to be performed after the second cooking step.

3 . The device as claimed in claim 2 , wherein, the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

determine whether to change the recipe information based on the context information and the recipe information,

in response to determining that the recipe information is changed, obtain changed recipe information,

identify a third cooking step and the at least one cooking step to be performed after the third cooking step among the plurality of cooking steps included in the changed recipe information, based on the context information, and

load from the first memory into the second memory, based on the changed recipe information and the resource of the electronic device, at least one third neural network model among the plurality of neural network models, the at least one third neural network model corresponding to the third cooking step and the at least one cooking step to be performed after the third cooking step.

4 . The device as claimed in claim 3 , wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

determine whether an intermediate cooking step exists between the first cooking step and the second cooking step,

in response to determining that no intermediate cooking step exists between the first cooking step and the second cooking step, determine that the recipe information is not changed,

in response to determining that the intermediate cooking step exists between the first cooking step and the second cooking step, determine whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step, and

in response to determining that the intermediate cooking step between the first cooking step and the second cooking step is the cooking step related to the next cooking step after the second cooking step, determine that the recipe information is changed.

5 . The device as claimed in claim 1 , wherein the neural network model change event corresponds to detection of a new cookware not included in the recipe information, and

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

based on the detection of the new cookware not included in the recipe information, change the recipe information based on the new cookware, and

load from the first memory to the second memory, based on the changed recipe information and the resource of the electronic device, at least one fourth neural network model among the plurality of neural network models, the at least one fourth neural network model corresponding to a fourth cooking step and the at least one cooking step to be performed after the fourth cooking step.

6 . The device as claimed in claim 1 , wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

identify the first cooking step included in the recipe information,

obtain information on a probability that one or more cooking steps are performed after the first cooking step based on the recipe information, and

identify the at least one cooking step among next cooking steps after the first cooking step, based on the information on the probability that the one or more cooking steps are performed after the first cooking step.

7 . The device as claimed in claim 6 , wherein the recipe information includes a knowledge graph showing a recipe corresponding to the selected cooking menu,

wherein the knowledge graph includes a plurality of nodes representing the plurality of cooking steps for completing the cooking menu based on the recipe and a plurality of edges representing a sequential relationship between the plurality of cooking steps, and

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to obtain the information on the probability that the one or more cooking steps are performed after the first cooking step based on a distance between a node representing the first cooking step and nodes representing the next cooking steps after the first cooking step.

8 . The device as claimed in claim 1 , wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

determine a weight value of each of the plurality of first neural network models based on the information on a probability that the at least one cooking step is performed after the first cooking step;

sequentially load, from the first memory into the second memory, each of the plurality of first neural network models based on the determined weight values; and

obtain the context information on the cooking situation indicated in the sensing data by inputting the sensing data to the plurality of sequentially loaded first neural network models.

9 . The device as claimed in claim 1 , wherein the at least one sensor comprises an illuminance sensor for sensing an illuminance value around the electronic device, and

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to load, from the first memory into the second memory, the at least one first neural network model among the plurality of neural network models, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step, based on at least one of a type of the at least one sensor and the illuminance value obtained from the illuminance sensor.

10 . A method of controlling an electronic device, the method comprising:

in response to receiving a user input selecting a cooking menu, obtaining recipe information corresponding to the cooking menu;

identifying, from among a plurality of cooking steps included in the recipe information, a first cooking step and at least one cooking step to be performed after the first cooking step;

loading, based on the recipe information and a resource of the electronic device, at least one first neural network model among a plurality of neural network models from a first memory of the electronic device into a second memory of at least one processor of the electronic device, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step;

obtaining sensing data from at least one sensor, the sensing data corresponding to at least the first cooking step;

obtaining context information on a cooking situation indicated in the sensing data by inputting the sensing data to the at least one first neural network model;

controlling an operation of the electronic device based on the recipe information and the context information;

detecting a neural network model change event based on the context information; and

in response to detection of the neural network model change event, loading, from the first memory into the second memory, at least one second neural network model among the plurality of neural network models based on the context information and resource information of the electronic device,

wherein the loading the at least one first neural network model comprises, in response to a determination that a plurality of first neural network models exist:

identifying a resource required to execute each of the plurality of first neural network models, and

determining a number of neural network models among the plurality of first neural network models to be simultaneously loaded, based on the resource of the electronic device and the resource required to execute each of the plurality of first neural network models,

wherein each first neural network model of the plurality of first neural network models corresponds to the first cooking step and the at least one cooking step to be performed after the first cooking step, and

wherein the plurality of first neural network models are included in the plurality of neural network models.

11 . The method as claimed in claim 10 , wherein the neural network model change event corresponds to an event in which performance of a second cooking step different from the first cooking step is detected, and

wherein the loading of the at least one second neural network model comprises:

identifying, based on the context information, the second cooking step and the at least one cooking step to be performed after the second cooking step among the plurality of cooking steps included in the recipe information; and

loading from the first memory into the second memory, based on the recipe information and the resource of the electronic device, the at least one second neural network model among the plurality of neural network models, the at least one second neural network model corresponding to the second cooking step and the at least one cooking step to be performed after the second cooking step.

12 . The method as claimed in claim 11 , wherein the loading the at least one second neural network model comprises:

determining whether to change the recipe information based on the context information and the recipe information;

in response to determining that the recipe information is changed, obtaining changed recipe information;

identifying, based on the context information, a third cooking step and the at least one cooking step to be performed after the third cooking step among the plurality of cooking steps included in the changed recipe information; and

loading from the first memory into the second memory, based on the changed recipe information and the resource of the electronic device, at least one third neural network model among the plurality of neural network models, the at least one third neural network model corresponding to the third cooking step and the at least one cooking step to be performed after the third cooking step.

13 . The method as claimed in claim 12 , wherein the determining whether to change the recipe information comprises:

determining whether an intermediate cooking step exists between the first cooking step and the second cooking step;

in response to determining that no intermediate cooking step exists between the first cooking step and the second cooking step, determining that the recipe information is not changed;

in response to determining that the intermediate cooking step exists between the first cooking step and the second cooking step, determining whether the intermediate cooking step between the first cooking step and the second cooking step is a cooking step related to a next cooking step after the second cooking step; and

in response to determining that the cooking step between the first cooking step and the second cooking step is the cooking step related to the next cooking step after the second cooking step, determining that the recipe information is changed.

14 . The method as claimed in claim 10 , wherein the neural network model change event corresponds to a detection of a new cookware not included in the recipe information, and

wherein the loading the at least one second neural network model comprises:

in response to the detection of the new cookware not included in the recipe information, changing the recipe information based on the new cookware; and

loading from the first memory into the second memory, based on the changed recipe information and the resource of the electronic device, at least one fourth neural network model among the plurality of neural network models, the at least one fourth neural network model corresponding to a fourth cooking step and the at least one cooking step to be performed after the fourth cooking step.

15 . An electronic device comprising:

at least one sensor;

a first memory storing a plurality of neural network models and one or more instructions; and

at least one processor comprising a second memory, the at least one processor operatively coupled to the first memory,

wherein the one or more instructions, when executed by the at least one processor, cause the electronic device to:

in response to reception of a user input selecting a cooking menu, obtain recipe information corresponding to the cooking menu,

identify a first cooking step and at least one cooking step to be performed after the first cooking step among a plurality of cooking steps included in the recipe information,

load from the first memory into the second memory, based on the recipe information and a resource of the electronic device, at least one first neural network model among the plurality of neural network models, the at least one first neural network model corresponding to the first cooking step and the at least one cooking step to be performed after the first cooking step,

obtain sensing data from the at least one sensor, the sensing data corresponding to the first cooking step,

obtain context information on a cooking situation indicated in the sensing data by inputting the sensing data to the at least one first neural network model,

control an operation of the electronic device based on the recipe information and the context information,

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to, in response to a determination that a plurality of first neural network models exist:

identify a resource required to execute each of the plurality of first neural network models, and

determine a number of neural network models among the plurality of first neural network models to be simultaneously loaded, based on the resource of the electronic device and the resource required to execute each of the plurality of first neural network models,

wherein each first neural network model in the plurality of first neural network models corresponds the first cooking step and the at least one cooking step to be performed after the first cooking step, and

wherein the plurality of first neural network models are included in the plurality of neural network models.

16 . The device as claimed in claim 15 , wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

identify the first cooking step included in the recipe information,

obtain information on a probability that one or more cooking steps are performed after the first cooking step based on the recipe information, and

identify, based on the information on the probability that the one or more cooking steps are performed after the first cooking step, the at least one cooking step among next cooking steps after the first cooking step.

17 . The device as claimed in claim 16 , wherein the recipe information includes a knowledge graph showing a recipe corresponding to the selected cooking menu,

wherein the knowledge graph includes a plurality of nodes representing the plurality of cooking steps for completing the cooking menu based on the recipe and a plurality of edges representing a sequential relationship between the plurality of cooking steps, and

wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to obtain the information on the probability that the one or more cooking steps are performed after the first cooking step based on a distance between a node representing the first cooking step and nodes representing the next cooking steps after the first cooking step.

18 . The device as claimed in claim 15 , wherein the one or more instructions, when executed by the at least one processor, further cause the electronic device to:

determine a weight value of each of the plurality of first neural network models based on information on a probability that the at least one cooking step is performed after the first cooking step;

sequentially load, from the first memory into the second memory, each of the plurality of first neural network models based on the determined weight values; and

obtain the context information on the cooking situation indicated in the sensing data by inputting the sensing data to the plurality of sequentially loaded first neural network models.