Segmenting food items from background in a cooking apparatus

1 . A method comprising:

accessing, by a computing device, a thermal image of a space within a cooking apparatus, wherein the space contains a food item;

identifying, from the thermal image, a temperature associated with each of a plurality of pixels in the thermal image;

determining a temperature threshold and a heating-rate threshold for segmenting the food item in the space from the food item's environment by:

determining, from the temperature associated with each of the plurality of pixels in the thermal image and prior to segmenting the food item from the food item's environment, a temperature distribution identifying a number of pixels in the thermal image that correspond to each of a plurality of separate temperature values, by determining, for each of the plurality of separate temperature values, a total number of pixels in the thermal image having a temperature that is within that respective temperature value;

determining, from the temperature distribution, at least a pair of peaks in the temperature distribution, each peak having a distinct temperature value;

determining, based on the determined temperature distribution and prior to segmenting the food item from the food item's environment, a temperature threshold that separates one of the at least one pair of peaks in the temperature distribution;

heating, using the cooking apparatus, the food or the space within the cooking apparatus;

accessing a second thermal image of the space after applying heat to the food or to the space;

determining, based on the thermal image of the space before heating and the second thermal image of the space after heating, a heating rate associated with each of the plurality of pixels in the thermal image of the space; and

determining, from a distribution of heating rates associated with the plurality of pixels of the thermal image, a heating-rate threshold; and

segmenting the food item in the space from the food item's environment by, for each of the plurality of pixels in the thermal image, identifying the pixel as associated with the food item or as associated with the food item's environment based on a combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

2 . The method of claim 1 , wherein the plurality of pixels in the thermal image includes every pixel in the thermal image.

3 . The method of claim 1 , wherein the food item's environment comprises one or more of: (1) an interior of the cooking apparatus or (2) a container within the cooking apparatus.

4 . The method of claim 1 , further comprising:

accessing an initial image of the space within the cooking apparatus during a time when the space does not contain any food item;

detecting a start of a cooking process of the cooking apparatus; and

capturing a subsequent image of the space within the cooking apparatus after the cooking process is started, wherein segmenting the food item in the space from the food item's environment is further based on the initial image of the space and on the subsequent image of the space after the cooking process is started.

5 . The method of claim 4 , wherein the initial image of the space and the subsequent image of the space comprise thermal images.

6 . The method of claim 4 , wherein the initial image of the space and the subsequent image of the space comprise optical images.

7 . The method of claim 1 , further comprising:

applying heat, using the cooking apparatus, to the food or to the space within the cooking apparatus;

accessing an image of the space after applying heat; and

re-segmenting, based on the image of the space after applying heat, the food item in the space from the food item's environment.

8 . The method of claim 1 , wherein:

segmenting the food item comprises: when both the pixel's associated temperature relative to the temperature threshold and the heating rate associated with that pixel identify the pixel as a food-item pixel, then identifying the pixel as a food-item pixel, and otherwise identifying the pixel as a background pixel.

9 . One or more non-transitory computer readable storage media embodying instructions and coupled to one or more processors that are operable to execute the instructions to:

access a thermal image of a space within a cooking apparatus, wherein the space contains a food item; and

identify, from the thermal image, a temperature associated with each of a plurality of pixels in the thermal image;

determine a temperature threshold and a heating-rate threshold for segmenting the food item in the space from the food item's environment by:

determining, from the temperature associated with each of the plurality of pixels in the thermal image and prior to segmenting the food item from the food item's environment, a temperature distribution identifying a number of pixels in the thermal image that correspond to each of a plurality of separate temperature values, by determining, for each of the plurality of separate temperature values, a total number of pixels in the thermal image having a temperature that is within that respective temperature value;

determining, from the temperature distribution, at least a pair of peaks in the temperature distribution, each peak having a distinct temperature value;

determining, based on the determined temperature distribution and prior to segmenting the food item from the food item's environment, a temperature threshold that separates one of the at least one pair of peaks in the temperature distribution;

heating, using the cooking apparatus, the food or the space within the cooking apparatus;

accessing a second thermal image of the space after applying heat to the food or to the space;

determining, based on the thermal image of the space before heating and the second thermal image of the space after heating, a heating rate associated with each of the plurality of pixels in the thermal image of the space; and

determining, from a distribution of heating rates associated with the plurality of pixels of the thermal image, a heating-rate threshold

segment the food item in the space from the food item's environment by, for each of the plurality of pixels in the thermal image, identifying the pixel as associated with the food item or as associated with the food item's environment based on a combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

10 . A system comprising one or more processors and a non-transitory computer readable storage media embodying instructions coupled to the one or more processors, the one or more processors operable to execute the instructions to:

access a thermal image of a space within a cooking apparatus, wherein the space contains a food item; and

identify, from the thermal image, a temperature associated with each of a plurality of pixels in the thermal image;

determine a temperature threshold and a heating-rate threshold for segmenting the food item in the space from the food item's environment by:

determining, from the temperature associated with each of the plurality of pixels in the thermal image and prior to segmenting the food item from the food item's environment, a temperature distribution identifying a number of pixels in the thermal image that correspond to each of a plurality of separate temperature values, by determining, for each of the plurality of separate temperature values, a total number of pixels in the thermal image having a temperature that is within that respective temperature value;

determining, from the temperature distribution, at least a pair of peaks in the temperature distribution, each peak having a distinct temperature value;

determining, based on the determined temperature distribution and prior to segmenting the food item from the food item's environment, a temperature threshold that separates one of the at least one pair of peaks in the temperature distribution;

heating, using the cooking apparatus, the food or the space within the cooking apparatus;

accessing a second thermal image of the space after applying heat to the food or to the space;

determining, based on the thermal image of the space before heating and the second thermal image of the space after heating, a heating rate associated with each of the plurality of pixels in the thermal image of the space; and

determining, from a distribution of heating rates associated with the plurality of pixels of the thermal image, a heating-rate threshold

segment the food item in the space from the food item's environment by, for each of the plurality of pixels in the thermal image, identifying the pixel as associated with the food item or as associated with the food item's environment based on a combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

11 . The method of claim 1 , wherein:

the temperature distribution comprises a multimodal distribution;

the temperature threshold comprises a plurality of temperature thresholds, each separating a particular pair of peaks in the multimodal temperature distribution; and

the method further comprises identifying the pixel as associated with the food item or as associated with the food item's environment based on that pixel's associated temperature relative to at least one of the temperature thresholds in the temperature distribution.

12 . The system of claim 10 , wherein:

the temperature distribution comprises a multimodal distribution;

the temperature threshold comprises a plurality of temperature thresholds, each separating a particular pair of peaks in the multimodal temperature distribution; and

the system further comprises one or more processors operable to execute the instructions to identify the pixel as associated with the food item or as associated with the food item's environment based on that pixel's associated temperature relative to at least one of the temperature thresholds in the temperature distribution.

13 . The system of claim 10 , further comprising the cooking apparatus, wherein:

the cooking apparatus comprises a microwave; and

the microwave comprises the one or more processors and the non-transitory computer readable storage media.

14 . The method of claim 1 , wherein segmenting the food item based on the combination comprises segmenting the food item based on a weighted combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

15 . The method of claim 14 , wherein each weight in the weighted combination is based on a certainty associated with either that pixel's associated temperature relative to the temperature threshold or with that pixel's associated heating rate relative to the heating-rate threshold.

16 . The media of claim 9 , wherein segmenting the food item based on the combination comprises segmenting the food item based on a weighted combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

17 . The media of claim 16 , wherein each weight in the weighted combination is based on a certainty associated with either that pixel's associated temperature relative to the temperature threshold or with that pixel's associated heating rate relative to the heating-rate threshold.

18 . The system of claim 10 , wherein segmenting the food item based on the combination comprises segmenting the food item based on a weighted combination of (1) that pixel's associated temperature relative to the temperature threshold in the temperature distribution and (2) that pixel's associated heating rate relative to the heating-rate threshold in the distribution of heating rates.

19 . The system of claim 18 , wherein each weight in the weighted combination is based on a certainty associated with either that pixel's associated temperature relative to the temperature threshold or with that pixel's associated heating rate relative to the heating-rate threshold.

20 . The system of claim 13 , wherein the cooking apparatus further comprises a geometric thermal pattern on an interior of the cooking apparatus.