Granted Patent B2

US 12,320,580 · App. 18/312,161 · Granted Jun 3, 2025

Inventors: Jason Andrew May (Louisville, KY); Brent Alden Junge (Louisville, KY); Michael Goodman Schroeder (Crestwood, KY)

Assignee: Haier US Appliance Solutions, Inc.

F25D29/00F25D23/062G06T7/74H04N5/77H04N23/60H04R1/08

Abstract

Claims (114)

1. A method of operating a refrigerator appliance comprising a camera module and a microphone within a chilled chamber defined by a cabinet, the method comprising:

initiating a capture sequence at the camera module and the microphone;

determining an internal location of a first stored item within the chilled chamber based on a two-dimensional image and a sound recording of the capture sequence; and

recording the internal location of the first stored item,

wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score.

2. The method of claim 1 , wherein the determined internal location is based on an amplitude characteristic of the sound recording.

3. The method of claim 1 , wherein the determined internal location is based on a frequency characteristic of the sound recording.

4. The method of claim 1 , further comprising:

determining a mass attribute of the first stored item based on the sound recording; and

recording the mass attribute of the first stored item,

wherein the determined mass attribute is based on a frequency characteristic of the sound recording.

5. The method of claim 1 , wherein the microphone comprises a plurality of microphones spaced apart within the chilled chamber,

wherein the sound recording comprises a plurality of sound recordings each corresponding to a discrete microphone of the plurality of microphones,

wherein determining the internal location comprises identifying a placement event of the first stored item within the plurality of sound recordings and determining latency of the placement event between the plurality of sound recordings, and

wherein the determined internal location is based on the determined latency.

6. The method of claim 1 , wherein determining the visual location score comprises identifying the first stored item within the two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises matching the identified first stored item with the identified placement event.

7. The method of claim 1 , wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence,

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds the first two-dimensional image, and

assigning greater weight to the second visual location score than the first visual location score in reconciliation based on the compared timing.

8. The method of claim 1 , wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording,

wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence, and

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image, the second visual location score being below a set score threshold,

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds or accompanies the placement event, and

assigning greater weight to the audio location score than the second visual location score in reconciliation based on the compared timing.

9. A method of operating a refrigerator appliance comprising a camera module and a microphone within a chilled chamber defined by a cabinet, the method comprising:

initiating a capture sequence at the camera module and the microphone;

determining an internal location of a first stored item within the chilled chamber based on a two-dimensional image and a sound recording of the capture sequence; and

recording the internal location of the first stored item,

wherein the determined internal location is based on an amplitude characteristic of the sound recording.

10. The method of claim 9 , further comprising:

determining a mass attribute of the first stored item based on the sound recording; and

recording the mass attribute of the first stored item,

wherein the determined mass attribute is based on a frequency characteristic of the sound recording.

11. The method of claim 9 , wherein the microphone comprises a plurality of microphones spaced apart within the chilled chamber,

wherein the sound recording comprises a plurality of sound recordings each correspond to a discrete microphone of the plurality of microphones,

wherein determining the internal location comprises identifying a placement event of the first stored item within the plurality of sound recordings and determining latency of the placement event between the plurality of sound recordings, and

wherein the determined internal location is based on the determined latency.

12. The method of claim 9 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein determining the visual location score comprises identifying the first stored item within the two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises matching the identified first stored item with the identified placement event.

13. The method of claim 9 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence,

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds the first two-dimensional image, and

assigning greater weight to the second visual location score than the first visual location score in reconciliation based on the compared timing.

14. The method of claim 9 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording,

wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence, and

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image, the second visual location score being below a set score threshold,

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds or accompanies the placement event, and

assigning greater weight to the audio location score than the second visual location score in reconciliation based on the compared timing.

15. A method of operating a refrigerator appliance comprising a camera module and a microphone within a chilled chamber defined by a cabinet, the method comprising:

initiating a capture sequence at the camera module and the microphone;

determining an internal location of a first stored item within the chilled chamber based on a two-dimensional image and a sound recording of the capture sequence; and

recording the internal location of the first stored item,

wherein the microphone comprises a plurality of microphones spaced apart within the chilled chamber,

wherein the sound recording comprises a plurality of sound recordings each corresponding to a discrete microphone of the plurality of microphones,

wherein determining the internal location comprises identifying a placement event of the first stored item within the plurality of sound recordings and determining latency of the placement event between the plurality of sound recordings, and

wherein the determined internal location is based on the determined latency.

16. The method of claim 15 , wherein the determined internal location is based on a frequency characteristic of the sound recording.

17. The method of claim 15 , further comprising:

determining a mass attribute of the first stored item based on the sound recording; and

recording the mass attribute of the first stored item,

wherein the determined mass attribute is based on a frequency characteristic of the sound recording.

18. The method of claim 15 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein determining the visual location score comprises identifying the first stored item within the two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises matching the identified first stored item with the identified placement event.

19. The method of claim 15 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence,

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording, and

wherein reconciling the determined visual location score with the determined audio location score comprises

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds the first two-dimensional image, and

assigning greater weight to the second visual location score than the first visual location score in reconciliation based on the compared timing.

20. The method of claim 15 , wherein determining the internal location comprises

determining a visual location score of the first stored item based on the two-dimensional image,

determining an audio location score of the first stored item based on the sound recording, and

reconciling the determined visual location score with the determined audio location score,

wherein determining the audio location score comprises identifying a placement event of the first stored item within the sound recording,

wherein the two-dimensional image comprises a first two-dimensional image and a second two-dimensional image captured subsequently to the first two-dimensional image during the capture sequence, and

wherein determining the visual location score comprises determining a first visual location score based on the first two-dimensional image and determining a second visual location score based on the second two-dimensional image, the second visual location score being below a set score threshold,

comparing timing of the first and second two-dimensional images with the placement event, comparing comprising determining the first two-dimensional image precedes the placement event and determining the second two-dimensional image succeeds or accompanies the placement event, and

assigning greater weight to the audio location score than the second visual location score in reconciliation based on the compared timing.

Assignments (1)

Continuity (1)

References Cited (12)