System and method for artificial intelligence (AI)-powered food classification and consumption tracking

1 . An apparatus, comprising:

a scale with a plurality of weight sensors,

a 2-dimensional (2D) camera,

a 3-dimensional (3D) camera, and

one or more processors and one or more non-transitory computer-readable memories coupled to the one or more processors and configured with instructions executable by the one or more processors to perform operations comprising:

in response to a container filled with food being placed on the scale:

obtaining, from the scale, a first weight measurement of the container;

capturing, using the 2D camera, a 2D image of the container;

capturing, using the 3D camera, a point cloud of the container;

extracting a first set of features of the container from the 2D image of the container;

scanning, from a top of the point cloud and moving downward, each plane of data points in the point cloud to extract a second set of features of the container;

identifying the container based on the first set of features and the second set of features;

obtaining a pre-stored second weight measurement corresponding to the identified container, wherein the second weight measurement was measured when the identified container is empty; and

obtaining a weight measurement of the food in the container based on the first weight measurement and the second weight measurement.

2 . The apparatus of claim 1 , wherein the extracting the second set of features of the container from the point cloud comprises:

detecting a rim of the container from the downward scanning of the point cloud.

3 . The apparatus of claim 2 , wherein the detecting the rim of the container from the scanning comprises:

in response to hitting a first plane of data points with identical depth values, determining that the first plane of data points represents the rim of the container.

4 . The apparatus of claim 2 , wherein the operations further comprising:

pre-registering shapes of rims of a plurality of containers to be used with the apparatus; and

the detecting the rim of the container from the scanning comprises:

in response to (1) hitting a first plane of data points with identical depth values and (2) the first plane of data points matching with one of the pre-registered shapes of rims, determining that the first plane of data points represents the rim of the container.

5 . The apparatus of claim 1 , wherein the operations further comprising:

determining an orientation of the container in the 2D image; and

rotating the 2D image of the container to align the orientation of the container with reference axis of the 2D camera.

6 . The apparatus of claim 5 , wherein the extracting the first set of features of the container from the 2D image of the container comprises:

generating a bounding box surrounding the rotated 2D image of the container; and

extracting, using a first machine learning model, the first set of features of the container within the bounding box.

7 . The apparatus of claim 1 , wherein the first set of features of the container comprises texture information, color, size, or shape.

8 . The apparatus of claim 1 , wherein the second set of features of the container comprises a shape of a rim of the container, depth information of the container, or volume.

9 . The apparatus of claim 1 , wherein the extracting the second set of features of the container from the point cloud comprises:

scanning, from the top of the point cloud and moving downward, the each plane of data points in the point cloud;

in response to that (1) a plane of data points has a cluster of data points with identical depth values and (2) a shape formed by the cluster of data points does not match with any of pre-registered shapes of rims, determining that the cluster of data points belong to an object extended above a rim of the container; and

continuing the scanning by ignoring the cluster of data points.

10 . The apparatus of claim 1 , wherein the extracting the first set of features of the container from the 2D image of the container comprises:

detecting an object extended over a rim of the container; and

erasing the object from the 2D image before detecting the first set of features.

11 . The apparatus of claim 1 , wherein the operations further comprise:

identifying, using a local food classification model, the food in the container based on the first set of features as one of a plurality of pre-registered food items, wherein the local food classification model is machine learning model downloaded from a server.

12 . The apparatus of claim 11 , further comprising:

a controller configured to:

in response to failure to identify the food as one of the plurality of pre-registered food items, display the 2D image of the food and a plurality of food labels corresponding to the plurality of pre-registered food items;

receive a user-selected food label from plurality of food labels; and

retrain the local food classification model based on the user-selected food label and the 2D image for subsequent food identification at the apparatus.

13 . The apparatus of claim 12 , wherein the operations further comprise:

uploading the user-selected food label and the 2D image to the server, for the server to update the machine learning model on the server, and broadcast the updated machine learning model to other apparatuses.

14 . A system for food classification, comprising:

a plurality of food scanning apparatuses deployed at different areas for scanning food containers, each food scanning apparatus comprising a scale, a 3D camera, and one or more processors,

wherein the one or more processors in the food scanning apparatus are configured to:

in response to a container filled with food being placed on the scale, obtain, from the scale, a first weight measurement of the container filled with food;

capture, using the 3D camera, a point cloud of the container;

determine a rim shape and a volume of the container based on the point cloud of the container;

identify the container based on the rim shape and the volume of the container;

obtain a pre-stored second weight measurement corresponding to the identified container, wherein the second weight measurement was measured when the identified container is empty; and

obtain a weight measurement of the food in the container based on the first weight measurement and the second weight measurement.

15 . The system of claim 14 , wherein the food scanning apparatus further comprises:

a 2D camera, configured to capture a 2D image of the container; and

a server,

wherein the one or more processors in the food scanning apparatus are further configured to:

extract a first set of features of the container from the 2D image of the container; and

identify, using a local food classification model stored on the food scanning apparatus, the food in the container based on the first set of features as one of a plurality of pre-registered food items, wherein the local food classification model is machine learning model downloaded from the server.

16 . The system of claim 15 , wherein the one or more processors in the food scanning apparatus are further configured to:

in response to failure to identify the food as one of the plurality of pre-registered food items, display the 2D image of the food and a plurality of food labels corresponding to the plurality of pre-registered food items;

receive a user-selected food label from plurality of food labels; and

retrain the local food classification model with the user-selected food label and the 2D image for subsequent food identification at the food scanning apparatus.

17 . The system of claim 16 , wherein the one or more processors in the food scanning apparatus are further configured to:

upload the user-selected food label and the 2D image to the server, for the server to update the machine learning model on the server.

18 . The system of claim 17 , wherein the server is further configured to:

broadcast the updated machine learning model to other food scanning apparatuses.

19 . The system of claim 15 , wherein to extract the first set of features of the container from the 2D image of the container, the one or more processors in the food scanning apparatus are further configured to:

determine an orientation of the container in the 2D image;

rotate the 2D image of the container to align the orientation of the container with reference axis of the 2D camera;

generate a bounding box surrounding the rotated 2D image of the container; and

extract, using a first machine learning model, the first set of features of the container within the bounding box.

20 . A computer-implemented method, comprising:

obtaining, from a scale, a first weight measurement of a container filled with food;

capturing, using a 2D camera, a 2D image of the container;

capturing, using a 3D camera, a point cloud of the container;

extracting a first set of features of the container from the 2D image of the container;

scanning, from a top of the point cloud and moving downward, each plane of data points in the point cloud to detect a shape of a rim of the container;

identifying the container based on the first set of features and the shape of the rim of the container;

obtaining a pre-stored second weight measurement corresponding to the identified container, wherein the second weight measurement was measured when the identified container is empty; and

obtaining a weight measurement of the food in the container based on the first weight measurement and the second weight measurement.