Automated item attribution and condition detection system
An electronic commerce platform application server comprises an attribute generator application, condition detector application, and a machine learning system. During operation, a device collects sensor data, such as images and/or 3D point clouds, of an item to add to inventory. The attribute generator and condition detector applications receive the sensor data from the device over a network. The attribute generator and condition detector applications process the sensor data using the ML system and predict attributes and condition of the item. In one embodiment, the ML system includes a vision encoder and one or more large language models (LLMs) prompted to generate a description of the item from an image. The ML system also includes one or more neural networks trained to detect condition information, including condition description, bounding boxes identifying defects, high wear regions, and condition classification categories. The ML systems provide scalable and efficient addition of new inventory.
1 . A method comprising:
receiving, at an application server over a network, an image of an item from a device comprising an image sensor;
supplying the image to a vision encoder comprising a vision transformer and a querying transformer;
generating a first prompt requesting a description of the item;
supplying an output of the vision encoder and the first prompt to a first large language model (LLM) to generate a text description of the item;
generating a second prompt requesting attribute values;
supplying the text description and the second prompt to a second LLM to generate attribute information of the item;
predicting condition information of the item from the image using a neural network trained to detect defects or wear regions;
predicting a bounding box associated with the condition information, wherein the bounding box identifies at least one of a defect or a high wear region on the item; and
ranking one or more condition detection regions based on return history information collected by an electronic commerce platform, wherein the return history information indicates items that were returned by buyers due to condition-related issues.
2 . The method of claim 1 , further comprising:
prompting a user to obtain additional images of one or more regions of interest based on the predicted condition information.
3 . The method of claim 1 , further comprising:
storing the predicted condition information and presenting the predicted condition information to a user.
4 . The method of claim 1 , wherein a single linear projection layer is disposed between the vision encoder and the first LLM.
5 . The method of claim 1 , wherein when a high wear region is identified, a user is instructed to capture additional images of the high wear region, and wherein the additional images are presented to potential purchasers.
6 . The method of claim 1 , further comprising:
categorizing the item based on the condition information into a condition category.
7 . The method of claim 1 , wherein the neural network is trained using labeled image data that includes a bounding box surrounding a defect and a label describing the defect.
8 . The method of claim 6 , wherein the condition category comprises: pristine and good.
9 . The method of claim 6 , further comprising:
supplying the item to an intake pipeline based on the assigned condition category.
10 . The method of claim 1 , wherein the ranking based on return history information assists in minimizing returns of items sold through the electronic commerce platform.
11 . The method of claim 1 , wherein the predicted condition information assists in generating a pricing estimate for the item.