Systems and methods for dynamically ingesting and monitoring online products
A system includes one or more memory devices storing instructions, and one or more processors configured to execute the instructions to perform the steps of a method to dynamically ingest and monitor online products. The method may include receiving access to a webpage, determining whether the webpage includes a predetermined template, and in response to receiving the predetermined template, ingesting one or more product images and product metadata from the webpage. If no template is found, the system may request search engine optimization metadata to determine if at least one image is of a product before initializing a machine learning algorithm to determine whether the images correspond to product images. If no associated metadata is found, the system may request manual input of associated metadata before displaying, in real-time and in a standardized format, the ingested photos and metadata, allowing a user to select photos representing products of interest.
1. A method comprising:
receiving a request to access a webpage, wherein the webpage comprises product metadata;
analyzing the webpage, via a trained machine learning algorithm, to (i) determine whether one or more images on the webpage correspond to one or more product images and (ii) determine whether the webpage comprises product metadata corresponding to the one or more product images;
responsive to determining (i) that the one or more images on the webpage correspond to one or more product images and (ii) that the one or more product images are missing product metadata associated with the one or more product images:
prompting a user to manually identify one or more relevant fields on the webpage corresponding to the product metadata associated with the one or more product images; and
ingesting the one or more product images and the product metadata associated with the one or more product images from the webpage based on the trained machine learning algorithm analysis and the manual identification;
displaying the ingested one or more product images and the product metadata associated with the ingested one or more product images in a standardized format independent of a format of the webpage;
receiving a user input based on an interaction with a first user input mechanism associated with a first ingested product image of the ingested one or more product images and a first ingested product metadata associated with the first ingested product image, the user input indicative of a selection of the first ingested product image and the first ingested product metadata;
saving the first ingested product image and the first ingested product metadata into a product repository; and
dynamically monitoring the first ingested product metadata stored in the product repository for changes.
2. The method of claim 1 , wherein the one or more product images further comprise vehicle images and the trained machine learning algorithm is iteratively retrained based on the manual identification of the product metadata associated with the one or more product images to recognize the one or more relevant fields on the webpage as indicative of the product metadata.
3. The method of claim 2 , wherein the product metadata comprises a make, a model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof.
4. The method of claim 1 , further comprising:
responsive to determining (i) that the one or more images on the webpage do not correspond to one or more product images and (ii) that the webpage does not comprise the product metadata associated with one or more product images, generating an error message indicating that no product images have been identified; and
responsive to determining (i) that the one or more images on the webpage do not correspond to one or more product images and (ii) that the webpage comprises the product metadata, generating an error message indicating that no product images corresponding to the product metadata have been identified.
5. The method of claim 1 , wherein displaying the ingested one or more product images and the product metadata associated with the one or more product images in the standardized format further comprises displaying the product metadata associated with each of the ingested one or more product images proximate to each of the ingested one or more product images.
6. The method of claim 1 , further comprising generating a product recommendation based on the selection of the first ingested product image and the first ingested product metadata.
7. The method of claim 1 , wherein the product metadata associated with the one or more product images may be based in part on the trained machine learning algorithm analysis determining a make, a model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof from processing the first ingested product image.
8. A method comprising:
receiving a request to access a webpage, wherein the webpage comprises product metadata;
analyzing the webpage, via a trained machine learning algorithm, to (i) determine that one or more images on the webpage correspond to one or more product images and (ii) determine that the webpage comprises product metadata corresponding to the one or more product images;
ingesting the one or more product images and the product metadata corresponding to the one or more product images from the webpage based on the trained machine learning algorithm analysis;
standardizing the ingested one or more product images and the product metadata associated with the one or more product images by identifying a first ingested product metadata for each of the ingested one or more product images;
displaying the ingested one or more product images and the product metadata associated with the ingested one or more product images in a standardized format independent of a format of the webpage by displaying the identified first product metadata proximate to each of the ingested one or more product images;
receiving a user input based on an interaction with a first user input mechanism associated with a first ingested product image of the ingested one or more product images and a first ingested product metadata associated with the first ingested product image, the user input indicative of a selection of the first ingested product image and the first ingested product metadata;
saving the first ingested product image and the first ingested product metadata into a product repository; and
dynamically monitoring the first ingested product metadata stored in the product repository for changes.
9. The method of claim 8 , wherein the one or more product images and the product metadata further comprise vehicle images.
10. The method of claim 9 , wherein the product metadata comprises a vehicle make, a vehicle model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof.
11. The method of claim 8 , further comprising:
responsive to determining (i) that the one or more images on the webpage do not correspond to one or more product images and (ii) that the webpage does not comprise the product metadata associated with one or more product images, generating an error message indicating that no product images have been identified; and
responsive to determining (i) that the one or more images on the webpage do not correspond to one or more product images and (ii) that the webpage comprises the product metadata, generating an error message indicating that no product images corresponding to the product metadata have been identified.
12. The method of claim 8 , further comprising an indication on the webpage for whether the first ingested product image has already been saved into the product repository.
13. The method of claim 8 , further comprising generating a product recommendation based on the selection of the first ingested product image and the first ingested product metadata.
14. The method of claim 8 , wherein the product metadata may be based in part on the trained machine learning algorithm analysis determining a make, a model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof from processing the first ingested product image.
15. A method for dynamically ingesting and monitoring online products, comprising:
receiving a request to access a webpage, wherein the webpage comprises webpage metadata;
determining whether the webpage comprises a predetermined template, the predetermined template comprising instructions for locating and ingesting one or more product images and an associated product metadata of the webpage metadata;
responsive to receiving the predetermined template:
ingesting the one or more product images and the product metadata associated with the one or more product images from the webpage; and
displaying the ingested one or more product images and the product metadata associated with the ingested one or more product images in a standardized format independent of a format of the webpage;
receiving a user input based on an interaction with a first user input mechanism associated with a first ingested product image and a first ingested product metadata associated with the first ingested product image, the user input indicative of a selection of the first ingested product image and the first ingested product metadata;
saving the first ingested product image and the first ingested product metadata into a product repository; and
dynamically monitoring the first ingested product metadata stored in the product repository for changes.
16. The method of claim 15 , wherein the one or more product images further comprise vehicle images and the associated product metadata further comprises a vehicle make, a vehicle model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof.
17. The method of claim 15 , wherein the predetermined template comprises dynamic queries that may be in the form of regular expression, DOM query, or XPath query.
18. The method of claim 15 , wherein displaying the ingested one or more product images and the product metadata associated with the one or more product images in the standardized format further comprises displaying the product metadata associated with each of the ingested one or more product images proximate to each of the ingested one or more product images.
19. The method of claim 15 , further comprising generating a product recommendation based on the selection of the first ingested product image and the first ingested product metadata.
20. The method of claim 15 , wherein the associated product metadata may be based in part on the trained machine learning algorithm analysis determining a make, a model, a year, a VIN, a product price, a mileage, a product location, a shipping fee, a webpage URL, or some combinations thereof from processing the first ingested product image.