Locating a center point of a lug nut for an automated vehicle wheel removal system
Described is a system (and method) for locating a center point of a lug nut for an automated vehicle wheel removal system. To improve the accuracy of the center point, the system may perform machine learning inferences using two-dimensional (2D) and three-dimensional (3D) image data. The system may process a 2D image to infer an initial center point, and potentially improve the accuracy by leveraging a 3D image. More particularly, the system may process a 3D image to infer a location of one or more edges (or edge points) around the perimeter of the lug nut and measure a set of distances between the initial center point and the located set of edges. The system may then refine (or adjust) the center point based on such measurements.
1. A system for locating a center of a lug nut comprising:
one or more processors; and
a non-transitory computer-readable medium storing a plurality of instructions, which when executed, cause the one or more processors to:
obtain a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts;
determine an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model;
obtain a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut;
locate a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model;
determine a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges;
update the initial center point of the first lug nut based on the determined set of distances; and
map a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point.
2. The system of claim 1 , wherein updating the initial center point of the first lug nut based on the determined set of distances includes:
determining an average distance of the determined set of distances; and
updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance.
3. The system of claim 1 , wherein locating the set of edges includes:
determining a confidence score for each edge of the set of edges; and
retaining only those edges satisfying a predetermined confidence score as part of the located set of edges.
4. The system of claim 1 , wherein locating the set of edges includes:
determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and
locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth.
5. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
direct a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel.
6. The system of claim 1 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine one or more dimensions of the first lug nut based on the updated center point and the determined set of distances.
7. The system of claim 6 , wherein the plurality of instructions, when executed, further cause the one or more processors to:
determine a lug nut size based on the determined one or more dimensions of the first lug nut; and
direct a robotic apparatus to select a socket corresponding the determined lug nut size as part of a sequence of operations to remove the first lug nut from the vehicle wheel.
8. A computer-implemented method for locating a center of a lug nut comprising:
obtaining a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts;
determining an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model;
obtaining a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut;
locating a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model;
determining a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges;
updating the initial center point of the first lug nut based on the determined set of distances; and
mapping a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point.
9. The method of claim 8 , wherein updating the initial center point of the first lug nut based on the determined set of distances includes:
determining an average distance of the determined set of distances; and
updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance.
10. The method of claim 8 , wherein locating the set of edges includes:
determining a confidence score for each edge of the set of edges; and
retaining only those edges satisfying a predetermined confidence score as part of the located set of edges.
11. The method of claim 8 , wherein locating the set of edges includes:
determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and
locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth.
12. The method of claim 8 , further comprising:
directing a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel.
13. A non-transitory computer-readable medium storing instructions which, when executed by one or more processors of a system, cause the system to perform operations comprising:
obtaining a two-dimensional image of a vehicle wheel, the vehicle wheel including a set of lug nuts;
determining an initial center point of at least a first lug nut by processing the two-dimensional image using a first machine learning model;
obtaining a three-dimensional image of at least a portion of the vehicle wheel including the first lug nut;
locating a set of edges at various points around a perimeter of the first lug nut by processing the three-dimensional image using a second machine learning model;
determining a set of distances between a point within the three-dimensional image that corresponds to the initial center point and the located set of edges; and
updating the initial center point of the first lug nut based on the determined set of distances and
mapping a set of two-dimensional coordinates representing the initial center point to a set of three-dimensional coordinates representing the point within the three-dimensional image that corresponds to the initial center point.
14. The non-transitory computer-readable medium of claim 13 , wherein updating the initial center point of the first lug nut based on the determined set of distances includes:
determining an average distance of the determined set of distances; and
updating the center point such that a distance from the updated center point to each of the set of edges corresponds to the determined average distance.
15. The non-transitory computer-readable medium of claim 13 , wherein locating the set of edges includes:
determining a confidence score for each edge of the set of edges; and
retaining only those edges satisfying a predetermined confidence score as part of the located set of edges.
16. The non-transitory computer-readable medium of claim 13 , wherein locating the set of edges includes:
determining a change of depth between a plurality of points proximate to the various points around the perimeter of the first lug nut; and
locating the set of edges at the various points around the perimeter of the first lug nut based on the determined change of depth.
17. The non-transitory computer-readable medium of claim 13 , storing further instructions which, when executed by the one or more processors of the system, cause the system to perform further operations comprising:
directing a robotic apparatus to maneuver to a position based on the updated center point of the first lug nut as part of a sequence of operations to remove the first lug nut from the vehicle wheel.