Virtual shoe try-on

1. A method for fitting a virtual shoe onto a foot, the method comprising:

receiving, from a database, a three-dimensional model of a shoe, a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe;

predicting, using a machine learning model and based on an image of a foot, a first point on the foot, a second point on the foot, and a third point on the foot;

determine whether the first point, the second point, or the third point are available;

in an instance in which at least one of the first point, the second point, or the third point are not available;

predicting a transformation matrix based on the first annotated point, the second annotated point, the third annotated point, and a plurality of annotated points for a standard shoe model, and

applying the transformation matrix to the three-dimensional model of the shoe to produce a transformed model of the shoe, wherein a two-dimensional model of the shoe is generated based on the transformed model of the shoe;

in an instance in which the first point, the second point, and the third point are available:

pairing the first annotated point with the first point to produce a first annotated point-first point pair, the second annotated point with the second point to produce a second annotated point-second point pair, and the third annotated point with the third point to produce a third annotated point-third point pair;

determining a scale for the three-dimensional model of the shoe and a translation for the three-dimensional model of the shoe based on the first annotated point-first point pair, the second annotated point-second point pair, and the third annotated point-third point pair;

applying the scale and the translation to the three-dimensional model of the shoe; and

generating, a two-dimensional model of the shoe using the three-dimensional model of the shoe after applying the scale and translation to the three-dimensional model of the shoe; and

superimposing the two-dimensional model of the shoe onto the image of the foot.

2. The method of claim 1 , wherein the first annotated point is at a first location on an insole of the three-dimensional model of the shoe corresponding to a heel, the second annotated point is at a second location of the insole corresponding to a protrusion of a first metatarsophalangeal joint, and the third annotated point is at a third location of the insole corresponding to a protrusion of a second metatarsophalangeal joint.

3. The method of claim 1 , further comprising:

predicting, using the machine learning model and based on the image of the foot, a predicted orientation of the foot; and

rotating the three-dimensional model of the shoe based on the predicted orientation.

4. The method of claim 1 , wherein first point corresponds to a heel of the foot, the second point corresponds to a protrusion of a first metatarsophalangeal joint of the foot, and the third point corresponds to a protrusion of a second metatarsophalangeal joint of the foot.

5. A method comprising:

receiving, from a database, a three-dimensional model of a shoe with a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe;

predicting, using a machine learning model and based on an image of a foot, one or more features of the foot, wherein the one or more features comprise a first point on the foot, a second point on the foot, and a third point on the foot;

determine whether at least one of the first point, the second point, or the third point are available;

in an instance in which at least one of the first point, the second point, or the third point are not available:

predicting a transformation matrix based on the first annotated point, the second annotated point, the third annotated point, and a plurality of annotated points for a standard shoe model, and

applying the transformation matrix to the three-dimensional model of the shoe to produce a transformed model of the shoe, wherein a two-dimensional model of the shoe is generated based on the transformed model of the shoe;

in an instance in which the first point, the second point, and the third point are available:

generating, based on the first annotated point, the second annotated point, the third annotated point, and the one or more features, a two-dimensional model of the shoe using the three-dimensional model of the shoe, wherein the generating the two-dimensional model of the shoe comprises scaling and translating the three-dimensional model of the shoe based on the first annotated point, the second annotated point, the third annotated point, the first point, the second point, and the third point; and

annotated point, the third annotated point, and a plurality of annotated points for a standard shoe;

superimposing the two-dimensional model of the shoe onto the image of the foot.

6. The method of claim 5 , wherein scaling and translating the three-dimensional model of the shoe comprises pairing the first annotated point with the first point to produce a first annotated point-first point pair, the second annotated point with the second point to produce a second annotated point-second point pair, and the third annotated point with the third point to produce a third annotated point-third point pair, wherein scaling the translating the three-dimensional model of the shoe is based on the first annotated point-first point pair, the second annotated point-second point pair, and the third annotated point-third point pair.

7. The method of claim 5 , further comprising:

predicting, using the machine learning model and based on the image of the foot, a predicted orientation of the foot; and

rotating the three-dimensional model of the shoe based on the predicted orientation.

8. The method of claim 5 , wherein first point corresponds to a heel of the foot, the second point corresponds to a protrusion of a first metatarsophalangeal joint of the foot, and the third point corresponds to a protrusion of a second metatarsophalangeal joint of the foot.

9. The method of claim 5 , wherein scaling and translating the three-dimensional model of the shoe comprises adjusting a size and a position of the three-dimensional model of the shoe to reduce a first distance between the first annotated point and the first point, a second distance between the second annotated point and the second point, and a third distance between the third annotated point and the third point.

10. The method of claim 5 , wherein the first annotated point is at a first location on an insole of the three-dimensional model of the shoe corresponding to a heel, the second annotated point is at a second location of the insole corresponding to a protrusion of a first metatarsophalangeal joint, and the third annotated point is at a third location of the insole corresponding to a protrusion of a second metatarsophalangeal joint.

11. A system comprising:

a memory; and

a processor communicatively coupled to the memory, the processor configured to:

receive, from a database, a three-dimensional model of a shoe with a first annotated point on the three-dimensional model of the shoe, a second annotated point on the three-dimensional model of the shoe, and a third annotated point on the three-dimensional model of the shoe;

predict, using a machine learning model and based on an image of a foot, one or more features of the foot, wherein the one or more features comprise a first point on the foot, a second point on the foot, and a third point on the foot;

determine whether at least one of the first point, the second point, or the third point are available;

in an instance in which at least one of the first point, the second point, or the third point are not available:

predict a transformation matrix based on the first annotated point, the second annotated point, the third annotated point, and a plurality of annotated points for a standard shoe model, and

apply the transformation matrix to the three-dimensional model of the shoe to produce a transformed model of the shoe, wherein a two-dimensional model of the shoe is generated based on the transformed model of the shoe;

in an instance in which the first point, the second point, and the third point are available:

generate, based on the first annotated point, the second annotated point, the third annotated point, and the one or more features, a two-dimensional model of the shoe using the three-dimensional model of the shoe, wherein the generating the two-dimensional model of the shoe comprises scaling and translating the three-dimensional model of the shoe based on the first annotated point, the second annotated point, the third annotated point, the first point, the second point, and the third point; and

superimpose the two-dimensional model of the shoe onto the image of the foot.

12. The system of claim 11 , wherein scaling and translating the three-dimensional model of the shoe comprises pairing the first annotated point with the first point to produce a first annotated point-first point pair, the second annotated point with the second point to produce a second annotated point-second point pair, and the third annotated point with the third point to produce a third annotated point-third point pair, wherein scaling the translating the three-dimensional model of the shoe is based on the first annotated point-first point pair, the second annotated point-second point pair, and the third annotated point-third point pair.

13. The system of claim 11 , wherein the processor is further configured to:

predict, using the machine learning model and based on the image of the foot, a predicted orientation of the foot; and

rotate the three-dimensional model of the shoe based on the predicted orientation.

14. The system of claim 11 , wherein first point corresponds to a heel of the foot, the second point corresponds to a protrusion of a first metatarsophalangeal joint of the foot, and the third point corresponds to a protrusion of a second metatarsophalangeal joint of the foot.

15. The system of claim 11 , wherein scaling and translating the three-dimensional model of the shoe comprises adjusting a size and a position of the three-dimensional model of the shoe to reduce a first distance between the first annotated point and the first point, a second distance between the second annotated point and the second point, and a third distance between the third annotated point and the third point.

16. The system of claim 11 , wherein the first annotated point is at a first location on an insole of the three-dimensional model of the shoe corresponding to a heel, the second annotated point is at a second location of the insole corresponding to a protrusion of a first metatarsophalangeal joint, and the third annotated point is at a third location of the insole corresponding to a protrusion of a second metatarsophalangeal joint.