Granted Patent B2

US 12,620,216 · App. 18/400,677 · Granted May 5, 2026

Inventors: Pavlo Chemerys (Amsterdam, NL); Colin Eles (Marina del Rey, CA); Ju Hu (Los Angeles, CA); Qing Jin (Palo Alto, CA); Yanyu Li (Malden, MA); Ergeta Muca (Long Island City, NY); Jian Ren (Marina Del Ray, CA); Dhritiman Sagar (Marina del Rey, CA); Aleksei Stoliar (Marina del Rey, CA); Sergey Tulyakov (Santa Monica, CA); Huan Wang (Somerville, MA)

Assignee: SNAP INC.

G06V10/82G06N3/0455G06N20/00G06T5/60G06T5/70G06T11/00G10L15/1815G10L15/22G06T2200/24G06T2207/20081G06T2207/20084

Abstract

Claims (57)

1 . A system comprising:

at least one processor; and

at least one memory component storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:

identifying an autoencoder for a latent diffusion machine learning model, the latent diffusion machine learning model trained to receive text as input and output an image based on the received text;

identifying a number of channels in a decoder of the autoencoder, the decoder configured to receive latent features as input and to output images;

identifying a first performance characteristic of the decoder;

changing a node topology of the decoder based on the first performance characteristic to generate an updated decoder; and

retraining the latent diffusion machine learning model using the updated decoder by performing operations comprising:

inputting latent features into the updated decoder;

receiving an outputted image from the updated decoder; and

updating one or more weights of the updated decoder based on assessment of the outputted image.

2 . The system of claim 1 , wherein the first performance characteristic includes a magnitude of the output for a particular channel.

3 . The system of claim 2 , wherein changing the node topology includes:

determining that the magnitude for the particular channel is above a minimum threshold; and

changing the node topology for the particular channel based on the determination that the magnitude is above the minimum threshold.

4 . The system of claim 1 , wherein the first performance characteristic includes a reconstruction loss that measures a loss of reconstruction by the autoencoder converting text to image data.

5 . The system of claim 1 , wherein changing the node topology includes removing one or more channels of the decoder.

6 . The system of claim 1 , the operations further comprising:

identifying the first performance characteristic for each channel in the decoder, and wherein changing the node topology of the decoder comprises removing half of the channels with lowest values for the first performance characteristics.

7 . The system of claim 1 , wherein changing the node topology includes reducing channel dimensions in at least one layer of the decoder to obtain a compressed decoder, wherein the compressed decoder processes latent features with lower latency and fewer parameters than to decoder.

8 . The system of claim 1 , wherein changing the node topology includes removing or adding one or more channels of the decoder until two or more performance thresholds are met.

9 . The system of claim 8 , the operations further comprising:

removing one or more channels in response to a first performance threshold being met, and adding one or more channels in response to a second performance threshold being met.

10 . The system of claim 8 , the operations further comprising:

repeatedly adding and removing channels of the decoder until two performance thresholds are met.

11 . The system of claim 8 , wherein adding the one or more channels of the decoder includes identifying one or more existing channels of the decoder that meet a performance threshold for a second performance characteristic, and copying the one or more existing channels of the decoder to add as new channels to the decoder.

12 . The system of claim 11 , wherein the first performance characteristic and second performance characteristic are of a same type.

13 . The system of claim 11 , wherein the first performance characteristic and second performance characteristic are of different types.

14 . The system of claim 1 , wherein the assessment of the outputted image comprises comparing the outputted image of the updated decoder with an outputted image of the decoder, wherein the outputted image of the updated decoder and the outputted image of the decoder are generated using the same input data.

15 . The system of claim 14 , the operations further comprising:

generating latent features by inputting random noise data into the latent diffusion machine learning model, wherein data that is input into the updated decoder and the decoder includes the latent features, the same input data including the random noise data.

16 . The system of claim 14 , wherein comparing the outputted image of the updated decoder with an outputted image of the decoder includes determining a mean squared error between the outputted image of the updated decoder and the outputted image of the decoder, and the operations further comprise:

updating one or more weights of the updated decoder based on the mean squared error; and

repeatedly inputting random noise data into the updated decoder with the updated weights, comparing the outputted image of the updated decoder with an outputted image of the decoder, and updating the weights of the updated decoder until the mean squared error between the outputted images meets a mean squared error threshold.

17 . The system of claim 1 , wherein the outputted image is a frame of a video, wherein the operations performed by the at least one processor are repeated to generate other frames for the video.

18 . The system of claim 1 , the operations further comprising:

generating a prompt based on user interaction data;

inputting the prompt into a latent feature generator of the latent diffusion machine learning model causing the latent feature generator to output latent features that are inputted into the updated decoder with the updated weights; and

receiving an image generated by the updated decoder based on the latent features inputted into the updated decoder with the updated weights.

19 . A method comprising:

identifying an autoencoder for a latent diffusion machine learning model, the latent diffusion machine learning model trained to receive text as input and output an image based on the received text;

identifying a number of channels in a decoder of the autoencoder, the decoder configured to receive latent features as input and to output images;

identifying a first performance characteristic of the decoder;

changing a node topology of the decoder based on the first performance characteristic to generate an updated decoder; and

retraining the latent diffusion machine learning model using the updated decoder by performing operations comprising:

inputting latent features into the updated decoder;

receiving an outputted image from the updated decoder; and

updating one or more weights of the updated decoder based on assessment of the outputted image.

20 . A non-transitory computer-readable storage medium, the computer-readable storage medium including instructions that when executed by a computer, cause the computer to:

identifying an autoencoder for a latent diffusion machine learning model, the latent diffusion machine learning model trained to receive text as input and output an image based on the received text;

identifying a number of channels in a decoder of the autoencoder, the decoder configured to receive latent features as input and to output images;

identifying a first performance characteristic of the decoder;

changing a node topology of the decoder based on the first performance characteristic to generate an updated decoder; and

retraining the latent diffusion machine learning model using the updated decoder by performing operations comprising:

inputting latent features into the updated decoder;

receiving an outputted image from the updated decoder; and

updating one or more weights of the updated decoder based on assessment of the outputted image.

Assignments (1)

Continuity (2)

References Cited (385)