Scene-aware generation of music visualizations

1 . A method comprising:

receiving a projection surface image;

generating a constraint image based on the projection surface image, wherein the constraint image comprises a detected edge from the projection surface image;

outputting, using a speaker of a projector device, sound representing a music track;

determining, based on beat detection data for the music track, a first set of timestamps, the first set of timestamps including a first timestamp and a second timestamp;

generating, at a remote system using a machine learning model, first image data representing a start image for a first time corresponding to the first timestamp, wherein a shape of the first image data is constrained according to the constraint image;

generating, at the remote system using the machine learning model and the first image data, second image data representing an end image for a second time corresponding to a time immediately prior to the second timestamp; wherein a shape of the second image data is constrained according to the constraint image;

sending, from the remote system to the projector device, the first image data and the second image data;

generating, at the projector device using the machine learning model and the first image data and the second image data, third image data representing one or more interpolated images generated based on the start image and the end image; and

projecting, using a projector of the projector device and the first image data, the second image data, and the third image data, a music visualization time-aligned to the music track and shaped according to the constraint image, the music visualization comprising the start image, the end image, and the one or more interpolated images.

2 . The method of claim 1 , wherein the generating of the second image data involves an image to image generation approach using a latent diffusion model.

3 . The method of claim 1 , wherein the generating of the first image data involves a text to image generation approach using a latent diffusion model.

4 . The method of claim 1 , wherein the generating of the third image data involves an interpolation approach using a latent diffusion model.

5 . The method of claim 1 ,

wherein the generating of the second image data involves an image to image generation approach using a latent diffusion model;

wherein the generating of the first image data involves a text to image generation approach using a latent diffusion model; and

wherein the generating of the third image data involves an interpolation approach using a latent diffusion model;

whereby using beat detection data to determine timestamps for use in defining segments of a music track enables parallel processing and generation of the initial image and final image of an image sub-sequence at the remote system but generation of intermediate images using interpolation at the projector device.

6 . The method of claim 1 , wherein the projector device is mounted to a ceiling.

7 . The method of claim 1 , wherein the projecting of the music visualization comprises projecting the initial image at a time corresponding to the first time.

8 . The method of claim 1 , wherein the projecting of the music visualization comprises projecting the final image at a time corresponding to the second time.

9 . A method comprising:

outputting, using a speaker of a projector device, sound representing a music track;

determining, based on beat detection data for the music track, a first set of timestamps, the first set of timestamps including a first timestamp and a second timestamp;

generating, at a remote system using a machine learning model, first image data representing a start image for a first time segment associated with the first timestamp, wherein generating the first image data uses an edge constraint from a constraint image;

generating, at the remote system using the machine learning model and the first image data, second image data representing an end image for the first time segment, wherein generating the second image data uses the edge constraint from the constraint image;

sending, from the remote system to the projector device, the first image data and the second image data;

generating, at the projector device using the machine learning model and the first image data and the second image data, third image data representing one or more interpolated images generated based on the start image and the end image; and

projecting, using a projector of the projector device and the first image data, the second image data, and the third image data, a music visualization time-aligned to the music track, the music visualization comprising the start image, the end image, and the one or more interpolated images, wherein the music visualization is shaped according to the edge constraint from the constraint image.

10 . The method of claim 9 , wherein the generating of the second image data involves an image to image generation approach using a latent diffusion model.

11 . The method of claim 9 , wherein the generating of the first image data involves a text to image generation approach using a latent diffusion model.

12 . The method of claim 9 , wherein the generating of the third image data involves an interpolation approach using a latent diffusion model.

13 . The method of claim 9 ,

wherein the generating of the second image data involves an image to image generation approach using a latent diffusion model;

wherein the generating of the first image data involves a text to image generation approach using a latent diffusion model; and

wherein the generating of the third image data involves an interpolation approach using a latent diffusion model;

whereby using beat detection data to determine timestamps for use in defining segments of a music track enables parallel processing and generation of the initial image and final image of an image sub-sequence at the remote system but generation of intermediate images using interpolation at the projector device.

14 . The method of claim 9 , wherein the method comprises generating the beat detection data.

15 . The method of claim 9 , wherein the method comprises accessing the beat detection data at the remote system.

16 . The method of claim 9 , wherein the method comprises generating the beat detection data at the remote system.

17 . The method of claim 9 , wherein the projector device is mounted to a ceiling.

18 . A method comprising:

receiving a music sample, a beat detection analysis of the music sample, and a characteristic analysis of the music sample, wherein the beat detection analysis of the music sample comprises a beat timestamp, wherein the characteristic analysis of the music sample comprises a first characteristic timestamp, a first characteristic, a second characteristic timestamp and a second characteristic;

interpolating the first characteristic timestamp and the second characteristic timestamp using the beat timestamp to derive a beat characteristic;

generating a positive prompt, wherein the positive prompt comprises the beat characteristic;

generating, using a latent diffusion model and the positive prompt, a first keyframe and a second keyframe, wherein generating the first keyframe and the second keyframe uses an edge constraint from a constraint image; and

generating a music visualization video comprising the first keyframe, the second keyframe, and the music sample, wherein the first keyframe is timed to occur at the beat timestamp, wherein the music visualization video is shaped according to the edge constraint from the constraint image.

19 . The method of claim 18 , wherein the method comprises:

generating, by an interpolation model, an interpolation image that interpolates the first keyframe and the second keyframe, wherein the interpolation model is executed by a first device, wherein the latent diffusion model is executed using a second device, wherein the music visualization video further comprises the interpolation image.

20 . The method of claim 18 , further comprising:

generating a text-to-image prompt comprising the first characteristic, wherein the positive prompt comprises the text-to-image prompt, wherein the first keyframe corresponds to the text-to-image prompt; and

generating a image-to-image prompt comprising the first characteristic and an indication of the first keyframe, wherein the positive prompt comprises the image-to-image prompt;

wherein the second keyframe corresponds to the image-to-image prompt.