Systems and methods for generating voxel-based three-dimensional objects in a virtual space based on natural language processing (NLP) of a user-provided description

1. A system configured to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the system comprising:

electronic storage configured to electronically store information, wherein the stored information includes a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object and an unannotated object that lacks a description;

one or more hardware processors configured by machine-readable instructions to:

obtain the set of three-dimensional objects;

convert one or more three-dimensional objects in the set into one or more objects using a three-dimensional voxelized format that natively supports voxels, responsive to the one or more three-dimensional objects in the set using three-dimensional formats that fail to support voxels, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;

encode, using an autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;

sample from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector having a particular dimensionality is created based on an encoding of the first voxelized object;

decode, using the decoder, the first sampled multi-dimensional vector into a first decoded voxelized three-dimensional object;

determine loss information for the first decoded voxelized three-dimensional object by comparison to the first voxelized object;

use the determined loss information to train the autoencoder;

fine-tune a pretrained text-based system to generate multi-dimensional vectors that are subsequently decoded into decoded voxelized three-dimensional objects, having the particular dimensionality, from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;

receive, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;

generate, using the fine-tuned pretrained text-based system, a multi-dimensional vector from the user-provided text in the user input; and

decode, using the decoder, the generated multi-dimensional vector into a voxelized three-dimensional object that is based on the user-provided text.

2. The system of claim 1 , wherein the one or more hardware processors are further configured to present, to the user, the voxelized three-dimensional object that is based on the user-provided text.

3. The system of claim 1 , wherein the autoencoder is a variational autoencoder.

4. The system of claim 1 , wherein fine-tuning the pretrained text-based system includes determining secondary loss information that is used for back-propagation during the fine-tuning.

5. The system of claim 1 , wherein the set of three-dimensional objects includes an annotated subset of annotated objects including the annotated object and an unannotated subset of unannotated objects including the unannotated object, and wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

6. The system of claim 1 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

7. The system of claim 1 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

8. The system of claim 7 , wherein the decoder is used as the generator of the Generative Adversarial Network.

9. The system of claim 8 , wherein the discriminator of the Generative Adversarial Network distinguishes between objects from the set of three-dimensional objects and the decoded voxelized three-dimensional objects as decoded by the decoder.

10. The system of claim 1 , wherein the pretrained text-based system is Generative Pretrained Transformer 3 (GPT-3).

11. A method to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the method being implemented in a computer system, the method comprising:

obtaining a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object and an unannotated object that lacks a description;

converting one or more three-dimensional objects in the set into one or more objects using a three-dimensional voxelized format that natively supports voxels, responsive to the one or more three-dimensional objects in the set using three-dimensional formats that fail to support voxels, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;

encoding, using an autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;

sampling from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector having a particular dimensionality is created based on an encoding of the first voxelized object;

decoding, using the decoder, the first sampled multi-dimensional vector into a first decoded voxelized three-dimensional object;

determining loss information for the first decoded voxelized three-dimensional object by comparison to the first voxelized object;

using the determined loss information to train the autoencoder;

fine-tuning a pretrained text-based system to generate multi-dimensional vectors that are subsequently decoded into decoded voxelized three-dimensional objects, having the particular dimensionality, from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;

receiving, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;

generating, using the fine-tuned pretrained text-based system, a multi-dimensional vector from the user-provided text in the user input; and

decoding, using the decoder, the generated multi-dimensional vector into a voxelized three-dimensional object that is based on the user-provided text.

12. The method of claim 11 , further comprising:

presenting, to the user, the voxelized three-dimensional object that is based on the user-provided text.

13. The method of claim 11 , wherein the autoencoder is a variational autoencoder.

14. The method of claim 11 , wherein fine-tuning the pretrained text-based system includes determining secondary loss information that is used for back-propagation during the fine-tuning.

15. The method of claim 11 , wherein the set of three-dimensional objects includes an annotated subset of annotated objects including the annotated object and an unannotated subset of unannotated objects including the unannotated object, and wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

16. The method of claim 11 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

17. The method of claim 11 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

18. The method of claim 17 , wherein the decoder is used as the generator of the Generative Adversarial Network.

19. The method of claim 18 , wherein the discriminator of the Generative Adversarial Network distinguishes between objects from the set of three-dimensional objects and the decoded voxelized three-dimensional objects as decoded by the decoder.

20. The method of claim 1 , wherein the pretrained text-based system is Generative Pretrained Transformer 3 (GPT-3).