Systems and methods for generating voxel-based three-dimensional objects in a virtual space based on natural language processing (NLP) of a user-provided description

1. A system configured to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the system comprising:

electronic storage configured to electronically store information, wherein the stored information includes a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object, wherein the annotated object uses a first three-dimensional format; and

one or more hardware processors configured by machine-readable instructions to:

obtain the set of three-dimensional objects, wherein the set of three-dimensional objects includes voxelized three-dimensional objects, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;

encode, using a variational autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;

create sampled multi-dimensional vectors having a particular dimensionality by sampling from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector is created based on an encoding of the first voxelized object;

decode, using the decoder, the sampled multi-dimensional vectors into decoded voxelized three-dimensional objects;

determine loss information for the decoded voxelized three-dimensional objects by comparing the decoded voxelized three-dimensional objects to corresponding individual voxelized three-dimensional objects;

use the loss information to train the variational autoencoder;

fine-tune a pretrained text-based system to generate multi-dimensional vectors having the particular dimensionality from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;

receive, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;

generate, using the fine-tuned pretrained text-based system, a user-provided multi-dimensional vector from the user-provided text in the user input;

decode, using the decoder, the user-provided multi-dimensional vector into a first voxelized three-dimensional object; and

present, to the user, the first voxelized three-dimensional object.

2. The system of claim 1 , wherein the set of three-dimensional objects includes one or more objects in one or more three-dimensional formats that fail to support voxels, wherein the one or more three-dimensional formats that fail to support voxels include one or both of a vector-based three-dimensional format and a polygon-based three-dimensional format.

3. The system of claim 1 , wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

4. The system of claim 1 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

5. The system of claim 1 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

6. The system of claim 5 , wherein the decoder is used as the generator of the Generative Adversarial Network.

7. The system of claim 6 , wherein the discriminator of the Generative Adversarial Network distinguishes between objects from the set of three-dimensional objects and the decoded voxelized three-dimensional objects as decoded by the decoder.

8. The system of claim 1 , wherein the pretrained text-based system is Generative Pretrained Transformer 3 (GPT-3).

9. The system of claim 1 , wherein the user input is received through a user interface, and wherein the first voxelized three-dimensional object is presented through the user interface.

10. A method to use natural language processing (NLP) to automatically generate three-dimensional objects in a virtual space, the method being implemented in a computer system, the method comprising:

obtaining a set of three-dimensional objects, wherein the set includes an annotated object that includes a textual description of the annotated object, wherein the annotated object uses a first three-dimensional format, wherein the set of three-dimensional objects includes voxelized three-dimensional objects, wherein the voxelized three-dimensional objects include a first voxelized object based on the annotated object;

encoding, using a variational autoencoder that includes an encoder and a decoder, the voxelized three-dimensional objects into pairs of multi-dimensional vectors, wherein individual pairs include a vector of means and a vector of standard deviations;

creating sampled multi-dimensional vectors having a particular dimensionality by sampling from the individual pairs of multi-dimensional vectors such that a first sampled multi-dimensional vector is created based on an encoding of the first voxelized object;

decoding, using the decoder, the sampled multi-dimensional vectors into decoded voxelized three-dimensional objects;

determining loss information by comparing the decoded voxelized three-dimensional objects to corresponding individual voxelized three-dimensional objects;

using the loss information to train the variational autoencoder, including the decoder;

fine-tuning a pretrained text-based system to generate multi-dimensional vectors having the particular dimensionality from textual descriptions included in annotated objects in the set of three-dimensional objects, wherein one or more prompts for the pretrained text-based system are based on a combination of the textual description of the annotated object and the first sampled multi-dimensional vector;

receiving, from a user, a user input, wherein the user input includes user-provided text describing a three-dimensional object;

generating, using the fine-tuned pretrained text-based system, a user-provided multi-dimensional vector from the user-provided text in the user input;

decoding, using the decoder, the user-provided multi-dimensional vector into a first voxelized three-dimensional object; and

presenting, to the user, the first voxelized three-dimensional object.

11. The method of claim 10 , wherein the set of three-dimensional objects includes one or more objects in one or more three-dimensional formats that fail to support voxels, wherein the one or more three-dimensional formats that fail to support voxels include one or both of a vector-based three-dimensional format and a polygon-based three-dimensional format.

12. The method of claim 10 , wherein the three-dimensional voxelized format that natively supports voxels is selected from one of .OBJ, .JSON, .XML, .SCHEMATIC, .SCAD, .STL, .QB, and .VOX.

13. The method of claim 10 , wherein the loss information includes reconstruction loss and Kullback-Leibler loss, wherein the Kullback-Leibler loss is based on a sum of Kullback-Leibler divergences.

14. The method of claim 10 , wherein the sampled multi-dimensional vectors are decoded using a Generative Adversarial Network that includes a generator and a discriminator.

15. The method of claim 14 , wherein the decoder is used as the generator of the Generative Adversarial Network.

16. The method of claim 15 , wherein the discriminator of the Generative Adversarial Network distinguishes between objects from the set of three-dimensional objects and the decoded voxelized three-dimensional objects as decoded by the decoder.

17. The method of claim 10 , wherein the pretrained text-based system is Generative Pretrained Transformer 3 (GPT-3).

18. The method of claim 10 , wherein the user input is received through a user interface, and wherein the user-provided voxelized three-dimensional object is presented through the user interface.