Building pre-trained contextual embeddings for programming languages using specialized vocabulary

1. A method for building pre-trained contextual embeddings, the method comprising:

collecting programming code;

loading and preparing the programming code using a specialized programming language keywords-based vocabulary;

creating contextual embeddings for the programming code using the specialized programming language keywords-based vocabulary;

determining a context for the programming code based on the contextual embeddings, wherein the contextual embeddings are associated with one or more vectors;

using natural language processing (NLP) to perform language modeling to initialize the one or more vectors based off the words in the programming code;

extracting one or more tokens in the programming code to identify word contexts in the programming code; and

storing the contextual embeddings,

wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use pre-trained models, fine tuning models or machine learning models.

2. The method of claim 1 , wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use with programming languages in conjunction with configuration files and vocabulary files.

3. The method of claim 1 , wherein the programming code consists of source code.

4. The method of claim 1 , wherein the loading and preparing the programming code includes loading the programming code into a data loader and converting the programming code into a required form.

5. The method of claim 1 , wherein the creating contextual embeddings includes creating character tokens for the programming code using byte pair encoding (BPE).

6. The method of claim 1 , wherein the contextual embeddings are pre-trained for the programming code.

7. A computer system for building pre-trained contextual embeddings, comprising:

one or more processors, one or more computer-readable memories, one or more computer-readable tangible storage media, and program instructions stored on at least one of the one or more computer-readable tangible storage media for execution by at least one of the one or more processors via at least one of the one or more computer-readable memories, wherein the computer system is capable of performing a method comprising:

collecting programming code;

loading and preparing the programming code using a specialized programming language keywords-based vocabulary;

creating contextual embeddings for the programming code using the specialized programming language keywords-based vocabulary;

determining a context for the programming code based on the contextual embeddings, wherein the contextual embeddings are associated with one or more vectors;

using natural language processing (NLP) to perform language modeling to initialize the one or more vectors based off the words in the programming code;

extracting one or more tokens in the programming code to identify word contexts in the programming code; and

storing the contextual embeddings,

wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use pre-trained models, fine tuning models or machine learning models.

8. The computer system of claim 7 , wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use with programming languages in conjunction with configuration files and vocabulary files.

9. The computer system of claim 7 , wherein the programming code consists of source code.

10. The computer system of claim 7 , wherein the loading and preparing the programming code includes loading the programming code into a data loader and converting the programming code into a required form.

11. The computer system of claim 7 , wherein the creating contextual embeddings includes creating character tokens for the programming code using byte pair encoding (BPE).

12. The computer system of claim 7 , wherein the contextual embeddings are pre-trained for the programming code.

13. A computer program product for building pre-trained contextual embeddings, comprising:

one or more computer-readable tangible storage media and program instructions stored on at least one of the one or more computer-readable tangible storage media, the program instructions executable by a processor to cause the processor to perform a method comprising:

collecting programming code;

loading and preparing the programming code using a specialized programming language keywords-based vocabulary;

creating contextual embeddings for the programming code using the specialized programming language keywords-based vocabulary;

determining a context for the programming code based on the contextual embeddings, wherein the contextual embeddings are associated with one or more vectors;

using natural language processing (NLP) to perform language modeling to initialize the one or more vectors based off the words in the programming code;

extracting one or more tokens in the programming code to identify word contexts in the programming code; and

storing the contextual embeddings,

wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use pre-trained models, fine tuning models or machine learning models.

14. The computer program product of claim 13 , wherein the contextual embeddings are stored as pre-trained contextual embeddings that are built to use with programming languages in conjunction with configuration files and vocabulary files.

15. The computer program product of claim 13 , wherein the programming code consists of source code.

16. The computer program product of claim 13 , wherein the loading and preparing the programming code includes loading the programming code into a data loader and converting the programming code into a required form.

17. The computer program product of claim 13 , wherein the creating contextual embeddings includes creating character tokens for the programming code using byte pair encoding (BPE).