Detection and filtration of phantom packages in large language model generated code

1 . A computer implemented method executed using one or more hardware processors, for detecting and filtering phantom packages in code generated by large language models, the method comprising:

generating a first set of code using a large language model;

comparing packages used in the first set of code to certain packages contained in one or more known lists, wherein the comparing of the packages used in the first set of code to the certain packages in the one or more known lists comprises comparing the packages used in the first set of code to first packages contained in a list of malicious packages and comparing the packages used in the first set of code to second packages contained in a list of permissible packages;

identifying a list of phantom packages based on the comparing;

prompting the large language model to generate a second set of code;

analyzing the second set of code based on the list of phantom packages, wherein the analyzing of the second set of code based on the list of phantom packages comprises comparing respective packages in the second set of code to given packages contained in a list of malicious packages and to identified packages contained in the list of phantom packages; and

based on the analyzing, automatically performing an action that results in the second set of code not being executed within a computing environment.

2 . The method of claim 1 , wherein the comparing of the packages used in the first set of code to the certain packages in the one or more known lists comprises identifying respective packages used in the first set of code based on one or more of:

a regular expression search;

a string search; or

a syntax tree generated based on the first set of code.

3 . The method of claim 1 , wherein the identifying the list of phantom packages comprises identifying particular packages contained neither in a list of malicious packages nor in a list of permissible packages and adding the particular packages to the list of phantom packages.

4 . The method of claim 1 , wherein the automatically performing the action that results in the second set of code not being executed within the computing environment is based on evaluating a risk score computed based on the analyzing.

5 . The method of claim 4 , wherein the evaluating of the risk score comprises determining whether packages used in the second set of code that are contained in one or more of a list of malicious packages or the list of phantom packages exceeds a threshold number of packages.

6 . The method of claim 1 , further comprising prompting the large language model to generate new code based on the analyzing.

7 . The method of claim 6 , wherein the prompting of the large language model to generate new code based on the analyzing comprises instructing the large language model not to use one or more packages contained in the list of phantom packages when generating the new code.

8 . The method of claim 1 , wherein the analyzing of the second set of code based on the list of phantom packages comprises detecting that one or more identified packages contained in the list of phantom packages were used in more than a threshold number of sets of code generated by the large language model and performing an action based on the detecting.

9 . A system for detecting and filtering phantom packages in code generated by large language models, comprising:

one or more processors; and

a memory comprising instructions that, when executed by the one or more processors, cause the system to:

generating a first set of code using a large language model;

comparing packages used in the first set of code to certain packages contained in one or more known lists, wherein the comparing of the packages used in the first set of code to the certain packages in the one or more known lists comprises comparing the packages used in the first set of code to first packages contained in a list of malicious packages and comparing the packages used in the first set of code to second packages contained in a list of permissible packages;

identifying a list of phantom packages based on the comparing;

prompting the large language model to generate a second set of code;

analyzing the second set of code based on the list of phantom packages, wherein the analyzing of the second set of code based on the list of phantom packages comprises comparing respective packages in the second set of code to given packages contained in a list of malicious packages and to identified packages contained in the list of phantom packages; and

based on the analyzing, automatically performing an action that results in the second set of code not being executed within a computing environment.

10 . The system of claim 9 , wherein the comparing of the packages used in the first set of code to the certain packages in the one or more known lists comprises identifying respective packages used in the first set of code based on one or more of:

a regular expression search;

a string search; or

a syntax tree generated based on the first set of code.

11 . The system of claim 9 , wherein the identifying the list of phantom packages comprises identifying particular packages contained neither in a list of malicious packages nor in a list of permissible packages and adding the particular packages to the list of phantom packages.

12 . The system of claim 9 , wherein the automatically performing the action that results in the second set of code not being executed within the computing environment is based on evaluating a risk score computed based on the analyzing.

13 . The system of claim 12 , wherein the evaluating of the risk score comprises determining whether packages used in the second set of code that are contained in one or more of a list of malicious packages or the list of phantom packages exceeds a threshold number of packages.

14 . The system of claim 9 , wherein the instructions, when executed by the one or more processors, further cause the system to prompt the large language model to generate new code based on the analyzing.

15 . The system of claim 14 , wherein the prompting of the large language model to generate new code based on the analyzing comprises instructing the large language model not to use one or more packages contained in the list of phantom packages when generating the new code.

16 . A non-transitory computer readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to:

generating a first set of code using a large language model;

comparing packages used in the first set of code to certain packages contained in one or more known lists, wherein the comparing of the packages used in the first set of code to the certain packages in the one or more known lists comprises comparing the packages used in the first set of code to first packages contained in a list of malicious packages and comparing the packages used in the first set of code to second packages contained in a list of permissible packages;

identifying a list of phantom packages based on the comparing;

prompting the large language model to generate a second set of code;

analyzing the second set of code based on the list of phantom packages, wherein the analyzing of the second set of code based on the list of phantom packages comprises comparing respective packages in the second set of code to given packages contained in a list of malicious packages and to identified packages contained in the list of phantom packages; and

based on the analyzing, automatically performing an action that results in the second set of code not being executed within a computing environment.