Automatic test generation for impact of patching software package vulnerabilities

1 . A method comprising:

determining a set of versions of a software package;

running a test having a plurality of components on each version of the set of versions of the software package;

generating a matrix comprising a plurality of vectors, each vector corresponding to a given published version of the software package and having a cell for each component of the plurality of components of the test, the cell populated with a result for that component from the test, wherein each vector of the plurality of vectors is different from each other vector of the plurality of vectors;

identifying a patch for a given version of the software package;

running the test on the patch, an output of the test comprising a vector of results from the test on the patch for each component of the test; and

responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, determining that the patch is usable to patch the given version of the software package.

2 . The method of claim 1 , further comprising determining the test by:

running each version of the set of versions of the software package through a candidate test;

determining that at least two resulting vectors from the candidate test for at least two versions of the set of versions match; and

selecting a different candidate test until each resulting vector is unique relative to each other resulting vector.

3 . The method of claim 2 , wherein the candidate test is extracted from the software package.

4 . The method of claim 2 , wherein the candidate test is generated by a large language model.

5 . The method of claim 1 , further comprising, further responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, applying the patch to the given version of the software package.

6 . The method of claim 1 , wherein the patch is generated automatically responsive to detecting a vulnerability.

7 . The method of claim 6 , wherein the patch is generated based on an existing patch that was also generated to address the vulnerability, and wherein the patch impacts less code than the existing patch when applied to the given version of the software package.

8 . A non-transitory computer-readable medium comprising memory with instructions encoded thereon that, when executed by one or more processors, cause the one or more processors to perform operations, the instructions comprising instructions to:

determine a set of versions of a software package;

run a test having a plurality of components on each version of the set of versions of the software package;

generate a matrix comprising a plurality of vectors, each vector corresponding to a given published version of the software package and having a cell for each component of the plurality of components of the test, the cell populated with a result for that component from the test, wherein each vector of the plurality of vectors is different from each other vector of the plurality of vectors;

identify a patch for a given version of the software package;

run the test on the patch, an output of the test comprising a vector of results from the test on the patch for each component of the test; and

responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, determine that the patch is usable to patch the given version of the software package.

9 . The non-transitory computer-readable medium of claim 8 , the instructions further comprising determining the test by:

running each version of the set of versions of the software package through a candidate test;

determining that at least two resulting vectors from the candidate test for at least two versions of the set of versions match; and

selecting a different candidate test until each resulting vector is unique relative to each other resulting vector.

10 . The non-transitory computer-readable medium of claim 9 , wherein the candidate test is extracted from the software package.

11 . The non-transitory computer-readable medium of claim 9 , wherein the candidate test is generated by a large language model.

12 . The non-transitory computer-readable medium of claim 8 , the instructions further comprising instructions to, further responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, apply the patch to the given version of the software package.

13 . The non-transitory computer-readable medium of claim 8 , wherein the patch is generated automatically responsive to detecting a vulnerability.

14 . The non-transitory computer-readable medium of claim 13 , wherein the patch is generated based on an existing patch that was also generated to address the vulnerability, and wherein the patch impacts less code than the existing patch when applied to the given version of the software package.

15 . A system comprising:

memory with instructions encoded thereon; and

one or more processors that, when executing the instructions, are caused to perform operations comprising:

determining a set of versions of a software package;

running a test having a plurality of components on each version of the set of versions of the software package;

generating a matrix comprising a plurality of vectors, each vector corresponding to a given published version of the software package and having a cell for each component of the plurality of components of the test, the cell populated with a result for that component from the test, wherein each vector of the plurality of vectors is different from each other vector of the plurality of vectors;

identifying a patch for a given version of the software package;

running the test on the patch, an output of the test comprising a vector of results from the test on the patch for each component of the test; and

responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, determining that the patch is usable to patch the given version of the software package.

16 . The system of claim 15 , the operations further comprising determining the test by:

running each version of the set of versions of the software package through a candidate test;

determining that at least two resulting vectors from the candidate test for at least two versions of the set of versions match; and

selecting a different candidate test until each resulting vector is unique relative to each other resulting vector.

17 . The system of claim 16 , wherein the candidate test is extracted from the software package.

18 . The system of claim 16 , wherein the candidate test is generated by a large language model.

19 . The system of claim 15 , the operations further comprising, further responsive to determining that the vector of results from the test on the patch matches a given vector corresponding to the given version of the software package, applying the patch to the given version of the software package.

20 . The system of claim 15 , wherein the patch is generated automatically responsive to detecting a vulnerability.