Systems and methods for evaluating a source code scanner

1. A method for evaluating a source code scanner, the method comprising:

determining, by a code evaluation computing device, control flow and data flow in source code;

inserting, by the code evaluation computing device, one or more exemplary code snippets or one or more irregular code snippets into one or more areas in the source code comprising one or more critical functional areas in the source code to produce modified source code, wherein the one or more critical functional areas in the source code are based on the determined control flow and data flow;

sending, by the code evaluation computing device, the modified source code to a source code scanner for analysis;

obtaining, by the code evaluation computing device, an issue list, generated by the source code scanner after the source code scanner has analyzed the modified source code, wherein the issue list comprises code segments comprising security defects identified by the source code scanner, reasons for the security defects, and locations of the security defects in the modified source code;

comparing, by the code evaluation computing device, the code segments in the issue list to the one or more exemplary code snippets or the one or more irregular code snippets; and

generating, by the code evaluation computing device, a plurality of metrics, indicating quality of the source code scanner, based on the comparison.

2. The method of claim 1 , wherein the one or more exemplary code snippets do not comprise security defects or the one or more irregular code snippets comprise one or more security defects.

3. The method of claim 1 , further comprising:

inserting, by the code evaluation computing device, at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the source code based on the determined control flow and data flow of the source code.

4. The method of claim 1 , wherein the further comprising:

inserting, by the code evaluation computing device, at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into random locations in the source code.

5. The method of claim 1 , wherein the plurality of metrics for the source code scanner comprises a false positive rate or a false negative rate.

6. The method of claim 1 , wherein the inserting further comprises:

determining, by the code evaluation computing device, when one or more areas in the source code comprise an evaluation of run-time variables; and

inserting, by the code evaluation computing device, at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the areas in the source code when the source code is determined to comprise an evaluation of run-time variables.

7. An evaluation computing apparatus comprising:

a processor; and

a memory coupled to the processor which is configured to be capable of programmed instructions, which comprise the programmed instructions stored in the memory to:

determine control flow and data flow in source code;

insert one or more exemplary code snippets or one or more irregular code snippets into one or more areas in the source code comprising one or more critical functional areas in the source code to produce modified source code, wherein the one or more critical functional areas in the source code are based on the determined control flow and data flow;

send the modified source code to a source code scanner for analysis;

obtain an issue list, generated by the source code scanner after the source code scanner has analyzed the modified source code, wherein the issue list comprises code segments comprising security defects identified by the source code scanner, reasons for the security defects, and locations of the security defects in the modified source code;

compare the code segments in the issue list to the one or more exemplary code snippets or the one or more irregular code snippets; and

generate a plurality of metrics, indicating quality of the source code scanner, based on the comparison.

8. The apparatus according to claim 7 , wherein the one or more exemplary code snippets do not comprise security defects or the one or more irregular code snippets comprise one or more security defects.

9. The apparatus according to claim 7 , wherein the processor is further configured to be capable of executing programmed instructions, which comprise the programmed instructions stored in the memory to:

insert at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the source code based on the control flow and data flow of the source code.

10. The apparatus according to claim 7 , wherein the configured to be capable of executing programmed instructions for the inserting, which comprise the programmed instructions stored in the memory to:

insert at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into random locations in the source code.

11. The apparatus according to claim 7 , wherein the plurality of metrics for the source code scanner comprises a false positive rate or a false negative rate.

12. The device of claim 7 , wherein the processor is further configured to be capable of executing programmed instructions for the inserting, which comprise the programmed instructions stored in the memory to:

determine when one or more areas in the source code comprise an evaluation of run-time variables; and

insert at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the areas in the source code when the source code is determined to comprise an evaluation of run-time variables.

13. A non-transitory computer-readable medium having stored thereon instructions for evaluating a source code scanner comprising machine executable instructions which when executed by at least one processor, causes the processor to perform steps comprising:

determining data flow and control flow in source code;

inserting one or more exemplary code snippets or one or more irregular code snippets into one or more areas in the source code comprising one or more critical functional areas in the source code to produce a modified source code, wherein the one or more critical functional areas in the source code are based on the determined control flow and data flow;

sending the modified source code to a source code scanner for analysis;

obtaining an issue list, generated by the source code scanner after the source code scanner has analyzed the modified source code, wherein the issue list comprises code segments comprising security defects identified by the source code scanner, reasons for the security defects, and locations of the security defects in the modified source code;

comparing the code segments in the issue list to the one or more exemplary code snippets or the one or more irregular code snippets; and

generating a plurality of metrics, indicating quality of the source code scanner, based on the comparison.

14. The non-transitory computer-readable medium according to claim 13 , wherein the one or more exemplary code snippets do not comprise security defects or the one or more irregular code snippets comprise one or more security defects.

15. The non-transitory computer-readable medium according to claim 13 , further having stored thereon instructions that when executed by the processor cause the processor to perform steps further comprising:

inserting at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the source code based on the control flow and data flow of the source code.

16. The non-transitory computer-readable medium according to claim 13 , further having stored thereon instructions that when executed by the processor cause the processor to perform steps further comprising:

inserting at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into random locations in the source code.

17. The non-transitory computer-readable medium according to claim 13 , wherein the plurality of metrics for the source code scanner comprises a false positive rate or a false negative rate.

18. The non-transitory computer-readable medium according to claim 13 , further having stored thereon instructions for the inserting that when executed by the processor cause the processor to perform steps further comprising:

determining when one or more areas in the source code comprise an evaluation of run-time variables; and

inserting at least one of the one or more exemplary code snippets or at least one of the one or more irregular code snippets into the areas in the source code when the source code is determined to comprise an evaluation of run-time variables.