Mining code expressions for data analysis
Techniques for computer software code analysis are disclosed. One or more data flows are generated, based on analyzing software code using static analysis. A data object is identified in the software code using the one or more data flows, the data object relating to a structured dataset. A correspondence between a code expression in the software code and a characteristic of the structured dataset is identified, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows. The code expression for the structured dataset is analyzed, based on the correspondence, including at least one of: (i) generating a software code recommendation engine based on the code expression and the structured dataset, or (ii) generating one or more lambda expressions for application to the structured dataset, based on the code expression.
1. A computer-implemented method comprising:
generating one or more data flows based on analyzing software code using static analysis;
identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset;
determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows; and
analyzing the code expression for the structured dataset, based on the correspondence, comprising:
generating one or more lambda expressions for application to the structured dataset, based on the code expression; and
applying the one or more lambda expressions to the structured dataset, comprising:
executing lambda expression software code that takes at least a portion of the structured dataset as an input parameter and generates an output,
wherein the lambda expression software code is executed and the code expression for the structured dataset is not executed.
2. The method of claim 1 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
identifying the one or more reads from the data object, based on the one or more data flows; and
identifying one or more writes relating to the one or more reads from the data object, based on the one or more data flows.
3. The method of claim 2 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
normalizing the code expression based on the one or more data flows.
4. The method of claim 3 , wherein normalizing the code expression based on the one or more data flows is based on at least one of: (i) a call path relating to the one or more data flows or (ii) type inference relating to the one or more data flows.
5. The method of claim 1 , wherein analyzing the code expression for the structured dataset further comprises generating a software code recommendation engine based on the code expression and the structured dataset.
6. The method of claim 5 , wherein the software code recommendation engine correlates the code expression and a column name relating to the structured dataset.
7. The method of claim 5 , wherein generating the software code recommendation engine comprises grouping a plurality of computations relating to the code expression based on an operation relating to each of the plurality of computations.
8. The method of claim 1 , wherein analyzing the code expression for the structured dataset further comprises:
determining a value for the code expression based on evaluating the one or more lambda expressions using automated machine learning (AutoML).
9. The method of claim 1 , wherein the structured dataset comprises at least one of a comma-separated-value (CSV) file or a javascript object notation (JSON) data structure.
10. A system, comprising:
a processor; and
a memory having instructions stored thereon which, when executed on the processor, performs operations comprising:
generating one or more data flows based on analyzing software code using static analysis;
identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset;
determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows; and
analyzing the code expression for the structured dataset, based on the correspondence, comprising:
generating one or more lambda expressions for application to the structured dataset, based on the code expression; and
applying the one or more lambda expressions to the structured dataset, comprising:
executing lambda expression software code that takes at least a portion of the structured dataset as an input parameter and generates an output,
wherein the lambda expression software code is executed and the code expression for the structured dataset is not executed.
11. The system of claim 10 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
identifying the one or more reads from the data object, based on the one or more data flows; and
identifying one or more writes relating to the one or more reads from the data object, based on the one or more data flows.
12. The system of claim 11 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
normalizing the code expression based on the one or more data flows.
13. The system of claim 10 , wherein analyzing the code expression for the structured dataset further comprises generating a software code recommendation engine based on the code expression and the structured dataset, and
wherein the software code recommendation engine correlates the code expression and a column name relating to the structured dataset.
14. A computer program product comprising:
a computer-readable storage medium having computer-readable program code embodied therewith, the computer-readable program code executable by one or more computer processors to perform operations, comprising:
generating one or more data flows based on analyzing software code using static analysis;
identifying a data object in the software code using the one or more data flows, the data object relating to a structured dataset;
determining a correspondence between a code expression in the software code and a characteristic of the structured dataset, based on analyzing one or more reads from and one or more writes to the data object using the one or more data flows; and
analyzing the code expression for the structured dataset, based on the correspondence, comprising:
generating one or more lambda expressions for application to the structured dataset, based on the code expression; and
applying the one or more lambda expressions to the structured dataset, comprising:
executing lambda expression software code that takes at least a portion of the structured dataset as an input parameter and generates an output,
wherein the lambda expression software code is executed and the code expression for the structured dataset is not executed.
15. The computer program product of claim 14 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
identifying the one or more reads from the data object, based on the one or more data flows; and
identifying one or more writes relating to the one or more reads from the data object, based on the one or more data flows.
16. The computer program product of claim 15 , wherein determining the correspondence between the code expression in the software code and the characteristic of the structured dataset, based on analyzing the one or more reads from and one or more writes to the data object using the one or more data flows further comprises:
normalizing the code expression based on the one or more data flows.
17. The computer program product of claim 14 , wherein analyzing the code expression for the structured dataset further comprises generating a software code recommendation engine based on the code expression and the structured dataset, and
wherein the software code recommendation engine correlates the code expression and a column name relating to the structured dataset.