Automated data extractor code generation and implementation
A method includes obtaining, by a processing device, web traffic including a set of messages associated with a target platform, analyzing, by the processing device, the web traffic to obtain a set of data extractor code candidates for extracting a set of data from the target platform, and selecting, by the processing device, data extractor code from the set of data extractor code candidates.
1. A method comprising:
obtaining, by a processing device, a web traffic representation identifying at least one relationship between at least one pair of messages sent between a target platform and a client platform;
generating, by the processing device based on the web traffic representation, a set of data extractor code candidates, each data extractor code candidate of the set of data extractor code candidates comprising source code for extracting a set of data from the target platform;
assigning, by the processing device to each data extractor code candidate of the set of data extractor code candidates, a respective value derived from a set of metrics;
ranking, by the processing device, each data extractor code candidate of the set of data extractor code candidates in accordance with the respective value assigned to the data extractor code candidate;
selecting, by the processing device based on the ranking, data extractor code as a highest ranked data extractor code candidate from the set of data extractor code candidates; and
causing, by the processing device, the set of data to be extracted from the target platform using the data extractor code.
2. The method of claim 1 , further comprising:
obtaining, by the processing device, web traffic from one or more sources related to navigation to the target platform, the web traffic comprising a set of messages sent between the target platform and the client platform, the set of messages comprising the at least one pair of messages; and
generating, by the processing device, the web traffic representation based on the web traffic;
wherein the set of messages comprises at least one of: a set of requests from the client platform to the target platform, or a set of responses from the target platform to the client platform.
3. The method of claim 2 , wherein obtaining the web traffic comprises capturing the web traffic utilizing a web traffic recorder to generate a web traffic recording.
4. The method of claim 1 , wherein the web traffic representation comprises an abstract syntax tree.
5. The method of claim 2 , wherein generating the web traffic representation comprises:
generating a graph of the web traffic comprising a set of vertices and a set of edges, each vertex of the set of vertices representing a respective pair of messages and each edge of the set of edges representing a respective relationship between vertices; and
transforming the graph into the web traffic representation.
6. The method of claim 1 , wherein the set of metrics comprises at least one of: a data completeness metric associated with a number of fields that can be extracted by each data extractor code candidate, a complexity metric associated with a number of declarations for each data extractor code candidate, or a simplicity metric associated with the number of declarations for each data extractor code candidate.
7. The method of claim 1 , further comprising:
causing, by the processing device, the set of data to be shared with an application.
8. A system comprising:
a memory; and
a processing device, operatively coupled to the memory, to perform operations comprising:
obtaining a web traffic representation identifying at least one relationship between at least one pair of messages sent between a target platform and a client platform;
generating, based on the web traffic representation, a set of data extractor code candidates, each data extractor code candidate of the set of data extractor code candidates comprising source code for extracting a set of data from the target platform;
assigning, to each data extractor code candidate of the set of data extractor code candidates, a respective value derived from a set of metrics;
ranking each data extractor code candidate of the set of data extractor code candidates in accordance with the respective value assigned to the data extractor code candidate;
selecting, based on the ranking, data extractor code as a highest ranked data extractor code candidate from the set of data extractor code candidates; and
causing the set of data to be extracted from the target platform using the data extractor code.
9. The system of claim 8 , wherein:
the operations further comprise:
obtaining web traffic from one or more sources related to navigation to the target platform, the web traffic comprising a set of messages sent between the target platform and the client platform, the set of messages comprising the at least one pair of messages; and
generating the web traffic representation based on the web traffic; and
the set of messages comprises at least one of: a set of requests from the client platform to the target platform, or a set of responses from the target platform to the client platform.
10. The system of claim 9 , wherein obtaining the web traffic comprises capturing the web traffic utilizing a web traffic recorder to generate a web traffic recording.
11. The system of claim 9 , wherein generating the web traffic representation further comprises:
generating a graph of the web traffic comprising a set of vertices and a set of edges, each vertex of the set of vertices representing a respective pair of messages and each edge of the set of edges representing a respective relationship between vertices; and
transforming the graph into the web traffic representation, wherein the web traffic representation comprises an abstract syntax tree.
12. The system of claim 8 , wherein the set of metrics comprises at least one of: a data completeness metric associated with a number of fields that can be extracted by each data extractor code candidate, a complexity metric associated with a number of declarations for each data extractor code candidate, or a simplicity metric associated with the number of declarations for each data extractor code candidate.
13. The system of claim 8 , wherein the operations further comprise:
causing the set of data to be shared with an application.
14. A non-transitory computer-readable storage medium comprising instructions that, when executed by a processing device, cause the processing device to perform operations comprising:
obtaining a web traffic representation identifying at least one relationship between at least one pair of messages sent between a target platform and a client platform;
generating, based on the web traffic representation, a set of data extractor code candidates, each data extractor code candidate of the set of data extractor code candidates comprising source code for extracting a set of data from the target platform;
assigning, to each data extractor code candidate of the set of data extractor code candidates, a respective value derived from a set of metrics;
ranking each data extractor code candidate of the set of data extractor code candidates in accordance with the respective value assigned to the data extractor code candidate;
selecting, based on the ranking, data extractor code as a highest ranked data extractor code candidate from the set of data extractor code candidates; and
causing the set of data to be extracted from the target platform using the data extractor code.
15. The non-transitory computer-readable storage medium of claim 14 , wherein:
the operations further comprise:
obtaining the web traffic from one or more sources related to navigation to the target platform, the web traffic comprising a set of messages sent between the target platform and the client platform, the set of messages comprising the at least one pair of messages; and
generating the web traffic representation based on the web traffic; and
the set of messages comprises at least one of: a set of requests from the client platform to the target platform, or a set of responses from the target platform to the client platform.
16. The non-transitory computer-readable storage medium of claim 15 , wherein obtaining the web traffic comprises capturing the web traffic utilizing a web traffic recorder to generate a web traffic recording.
17. The non-transitory computer-readable storage medium of claim 15 , wherein generating the web traffic representation further comprises:
generating a graph of the web traffic comprising a set of vertices and a set of edges, each vertex of the set of vertices representing a respective pair of messages and each edge of the set of edges representing a respective relationship between vertices; and
transforming the graph into the web traffic representation, wherein the web traffic representation comprises an abstract syntax tree.
18. The non-transitory computer-readable storage medium of claim 14 , wherein the operations further comprise:
causing the set of data to be shared with an application.