Distributed data processing for machine learning
Embodiments of the present invention are directed to facilitating distributed data processing for machine learning. In accordance with aspects of the present disclosure, a set of commands in a query to process at an external computing service is identified. For each command in the set of commands, at least one compute unit including at least one operation to perform at the external computing service is identified. Each of the at least one compute unit associated with each command is analyzed to identify an optimized manner in which to execute the set of commands at the external computing service. An indication of the optimized manner in which to execute the set of commands and a corresponding set of data is provided to the external computing service to utilize for executing the set of commands at the external computing service.
1. A computer-implemented method comprising:
identifying, at a data-processing system, a set of commands in a query to process at an external computing service;
for each command in the set of commands, identifying at least one compute unit including at least one operation to perform at the external computing service;
analyzing each of the at least one compute unit associated with each command to identify a manner in which to execute the set of commands at the external computing service to reduce memory consumption or improve execution time, the manner being identified based on information accessible via the data-processing system and inaccessible to the external computing service, wherein the manner comprises at least one of a compute unit reordering, a merger of multiple compute units, an aggregation of multiple compute units, a removal of a compute unit, a parallel execution of multiple compute units, or a combination thereof; and
providing, to the external computing service, an indication of the manner in which to execute the set of commands and a corresponding set of data to utilize for executing the set of commands at the external computing service.
2. The computer-implemented method of claim 1 , wherein the external computing service is separate from a data-processing system that receives the query to search for events in a field-searchable data store, wherein each event includes a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment, and wherein each event is associated with a timestamp extracted from the raw machine data associated with that event.
3. The computer-implemented method of claim 1 , wherein the corresponding set of data comprises a set of events, wherein each event includes a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment, and wherein each event is associated with a timestamp extracted from the raw machine data associated with that event.
4. The computer-implemented method of claim 1 further comprising receiving the query from a client computing device.
5. The computer-implemented method of claim 1 , wherein the set of commands identified in the query correspond with an external processing indicator.
6. The computer-implemented method of claim 1 , wherein the set of commands identified in the query correspond with an external processing indicator that provides an indication to process the set of commands in the external computing service.
7. The computer-implemented method of claim 1 , wherein the set of commands identified in the query correspond with an external processing indicator that provides an indication to process the set of commands in the external computing service, the external processing indicator comprising a command, a symbol, a syntax, or combination thereof.
8. The computer-implemented method of claim 1 , wherein the set of commands identified in the query consist of a portion of commands included in the query.
9. The computer-implemented method of claim 1 , wherein for each command in the set of commands, the at least one compute unit further includes an indication of one or more fields to utilize in performing the at least one operation.
10. The computer-implemented method of claim 1 , further comprising identifying one or more required fields for performing the at least one operation and using the one or more required fields to identify the at least one compute unit.
11. The computer-implemented method of claim 1 further comprising aggregating each of the at least one compute units associated with each command.
12. The computer-implemented method of claim 1 , wherein the manner is identified in accordance with a preference to optimize memory consumption.
13. The computer-implemented method of claim 1 , wherein the manner is identified in accordance with a preference to optimize execution time.
14. The computer-implemented method of claim 1 , wherein the manner comprises an external command pipeline indicating machine learning commands to perform at the external computing service and a set of metadata indicating operations to perform to prepare data for performing the machine learning commands.
15. The computer-implemented method of claim 1 , further comprising obtaining the set of data based on the query.
16. The computer-implemented method of claim 1 , further comprising obtaining the set of data based on the query, wherein the set of data comprises chunks of events in an input buffer.
17. The computer-implemented method of claim 1 , wherein the external computing service utilizes the indication of the manner and the corresponding set of data to execute the set of commands.
18. The computer-implemented method of claim 1 , wherein the external computing services:
obtains the set of data based on the query, the set of data comprising chunks of data;
stores the chunks of data in a plurality of partitions in a columnar format;
performs column-based operations on the chunks of data to obtain data having a same schema.
19. The computer-implemented method of claim 1 , wherein the external computing service is selected from among a set of candidate external computing services.
20. The computer-implemented method of claim 1 , wherein the external computing service is selected from among a set of candidate external computing services based on optimization of the external computing service to execute the set of commands compared to the other external computing services in the set of candidate external computing services.
21. One or more computer-readable storage media having instructions stored thereon, wherein the instructions, when executed by a computing device, cause the computing device to:
identifying, at a data-processing system, a set of commands in a query to process at an external computing service;
for each command in the set of commands, identifying at least one compute unit including at least one operation to perform at the external computing service;
analyzing each of the at least one compute unit associated with each command to identify a manner in which to execute the set of commands at the external computing service to reduce memory consumption or improve execution time, the manner being identified based on information accessible via the data-processing system and inaccessible to the external computing service, wherein the manner comprises at least one of a compute unit reordering, a merger of multiple compute units, an aggregation of multiple compute units, a removal of a compute unit, a parallel execution of multiple compute units, or a combination thereof; and
providing, to the external computing service, an indication of the manner in which to execute the set of commands and a corresponding set of data to utilize for executing the set of commands at the external computing service.
22. A computing device comprising:
one or more processors; and
a memory coupled with the one or more processors, the memory having instructions stored thereon, wherein the instructions, when executed by the one or more processors, cause the computing device to:
identifying, at a data-processing system, a set of commands in a query to process at an external computing service;
for each command in the set of commands, identifying at least one compute unit including at least one operation to perform at the external computing service;
analyzing each of the at least one compute unit associated with each command to identify a manner in which to execute the set of commands at the external computing service to reduce memory consumption or improve execution time, the manner being identified based on information accessible via the data-processing system and inaccessible to the external computing service, wherein the manner comprises at least one of a compute unit reordering, a merger of multiple compute units, an aggregation of multiple compute units, a removal of a compute unit, a parallel execution of multiple compute units, or a combination thereof; and
providing, to the external computing service, an indication of the manner in which to execute the set of commands and a corresponding set of data to utilize for executing the set of commands at the external computing service.
23. The computing device of claim 22 , wherein the corresponding set of data comprises a set of events, wherein each event includes a portion of raw machine data that reflects activity in an information technology environment and that is produced by a component of that information technology environment, and wherein each event is associated with a timestamp extracted from the raw machine data associated with that event.
24. The computing device of claim 22 further comprising receiving the query from a client computing device.
25. The computing device of claim 22 , wherein the set of commands identified in the query correspond with an external processing indicator.
26. The computing device of claim 22 , wherein the set of commands identified in the query correspond with an external processing indicator that provides an indication to process the set of commands in the external computing service.
27. The computing device of claim 22 , wherein the set of commands identified in the query correspond with an external processing indicator that provides an indication to process the set of commands in the external computing service, the external processing indicator comprising a command, a symbol, a syntax, or combination thereof.
28. The computing device of claim 22 , wherein the set of commands identified in the query comprise a portion of commands included in the query.
29. The computing device of claim 22 , wherein for each command in the set of commands, the at least one compute unit further includes an indication of one or more fields to utilize in performing the at least one operation.