Producer/consumer optimization
View Patent ↗Systems and methods facilitate efficient data processing in a computer environment. Data producers and consumers are considered in aggregate rather than in isolation. In one instance, interaction between data producers and consumers is improved by integrating producers and consumers. Optimization can subsequently be performed over the combination to produce synergistic results.
1. A code optimization system, comprising at least one processor coupled to at least one machine-readable storage medium storing instructions executable by the at least one processor to implement:
an interface component configured to identify a programmatic data producer and consumer;
an optimization component configured to optimize interaction between the producer and the consumer;
an analysis component configured to analyze actions of the consumer;
a modification component configured to modify the producer as a function of the consumer analysis; and
an integration component configured to combine activity of the consumer with activity of the producer to generate integrated code that includes activity of the producer and at least a portion of activity of the consumer, wherein the combining activity of the consumer with activity of the producer comprises:
creating a filtered object;
passing the filtered object into a loop of the consumer activity; and
integrating the loop of the consumer activity into a query of the producer activity to fuse the producer activity and the consumer activity by removing the loop and altering the query of the producer activity so that consumer activity is a query operation of the producer activity.
2. The system of claim 1 , wherein the integration component is configured to integrate the producer and consumer into a single unit of code.
3. The system of claim 2 , further comprising a component configured to determine a parallel execution strategy for the single unit.
4. The system of claim 2 , wherein the producer and/or the consumer includes a database query.
5. The system of claim 2 , wherein the producer and/or the consumer includes a language-integrated query.
6. The system of claim 5 , further comprising a component configured to generate a query plan to extend parallelism between query operations and consumer code.
7. The system of claim 6 , further comprising a component configured to determine cost associated with the consumer code to facilitate pipelining.
8. The system of claim 1 , wherein the modification component is configured to modify the producer by altering a query associated with the producer.
9. The system of claim 1 , wherein the components form part of a compiler.
10. The system of claim 1 , wherein the components form part of an application programming interface.
11. A computer-implemented method of processing data, comprising:
identifying a programmatic data producer and consumer;
optimizing interaction between the producer and the consumer;
analyzing actions of the consumer;
modifying the producer as a function of the consumer analysis; and
combining activity of the consumer with activity of the producer to generate integrated code that includes activity of the producer and at least a portion of activity of the consumer, wherein the combining activity of the consumer with activity of the producer comprises:
creating a filtered object;
passing the filtered object into a consumer loop; and
integrating the loop of the consumer activity into a query of the producer activity to fuse the producer activity and the consumer activity by removing the loop and altering the query of the producer activity so that consumer activity is a query operation of the producer activity.
12. A computer-readable storage medium storing instructions, the instructions to, if executed by a computing device, cause the computing device to perform operations comprising:
identifying a programmatic data producer and consumer;
optimizing interaction between the producer and the consumer;
analyzing actions of the consumer;
modifying the producer as a function of the consumer analysis; and
combining activity of the consumer with activity of the producer to generate integrated code that includes activity of the producer and at least a portion of activity of the consumer, wherein the combining activity of the consumer with activity of the producer comprises:
creating a filtered object;
passing the filtered object into a consumer loop; and
integrating the loop of the consumer activity into a query of the producer activity to fuse the producer activity and the consumer activity by removing the loop and altering the query of the producer activity so that consumer activity is a query operation of the producer activity.