IP Library › Granted Patent US 12688023
Granted Patent B2
US 12688023 · App. 18/407,853 · Granted Jul 21, 2026

Adding operators to code based on source data characteristics and a target runtime configuration

Inventor: Krishanu Chattopadhyay (Sunnyvale, CA)
Assignee: Oracle International Corporation
G06F8/447G06F8/433G06F8/443
View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 12688023
App. No.
18/407,853
Granted
Jul 21, 2026
Kind
B2
Abstract

Systems and methods for converting data flow to data processing code. One example system includes an electronic processor configured to receive a data flow for processing a set of source data on a target runtime, determine a characteristic associated with the set of source data, determine a target configuration of the target runtime, generate data processing code at least by adding an operator to the data flow at a point based at least on the characteristic associated with the set of source data and the target configuration of the target runtime, and output the data processing code to a compiler for generation of machine executable code.

Claims (50)

1 . A method comprising:

identifying a sequence of data processing activities for processing a set of source data in a target processing environment;

determining one or more characteristics associated with the set of source data, the one or more characteristics including a size, a location, a type, a number of rows, and a size of intersection for a unique key;

determining a target configuration of the target processing environment, the target configuration including a number of servers, a number of central processing units, a number of virtual machines, a cluster topology, a type of central processing units, a type of virtual machines, bandwidth limitations, a data storage performance, and combinations thereof;

selecting a point in the sequence of data processing activities based on the one or more characteristics associated with the set of source data and the target configuration of the target processing environment;

generating data processing code that adds an operator to the sequence of data processing activities at the point based on at least on the one or more characteristics associated with the set of source data and the target configuration of the target processing environment that optimizes processing efficiency or cost efficiency of the sequence of data processing activities; and

outputting the data processing code to a compiler for generation of machine executable code that adds the operator to the sequence of data processing activities at the point,

wherein the method is performed by at least one device including a hardware processor.

2 . The method of claim 1 , wherein the sequence of data processing activities comprises at least one selected from a group consisting of a data processing design and data processing code.

3 . The method of claim 1 , wherein the sequence of data processing activities comprises a directed acyclic graph (DAG).

4 . The method of claim 1 , wherein the operator is configured to persist a content element associated with the set of source data in a cloud object store or a persistent cache.

5 . The method of claim 1 , wherein the sequence of data processing activities includes a repetitive data processing operation, and wherein selecting the point in the sequence of data processing activities corresponds to a location the repetitive data processing operation.

6 . The method of claim 5 , wherein the repetitive data processing operation performs a same operation at least twice, and wherein the operator added at the point is configured as a target for an upstream operator and as a source for a downstream operator associated with the operation.

7 . The method of claim 1 , wherein the sequence of data processing activities includes writing a data set to a plurality of output partitions.

8 . The method of claim 7 , wherein the operator added to the sequence of data processing activities comprises a staging operator configured to reduce a size of clusters required to write the data set to the plurality of output partitions compared to a size of clusters required to process the data set.

9 . The method of claim 1 , wherein adding the operator to the sequence of data processing activities maintains semantic equivalence of the sequence of data processing activities.

10 . The method of claim 1 , wherein at least one of the operator comprises a staging operator configured to reduce a cost associated with executing the sequence of data processing activities in the target processing environment.

11 . A system comprising:

a central processing unit (CPU) and computer executable instructions stored in a memory that, when executed by the CPU, are configured to:

identify a sequence of data processing activities for processing a set of source data in a target processing environment;

determine one or more characteristics associated with the set of source data, the one or more characteristics including a size, a location, a type, a number of rows, and a size of intersection for a unique key;

determine a target configuration of the target processing environment, the target configuration including a number of servers, a number of central processing units, a number of virtual machines, a cluster topology, a type of central processing units, a type of virtual machines, bandwidth limitations, a data storage performance, and combinations thereof;

select a point in the sequence of data processing activities based on the one or more characteristics associated with the set of source data and the target configuration of the target processing environment;

generate data processing code that adds an operator to the sequence of data processing activities at the point based on at least on the one or more characteristics a characteristic associated with the set of source data and the target configuration of the target processing environment that optimizes processing efficiency or cost efficiency of the sequence of data processing activities; and

output the data processing code to a compiler for generation of machine executable code that adds the operator to the sequence of data processing activities at the point.

12 . The system of claim 1 , wherein the sequence of data processing activities comprises at least one selected from a group consisting of a data processing design and data processing code.

13 . The system of claim 1 , wherein the sequence of data processing activities comprises a directed acyclic graph (DAG).

14 . The system of claim 1 , wherein the operator is configured to persist a content element associated with the set of source data in a cloud object store or a persistent cache.

15 . The system of claim 1 , wherein the sequence of data processing activities includes a repetitive data processing operation, and wherein selecting the point in the sequence of data processing activities corresponds to a location the repetitive data processing operation.

16 . The system of claim 15 , where the repetitive data processing operation performs a same operation at least twice, and wherein the operator added at the point is configured as a target for an upstream operator and as a source for a downstream operator associated with the operation.

17 . The system of claim 1 , wherein the sequence of data processing activities includes writing a data set to a plurality of output partitions.

18 . The system of claim 17 , wherein the operator added to the sequence of data processing activities comprises a staging operator configured to reduce a size of clusters required to write the data set to the plurality of output partitions compared to a size of clusters required to process the data set.

19 . The system of claim 1 , wherein adding the operator to the sequence of data processing activities maintains semantic equivalence of the sequence of data processing activities.

20 . The system of claim 1 , wherein operator comprises a staging operator configured to reduce a cost associated with executing the sequence of data processing activities in the target processing environment.

21 . One or more non-transitory computer-readable media storing program instructions that, when executed by one or more hardware processors, cause performance of operations comprising:

identifying a sequence of data processing activities for processing a set of source data in a target processing environment;

determining one or more characteristics associated with the set of source data, the one or more characteristics including a size, a location, a type, a number of rows, and a size of intersection for a unique key;

determining a target configuration of the target processing environment, the target configuration including a number of servers, a number of central processing units, a number of virtual machines, a cluster topology, a type of central processing units, a type of virtual machines, bandwidth limitations, a data storage performance, and combinations thereof;

selecting a point in the sequence of data processing activities based on the one or more characteristics associated with the set of source data and the target configuration of the target processing environment;

generating data processing code that adds an operator to the sequence of data processing activities at the point based on at least on the one or more characteristics associated with the set of source data and the target configuration of the target processing environment that optimizes processing efficiency or cost efficiency of the sequence of data processing activities; and

outputting the data processing code to a compiler for generation of machine executable code that adds the operator to the sequence of data processing activities at the point.

22 . The one or more non-transitory computer-readable media of claim 21 , wherein the sequence of data processing activities comprises at least one selected from a group consisting of a data processing design and data processing code.

23 . The one or more non-transitory computer-readable media of claim 21 , wherein the sequence of data processing activities comprises a directed acyclic graph (DAG).

24 . The one or more non-transitory computer-readable media of claim 21 , wherein the operator is configured to persist a content element associated with the set of source data in a cloud object store or a persistent cache.

25 . The one or more non-transitory computer-readable media of claim 21 , wherein the sequence of data processing activities includes a repetitive data processing operation, and wherein selecting the point in the sequence of data processing activities corresponds to a location the repetitive data processing operation.

26 . The one or more non-transitory computer-readable media of claim 25 , wherein the repetitive data processing operation performs a same operation at least twice, and wherein the operator added at the point is configured as a target for an upstream operator and as a source for a downstream operator associated with the operation.

27 . The one or more non-transitory computer-readable media of claim 21 , wherein the sequence of data processing activities includes writing a data set to a plurality of output partitions.

28 . The one or more non-transitory computer-readable media of claim 27 , wherein the operator added to the sequence of data processing activities comprises a staging operator configured to reduce a size of clusters required to write the data set to the plurality of output partitions compared to a size of clusters required to process the data set.

29 . The one or more non-transitory computer-readable media of claim 21 , wherein adding the operator to the sequence of data processing activities maintains semantic equivalence of the sequence of data processing activities.

30 . The one or more non-transitory computer-readable media of claim 21 , wherein at least one of the operator comprises a staging operator configured to reduce a cost associated with executing the sequence of data processing activities in the target processing environment.