Compilation system and method

1 . A system capable of compiling a source program that comprises a plurality of source operations and one or more loops and/or branches, thereby defining a non-linear sequence of execution of the source operations, wherein the system comprises:

a dispatcher configured to step through the sequence including looping through the loops and/or branching at the branches, and to map each source operation to a respective function call each time encountered during the stepping through of the sequence;

a trace buffer;

an interceptor configured to intercept the function calls from the dispatcher, convert each into a corresponding trace entry comprising a set of one or more trace operations, and insert the trace entries one after another into the trace buffer to thereby form a linear trace of the source program; and

a trace manager configured to transfer each of a succession of trace instances from the trace buffer to at least one compiler for compilation into corresponding compiled code, each trace instance comprising a current instance of the linear trace in the trace buffer or a portion thereof, the trace manager being arranged to receive back the compiled code compiled from each trace instance, and send the compiled code to at least one destination device to be executed.

2 . The system of claim 1 , wherein the trace manager is configured to:

for at least a first one or more of the trace instances, collect statistics on one or more of:

a compilation time of the compilation of the first one or more of the trace instances,

an execution time of the corresponding compiled code,

a frequency of repetition of one or more trace instances,

a memory utilization of the corresponding compiled code, or

a device health of the destination device used to run the corresponding compiled code; and

based on said statistics, select which of multiple compilers to use as the at least one compiler to compile a second one or more of the trace instances, or to select a tier of a tiered optimization algorithm of the at least one compiler used to compile the second one or more of the trace instances.

3 . The system of claim 1 , wherein the trace manager is configured to:

monitor resource usage on multiple candidate devices; and

based thereon select which to use as the at least one destination device, or whether to migrate operand data from one of the candidate devices to the destination device, the operand data being data to be operated upon by the compiled code.

4 . The system of claim 1 , wherein the trace manager is configured to:

cache the compiled code corresponding to at least some of the trace instances in a cache;

detect in the trace buffer a most-recent run of contiguous trace operations which matches an initial, contiguous subset of the trace operations from a previously compiled trace instance; and

based thereon, speculatively send a copy of the compiled code corresponding to the previously compiled trace instance from the cache to the at least one destination device for speculative execution.

5 . The system of claim 1 , wherein each of some or all of the trace operations records whether or not any variable operated upon by that trace operation is observable, thereby enabling the at least one compiler to optimize compilation according to whether the variable is observable.

6 . The system of claim 1 , wherein the trace manager is configured to monitor the trace in the trace buffer, and perform the transfer of a trace instance from the trace buffer for compilation in response to the trace reaching a predetermined size.

7 . The system of claim 1 , wherein the trace manager is configured to:

cache the compiled code corresponding to at least some of the trace instances in a cache;

monitor the trace in the trace buffer;

detect in the trace buffer a repeated trace instance, being a portion of the trace comprising the same trace operations, with the same or different variable values, as a previous one of the trace instances which has already been compiled; and

in response to detecting the repeated trace instance, one or both of:

send the compiled code corresponding to the previous one of the trace instances to at least one destination device from the cache to be executed, and

transfer the repeated trace instance to be compiled by a higher power compiler than the previous trace instance, or by a higher tier of a tiered compilation algorithm than the previous one of the trace instances.

8 . The system of claim 7 , wherein the trace manager is configured to perform said detection based on a central service logging pre-compiled trace instances from a plurality of systems of different parties.

9 . The system of claim 1 , wherein the trace manager is configured to monitor the linear trace in the trace buffer, and perform the transfer of a trace instance from the trace buffer for compilation based on a heuristic rule applied to the trace in the trace buffer.

10 . The system of claim 1 , wherein the trace manager is configured to monitor the linear trace in the trace buffer, and to perform the transfer of a trace instance from the trace buffer for compilation when an observable event occurs in the trace.

11 . The system of claim 1 , wherein the source program comprises a dynamic machine learning model.

12 . The system of claim 1 , wherein at least some of the source operations comprise tensor operations.

13 . The system of claim 1 , wherein the source program is written in Python, and at least some of the source operations are functions of one or more PyTorch libraries.

14 . A method of compiling a source program that comprises a plurality of source operations and one or more loops and/or branches, thereby defining a non-linear sequence of execution of the source operations; wherein the method comprises:

stepping through the sequence including looping through the loops and/or branching at the branches, and mapping each source operation to a respective function call each time encountered during the stepping through of the sequence;

intercepting the function calls, converting each into a corresponding trace entry comprising a set of one or more trace operations, and inserting the trace entries one after another into a trace buffer to thereby form a linear trace of the source program; and

transferring each of a succession of trace instances from the trace buffer to at least one compiler for compilation into corresponding compiled code, each trace instance comprising a current instance of the linear trace in the trace buffer or a portion thereof,

receiving the compiled code compiled from each trace instance; and

sending the compiled code to at least one destination device to be executed.

15 . The method of claim 14 , further comprising:

collecting statistics on:

a compilation time of the compilation of a first trace instance of the trace instances,

an execution time of the corresponding compiled code,

a frequency of repetition of the first trace instance,

a memory utilization of the corresponding compiled code, or

a device health of the destination device used to run the corresponding compiled code; and

based on the collected statistics, select which of multiple compilers to use as a compiler to compile a second trace instance.

16 . The method of claim 14 , further comprising:

monitoring resource usage on candidate devices; and

based thereon, select which of the candidate devices to use as the destination device.

17 . The method of claim 14 , further comprising:

caching the compiled code corresponding to the trace instances in a cache;

detecting, in the trace buffer, a most-recent run of contiguous trace operations which matches an initial, contiguous subset of the trace operations from a previously compiled trace instance; and

based thereon, speculatively sending a copy of the compiled code corresponding to the previously compiled trace instance from the cache to the destination device for speculative execution.

18 . The method of claim 14 , wherein transferring each of the succession of the trace instances from the trace buffer to the at least one compiler comprises:

monitoring the trace in the trace buffer, and

performing the transfer of a trace instance from the trace buffer for compilation in response to the trace reaching a predetermined size.

19 . The method of claim 14 , further comprising:

caching the compiled code corresponding to the trace instances in a cache;

monitoring the trace in the trace buffer;

detecting in the trace buffer a repeated trace instance with the same or different variable values, the repeated trace instance being a portion of the trace comprising the same trace operations as a previous one of the trace instances which has already been compiled; and

in response to detecting the repeated trace instance:

sending the compiled code corresponding to the previous one of the trace instances to the destination device from the cache to be executed, or

transferring the repeated trace instance to be compiled by a higher power compiler than the previous one of the trace instances, or by a higher tier of a tiered compilation algorithm than the previous trace instance.

20 . A software toolchain for use in compiling a source program that comprises a plurality of source operations and one or more loops and/or branches, thereby defining a non-linear sequence of execution of the source operations; wherein the software toolchain is embodied on a non-transitory computer-readable storage and is configured so as when run on a host computer to perform operations of:

stepping through the sequence including looping through the loops and/or branching at the branches, and mapping each source operation to a respective function call each time encountered during the stepping through of the sequence;

intercepting the function calls, converting each into a corresponding trace entry comprising a set of trace operations, and inserting the trace entries one after another into a trace buffer to thereby form a linear trace of the source program; and

transferring each of a succession of trace instances from the trace buffer to at least one compiler for compilation into a corresponding compiled code, each trace instance comprising a current instance of the linear trace in the trace buffer or a portion thereof,

receiving the compiled code compiled from each trace instance; and

sending the compiled code to a destination device to be executed.