Language interoperable runtime adaptable data collections
Adaptive data collections may include various type of data arrays, sets, bags, maps, and other data structures. A simple interface for each adaptive collection may provide access via a unified API to adaptive implementations of the collection. A single adaptive data collection may include multiple, different adaptive implementations. A system configured to implement adaptive data collections may include the ability to adaptively select between various implementations, either manually or automatically, and to map a given workload to differing hardware configurations. Additionally, hardware resource needs of different configurations may be predicted from a small number of workload measurements. Adaptive data collections may provide language interoperability, such as by leveraging runtime compilation to build adaptive data collections and to compile and optimize implementation code and user code together. Adaptive data collections may also provide language-independent such that implementation code may be written once and subsequently used from multiple programming languages.
1. A computer-implemented method, comprising:
configuring a data collection according to a first configuration to provide a first data functionality of the data collection, the data collection comprising data and one or more methods of a language-independent application programming interface (API), wherein the first data functionality comprises access to at least a portion of the one or more methods of the language-independent API, and wherein the first configuration is selected based at least in part on one or more predicted resource requirements of a workload;
collecting, subsequent to the configuring, performance information during execution of the workload within a platform-independent virtual environment, the execution comprising executing the one or more methods of the data collection to access the data of the data collection; and
reconfiguring the data collection according to a second configuration to provide a second data functionality of the data collection, wherein the second data functionality comprises access to at least another portion of the one or more methods of the language-independent API different from the portion of the one or more methods of the language-independent API of the first data functionality, and wherein the second configuration is selected based at least in part on the collected performance information.
2. The computer-implemented method of claim 1 , wherein the one or more methods of the language-independent API are developed using a first programming language, and wherein the execution of the workload is performed by an application developed using a second programming language different from the first programming language.
3. The computer-implemented method of claim 1 , wherein the method is implemented by a computer including multiple sockets each including multiple processor cores.
4. The computer-implemented method of claim 1 , wherein the first data functionality and the second data functionality individually comprise one or more of:
an operating system default NUMA-aware data placement for the data of the data collection;
a single socket NUMA-aware data placement for the data of the data collection;
an interleaved NUMA-aware data placement for the data of the data collection;
a replicated NUMA-aware data placement for the data of the data collection;
a compression scheme for the data of the data collection;
an indexing scheme for data elements of the data collection; or
a data synchronization scheme for the data collection.
5. The computer-implemented method of claim 1 , further comprising determining the one or more predicted resource requirements based at least in part on one or more of:
a specification of a computer implementing the method, comprising at least one of:
a size of a memory, a maximum bandwidth between hardware components of the computer, or a maximum computing capacity available on each of the processor cores of the computer;
a specification of performance characteristics of the data collection based at least in part on one or more performance counters of the computer; or
information collected during one or more profiling runs of the workload.
6. The computer-implemented method of claim 1 , wherein the data collection is configured to organize the data of the data collection as one of a bag, a set, an array, or a map.
7. The computer-implemented method of claim 1 , further comprising:
selecting the first configuration, the selecting comprising:
assigning the first configuration from among a first candidate configuration and a second candidate configuration based at least in part on, the one or more predicted resource requirements of the workload, the first candidate configuration specifying an uncompressed data functionality and the second candidate configuration specifying a compressed data functionality, wherein the uncompressed data functionality specifies an uncompressed data layout for the data of the data collection, and wherein the compressed data functionality specifies a compressed data layout for the data of the data collection.
8. A system, comprising:
a computing device comprising one or more processors and a memory coupled to the one or more processors, the memory comprising program instructions executable by the one or more processors to implement a platform-independent virtual environment configured to:
configure a data collection according to a first configuration to provide a first data functionality of the data collection, the data collection comprising data and one or more methods of a language-independent application programming interface (API), wherein the first data functionality comprises access to a portion of the one or more methods of the language-independent API, and wherein the first configuration is selected based at least in part on one or more predicted resource requirements of a workload;
collect, subsequent to the configuring, performance information during execution of the workload, the execution comprising executing the one or more methods of the data collection to access the data of the data collection; and
reconfigure the data collection according to a second configuration to provide a second data functionality of the data collection, wherein the second data functionality comprises access to another portion of the one or more methods of the language-independent API different from the portion of the one or more methods of the language-independent API of the first data functionality, and wherein the second configuration is selected based at least in part on the collected performance information.
9. The system of claim 8 , wherein the one or more methods of the language-independent API are developed using a first programming language, and wherein the execution of the workload is performed by an application developed using a second programming language different from the first programming language.
10. The system of claim 8 , wherein the computing device comprises multiple sockets each including multiple processor cores.
11. The system of claim 8 , wherein the first data functionality and the second data functionality individually comprise one or more of:
an operating system default NUMA-aware data placement for the data of the data collection;
a single socket NUMA-aware data placement for the data of the data collection;
an interleaved NUMA-aware data placement for the data of the data collection;
a replicated NUMA-aware data placement for the data of the data collection;
a compression scheme for the data of the data collection;
an indexing scheme for data elements of the data collection; or
a data synchronization scheme for the data collection.
12. The system of claim 8 , the platform-independent virtual environment further configured to determine the one or more predicted resource requirements based at least in part on one or more of:
a specification of a computer implementing the method, comprising at least one of:
a size of a memory, a maximum bandwidth between hardware components of the computer, or a maximum computing capacity available on each of the processor cores of the computer;
a specification of performance characteristics of the data collection based at least in part on one or more performance counters of the computer; or
information collected during one or more profiling runs of the workload.
13. The system of claim 8 , wherein the data collection is configured to organize the data of the data collection as one of a bag, a set, an array, or a map.
14. The system of claim 8 , the platform-independent virtual environment further configured to:
select the first configuration from among a first candidate configuration and a second candidate configuration based at least in part on, the one or more predicted resource requirements of the workload, the first candidate configuration specifying an uncompressed data functionality and the second candidate configuration specifying a compressed data functionality, wherein the uncompressed data functionality specifies an uncompressed data layout for the data of the data collection, and wherein the compressed data functionality specifies a compressed data layout for the data of the data collection.
15. One or more non-transitory computer readable storage media storing program instructions that, when executed on or across one or more processors, cause the one or more processors to implement a platform-independent virtual environment to perform:
configuring a data collection according to a first configuration to provide a first data functionality of the data collection, the data collection comprising data and one or more methods of a language-independent application programming interface (API), wherein the first data functionality comprises access to a portion of the one or more methods of the language-independent API, and wherein the first configuration is selected based at least in part on one or more predicted resource requirements of a workload;
collecting, subsequent to the configuring, performance information during execution of the workload, the execution comprising executing the one or more methods of the data collection to access the data of the data collection; and
reconfiguring the data collection according to a second configuration to provide a second data functionality of the data collection, wherein the second data functionality comprises access to another portion of the one or more methods of the language-independent API different from the portion of the one or more methods of the language-independent API of the first data functionality, wherein the second configuration is selected based at least in part on the collected performance information.
16. The one or more non-transitory computer readable storage media as recited in claim 15 , wherein the one or more methods of the language-independent API are developed using a first programming language, and wherein the execution of the workload is performed by an application developed using a second programming language different from the first programming language.
17. The one or more non-transitory computer readable storage media as recited in claim 15 , wherein the method is implemented by a computer including multiple sockets each including multiple processor cores.
18. The one or more non-transitory computer readable storage media as recited in claim 15 , wherein the first data functionality and the second data functionality individually comprise one or more of:
an operating system default NUMA-aware data placement for the data of the data collection;
a single socket NUMA-aware data placement for the data of the data collection;
an interleaved NUMA-aware data placement for the data of the data collection;
a replicated NUMA-aware data placement for the data of the data collection;
a compression scheme for the data of the data collection;
an indexing scheme for data elements of the data collection; or
a data synchronization scheme for the data collection.
19. The one or more non-transitory computer readable storage media as recited in claim 15 , the platform-independent virtual environment further performing determining the one or more predicted resource requirements based at least in part on one or more of:
a specification of a computer implementing the method, comprising at least one of:
a size of a memory, a maximum bandwidth between hardware components of the computer, and a maximum computing capacity available on each of the processor cores of the computer;
a specification of performance characteristics of the data collection based at least in part on one or more performance counters of the computer; or
information collected during one or more profiling runs of the workload.
20. The one or more non-transitory computer readable storage media as recited in claim 15 , the platform-independent virtual environment further performing:
selecting the first configuration from among a first candidate configuration and a second candidate configuration based at least in part on, the one or more predicted resource requirements of the workload, the first candidate configuration specifying an uncompressed data functionality and the second candidate configuration specifying a compressed data functionality, wherein the uncompressed data functionality specifies an uncompressed data layout for the data of the data collection, and wherein the compressed data functionality specifies a compressed data layout for the data of the data collection.