Memory management through control of data processing tasks
A module manages usage of a memory in a computer by scheduling data processing tasks. The module determines whether a memory associated with a computer is available for performing a data processing task. In response to determining that the memory is not available for performing the data processing task, the module generates a factory object for the data processing task. The factory object specifies instructions to produce the data processing task. After the factory object is generated, the module receives a request to perform the data processing task. The module produces the data processing task by using the factory object in response to the request and determine again whether the memory is available for performing the data processing task. In response to determining that the memory is now available for performing the data processing task, the module performs the data processing task by using the memory.
1. A computer-implemented method for managing memory usage in a data analytics environment including a data analytics system connected to data sources, comprising:
accessing, by a computer system of the data analytics system, an analytics workflow comprising a sequence of tools, each tool in the sequence adapted to perform a data analytics function of a plurality of data analytics functions of the analytics workflow;
determining, at a tool of the sequence of tools in the analytics workflow at a first time, whether an amount of available memory associated with the computer system is sufficient for storing data records for performing a data processing task received from the data sources;
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, generating a factory object for the data processing task, the factory object specifying instructions to produce the data processing task and has a memory footprint that is smaller than a memory footprint of the data processing task;
registering, by the tool, the factory object with a scheduler of the data analytics system;
in response to detection of a free thread associated with the tool by the scheduler, requesting a data processing task from the factory object;
in response to the request, producing the data processing task by using the factory object;
determining, at a second time that is after the first time, that an amount available memory is sufficient for performing a reduced scope of the data processing task and scheduling the reduced scope of the data processing task with the scheduler, the determining comprising:
determining that an amount of memory required for performing the data processing task is more than the amount of available memory,
determining a smaller amount of memory required for performing the reduced scope of the data processing task,
determining that the smaller amount of memory required for the reduced scope of the data processing task is smaller than the amount of available memory, and
scheduling the reduced scope of the data processing task with the scheduler; and
in response to the scheduling, storing the data records using the amount of available memory and performing, by the computer system, the reduced scope of the data processing task to process the data records.
2. The computer-implemented method of claim 1 , further comprising:
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, alternatively placing the data processing task into a queue of data processing tasks; and
in response to determining, at the second time, that the amount of available memory is sufficient for performing a reduced scope of the data processing task, removing the data processing task from the queue of data processing tasks.
3. The computer-implemented method of claim 2 , further comprising:
determining, at a third time that falls between the first time and the second time, whether the amount of available memory is sufficient for performing the data processing task; and
in response to determining, at the third time, that the amount of available memory is not sufficient for performing the data processing task;
placing the data processing task back into the queue.
4. The computer-implemented method of claim 1 , wherein determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task comprises:
determining the amount of memory required for performing the data processing task;
determining an amount of memory that has not been reserved for other data processing tasks;
determining whether the amount of memory required for performing the data processing task is more than the amount of memory that has not been reserved for other data processing tasks; and
in response to determining that the amount of memory required for performing the data processing task is more than the amount of memory that has not been reserved for other data processing tasks, determining that the amount of available memory is not sufficient for performing the data processing task.
5. A computer system for managing memory usage in a data analytics environment including a data analytics system connected to data sources, the computer system comprising:
a computer processor; and
a non-transitory computer-readable storage medium storing executable computer program instructions, the computer program instructions comprising instructions that when executed cause the computer processor to perform operations comprising:
accessing an analytics workflow comprising a sequence of tools, each tool in the sequence adapted to perform a data analytics function of a plurality of data analytics functions of the analytics workflow;
determining, at a tool of the sequence of tools in the analytics workflow at a first time, whether an amount of available memory associated with the computer system is sufficient for storing data records for performing a data processing task received from the data sources;
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, generating a factory object for the data processing task, the factory object specifying instructions to produce the data processing task and has a memory footprint that is smaller than a memory footprint of the data processing task;
registering, by the tool, the factory object with a scheduler of the data analytics system;
in response to detection of a free thread associated with the tool by the scheduler, requesting a data processing task from the factory object;
in response to the request, producing the data processing task by using the factory object;
determining, at a second time that is after the first time, that an amount available memory is sufficient for performing a reduced scope of the data processing task and scheduling the reduced scope of the data processing task with the scheduler, the determining comprising:
determining that an amount of memory required for storing the data records of the data processing task is more than an amount of available memory,
determining a smaller amount of memory required for performing the reduced scope of the data processing task,
determining that the smaller amount of memory required for the reduced scope of the data processing task is smaller than the amount of available memory, and
scheduling the reduced scope of the data processing task with the scheduler; and
in response to the scheduling, storing the data records using the amount of available and performing the reduced scope of the data processing task to process the data records.
6. The computer system of claim 5 , wherein the operations further comprise:
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, alternatively placing the data processing task into a queue of data processing tasks; and
in response to determining, at the second time, that the amount of available memory is sufficient for performing a reduced scope of the data processing task, removing the data processing task from the queue of data processing tasks.
7. The computer system of claim 6 , wherein the operations further comprise:
determining, at a third time that falls between the first time and the second time, whether the amount of available memory is sufficient for performing the data processing task; and
in response to determining, at the third time, that the amount of available memory is not sufficient for performing the data processing task;
placing the data processing task back into the queue.
8. The computer system of claim 5 , wherein determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task comprises:
determining the amount of memory required for performing the data processing task;
determining an amount of memory that has not been reserved for other data processing tasks;
determining whether the amount of memory required for performing the data processing
task is more than the amount of memory that has not been reserved for other data processing tasks; and
in response to determining that the amount of memory required for performing the data
processing task is more than the amount of memory that has not been reserved for other data processing tasks, determining that the amount of available memory is not sufficient for performing the data processing task.
9. A non-transitory computer readable storage medium storing executable computer program instructions for managing memory usage in a data analytics environment including a data analytics system connected to data sources, the computer program instructions comprising instructions that when executed cause a computer processor to perform operations comprising:
accessing an analytics workflow comprising a sequence of tools, each tool in the sequence adapted to perform a data analytics function of a plurality of data analytics functions of the analytics workflow;
determining, at a tool of the sequence of tools in the analytics workflow at a first time, whether an amount of available memory associated with a computer is sufficient for storing data records for performing a data processing task received from the data sources;
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, generating a factory object for the data processing task, the factory object specifying instructions to produce the data processing task and has a memory footprint that is smaller than a memory footprint of the data processing task;
registering, by the tool, the factory object with a scheduler of the data analytics system;
in response to detection of a free thread associated with the tool by the scheduler, requesting a data processing task from the factory object;
in response to the request, producing the data processing task by using the factory object;
determining, at a second time that is after the first time, that an amount available memory is sufficient for performing a reduced scope of the data processing task and scheduling the reduced scope of the data processing task with the scheduler, the determining comprising:
determining that an amount of memory required for storing the data records of the data processing task is more than an amount of available memory,
determining a smaller amount of memory required for performing the reduced scope of the data processing task,
determining that the smaller amount of memory required for the reduced scope of the data processing task is smaller than the amount of memory available, and
scheduling the reduced scope of the data processing task with the scheduler; and
in response to the scheduling, storing the data records using the amount of available and performing the reduced scope of the data processing task to process the data records.
10. The non-transitory computer readable storage medium of claim 9 , wherein the operations further comprise:
in response to determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task, alternatively placing the data processing task into a queue of data processing tasks; and
in response to determining, at the second time, that the amount of available memory is sufficient for performing a reduced scope of the data processing task, removing the data processing task from the queue of data processing tasks.
11. The non-transitory computer readable storage medium of claim 9 , wherein determining, at the first time, that the amount of available memory is not sufficient for storing the data records for performing the data processing task comprises:
determining the amount of memory required for performing the data processing task;
determining an amount of memory that has not been reserved for other data processing tasks;
determining whether the amount of memory required for performing the data processing task is more than the amount of memory that has not been reserved for other data processing tasks; and
in response to determining that the amount of memory required for performing the data processing task is more than the amount of memory that has not been reserved for other data processing tasks, determining that the amount of available memory is not sufficient for performing the data processing task.