Slab memory allocator with dynamic buffer resizing
Embodiments for dynamically resizing buffers for a slab allocator process are described. The slab allocator informs the consumer that the memory buffer must be shrunk to a smaller size. A buffer allocation process dynamically reclaims portions of larger memory buffers to make room for a smaller allocation by shrinking data objects in larger slabs and returning slabs to reserve or free slab lists. Initially a large limit is set, and it is dynamically reduced once all the available memory is exhausted. This allows the slab allocator to adapt to the workload.
1. A computer-implemented method for dynamically allocating buffers for a slab allocator process, comprising:
maintaining a link list of all allocated buffers grouped by buffer size;
subdividing, by the slab allocator, a fixed size slab into a plurality of small data objects of a specific size;
setting an upper bound for a number of data elements in a record;
triggering, upon exceeding the upper bound, a reclaim callback mechanism to request a portion of the grouped allocated buffers back to the slab allocator;
selecting a largest buffer from the grouped allocated buffers for space reclamation; and
shrinking data objects in the largest buffer to fit a reserved slab and returning a slab of the largest buffer to a reserve slab pool and at least one other slab of the largest buffer to a free slab list.
2. The method of claim 1 wherein the buffers comprise large buffers and small buffers grouped into a large buffer group and a small buffer group.
3. The method of claim 2 wherein the shrinking shrinks the data objects in the largest buffer by one-half.
4. The method of claim 1 wherein the slab allocator allocates data objects to the buffers depending on data type.
5. The method of claim 1 wherein the data elements comprise tags identifying one or more related files, and wherein the buffers are allocated for a file selection application executed in a deduplication backup system.
6. The method of claim 3 wherein the files are related through one of: application, file size, creation time, client, or data type.
7. The method of claim 2 wherein the reclaim callback mechanism issues a callback to an application with an object pointer for the application to allocate a new object of a smaller size to fit into a small buffer.