Kernel-level monitoring for software applications
The systems and methods disclosed herein monitor application (e.g., artificial intelligence (AI) model) operations using interactions between the application and a kernel. The systems and methods disclosed herein intercept, using a kernel interface, one or more function invocations transmitted from the application (e.g., an AI model without model modification). Event record(s) are generated for one or more functions to define process identifiers, resource interaction types, timestamps, and/or resource identifiers. Observed pattern(s) for the application are identified by comparing current event record(s) with previous record(s), and the identified observed pattern(s) are evaluated against reference pattern(s) to generate score(s). Data packet(s) that indicate observed pattern(s), corresponding score(s), and/or cryptographic digital fingerprint(s) of the one or more functions are generated. The data packet(s) are transmitted to distributed ledgers for immutable storage.
1 . A system for managing resource allocation for artificial intelligence (AI) applications using kernel commands, comprising:
at least one hardware processor; and
at least one non-transitory memory storing instructions, which, when executed by the at least one hardware processor, cause the system to:
intercept, using a system hook associated with a kernel interface configured to manage communications with a kernel of an operating system, a function invocation transmitted from an AI application associated with AI application metadata, wherein the AI application metadata comprises at least one of: an application identifier, a resource priority level, or an allocation parameter, wherein the function invocation represents a request to execute a function set using the kernel, and wherein the system hook comprises an extended Berkeley Packet Filter (eBPF) program that is executed by the kernel to intercept the function invocation;
identify, using the system hook, a resource allocation request associated with the function invocation, wherein the resource allocation request indicates that the AI application has requested allocation of one or more resources accessible by the operating system;
map, using the AI application metadata, the AI application to an application-specific rule set that is maintained by the kernel based on the eBPF program, wherein the application-specific rule set defines a specific resource quota allocated for the AI application;
apply the application-specific rule set to the resource allocation request to generate one or more resource allocation metric values for the one or more resources requested by the AI application;
determine whether to execute the function set associated with the function invocation based on the one or more resource allocation metric values meeting an execution constraint set;
in response to determining that the one or more resource allocation metric values meet the execution constraint set, cause execution of the function set associated with the function invocation using the kernel of the operating system, wherein the kernel is configured to allocate the one or more resources to the AI application in accordance with the resource allocation request; and
in response to determining that the one or more resource allocation metric values fail to meet the execution constraint set, throttle the AI application by blocking at least a portion of the resource allocation request.
2 . The system of claim 1 , wherein the one or more resources include: a processor time, a memory allocation value, a storage capacity value, a network bandwidth value, or a graphics processing unit (GPU) utilization value.
3 . The system of claim 1 , wherein the system is further caused to:
determine at least one system load metric including one or more of: a Central Processing Unit (CPU) utilization value, an I/O congestion value, or a network saturation value; and
determine an aggregate load score using the at least one system load metric.
4 . The system of claim 3 , wherein the system is further caused to:
adjust a resource exchange rate for the one or more resources using the aggregate load score, wherein the resource exchange rate is mapped to one or more zones that each correspond to a different system load level.
5 . The system of claim 1 , wherein the eBPF program executes within kernel space to intercept the function invocation.
6 . A non-transitory computer-readable storage medium comprising instructions stored thereon for managing resource allocation for artificial intelligence (AI) applications using kernel commands, wherein the instructions when executed by at least one data processor of a system, cause the system to:
intercept, using a system hook associated with a kernel interface configured to manage communications with a kernel of an operating system, a function invocation transmitted from an AI application associated with AI application metadata, wherein the function invocation represents a request to execute a function set using the kernel, wherein the AI application metadata comprises at least one of: an application identifier, a resource priority level, or an allocation parameter, and wherein the system hook comprises an extended Berkeley Packet Filter (eBPF) program that is executed by the kernel to intercept the function invocation;
identify, using the system hook, a resource allocation request associated with the function invocation, wherein the resource allocation request indicates that the AI application has requested allocation of one or more resources accessible by the operating system;
associate, using the AI application metadata, the AI application to an application-specific rule set that is maintained by the kernel based on the eBPF program, wherein the application-specific rule set defines a specific resource quota allocated for the AI application;
apply the application-specific rule set to the resource allocation request to generate one or more resource allocation metric values for the one or more resources requested by the AI application;
determine whether to execute the function set associated with the function invocation based on the one or more resource allocation metric values satisfying an execution constraint set;
in response to determining that the one or more resource allocation metric values satisfy the execution constraint set, cause execution of the function set associated with the function invocation using the kernel of the operating system, wherein the kernel is configured to allocate the one or more resources to the AI application in accordance with the resource allocation request; and
in response to determining that the one or more resource allocation metric values fail to satisfy the execution constraint set, throttle the AI application by blocking at least a portion of the resource allocation request.
7 . The non-transitory computer-readable storage medium of claim 6 , wherein the eBPF program executes within kernel space to intercept the function invocation.
8 . The non-transitory computer-readable storage medium of claim 7 , wherein the eBPF program is attached to one or more kernel hooks including one or more of: a scheduler hook, a memory allocation hook, a block I/O hook, or a network hook.
9 . The non-transitory computer-readable storage medium of claim 6 , wherein the resource quota representing a maximum value of resource consumption for the AI application.
10 . The non-transitory computer-readable storage medium of claim 6 , wherein the execution constraint set includes a predefined threshold, and wherein the instructions further cause the system to:
generate a billing event when a resource usage value associated with the AI application exceeds the predefined threshold,
wherein the billing event includes one or more of: an agent identifier, the resource usage value, or a cost value corresponding to the resource usage value.
11 . The non-transitory computer-readable storage medium of claim 10 , wherein the instructions further cause the system to:
generate a cryptographic proof for the billing event by:
applying one or more hash operations to the billing event by generating a cryptographic hash of the billing event, and
linking the cryptographic hash to a previous hash of a previous billing event to form a hash chain.
12 . The non-transitory computer-readable storage medium of claim 6 , wherein the instructions further cause the system to:
store the one or more resource allocation metric values in a kernel-accessible data structure comprising one or more eBPF maps.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the instructions further cause the system to:
record a start timestamp when the AI application is scheduled to execute; and
determine an elapsed resource consumption value using a difference between a current timestamp and the start timestamp.
14 . A computer-implemented method for managing resource allocation for artificial intelligence (AI) applications using kernel commands, the computer-implemented method comprising:
intercepting, using a system hook associated with a kernel interface configured to manage communications with a kernel of an operating system, a function invocation transmitted from an AI application associated with AI application metadata, wherein the function invocation represents a request to execute a function set using the kernel, wherein the AI application metadata comprises at least one of: an application identifier, a resource priority level, or an allocation parameter, and wherein the system hook comprises an extended Berkeley Packet Filter (eBPF) program that is executed by the kernel to intercept the function invocation;
identifying, using the system hook, a resource allocation request associated with the function invocation, wherein the resource allocation request indicates that the AI application has requested allocation of one or more resources accessible by the operating system;
linking, based on the AI application metadata, the AI application to a criterion set that is maintained by the kernel based on the eBPF program, wherein the criterion set defines a specific resource quota allocated for the AI application;
applying the criterion set to the resource allocation request to generate one or more resource allocation metric values for the one or more resources requested by the AI application;
determining whether to cause execution of the function set associated with the function invocation based on the one or more resource allocation metric values satisfying an execution constraint set; and
in response to determining that the one or more resource allocation metric values satisfy the execution constraint set, cause execution of the function set associated with the function invocation using the kernel of the operating system, wherein the kernel is configured to allocate the one or more resources to the AI application in accordance with the resource allocation request; and
in response to determining that the one or more resource allocation metric values fail to satisfy the execution constraint set, throttle the AI application by blocking at least a portion of the resource allocation request.
15 . The computer-implemented method of claim 14 , wherein the eBPF program executing within kernel space.
16 . The computer-implemented method of claim 14 , wherein the criterion set is stored in a data structure maintained by the kernel that associates the AI application metadata with a maximum allowed resource consumption value for one or more resource types.
17 . The computer-implemented method of claim 14 , further comprising:
determining a resource delta value representing an incremental resource consumption by the AI application between two timestamps; and
updating a data structure accessible by the kernel with an indication of the resource delta value.
18 . The computer-implemented method of claim 14 , further comprising:
determining that the one or more resource allocation metric values fail to satisfy the execution constraint set; and
in response to determining that the one or more resource allocation metric values fail to satisfy the execution constraint set, generating a billing event, wherein the billing event indicates one or more of: an application identifier associated with the AI application, a resource usage value, or a cost value corresponding to the resource usage value.
19 . The computer-implemented method of claim 18 , further comprising:
generating a cryptographic hash of the billing event;
retrieving a previous hash value associated with a previous billing event; and
linking the billing event to the previous billing event by including the previous hash value in the billing event to generate a hash chain.
20 . The computer-implemented method of claim 14 , further comprising:
in response to determining that the one or more resource allocation metric values fail to satisfy the execution constraint set, managing the AI application by performing one or more of: causing a scheduling delay, causing transmission of a memory allocation failure, queuing a storage operation, or restricting a network bandwidth allocation.