Method to handle heterogeneous input/output (I/O) load for containers running in a virtualized environment
The disclosure provides a method for handling heterogeneous input/output (I/O) load for containers running in a virtualized environment. The method generally includes receiving, from an application running in a container, an I/O indicating to write data to a persistent volume backed by a virtual disk file in storage, determining a maximum number of in-flight write I/Os allowed for the persistent volume based on a share of a total write I/O bandwidth assigned to the virtual disk file and allocated to the persistent volume, determining a current number of in-flight write I/Os for the persistent volume, and determining whether the current number of in-flight write I/Os for the persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the persistent volume to determine whether the received I/O is to be rejected or processed.
1 . A method for writing data in a virtualized computing system, the method comprising:
receiving, from a first application running in a first container, a first input/output (I/O) to a first persistent volume, the first I/O indicating to write first data to a block of the first persistent volume associated with a logical block address (LBA), wherein the first persistent volume is backed by a first virtual disk file in storage;
determining a maximum number of in-flight write I/Os allowed for the first persistent volume, wherein the maximum number of in-flight write I/Os allowed for the first persistent volume is based on a first share of a total write I/O bandwidth assigned to the first virtual disk file, the first share being allocated to the first persistent volume;
determining a current number of in-flight write I/Os for the first persistent volume;
determining whether the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume;
when the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, rejecting the first I/O; and
when the current number of in-flight write I/Os for the first persistent volume is not greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, writing the first data to the block of the first persistent volume.
2 . The method of claim 1 , wherein determining the maximum number of in-flight write I/Os allowed for the first persistent volume comprises using an identifier of the first container and the first persistent volume as a key to search a container metadata table to determine the maximum number of in-flight write I/Os allowed for the first persistent volume, the container metadata table comprising:
a plurality of key-value mappings between a plurality of keys and a plurality of values,
the plurality of keys each being an identifier of a container and an identifier of a persistent volume associated with the container, and
the plurality of values indicating at least a number of in-flight write I/Os allowed for a corresponding persistent volume and container.
3 . The method of claim 2 , further comprising:
receiving a request to delete the first persistent volume;
deleting the first persistent volume based on the request; and
removing a key-value mapping in the plurality of key-value mappings have a key comprising an identifier of the first container and an identifier of the first persistent volume.
4 . The method of claim 1 , wherein determining a current number of in-flight write I/Os for the first persistent volume comprises accessing a counter associated with the first persistent volume, the counter being configured to track a number of in-flight write I/Os issued to the first persistent volume.
5 . The method of claim 4 , further comprising:
prior to writing the first data to the block of the first persistent volume, incrementing the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter; and
subsequent to writing the first data to the block of the first persistent volume, decrementing the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter.
6 . The method of claim 1 , wherein:
the first virtual disk file in the storage backs a plurality of persistent volumes, including the first persistent volume;
the total write I/O bandwidth assigned to the first virtual disk file is divided between the plurality of persistent volumes such that a corresponding percentage of the total write I/O bandwidth is allocated to each of the plurality of persistent volumes; and
the first share is a first percentage of the total write I/O bandwidth assigned to the first virtual disk file.
7 . The method of claim 1 , wherein an amount of the first share is based on an application type of the first application running in the first container.
8 . A system comprising:
one or more processors; and
at least one memory, the one or more processors and the at least one memory configured to:
receive, from a first application running in a first container, a first input/output (I/O) to a first persistent volume, the first I/O indicating to write first data to a block of the first persistent volume associated with a logical block address (LBA), wherein the first persistent volume is backed by a first virtual disk file in storage;
determine a maximum number of in-flight write I/Os allowed for the first persistent volume, wherein the maximum number of in-flight write I/Os allowed for the first persistent volume is based on a first share of a total write I/O bandwidth assigned to the first virtual disk file, the first share being allocated to the first persistent volume;
determine a current number of in-flight write I/Os for the first persistent volume;
determine whether the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume;
when the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, reject the first I/O; and
when the current number of in-flight write I/Os for the first persistent volume is not greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, write the first data to the block of the first persistent volume.
9 . The system of claim 8 , wherein to determine the maximum number of in-flight write I/Os allowed for the first persistent volume comprises to use an identifier of the first container and the first persistent volume as a key to search a container metadata table to determine the maximum number of in-flight write I/Os allowed for the first persistent volume, the container metadata table comprising:
a plurality of key-value mappings between a plurality of keys and a plurality of values,
the plurality of keys each being an identifier of a container and an identifier of a persistent volume associated with the container, and
the plurality of values indicating at least a number of in-flight write I/Os allowed for a corresponding persistent volume and container.
10 . The system of claim 9 , wherein the one or more processors and the at least one memory are further configured to:
receive a request to delete the first persistent volume;
delete the first persistent volume based on the request; and
remove a key-value mapping in the plurality of key-value mappings have a key comprising an identifier of the first container and an identifier of the first persistent volume.
11 . The system of claim 8 , wherein to determine a current number of in-flight write I/Os for the first persistent volume comprises to access a counter associated with the first persistent volume, the counter being configured to track a number of in-flight write I/Os issued to the first persistent volume.
12 . The system of claim 11 , wherein the one or more processors and the at least one memory are further configured to:
prior to writing the first data to the block of the first persistent volume, increment the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter; and
subsequent to writing the first data to the block of the first persistent volume, decrement the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter.
13 . The system of claim 8 , wherein:
the first virtual disk file in the storage backs a plurality of persistent volumes, including the first persistent volume;
the total write I/O bandwidth assigned to the first virtual disk file is divided between the plurality of persistent volumes such that a corresponding percentage of the total write I/O bandwidth is allocated to each of the plurality of persistent volumes; and
the first share is a first percentage of the total write I/O bandwidth assigned to the first virtual disk file.
14 . The system of claim 8 , wherein an amount of the first share is based on an application type of the first application running in the first container.
15 . A non-transitory computer-readable medium comprising instructions that, when executed by one or more processors of a computing system, cause the computing system to perform operations for writing data in a virtualized computing system, the operations comprising:
receiving, from a first application running in a first container, a first input/output (I/O) to a first persistent volume, the first I/O indicating to write first data to a block of the first persistent volume associated with a logical block address (LBA), wherein the first persistent volume is backed by a first virtual disk file in storage;
determining a maximum number of in-flight write I/Os allowed for the first persistent volume, wherein the maximum number of in-flight write I/Os allowed for the first persistent volume is based on a first share of a total write I/O bandwidth assigned to the first virtual disk file, the first share being allocated to the first persistent volume;
determining a current number of in-flight write I/Os for the first persistent volume;
determining whether the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume;
when the current number of in-flight write I/Os for the first persistent volume is greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, rejecting the first I/O; and
when the current number of in-flight write I/Os for the first persistent volume is not greater than or equal to the maximum number of in-flight write I/Os allowed for the first persistent volume, writing the first data to the block of the first persistent volume.
16 . The non-transitory computer-readable medium of claim 15 , wherein determining the maximum number of in-flight write I/Os allowed for the first persistent volume comprises using an identifier of the first container and the first persistent volume as a key to search a container metadata table to determine the maximum number of in-flight write I/Os allowed for the first persistent volume, the container metadata table comprising:
a plurality of key-value mappings between a plurality of keys and a plurality of values,
the plurality of keys each being an identifier of a container and an identifier of a persistent volume associated with the container, and
the plurality of values indicating at least a number of in-flight write I/Os allowed for a corresponding persistent volume and container.
17 . The non-transitory computer-readable medium of claim 16 , wherein the operations further comprise:
receiving a request to delete the first persistent volume;
deleting the first persistent volume based on the request; and
removing a key-value mapping in the plurality of key-value mappings have a key comprising an identifier of the first container and an identifier of the first persistent volume.
18 . The non-transitory computer-readable medium of claim 15 , wherein determining a current number of in-flight write I/Os for the first persistent volume comprises accessing a counter associated with the first persistent volume, the counter being configured to track a number of in-flight write I/Os issued to the first persistent volume.
19 . The non-transitory computer-readable medium of claim 18 , wherein the operations further comprise:
prior to writing the first data to the block of the first persistent volume, incrementing the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter; and
subsequent to writing the first data to the block of the first persistent volume, decrementing the number of in-flight write I/Os issued to the first persistent volume that are tracked by the counter.
20 . The non-transitory computer-readable medium of claim 15 , wherein:
the first virtual disk file in the storage backs a plurality of persistent volumes, including the first persistent volume;
the total write I/O bandwidth assigned to the first virtual disk file is divided between the plurality of persistent volumes such that a corresponding percentage of the total write I/O bandwidth is allocated to each of the plurality of persistent volumes; and
the first share is a first percentage of the total write I/O bandwidth assigned to the first virtual disk file.