STORAGE SYSTEMS FOR LARGE LANGUAGE MODEL FINE-TUNING
Data is received from an application, and it is determined whether the received data is to be used for fine-tuning a Large Language Model (LLM). The received data is stored in a primary storage or in a secondary storage based at least in part on whether the received data is to be used for fine-tuning the LLM. The secondary storage is configured to store data that is less frequently accessed than data stored in the primary storage. In response to determining that the received data is to be used for fine-tuning the LLM, the received data is stored in the secondary storage. In one aspect, a query for information is received that is associated with particular data stored in the secondary storage and the query is input into the LLM to provide the information without accessing the particular data stored.
1 . A method performed by a storage system including a primary storage and a secondary storage, the method comprising:
receiving data from an application executing on a host device of the storage system to store the received data;
determining whether the received data is to be used for fine-tuning a Large Language Model (LLM);
determining whether the received data is to be stored in the secondary storage or in the primary storage based at least in part on the determination on whether the received data is to be used for fine-tuning the LLM, wherein stored data is accessed faster by the host device from the primary storage than from the secondary storage; and
in response to determining that the received data is to be used for fine-tuning the LLM, storing the received data in the secondary storage.
2 . The method of claim 1 , further comprising:
receiving a query for information associated with particular data stored in the secondary storage; and
inputting the query into the LLM to provide the information without accessing the particular data.
3 . The method of claim 1 , further comprising:
receiving a query for information;
determining that the query for information is not associated with a data type used to fine-tune the LLM; and
in response to determining that the query for information is not associated with a data type used to fine-tune the LLM, accessing particular data associated with the information that is stored in the secondary storage or in the primary storage.
4 . The method of claim 1 , further comprising determining whether the received data is to be used for fine-tuning the LLM based on at least one characteristic of the received data.
5 . The method of claim 1 , wherein the secondary storage is a first partition of a storage device of the storage system and the primary storage is a second partition of the storage device, and wherein the secondary storage provides a higher data storage density than the primary storage.
6 . The method of claim 1 , wherein the primary storage includes one or more first storage devices of a first type and the secondary storage includes one or more second storage devices of a second type.
7 . The method of claim 1 , further comprising:
determining that a frequency of access of particular data stored in the secondary storage is greater than or equal to a threshold frequency of access; and
in response to determining that the frequency of access of the particular data is greater than or equal to the threshold frequency of access, migrating the particular data from the secondary storage to the primary storage.
8 . The method of claim 1 , wherein in response to determining that the received data is to be used for fine-tuning the LLM, the method further comprises:
temporarily storing the received data in an intermediate storage of the storage system for fine-tuning the LLM; and
storing the received data in the secondary storage after fine-tuning the LLM using the received data.
9 . The method of claim 8 , further comprising providing a batch of accumulated data stored in the intermediate storage to at least one processor for fine-tuning the LLM, the batch of accumulated data including the temporarily stored received data.
10 . A system, comprising:
a primary storage configured to store data; and
a secondary storage configured to store data that is less frequently accessed than data stored in the primary storage; and
at least one processor, individually or in combination, configured to:
receive data from an application to store the received data;
determine whether to use the received data to fine-tune a Large Language Model (LLM); and
in response to determining that the received data is to be used to fine-tune the LLM, store the received data in the secondary storage.
11 . The system of claim 10 , wherein the at least one processor, individually or in combination, is further configured to:
receive a query for information associated with particular data stored in the secondary storage;
input the query into the LLM to provide the information without accessing the particular data.
12 . The system of claim 10 , wherein the at least one processor, individually or in combination, is further configured to:
receive a query for information;
determine that the query for information is not associated with a data type used to fine-tune the LLM; and
in response to determining that the query for information is not associated with a data type used to fine-tune the LLM, access particular data associated with the information that is stored in the secondary storage or in the primary storage.
13 . The system of claim 10 , wherein the at least one processor, individually or in combination, is further configured to determine to use the received data to fine-tune the LLM based on at least one characteristic of the received data.
14 . The system of claim 10 , further comprising a storage device, wherein the secondary storage is a first partition of the storage device and the primary storage is a second partition of the storage device, and wherein the secondary storage provides a higher data storage density than the primary storage.
15 . The system of claim 10 , further comprising a plurality of storage devices, wherein the primary storage includes one or more first storage devices of the plurality of storage devices of a first type and the secondary storage includes one or more second storage devices of the plurality of storages devices of a second type.
16 . The system of claim 10 , wherein the at least one processor, individually or in combination, is further configured to:
determine that a frequency of access of particular data stored in the secondary storage is greater than or equal to a threshold frequency of access; and
in response to determining that the frequency of access of the particular data is greater than or equal to the threshold frequency of access, migrate the particular data from the secondary storage to the primary storage.
17 . The system of claim 10 , further comprising:
an intermediate storage configured to temporarily store the received data for fine-tuning the LLM; and
wherein the at least one processor, individually or in combination, is further configured to store the received data in the secondary storage after fine-tuning the LLM using the received data.
18 . The system of claim 17 , wherein the temporarily stored data forms part of a batch of accumulated data stored in the intermediate storage for fine-tuning the LLM.
19 . A system, comprising:
a primary storage configured to store data; and
a secondary storage configured to store data that is less frequently accessed than data stored in the primary storage; and
means for:
receiving data from an application executing on a host device to store the received data;
determining whether the received data is to be used for fine-tuning a Large Language Model (LLM);
determining whether the received data is to be stored in the secondary storage or in the primary storage based at least in part on the determination on whether the received data is to be used for fine-tuning the LLM; and
in response to determining that the received data is to be used for fine-tuning the LLM, storing the received data in the secondary storage.
20 . The system of claim 19 , further comprising means for:
receiving a query for information associated with particular data stored in the secondary storage;
inputting the query into the LLM to provide the information without accessing the particular data.