LLM context window optimized data distribution platform

1 . A system comprising:

one or more data stores comprising one or more non-transitory computer-readable mediums storing a collection of datasets in a catalog, each dataset associated with attributes; and

a data distribution platform powered by an artificial intelligence (AI) agent for facilitating exchanges of the datasets among users on the data distribution platform, the data distribution platform comprising one or more processors and memory storing executable instructions, wherein the executable instructions, when executed by the one or more processors, cause the one or more processors to:

extract the attributes of at least a subset of the collection of datasets that are associated with a target domain to form a collection of attributes;

verify that the collection of attributes fit within a context window limitation of a machine-learning (ML) language model powering the AI agent;

receive a query from a user indicating an intent to search for one or more datasets according to a project goal of the user;

responsive to determining that the collection of attributes do not fit within the context window limitation:

access metadata indicative of one or more subdomains of the target domain, wherein each subdomain is associated with a respective subset of the collection of attributes,

select a subdomain of the one or more subdomains based on the query, and

tune the ML model powered by the AI agent with the subset of the collection of attributes associated with the selected subdomain;

provide the query along with the collection of attributes, which are verified to fit within the context window limitation, to the AI agent; and

cause a user interface to display a recommendation of one or more candidates of datasets for the user based on a result generated by the AI agent.

2 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

responsive to determining that the collection of attributes do not fit within the context window limitation:

access metadata indicative of interactions of the AI agent with each of the datasets in the catalog;

select, based on the metadata, a collection of datasets that each meet an interaction threshold;

determine a set of attributes associated with each of the collection of datasets; and

verify that the set of attributes fit within the context window limitation.

3 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

select a subset of the collection of datasets from the catalog that collectively meet the collection of attributes;

determine a level of interaction associated with each of the subset of the collection of datasets, wherein each level of interaction represents an amount of interactions performed at one or more user devices with a respective dataset of the subset;

order the subset of the collection of datasets by level of interaction; and

responsive to receiving a call from the ML model for references associated with the collection of attributes, provide the ordered subset of the collection of datasets.

4 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

cause a user device to display an interactive element configured to receive an interaction indicative of approval or disapproval of the candidates by a user of the user device;

in response to receiving an interaction indicative of disapproval:

create a set of tuning data comprising the candidates and an indication of disapproval; and

tune the ML model with the tuning data.

5 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

select an index context of datasets of the catalog that each include at least one attribute of the collection; and

tune the ML model powered by the AI agent with the selected index context.

6 . The system of claim 1 , wherein the target domain is a medical field and includes a set of subdomains, each subdomain defining a sub-medical field included within the medical field.

7 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

determine a level of similarity between each of datasets associated with at least one attribute of the collection;

responsive to the level of similarity between a respective pair of datasets satisfying a similarity threshold:

combine the respective pair of datasets into an aggregated dataset, wherein the aggregated dataset includes the attributes of each of the pair of datasets;

generate an index context from the datasets associated with at least one attribute of the collection by:

removing each pair of datasets with a level of similarity that satisfied the similarity threshold from the datasets associated with at least one attribute of the collection; and

adding the aggregated datasets to the index context; and

in response to receiving a call from the ML model, provide the index context to the ML model.

8 . The system of claim 1 , wherein the executable instructions, when executed, further cause the one or more processors to:

responsive to determining that the subset of the collection of attributes associated with the selected subdomain do not fit within the context window limitation:

iteratively determining and selecting subdomains until the respective subset of attributes associated with a selected subdomain fit within the context window.

9 . A non-transitory computer-readable storage medium storing instructions, that when executed, cause a processor to:

extract attributes of at least a subset of a collection of datasets that are associated with a target domain to form a collection of attributes, the collection of datasets stored in a catalog, each dataset associated with attributes;

verify that the collection of attributes fit within a context window limitation of a machine-learning (ML) language model powering an artificial intelligence (AI) agent, the AI agent for facilitating exchanges of datasets among users on a data distribution platform;

receive a query from a user indicating an intent to search for one or more datasets according to a project goal of the user;

responsive to determining that the collection of attributes do not fit within the context window limitation:

access metadata indicative of one or more subdomains of the target domain, wherein each subdomain is associated with a respective subset of the collection of attributes,

select a subdomain of the one or more subdomains based on the query, and

tune the ML model powered by the AI agent with the subset of the collection of attributes associated with the selected subdomain;

provide the query along with the collection of attributes, which are verified to fit within the context window limitation, to the AI agent; and

cause a user interface to display a recommendation of one or more candidates of datasets for the user based on a result generated by the AI agent.

10 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

responsive to determining that the collection of attributes do not fit within the context window limitation:

access metadata indicative of interactions of the AI agent with each of the datasets in the catalog;

select, based on the metadata, a collection of datasets that each meet an interaction threshold;

determine a set of attributes associated with each of the collection of datasets; and

verify that the set of attributes fit within the context window limitation.

11 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

select a subset of the collection of datasets from the catalog that collectively meet the collection of attributes;

determine a level of interaction associated with each of the subset of the collection of datasets, wherein each level of interaction represents an amount of interactions performed at one or more user devices with a respective dataset of the subset;

order the subset of the collection of datasets by level of interaction; and

responsive to receiving a call from the ML model for references associated with the collection of attributes, provide the ordered subset of the collection of datasets.

12 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

cause a user device to display an interactive element configured to receive an interaction indicative of approval or disapproval of the candidates by a user of the user device;

in response to receiving an interaction indicative of disapproval:

create a set of tuning data comprising the candidates and an indication of disapproval; and

tune the ML model with the tuning data.

13 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

select an index context of datasets of the catalog that each include at least one attribute of the collection; and

tune the ML model powered by the AI agent with the selected index context.

14 . The non-transitory computer-readable storage medium of claim 9 , wherein the target domain is a medical field and includes a set of subdomains, each subdomain defining a sub-medical field included within the medical field.

15 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

determine a level of similarity between each of datasets associated with at least one attribute of the collection;

responsive to the level of similarity between a respective pair of datasets satisfying a similarity threshold:

combine the respective pair of datasets into an aggregated dataset, wherein the aggregated dataset includes the attributes of each of the pair of datasets;

generate an index context from the datasets associated with at least one attribute of the collection by:

removing each pair of datasets with a level of similarity that satisfied the similarity threshold from the datasets associated with at least one attribute of the collection; and

adding the aggregated datasets to the index context; and

in response to receiving a call from the ML model, provide the index context to the ML model.

16 . The non-transitory computer-readable storage medium of claim 9 , wherein the instructions, when executed, further cause the processor to:

responsive to determining that the subset of the collection of attributes associated with the selected subdomain do not fit within the context window limitation:

iteratively determining and selecting subdomains until the respective subset of attributes associated with a selected subdomain fit within the context window.

17 . A method comprising:

extracting attributes of at least a subset of a collection of datasets that are associated with a target domain to form a collection of attributes, the collection of datasets stored in a catalog, each dataset associated with attributes;

verifying that the collection of attributes fit within a context window limitation of a machine-learning (ML) language model powering an artificial intelligence (AI) agent, the AI agent for facilitating exchanges of datasets among users on a data distribution platform;

receiving a query from a user indicating an intent to search for one or more datasets according to a project goal of the user;

responsive to determining that the collection of attributes do not fit within the context window limitation:

accessing metadata indicative of one or more subdomains of the target domain, wherein each subdomain is associated with a respective subset of the collection of attributes,

selecting a subdomain of the one or more subdomains based on the query, and

tuning the ML model powered by the AI agent with the subset of the collection of attributes associated with the selected subdomain;

providing the query along with the collection of attributes, which are verified to fit within the context window limitation, to the AI agent; and

causing a user interface to display a recommendation of one or more candidates of datasets for the user based on a result generated by the AI agent.

18 . The method of claim 17 , further comprising:

responsive to determining that the collection of attributes do not fit within the context window limitation:

accessing metadata indicative of interactions of the AI agent with each of the datasets in the catalog;

selecting, based on the metadata, a collection of datasets that each meet an interaction threshold;

determining a set of attributes associated with each of the collection of datasets; and

verifying that the set of attributes fit within the context window limitation.