AI platform for processing and querying specified objects

1. A system, comprising:

a memory; and

a processor coupled to the memory and configured to process a specified set of files for an artificial intelligence (AI) querying service according to process that includes:

hierarchically parsing the specified set of files into a set of hierarchically connected data chunks;

generating metadata for each of the hierarchically connected data chunks, wherein the metadata includes hierarchical information;

processing the hierarchically connected data chunks and metadata with an embedding model to generate vector embeddings that include the hierarchical information;

generating textual summaries from the hierarchically connected data chunks; and

storing the vector embeddings, textual summaries, and hierarchically connected data chunks for the AI querying service.

2. The system of claim 1 , wherein the AI querying service processes a query according to a process that includes:

generating a query embedding from the query using the embedding model;

retrieving a subset of hierarchically connected data chunks based on the query embedding, wherein the retrieving utilizes metadata filtering and nearest neighbor retrieval;

selecting a subset of the textual summaries based on the query;

submitting the query, subset of textual summaries, and subset of hierarchically connected data chunks to a large language model (LLM); and

receiving a response from the LLM.

3. The system of claim 1 , wherein the textual summaries are generated using a large language model.

4. The system of claim 1 , further comprising an indexing system that links vector embeddings to the hierarchically connected data chunks.

5. The system of claim 2 , wherein the subset of textual summaries are selected based on relevance to the query.

6. The system of claim 2 , wherein the metadata filtering reduces a search space of hierarchically connected data chunks prior to implementing nearest neighbor retrieval.

7. The system of claim 6 , wherein the metadata filtering limits nearest neighbor retrieval to hierarchically connected data chunks at a common hierarchical level.

8. The system of claim 2 , wherein the query, subset of textual summaries, and subset of hierarchically connected data chunks are submitted to the LLM with a prompt.

9. The system of claim 1 , wherein the embedding model is trained using a loss function that uses a multiple negative ranking loss.

10. The system of claim 1 , wherein the specified set of files belong to a domain selected from a group consisting of: medical, legal, engineering, finance, and entertainment.

11. A method of processing files for an artificial intelligence (AI) querying service, the method comprising:

hierarchically parsing the files into a set of hierarchically connected data chunks;

generating metadata for each of the hierarchically connected data chunks, wherein the metadata includes hierarchical information;

processing the hierarchically connected data chunks and metadata with an embedding model to generate vector embeddings that include the hierarchical information;

generating textual summaries from the hierarchically connected data chunks; and

storing the vector embeddings, textual summaries, and hierarchically connected data chunks for the AI querying service.

12. The method of claim 11 , wherein the AI querying service processes a query according to a process that includes:

generating a query embedding from the query using the embedding model;

retrieving a subset of hierarchically connected data chunks based on the query embedding, wherein the retrieving utilizes metadata filtering and nearest neighbor retrieval;

selecting a subset of the textual summaries based on the query;

submitting the query, subset of textual summaries, and subset of hierarchically connected data chunks to a large language model (LLM); and

receiving and outputting a response from the LLM.

13. The method of claim 11 , wherein the textual summaries are generated using a large language model.

14. The method of claim 11 , further comprising generating indexes that link vector embeddings to the hierarchically connected data chunks.

15. The method of claim 12 , wherein the subset of textual summaries are selected based on relevance to the query.

16. The method of claim 12 , wherein the metadata filtering reduces a search space of hierarchically connected data chunks prior to implementing nearest neighbor retrieval.

17. The method of claim 16 , wherein the metadata filtering limits nearest neighbor retrieval to hierarchically connected data chunks at a common hierarchical level.

18. The method of claim 12 , wherein the query, subset of textual summaries, and subset of hierarchically connected data chunks are submitted to the LLM with a prompt.

19. The method of claim 11 , wherein the embedding model is trained using a loss function that uses a multiple negative ranking loss.

20. An artificial intelligence platform, comprising:

a memory; and

a processor coupled to the memory and configured to process a specified set of data objects for an AI querying service according to process that includes:

hierarchically parsing the specified set of objects into a set of hierarchically connected data chunks;

generating metadata for each of the hierarchically connected data chunks, wherein the metadata includes hierarchical information;

processing the hierarchically connected data chunks and metadata with an embedding model to generate vector embeddings that include the hierarchical information; and

storing the vector embeddings and hierarchically connected data chunks for the AI querying service;

wherein the AI querying service processes a query according to a process that includes:

generating a query embedding from the query using the embedding model;

retrieving a subset of hierarchically connected data chunks based on the query embedding;

submitting the query and subset of hierarchically connected data chunks to a large language model (LLM); and

receiving a response from the LLM.