Vector embedding compression
A method of database operations includes receiving a user query, generating a query vector embedding representative of the user query, querying a vector database using the query vector embedding, retrieving a first database vector of the plurality of database vectors based on the query and representative of a first data file corresponding to a first time and belonging to a first time-series data set, receiving a first plurality of delta encodings describing differences between vector representations of temporally-adjacent data files of the first time-series data set, identifying a second data file of the first time-series data set having a second vector representation that differs from the first database vector and corresponds to a second time, and retrieving the second data file from a database.
1. A method of database operations, the method comprising:
receiving, by a processor, a user query;
generating, by the processor, a query vector embedding representative of the user query;
querying, by the processor, a vector database using the query vector embedding, wherein:
the vector database comprises a plurality of database vectors representative of a plurality of data files;
each database vector is representative of one data file of the plurality of data files,
each data file of the plurality of data files belongs to one time-series data set of a plurality of time-series data sets, and
each data file of the plurality of data files corresponds to a first time;
retrieving, by the processor, a first database vector of the plurality of database vectors based on the query, the first database vector representative of a first data file belonging to a first time-series data set of the plurality of time-series data sets;
receiving, by the processor, a first plurality of delta encodings describing differences between vector representations of temporally-adjacent data files of the first time-series data set;
identifying, by the processor and based on the first plurality of delta encodings, a second data file of the first time-series data set having a second vector representation that differs from the first database vector, the second data file corresponding to a second time; and
retrieving, by the processor, the second data file of the first time-series data set from a database.
2. The method of claim 1 , wherein receiving the user query comprises receiving one or more first electronic signals representative of the user query from a user device, and further comprising transmitting one or more second electronic signals representative of the retrieved second data file to the user device.
3. The method of claim 2 , and further comprising:
retrieving the first data file from the database; and
transmitting one or more third electronic signals representative of the retrieved first data file to the user device.
4. The method of claim 3 , wherein identifying the second data file comprises identifying a delta encoding of the first plurality of delta encodings including at least one value greater than a threshold value.
5. The method of claim 3 , and further comprising:
identifying, by the processor and based on the first plurality of delta encodings, a third data file of the first time-series data set having a third vector representation that differs from the first vector representation, the third data file corresponding to a third time; and
retrieving, by the processor, the third data file of the first time-series data set from a database.
6. The method of claim 5 , and further comprising:
retrieving, by the processor, a fourth database vector of the plurality of database vectors based on the query, the fourth database vector representative of a fourth data file belonging to a second time-series data set of the plurality of time-series data sets;
receiving, by the processor, a second plurality of delta encodings describing differences between vector representations of temporally-adjacent data files of the second time-series data set;
identifying, by the processor and based on the second plurality of delta encodings, a fifth data file belonging to the second time-series data set and having a fifth vector representation that differs from the fourth database vector, the fifth data file corresponding to the second time; and
retrieving, by the processor, the fifth data file from the database.
7. The method of claim 6 , and further comprising transmitting one or more fourth electronic signals representative of the retrieved fifth data file to the user device.
8. The method of claim 7 , wherein the first time is before the second time.
9. The method of claim 8 , wherein the third time is one of before the first time and after the second time.
10. The method of claim 7 , wherein the second time is before the first time.
11. The method of claim 10 , wherein the third time is one of after the first time and before the second time.
12. A system for performing database operations, the system comprising:
a vector database comprising a plurality of database vectors representative of a plurality of data files, wherein:
each database vector is representative of one data file of the plurality of data files,
each data file of the plurality of data files belongs to one time-series data set of a plurality of time-series data sets, and
each data file of the plurality of data files corresponds to a first time;
a file database organizing the plurality of data files;
a processor; and
at least one memory encoded with instructions that, when executed, cause the processor to:
receive a user query;
generate a query vector embedding representative of the user query;
query the vector database using the query vector embedding;
retrieve a first database vector of the plurality of database vectors based on the query, the first database vector representative of a first data file belonging to a first time-series data set of the plurality of time-series data sets;
receive a first plurality of delta encodings describing differences between vector representations of temporally-adjacent data files of the first time-series data set;
identify, based on the first plurality of delta encodings, a second data file of the first time-series data set having a second vector representation that differs from the first database vector, the second data file corresponding to a second time; and
retrieve the second data file of the first time-series data set from the file database.
13. The system of claim 12 , wherein the instructions, when executed, cause the processor to receive the user query by receiving one or more first electronic signals representative of the user query from a user device, and wherein the instructions, when executed, further cause the processor to transmit one or more second electronic signals representative of the retrieved second data file to the user device.
14. The system of claim 13 , wherein the instructions, when executed, further cause the processor to:
retrieve the first data file from the database; and
transmit one or more third electronic signals representative of the retrieved first data file to the user device.
15. The system of claim 14 , wherein the instructions, when executed, cause the processor to identify the second data file by identifying a delta encoding of the first plurality of delta encodings including at least one value greater than a threshold value.
16. The system of claim 14 , wherein the instructions, when executed, further cause the processor to:
identify, based on the first plurality of delta encodings, a third data file of the first time-series data set having a third vector representation that differs from the first vector representation, the third data file corresponding to a third time; and
retrieve the third data file of the first time-series data set from a database.
17. The system of claim 16 , wherein the instructions, when executed, further cause the processor to:
retrieve a fourth database vector of the plurality of database vectors based on the query, the fourth database vector representative of a fourth data file belonging to a second time-series data set of the plurality of time-series data sets;
receive a second plurality of delta encodings describing differences between vector representations of temporally-adjacent data files of the second time-series data set;
identify a fifth data file belonging to the second time-series data set and having a fifth vector representation that differs from the fourth database vector, the fifth data file corresponding to the second time; and
retrieve the fifth data file from the database.
18. The system of claim 17 , and wherein the instructions, when executed, further cause the processor to transmit one or more fourth electronic signals representative of the retrieved fifth data file to the user device.
19. The system of claim 18 , wherein the first time is before the second time.
20. The system of claim 19 , wherein the third time is one of before the first time and after the second time.