Approximate query equivalence for feature stores in machine learning operations products
Methods, systems, apparatuses, and computer-readable storage mediums are descried for identifying a similarity between queries. An intermediate representation generator receives a set of queries from a repository, each query in the set of queries having generated a corresponding set of data stored in a data store. An intermediate representation is generated for each query, where the intermediate representation is characterized by a feature associated with text specified in the query. A similarity determiner determines similarity scores between pairs of intermediate representations. A pair of intermediate representations with a similarity score above a threshold is identified. An indication is generated that sets of data corresponding to queries corresponding to the intermediate representations are overlapping.
1 . A system for deduplicating data in a data store, comprising:
a processor; and
a memory that stores program code structured to cause the processor to:
generate an intermediate representation for each of a first query and a second query, each intermediate representation comprising multiple dimensions, each dimension corresponding to a weighted feature of the query, the first query having generated a first set of data in the data store and the second query having generated a second set of data in the data store;
generate a similarity score between the intermediate representation for the first query and the intermediate representation for the second query; and
remove one of the first set of data or the second set of data in the data store based at least on the similarity score.
2 . The system of claim 1 , wherein the intermediate representation for each query comprises a vector representation.
3 . The system of claim 1 , wherein for the first query, one of the weighted features is associated with text specified in the first query.
4 . The system of claim 1 , wherein for the first query, one of the weighted features is associated with a keyword specified in the first query.
5 . The system of claim 1 , wherein for the first query, one of the weighted features is associated with a function specified in the first query.
6 . The system of claim 1 , wherein for the first query, one of the weighted features is associated with a data source specified in the first query.
7 . The system of claim 1 , wherein the first set of data and the second set of data comprise machine-learning features.
8 . The system of claim 1 , wherein the program code is further structured to cause the processor to:
provide a notification to an author of at least one of the first query or the second query indicating that the first query and the second query are overlapping.
9 . A method for deduplicating data in a data store, comprising:
generating an intermediate representation for each of a first query and a second query, each intermediate representation comprising multiple dimensions, each dimension corresponding to a weighted feature of the query, the first query having generated a first set of data in the data store and the second query having generated a second set of data in the data store;
generating a similarity score between the intermediate representation for the first query and the intermediate representation for the second query; and
removing one of the first set of data or the second set of data in the data store based at least on the similarity score.
10 . The method of claim 9 , wherein the intermediate representation for each query comprises a vector representation.
11 . The method of claim 9 , wherein for the first query, one of the weighted features is associated with a keyword specified in the first query.
12 . The method of claim 9 , wherein for the first query, one of the weighted features is associated with a function specified in the first query.
13 . The method of claim 9 , wherein for the first query, one of the weighted features is associated with a data source specified in the first query.
14 . The method of claim 9 , wherein the first set of data and the second set of data comprise machine-learning features.
15 . The method of claim 9 , further comprising:
providing a notification to an author of at least one of the first query or the second query indicating that the first query and the second query are overlapping.
16 . A computer-readable storage medium having computer program code recorded thereon that when executed by at least one processor causes the at least one processor to perform a method comprising:
generating an intermediate representation for each of a first query and a second query, each intermediate representation comprising multiple dimensions, each dimension corresponding to a weighted feature of the query, the first query having generated a first set of data in the data store and the second query having generated a second set of data in the data store;
generating a similarity score between the intermediate representation for the first query and the intermediate representation for the second query; and
removing one of the first set of data or the second set of data in the data store based at least on the similarity score.
17 . The computer-readable storage medium of claim 16 , wherein the intermediate representation for each query comprises a vector representation.
18 . The computer-readable storage medium of claim 16 , wherein for the first query, one of the weighted features is associated with a keyword specified in the first query.
19 . The computer-readable storage medium of claim 16 , wherein for the first query, one of the weighted features is associated with a function specified in the first query.
20 . The computer-readable storage medium of claim 16 , wherein for the first query, one of the weighted features is associated with a data source specified in the first query.