Indexing a data corpus to a set of multidimensional points
Indexing a data corpus to a set of multidimensional points, including: generating a set of points in a multidimensional space; identifying, for each sample in a plurality of samples in a data corpus, a nearest point in the set of points; and generating an index mapping each sample with the nearest point in the set of points.
1. A method of indexing a data corpus to a set of multidimensional points, the method comprising:
generating a set of points comprising a Sobol sequence in a multidimensional space;
identifying, for each sample in a plurality of samples in a data corpus, a nearest point in the set of points;
generating an index mapping each sample with the nearest point in the Sobol sequence;
receiving a request for a number of samples from the data corpus;
selecting a subset of points from the Sobol sequence, wherein the subset of points includes a number of points equal to the number of samples, and wherein the subset of points are sequential from a beginning of the Sobol sequence;
providing, in response to the request and based on mappings in the index to the subset of points, a subset of the plurality of samples corresponding to the subset of points; and
generating one or more models by training the one or more models using the subset of the plurality of samples.
2. The method of claim 1 , wherein identifying the nearest point is based on a Euclidean distance.
3. The method of claim 1 , wherein the data corpus comprises one or more categorical fields, and wherein generating the index comprises generating a plurality of indexes each corresponding to a different permutation of categorical values for the one or more categorical fields.
4. The method of claim 1 , further comprising maintaining a data structure describing, for each point in the plurality of points, whether any sample of the plurality of samples is mapped to a particular point.
5. The method of claim 4 , wherein the data structure is one of a plurality of data structures each corresponding to a different permutation of categorical values for one or more categorical fields of the data corpus.
6. The method of claim 4 , further comprising causing one or models to be retrained in response to detecting, based on the data structure, a mapping to a previously unmapped point in the set of points.
7. The method of claim 1 , further comprising:
receiving a request for one or more reduced dimensionality samples;
mapping the set of points in the multidimensional space to another set of points in a lower dimensional space relative to the multidimensional space; and
selecting, based on the other set of points, the one or more samples from the data corpus.
8. A system for indexing a data corpus to a set of multidimensional points, the system configured to perform steps comprising:
generating a set of points comprising a Sobol sequence in a multidimensional space;
identifying, for each sample in a plurality of samples in a data corpus, a nearest point in the set of points;
generating an index mapping each sample with the nearest point in the Sobol sequence;
receiving a request for a number of samples from the data corpus;
selecting a subset of points from the Sobol sequence, wherein the subset of points includes a number of points equal to the number of samples, and wherein the subset of points are sequential from a beginning of the Sobol sequence;
providing, in response to the request and based on mappings in the index to the subset of points, a subset of the plurality of samples corresponding to the subset of points; and
generating one or more models by training the one or more models using the subset of the plurality of samples.
9. The system of claim 8 , wherein identifying the nearest point is based on a Euclidean distance.
10. The system of claim 8 , wherein the data corpus comprises one or more categorical fields, and wherein generating the index comprises generating a plurality of indexes each corresponding to a different permutation of categorical values for the one or more categorical fields.
11. The system of claim 8 , wherein the steps further comprise maintaining a data structure describing, for each point in the plurality of points, whether any sample of the plurality of samples is mapped to a particular point.
12. The system of claim 11 , wherein the data structure is one of a plurality of data structures each corresponding to a different permutation of categorical values for one or more categorical fields of the data corpus.
13. The system of claim 11 , wherein the steps further comprise causing one or models to be retrained in response to detecting, based on the data structure, a mapping to a previously unmapped point in the set of points.
14. The system of claim 8 , wherein the steps further comprise:
receiving a request for one or more reduced dimensionality samples;
mapping the set of points in the multidimensional space to another set of points in a lower dimensional space relative to the multidimensional space; and
selecting, based on the other set of points, the one or more samples from the data corpus.
15. A computer program product comprising a non-transitory computer readable medium storing computer program instructions for indexing a data corpus to a set of multidimensional points that, when executed, cause a computer system to perform steps comprising:
generating a set of points comprising a Sobol sequence in a multidimensional space;
generating an index mapping each sample with the nearest point in the Sobol sequence;
generating an index mapping each sample with the nearest point in the set of points;
receiving a request for a number of samples from the data corpus;
selecting a subset of points from the Sobol sequence, wherein the subset of points includes a number of points equal to the number of samples, and wherein the subset of points are sequential from a beginning of the Sobol sequence;
providing, in response to the request and based on mappings in the index to the subset of points, a subset of the plurality of samples corresponding to the subset of points; and
generating one or more models by training the one or more models using the subset of the plurality of samples.
16. The computer program product of claim 15 , wherein identifying the nearest point is based on a Euclidean distance.
17. The computer program product of claim 15 , wherein the data corpus comprises one or more categorical fields, and wherein generating the index comprises generating a plurality of indexes each corresponding to a different permutation of categorical values for the one or more categorical fields.
18. The computer program product of claim 15 , wherein the steps further comprise maintaining a data structure describing, for each point in the plurality of points, whether any sample of the plurality of samples is mapped to a particular point.
19. The computer program product of claim 18 , wherein the data structure is one of a plurality of data structures each corresponding to a different permutation of categorical values for one or more categorical fields of the data corpus.
20. The computer program product of claim 15 , wherein the steps further comprise causing one or models to be retrained in response to detecting, based on the data structure, a mapping to a previously unmapped point in the set of points.