Single-pass distributed sampling from block-partitioned matrices
A computer-implemented method is provided that includes identifying an input dataset formatted as an input matrix, the input matrix including a plurality of rows and a plurality of columns. The computer-implemented method also includes dividing the input matrix into a plurality of input matrix blocks. Further, the computer-implemented method includes distributing the input matrix blocks to a plurality of different machines across a distributed filesystem, and sampling, by at least two of the different machines in parallel, at least two of the input matrix blocks. Finally, the computer-implemented method includes generating at least one sample matrix based on the sampling of the at least two of the input matrix blocks.
1. A computer-implemented method, comprising:
identifying an input dataset formatted as an input matrix;
dividing the input matrix into a plurality of input matrix blocks;
sampling at least two of the input matrix blocks, including:
reading each sampled input matrix block by iterating over partitions of the sampled input matrix block,
for each of the partitions of the sampled input matrix block, determining occurrences of the partition in at least one sample matrix, and
for each occurrence of one of the partitions in the at least one sample matrix, emitting a record; and
generating at least one sample matrix, including:
grouping the emitted records,
sorting each of the groups of the emitted records, and
re-blocking each of the sorted groups of the emitted records.
2. The computer-implemented method of claim 1 , wherein the partitions comprise one of rows and columns.
3. The computer-implemented method of claim 1 , wherein each of the emitted records includes a sample matrix identifier that identifies a particular sample matrix of the at least one sample matrix, a vertical block index, a position, and a partial observation.
4. The computer-implemented method of claim 1 , wherein the emitted records are grouped based on sample matrix identifiers and vertical block indices of the emitted records.
5. The computer-implemented method of claim 1 , wherein each of the groups of the emitted records is sorted based on positions of the emitted records in the group.
6. The computer-implemented method of claim 1 , wherein the at least one sample matrix is generated based on the re-blocking of the sorted groups of the emitted records.
7. A computer program product for performing single-pass distributed sampling from block-partitioned matrices, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a processor to cause the processor to:
identify, by the processor, an input dataset formatted as an input matrix;
divide, by the processor, the input matrix into a plurality of input matrix blocks;
sampling, by the processor, at least two of the input matrix blocks, including:
reading each sampled input matrix block by iterating over partitions of the sampled input matrix block,
for each of the partitions of the sampled input matrix block, determining occurrences of the partition in at least one sample matrix, and
for each occurrence of one of the partitions in the at least one sample matrix, emitting a record; and
generate, by the processor, at least one sample matrix, including:
grouping the emitted records,
sorting each of the groups of the emitted records, and
re-blocking each of the sorted groups of the emitted records.
8. The computer program product of claim 7 , wherein the partitions comprise one of rows and columns.
9. The computer program product of claim 7 , wherein each of the emitted records includes a sample matrix identifier that identifies a particular sample matrix of the at least one sample matrix, a vertical block index, a position, and a partial observation.
10. The computer program product of claim 7 , wherein the emitted records are grouped based on sample matrix identifiers and vertical block indices of the emitted records.
11. The computer program product of claim 7 , wherein each of the groups of the emitted records is sorted based on positions of the emitted records in the group.
12. The computer program product of claim 7 , wherein the at least one sample matrix is generated based on the re-blocking of the sorted groups of the emitted records.
13. A system, comprising:
one or more processors and logic integrated with the processors, executable by the processors, or integrated with and executable by the processors, the logic being configured to:
identify an input dataset formatted as an input matrix;
divide the input matrix into a plurality of input matrix blocks;
sample at least two of the input matrix blocks, including:
reading each sampled input matrix block by iterating over partitions of the sampled input matrix block,
for each of the partitions of the sampled input matrix block, determining occurrences of the partition in at least one sample matrix, and
for each occurrence of one of the partitions in the at least one sample matrix, emitting a record; and
generate at least one sample matrix, including:
grouping the emitted records,
sorting each of the groups of the emitted records, and
re-blocking each of the sorted groups of the emitted records.
14. The system of claim 6 , wherein the partitions comprise one of rows and columns.
15. The system of claim 13 , wherein each of the emitted records includes a sample matrix identifier that identifies a particular sample matrix of the at least one sample matrix, a vertical block index, a position, and a partial observation.