Scalable recursive computation for pattern identification across distributed data processing nodes

1. A method comprising:

receiving results of two or more intermediate long-tail histogram computations performed on respective ones of a plurality of datasets in respective ones of a plurality of data zones associated with a plurality of distributed processing nodes configured to communicate over at least one network, wherein each of the plurality of datasets is locally accessible within at least one of the plurality of data zones by at least one distributed processing node in the plurality of distributed processing nodes, and wherein the results for a given one of the two or more intermediate long-tail histogram computations comprise information characterizing a local head, a local body and a local long tail of a given intermediate long-tail histogram computed for a given one of the plurality of datasets in a given one of the plurality of data zones associated with a given one of the plurality of distributed processing nodes;

wherein the given intermediate long-tail histogram computation comprises:

separating a corresponding one of the datasets into groups of data based on a particular type of long-tail histogram representation; and

representing each of the groups of data in a privacy-preserving profile comprising a local long-tail histogram;

wherein the local long-tail histogram is returned from a local one of the distributed processing nodes in a corresponding one of the data zones to an initiating distributed processing node;

performing at least one global long-tail histogram computation based at least in part on the results of the two or more intermediate long-tail histogram computations, wherein the at least one global long-tail histogram computation comprises determining at least one of a global head, a global body and a global long tail for a global long-tail histogram characterizing information across the plurality of datasets; and

utilizing results of the two or more intermediate long-tail histogram computations and the least one global long-tail histogram computation to identify one or more patterns in the plurality of datasets;

wherein the method is performed by at least one processing device comprising a processor coupled to a memory.

2. The method of claim 1 further comprising:

repeating at least a subset of the receiving, performing and utilizing in each of a plurality of iterations; and

passing a result of the at least one global long-tail histogram computation in a first one of the iterations as an input to the two or more intermediate long-tail histogram computations in a second one of the iterations.

3. The method of claim 2 wherein in the course of multiple iterations, items present in one or more local long tails of respective local long-tail histograms computed for respective ones of the data sets obtain statistical significance in one or more results of the at least one global long-tail histogram computation.

4. The method of claim 2 wherein in the course of multiple iterations, items present in one or more local long tails of respective local long-tail histograms become part of at least one of the global head and the global body of the at least one global long-tail histogram.

5. The method of claim 1 wherein the two or more intermediate long-tail histogram computations are initiated by an initiating distributed processing node.

6. The method of claim 5 wherein the initiating distributed processing node is configured to perform the at least one global long-tail histogram computation.

7. The method of claim 1 wherein the at least one global long-tail histogram computation is performed at a same one of the distributed processing nodes that performs one of the two or more intermediate long-tail histogram computations.

8. The method of claim 1 wherein the two or more intermediate long-tail histogram computations comprise respective local long-tail histogram computations performed using the datasets locally accessible in the respective data zones.

9. The method of claim 1 wherein the given intermediate long-tail histogram computation comprises:

generating a histogram comprising a plurality of slices;

sorting the slices in a designated order to obtain a sorted histogram;

determining a head index for the sorted histogram;

determining a body index for the sorted histogram;

calculating a tail summary for the sorted histogram for all items of the sorted histogram having a derived value falling within a tail portion as defined by the body index; and

returning as a result of the given intermediate long-tail histogram computation a long-tail histogram characterized by the head index, the body index and the tail summary.

10. The method of claim 9 wherein tail summary represents the local long tail of the long-tail histogram as a single entity on an x-axis, having a y-axis value that represents a summary of derived values for the local long tail.

11. The method of claim 1 wherein utilizing results of the two or more intermediate long-tail histogram computations and the at least one global long-tail histogram computation to identify patterns in the plurality of datasets comprising identifying at least one micro-pattern that was not a statistically significant pattern in any of the individual datasets.

12. The method of claim 1 wherein the initiating distributed processing node aggregates the local long-tail histograms of the respective datasets into uthe global long-tail histogram also having groups of data based on the particular type of long-tail histogram representation.

13. The method of claim 1 wherein the initiating distributed processing node further initiates one or more additional iterations of intermediate long-tail histogram computations in order to at least one of zoom in or zoom out of at least one of the groups of data of the global long-tail histogram and wherein results of intermediate long-tail histogram computations of the additional iterations are utilized to generate respective updated global long-tail histograms each having different groups of data.

14. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device causes said at least one processing device:

to receive results of two or more intermediate long-tail histogram computations performed on respective ones of a plurality of datasets in respective ones of a plurality of data zones associated with a plurality of distributed processing nodes configured to communicate over at least one network, wherein each of the plurality of datasets is locally accessible within at least one of the plurality of data zones by at least one distributed processing node in the plurality of distributed processing nodes, and wherein the results for a given one of the two or more intermediate long-tail histogram computations comprise information characterizing a local head, a local body and a local long tail of a given intermediate long-tail histogram computed for a given one of the plurality of datasets in a given one of the plurality of data zones associated with a given one of the plurality of distributed processing nodes;

wherein the given intermediate long-tail histogram computation comprises:

separating a corresponding one of the datasets into groups of data based on a particular type of long-tail histogram representation; and

representing each of the groups of data in a privacy-preserving profile comprising a local long-tail histogram;

wherein the local long-tail histogram is returned from a local one of the distributed processing nodes in a corresponding one of the data zones to an initiating distributed processing node;

to perform at least one global long-tail histogram computation based at least in part on the results of the two or more intermediate long-tail histogram computations, wherein the at least one global long-tail histogram computation comprises determining at least one of a global head, a global body and a global long tail for a global long-tail histogram characterizing information across the plurality of datasets; and

to utilize results of the two or more intermediate long-tail histogram computations and the least one global long-tail histogram computation to identify one or more patterns in the plurality of datasets.

15. The computer program product of claim 14 wherein at least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations and wherein a result of the at least one global long-tail histogram computation in a first one of the iterations is passed as an input to the two or more intermediate long-tail histogram computations in a second one of the iterations.

16. The computer program product of claim 14 wherein in the course of multiple iterations, items present in one or more local long tails of respective local long-tail histograms become part of at least one of the global head and the global body of the global long-tail histogram.

17. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

wherein said at least one processing device is configured:

to receive results of two or more intermediate long-tail histogram computations performed on respective ones of a plurality of datasets in respective ones of a plurality of data zones associated with a plurality of distributed processing nodes configured to communicate over at least one network, wherein each of the plurality of datasets is locally accessible within at least one of the plurality of data zones by at least one distributed processing node in the plurality of distributed processing nodes, and wherein the results for a given one of the two or more intermediate long-tail histogram computations comprise information characterizing a local head, a local body and a local long tail of a given intermediate long-tail histogram computed for a given one of the plurality of datasets in a given one of the plurality of data zones associated with a given one of the plurality of distributed processing nodes;

wherein the given intermediate long-tail histogram computation comprises:

separating a corresponding one of the datasets into groups of data based on a particular type of long-tail histogram representation; and

representing each of the groups of data in a privacy-preserving profile comprising a local long-tail histogram;

wherein the local long-tail histogram is returned from a local one of the distributed processing nodes in a corresponding one of the data zones to an initiating distributed processing node;

to perform at least one global long-tail histogram computation based at least in part on the results of the two or more intermediate long-tail histogram computations, wherein the at least one global long-tail histogram computation comprises determining at least one of a global head, a global body and a global long tail for a global long-tail histogram characterizing information across the plurality of datasets; and

to utilize results of the two or more intermediate long-tail histogram computations and the least one global long-tail histogram computation to identify one or more patterns in the plurality of datasets.

18. The apparatus of claim 17 wherein at least a subset of the receiving, performing and utilizing are repeated in each of a plurality of iterations and wherein a result of the at least one global long-tail histogram computation in a first one of the iterations is passed as an input to the two or more intermediate long-tail histogram computations in a second one of the iterations.

19. The apparatus of claim 17 wherein in the course of multiple iterations, items present in one or more local long tails of respective local long-tail histograms become part of at least one of the global head and the global body of the global long-tail histogram.