METHOD AND SYSTEM FOR DATA INDEXING AND REPORTING
The present teaching relates to system, method, and programming for indexing user records. A plurality of user records are obtained, wherein each user record is associated with a user identifier and one or more data-segments. For each data-segment, one or more user identifiers associated with the data-segment are determined, a relative distance of each user identifier with respect to a previous user identifier associated with the data-segment is computed, and a variable number of bits to be used to store the relative distance of each user identifier in a bit stream are determined based on a criterion associated with the relative distance. Offset values are generated for the one or more data-segments, wherein an offset value of a data-segment indicates an index of the bit stream where information of a first user identifier associated with the data-segment is stored.
1 . A method for responding to a query, comprising:
obtaining a query from a user;
assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;
obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;
transforming the received data into a column-based format;
transmitting the transformed data to the indexing cluster; and
generating, by the indexing cluster, a response to the query based on the transformed data.
2 . The method of claim 1 , further comprising:
storing each column of the data in the column-based format in a binary file.
3 . The method of claim 2 , further comprising:
processing only data stored in the binary files to generate the response.
4 . The method of claim 1 , wherein the data in the column-based format comprises user records, and wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments.
5 . The method of claim 4 , further comprising:
generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.
6 . The method of claim 1 , wherein the indexing cluster includes multiple index processing units.
7 . The method of claim 6 , wherein each of the index processing units processes a respective portion of the transformed data to generate the response.
8 . A non-transitory, computer-readable medium having information recorded thereon for advertising, wherein the information, when read by a machine, causes the machine to perform operations comprising:
obtaining a query from a user;
assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;
obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;
transforming the received data into a column-based format;
transmitting the transformed data to the indexing cluster; and
generating, by the indexing cluster, a response to the query based on the transformed data.
9 . The medium of claim 8 , wherein the operations further comprise:
storing each column of the data in the column-based format in a binary file.
10 . The medium of claim 9 , wherein the operations further comprise:
processing only data stored in the binary files to generate the response.
11 . The medium of claim 8 , wherein the data in the column-based format comprises user records, wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments.
12 . The medium of claim 11 , wherein the operations further comprise:
generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, and wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.
13 . The medium of claim 8 , wherein the indexing cluster includes multiple index processing units.
14 . The medium of claim 13 , wherein each of the index processing units processes a respective portion of the transformed data to generate the response.
15 . A system for advertising, the system comprising:
memory storing computer program instructions; and
one or more processors that, in response to executing the computer program instructions, effectuate operations comprising:
obtaining a query from a user;
assigning the query to an indexing cluster of a plurality of indexing clusters based on a processing workload of the indexing cluster;
obtaining data from one or more data networks based on a triggering condition or a predetermined schedule;
transforming the received data into a column-based format;
transmitting the transformed data to the indexing cluster; and
generating, by the indexing cluster, a response to the query based on the transformed data.
16 . The system of claim 15 , wherein the operations further comprise:
storing each column of the data in the column-based format in a binary file.
17 . The system of claim 16 , wherein the operations further comprise:
processing only data stored in the binary files to generate the response.
18 . The system of claim 15 , wherein the data in the column-based format comprises user records, wherein each user record is associated with one of a plurality of user identifiers and one or more of a plurality of data-segments.
19 . The system of claim 18 , wherein the operations further comprise:
generating an offset stream utilized to generate the response, wherein the offset stream includes offset values for the plurality of data-segments, and wherein an offset value of a corresponding data-segment indicates an index of a bit stream where information of a first user identifier of the plurality of user identifiers associated with the corresponding data-segment is stored.
20 . The system of claim 15 , wherein the indexing cluster includes multiple index processing units, wherein each of the index processing units processes a respective portion of the transformed data to generate the response.