Data transmission filtering through a customized filter list
A system and method are presented to improve the speed of shared network processing by filtering data using a filter list. The filter list filters the data submitted for analysis to the server system. Some data is processed locally, but the filtered data is processed at the server system. The local systems process similar data, but they use distinct data hierarchies to classify the data. Hierarchy designations are utilized in the filter list to filter the data. Rules establish a unified hierarchy that is translated to local hierarchies in the filter lists. Tags are established at the server for exceptions. Tags are translated into data identifiers for inclusion in the filter lists. The size of the filter list is reduced by avoiding redundancies between the tagged data identifiers and the local hierarchy designations. The filter list creation process also creates pre-analysis data at the server to assist in its analysis.
1 . A computer-implemented method comprising:
receiving, by one or more processors and from a local system, list group identifiers and item identifiers associated with the list group identifiers;
mapping, by the one or more processors, (i) the list group identifiers to unified group identifiers based at least in part on a set of mapping rules and (ii) the item identifiers to one or more tags based at least in part on a set of keyword rules;
accessing, by the one or more processors, a configuration file in a database that defines (i) one or more processing segments to be performed by the local system, (ii) a relevant subset of the unified group identifiers, and (iii) a relevant tag of the one or more tags;
generating, by the one or more processors and based at least in part on the configuration file, a filter list customized for the local system, wherein the filter list comprises (i) a subset of the list group identifiers that are associated with the relevant subset of the unified group identifiers, and (ii) a subset of the item identifiers identified based at least in part on the one or more processing segments;
transmitting, by the one or more processors and to the local system, the filter list, causing the local system to apply the filter list to pre-analysis data to generate filtered data;
responsive to receiving, by the one or more processors and from the local system, the filtered data, generating, by the one or more processors and based at least in part on the one or more processing segments and the filtered data, augmented data comprising (i) the filtered data and (ii) one or more item specific datasets that are relevant to the filtered data;
automatically updating, by the one or more processors, based at least in part on the augmented data, and for the local system, available benefits in the database; and
responsive to updating the database, triggering, by the one or more processors, a resource exchange with the local system.
2 . The computer-implemented method of claim 1 , further comprising:
responsive to detecting changes to data required by a server system associated with the local system, updating the filter list.
3 . The computer-implemented method of claim 1 , wherein the filtered data is received and processed by an analysis server that is distinct from a filter list server that generated the filter list.
4 . The computer-implemented method of claim 1 , wherein the one or more processing segments are defined based at least in part on configuration data comprising data required for the one or more processing segments.
5 . The computer-implemented method of claim 4 , wherein (a) the configuration data defines a first processing segment of the one or more processing segments that (i) includes a first unified group identifier of the unified group identifiers, and (ii) excludes a first tag of the one or more tags, and (b) the first processing segment is processed against a first subset of the filtered data that is both associated with the first unified group identifier and associated with any tag of the one or more tags that is different from the first tag.
6 . The computer-implemented method of claim 4 , wherein (a) the configuration data defines a second processing segment of the one or more processing segments that excludes a second unified group identifier of the unified group identifiers and includes a second tag of the one or more tags, and (b) the second processing segment is processed against a second subset of the filtered data that is both associated with the second unified group identifier and associated with any tag of the one or more tags that is different from the second tag.
7 . The computer-implemented method of claim 6 , wherein the filtered data includes summary data summarizing a plurality of items for a subset of the list group identifiers that are not identified in the filter list.
8 . The computer-implemented method of claim 4 , further comprising:
generating pre-analysis data for the one or more processing segments,
wherein the pre-analysis data comprises (a) one or more additional list group identifiers for local systems associated with the unified group identifiers, and (b) non-redundant item identifiers associated with the one or more tags and not associated with any of the relevant subset of the unified group identifiers for the one or more processing segments.
9 . The computer-implemented method of claim 8 , wherein generating the augmented data corresponding to the filtered data is further based at least in part on the pre-analysis data.
10 . The computer-implemented method of claim 9 , further comprising:
receiving, from a second local system, unfiltered data; and
processing the unfiltered data based at least in part on the pre-analysis data.
11 . The computer-implemented method of claim 1 , wherein (1) the list group identifiers comprise list departments and list sub-departments, (2) the unified group identifiers comprise one or more unified departments and one or more unified sub-departments, (3) at least one list sub-department is associated with one of the one or more unified departments, and (4) at least one list department is associated with one of the one or more unified sub-departments.
12 . The computer-implemented method of claim 11 , wherein the local system is associated with a location, the item identifiers are associated with one or more transaction items available at the location, and the list departments and list sub-departments are used to categorize the one or more transaction items.
13 . The computer-implemented method of claim 1 , wherein the subset of the item identifiers are associated with one or more unified group identifiers distinct from the relevant subset of the unified group identifiers.
14 . The computer-implemented method of claim 1 , wherein mapping the list group identifiers to unified group identifiers and (ii) the item identifiers to one or more tags comprises:
generating association data by associating the list group identifiers with the unified group identifiers based on the set of mapping rules,
storing the association data in a master list,
associating the item identifiers with a first subset of the one or more tags based on associations between the item identifiers and the list group identifiers,
storing the item identifiers and the first subset of the one or more tags in the master list,
identifying a subset of the item identifiers that match list group identifiers and key text found in the set of keyword rules,
associating the subset of the item identifiers with a second subset of the one or more tags based on the set of keyword rules, and
storing the subset of the item identifiers and the second subset of the one or more tags based in the master list.
15 . The computer-implemented method of claim 14 , wherein a first tag of the one or more tags and a second tag of the one or more tags are considered to conflict when both the first tag and the second tag cannot be associated with a single item identifier.
16 . The computer-implemented method of claim 15 , further comprising resolving a conflict when the first tag is associated with a first item identifier by a first set of rules and the second tag is associated with the first item identifier by a second set of rules, wherein the conflict is resolved by associating the first item identifier with the first tag when a first weight associated with the first set of rules is greater than a second weight associated with the second set of rules.
17 . The computer-implemented method of claim 16 , wherein one or more weights are assigned to a set of rules based at least in part on a specificity of the set of rules relative to one or more other sets of rules.
18 . The computer-implemented method of claim 12 , wherein the one or more processing segments are associated with one or more available alterations to the pre-analysis data.
19 . A system comprising one or more processors and at least one memory storing processor executable instructions that, when executed by the one or more processors, cause the one or more processors to:
receive, from a local system, list group identifiers and item identifiers associated with the list group identifiers;
map (i) the list group identifiers to unified group identifiers based at least in part on a set of mapping rules and (ii) the item identifiers to one or more tags based at least in part on a set of keyword rules;
access a configuration file in a database that defines (i) one or more processing segments to be performed by the local system, (ii) a relevant subset of the unified group identifiers, and (iii) a relevant tag of the one or more tags;
generate, based at least in part on the configuration file, a filter list customized for the local system, wherein the filter list comprises (i) a subset of the list group identifiers that are associated with the relevant subset of the unified group identifiers, and (ii) a subset of the item identifiers identified based at least in part on the one or more processing segments;
transmit, to the local system, the filter list, causing the local system to apply the filter list to pre-analysis data to generate filtered data;
responsive to receiving, from the local system, the filtered data, generate, based at least in part on the one or more processing segments and the filtered data, augmented data comprising (i) the filtered data and (ii) one or more item specific datasets that are relevant to the filtered data;
automatically update, for the local system and based at least in part on the augmented data, available benefits in the database; and
responsive to updating the database, trigger a resource exchange with the local system.
20 . One or more non-transitory computer-readable storage media including instructions that, when executed by one or more processors, cause the one or more processors to:
receive, from a local system, list group identifiers and item identifiers associated with the list group identifiers;
map (i) the list group identifiers to unified group identifiers based at least in part on a set of mapping rules and (ii) the item identifiers to one or more tags based at least in part on a set of keyword rules;
access a configuration file in a database that defines (i) one or more processing segments to be performed by the local system, (ii) a relevant subset of the unified group identifiers, and (iii) a relevant tag of the one or more tags;
generate, based at least in part on the configuration file, a filter list customized for the local system, wherein the filter list comprises (i) a subset of the list group identifiers that are associated with the relevant subset of the unified group identifiers, and (ii) a subset of the item identifiers identified based at least in part on the one or more processing segments;
transmit, to the local system, the filter list, causing the local system to apply the filter list to pre-analysis data to generate filtered data;
responsive to receiving, from the local system, the filtered data, generate, based at least in part on the one or more processing segments and the filtered data, augmented data comprising (i) the filtered data and (ii) one or more item specific datasets that are relevant to the filtered data;
automatically update, for the local system and based at least in part on the augmented data, available benefits in the database; and
responsive to updating the database, trigger a resource exchange with the local system.