Self-adaptive data source aggregation system and method
A data aggregation system for aggregation of data from at least two data sources includes: a data aggregation unit configured to determine a data aggregation action to be executed by the system, in response to a message received by at least one data source device, each message comprising data; a consistency checking unit configured to check the consistency of each message received from a data source device using one or more consistency rules and determine an action to be executed by said data aggregation unit depending on the consistency checking; and an accuracy metric determination unit configured to determine an accuracy metric for the executed action based on feedback values received from a set of at least one user device, said data aggregation unit comprising a rule updating unit configured to update said consistency rules based on the accuracy metrics determined for the executed actions.
1. A data aggregation system comprising:
a first data source computing device, and a second data source computing device;
a set of user computing devices; and
a data aggregation computing device having (i) an interface communicatively coupled with the first and second data source computing devices and the set of user computing devices, (ii) a memory storing a database, and (iii) a processor configured to:
receive, from the first data source computing device, a first message comprising first data corresponding to a target portion of the database;
receive, from the second data source computing device, a second message comprising second data corresponding to the target portion of the database, the second data conflicting with the first data;
in response to receiving the first and second messages, using (i) one or more consistency rules, and (ii) previously received data stored in the database to select
a data aggregation action to be executed, from (i) storing the first message and discarding the second message, or (ii) storing the second message and discarding the first message;
update the database according to the determined data aggregation action;
in response to updating the database, send respective notifications to each of the set of user computing devices, the notifications including a list of selectable feedback values;
determine an accuracy metric for the executed data aggregation action based on selected feedback values received from the set of user computing devices; and
update the one or more consistency rules based on the accuracy metric determined for the executed data aggregation action.
2. The data aggregation system of claim 1 , wherein the first data has a first source data format, the second data has a second source data format, and the database has a target data format; and
wherein the processor is further configured to:
parse the first and second messages in the respective first and second source data formats; and
translate at least a part of the first and second data into the target data format.
3. The data aggregation system of claim 1 , wherein the first message comprises an identifier identifying the first data source computing device.
4. The data aggregation system of claim 1 , wherein the processor is further configured to:
execute a value function using one or more parameters derived from the selected feedback values as inputs, and providing an output value; and
determine the accuracy metric for the executed data aggregation action based on the output value.
5. The data aggregation system of claim 4 , wherein the one or more parameters comprise a parameter representing a number of occurrences of a given feedback value received from the set of user devices.
6. The data aggregation system of claim 4 , wherein the given feedback value is a value indicating a rejection of the data aggregation action.
7. The data aggregation system of claim 4 , wherein the output value represents a reward value for the data aggregation action.
8. The data aggregation system of claim 1 , wherein the consistency rules define at least one of (i) data cleaning conditions, (ii) data merging conditions, (iii) data consolidation conditions, and (iv) received feedback values for previous data aggregation actions.
9. A method of aggregating data, wherein the method comprises:
storing, at a data aggregation computing device, a database;
receiving, from a first data source computing device, a first message including first data corresponding to a target portion of the database;
receiving, from a second data source computing device, a second message including second data corresponding to the target portion of the database, the second data conflicting with the first data;
in response to receiving the first and second messages, using (i) one or more consistency rules, and (ii) previously received data stored in the database to select
a data aggregation action to be executed, from (i) storing the first message and discarding the second message, or (ii) storing the second message and discarding the first message;
updating the database according to the determined data aggregation action;
in response to updating the database, sending respective notifications to each of a set of user computing devices, the notifications including a list of selectable feedback values;
determining an accuracy metric for the executed action based on selected feedback values received from the set of user computing devices; and
updating said consistency rules based on the accuracy metric determined for the executed data aggregation action.
10. The method of claim 9 , further comprising:
parsing the first message received from the first data source computing device in a first source data format, and parsing the second message received from the second data source computing device in a second source data format; and
translating at least a part of the first data and the second data into a target data format corresponding to the database.
11. The method of claim 9 , further comprising:
executing a value function using one or more parameters derived from the selected feedback values as inputs, and providing an output value; and
determining the accuracy metric for the executed data aggregation action based on the output value.
12. The method of claim 11 , wherein the one or more parameters comprise a parameter representing a number of occurrences of a given feedback value received from the set of user devices.
13. A data aggregation computing device, comprising:
an interface communicatively coupled with a first data source computing device, a second data source computing device, and a set of user computing devices;
a memory storing a database; and
a processor configured to:
receive, from the first data source computing device, a first message comprising first data corresponding to a target portion of the database;
receive, from the second data source computing device, a second message comprising second data corresponding to the target portion of the database, the second data conflicting with the first data:
in response to receiving the first aid second messages, using (i) one or more consistency rules, and (ii) previously received data stored in the database to select
a data aggregation action to be executed, from (i) storing the first message and discarding the second message, or (ii) storing the second message and discarding the first message;
update the database according to the determined data aggregation action;
in response to updating the database, send respective notifications to each of the set of user computing devices, the notifications including a list of selectable feedback values;
determine an accuracy metric for the executed data aggregation action based on selected feedback values received from the set of user computing devices; and
update the one or more consistency rules based on the accuracy metric determined for the executed data aggregation action.