Granted Patent B1

US 8,935,201 · App. 14/254,757 · Granted Jan 13, 2015

Abstract

Claims (56)

1. A computer system configured to obtain changed data from a data source, the computer system comprising:

one or more hardware computer processors configured to execute code in order to cause the system to:

obtain information indicating a plurality of groupings of data stored in one or more files or databases in a data source, wherein the plurality of groupings is based on date and time of timestamps of respective data items such that an uneven distribution of data items included in the plurality of groupings is provided, each of the groupings associated with a different data item attribute such that respective groupings each include data items having the corresponding data item attribute, wherein the information indicates a current quantity of data items included in each of the plurality of groupings at a current time, wherein the information is obtained using a query of a timestamp data item attribute;

access historical grouping information indicating a historical quantity of data items included in at least some of the plurality of groupings at a previous time, wherein for a first subset of the groupings a current quantity of data items at the current time and a historical quantity of data items are not equal, and for another subset of the groupings a current quantity of data items and a historical quantity of data items are equal;

compare the quantity of data items in the same groupings at the previous time and the current time;

determine, based on said comparing, that a first current quantity of data items included in a first grouping and a first historical quantity of data items included in a first historical grouping of data items are not equal, indicating that one or more data items of the first grouping have changed; and

in response to determining that the first current quantity of data items is higher or lower than the first historical quantity of data items:

access data items included in the first grouping from the data source;

compare the data items included in the first grouping to data items of a corresponding local version of the first grouping to determine which data items of the first grouping from the data source have changed;

extract the changed data items of the first grouping; and

forward the extracted changed data items to a destination system.

2. The system of claim 1 , wherein the plurality of groupings is based on the timestamps of the respective data items, wherein the timestamps indicate respective times at which data items were last updated.

3. The system of claim 2 , wherein the timestamps of the respective data items include a date and a time at which the respective data items were last updated, and wherein the plurality of groupings is based on only the date of the timestamps of the respective data items.

4. The system of claim 3 , wherein each of the plurality of groupings is associated with a different date.

5. The system of claim 3 , wherein the dates associated with each of the plurality of groupings are within a range of dates.

6. The system of claim 1 , wherein the corresponding local version of the first grouping comprises a copy of the data items included in the first grouping at a first time prior to said obtaining the information indicating the plurality of groupings of the data.

7. The system of claim 1 , wherein, in response to determining that the first current quantity of data items is higher than the first historical quantity of data items, the code is further configured to cause the system to:

identify the changed data items as added or updated data items; and

forward the changed data items to the destination system to be stored.

8. The system of claim 1 , wherein, in response to determining that the first current quantity of data items is lower than the first historical quantity of data items, the code is further configured to cause the system to:

identify the changed data items as deleted data items; and

forward the changed data items to the destination system to be removed.

9. The system of claim 1 , wherein the code is further configured to:

assign a unique identifier to each of the data items included in the first grouping; and

determine whether a first changed data item of the changed data items is a new data item or an updated data item based on the unique identifier associated with the first changed data item.

10. The system of claim 1 , wherein a data item included in the first grouping is a row in the one or more databases or a line in the one or more files.

11. The system of claim 1 , wherein the code is further configured to cause the system to:

obtain the information indicating the plurality of groupings of the data stored in the one or more files in the data source using a first adapter; and

obtain the information indicating the plurality of groupings of the data stored in the one or more databases in the data source using a second adapter.

12. The system of claim 1 , wherein the code is further configured to cause the system to obtain the information indicating the plurality of groupings of the data at an interval.

13. The system of claim 1 , wherein the code is further configured to cause the system to obtain the information indicating the plurality of groupings of the data in response to receiving a request from the destination system.

14. A method of obtaining changed data from a data source, the method comprising:

obtaining, by one or more hardware computer processors, information indicating a plurality of groupings of data stored in one or more files or databases in a data source, wherein the plurality of groupings is based on date and time of timestamps of respective data items such that an uneven distribution of data items included in the plurality of groupings is provided, each of the groupings associated with a different data item attribute such that respective groupings each include data items having the corresponding data item attribute, wherein the information indicates a current quantity of data items included in each of the plurality of groupings at a current time, wherein the information is obtained using a query of a timestamp data item attribute;

accessing, by the one or more hardware computer processors, historical grouping information indicating a historical quantity of data items included in at least some of the plurality of groupings at a previous time, wherein for a first subset of the groupings a current quantity of data items at the current time and a historical quantity of data items are not equal, and for another subset of the groupings a current quantity of data items and a historical quantity of data items are equal;

comparing, by the one or more hardware computer processors, the quantity of data items in the same groupings at the previous time and the current time;

determining, by the one or more hardware computer processors, based on said comparing, that a first current quantity of data items included in a first grouping and a first historical quantity of data items included in a first historical grouping of data items are not equal, indicating that one or more data items of the first grouping have changed; and

in response to determining that the first current quantity of data items is higher or lower than the first historical quantity of data items:

accessing, by the one or more hardware computer processors, data items included in the first grouping from the data source;

comparing, by the one or more hardware computer processors, the data items included in the first grouping to data items of a corresponding local version of the first grouping to determine which data items of the first grouping from the data source have changed;

extracting, by the one or more hardware computer processors, the changed data items of the first grouping; and

forwarding, by the one or more hardware computer processors, the extracted changed data items to a destination system.

15. The method of claim 14 , wherein the plurality of groupings is based on the timestamps of the respective data items, wherein the timestamps indicate respective times at which data items were last updated.

16. The method of claim 14 , wherein the corresponding local version of the first grouping comprises a copy of data items included in the first grouping at a first time prior to said obtaining the information indicating the plurality of groupings of the data.

17. The method of claim 14 , wherein a data item included in the first grouping is a row in the one or more databases or a line in the one or more files.

18. A non-transitory computer readable storage medium comprising instructions for obtaining changed data from a data source that cause a computer processor to:

obtain information indicating a plurality of groupings of data stored in one or more files or databases in a data source, wherein the plurality of groupings is based on date and time of timestamps of respective data items such that an uneven distribution of data items included in the plurality of groupings is provided, each of the groupings associated with a different data item attribute such that respective groupings each include data items having the corresponding data item attribute, wherein the information indicates a current quantity of data items included in each of the plurality of groupings at a current time, wherein the information is obtained using a query of a timestamp data item attribute;

access historical grouping information indicating a historical quantity of data items included in at least some of the plurality of groupings at a previous time, wherein for a first subset of the groupings a current quantity of data items at the current time and a historical quantity of data items are not equal, and for another subset of the groupings a current quantity of data items and a historical quantity of data items are equal;

compare the quantity of data items in the same groupings at the previous time and the current time;

determine, based on said comparing, that a first current quantity of data items included in a first grouping and a first historical quantity of data items included in a first historical grouping of data items are not equal, indicating that one or more data items of the first grouping have changed; and

in response to determining that the first current quantity of data items is higher or lower than the first historical quantity of data items:

access data items included in the first grouping from the data source;

compare the data items included in the first grouping to data items of a corresponding local version of the first grouping to determine which data items of the first grouping from the data source have changed;

extract the changed data items of the first grouping; and

forward the extracted changed data items to a destination system.

19. The computer readable storage medium of claim 18 , wherein the plurality of groupings is based on the timestamps of the respective data items, wherein the timestamps indicate respective times at which data items were last updated.

20. The computer readable storage medium of claim 18 , wherein the corresponding local version of the first grouping comprises a copy of data items included in the first grouping at a first time prior to said obtaining the information indicating the plurality of groupings of the data.

Assignments (8)