Method and apparatus for building index of source data
View Patent ↗An online index building operation is disclosed for building an index from source data with minimal loss of availability to the source data. The source data can be maintained in a relational database system, such as in a tablespace of a DB2® environment. The disclosed operation creates a consistent image of the source data as of a point-in-time and creates an index from the consistent image. Then, the disclosed operation repeats the acts of making the image consistent as of a subsequent point-in-time and updating the index to reflect the subsequent consistent image until substantially caught up with the current changes to the source data. If not caught up, the disclosed operation continues unless it is falling behind at which point the operation terminates. If it is caught up, the disclosed operation locks access to the source data, updates the image to reflect any final changes, updates the index, and allows access to the index.
1. An index building method, comprising:
creating a snapshot image of source data at a point-in-time;
making the snapshot image consistent with the source data as of the point-in-time;
creating the index from the consistent image;
making the consistent image consistent with the source data as of a subsequent point-in-time; and
updating the index to reflect the subsequent consistent image.
2. The method of claim 1 , wherein the act of creating the snapshot image of the source data at the point-in-time comprises creating the snapshot image in a manner that does not substantially block access to the source data.
3. The method of claim 1 , wherein the act of making the snapshot consistent with the source data as of the point-in-time comprises:
identifying one or more changes to the source data as of the point-in-time; and
updating the snapshot image to reflect the identified changes.
4. The method of claim 3 , wherein the acts of identifying, updating and creating are repeated during a single index building operation.
5. The method of claim 3 , wherein the acts of identifying the one or more changes and updating the snapshot image further comprises:
identifying the one or more changes to the source data as of the point-in-time that do not exist in the image;
updating the snapshot image to reflect the identified changes; and
removing all updates from the snapshot image made after the point-in-time.
6. The method of claim 5 , wherein the acts of identifying the one or more changes and updating the snapshot image further comprises:
identifying one or more in-flight units-of-work associated with the updates to the source data as of the point-in-time; and
removing each of the updates associated with each identified in-flight unit-of-work from the snapshot image.
7. The method of claim 1 , wherein the act of making the image consistent comprises obtaining one or more log records from a database management system log.
8. The method of claim 1 , wherein the act of making the image consistent comprises asynchronously capturing one or more log records associated with the source data.
9. The method of claim 1 , wherein the act of creating the index from the updated image comprises:
extracting one or more index keys from the consistent image; and
creating the index using the extracted index keys.
10. The method of claim 1 , further comprising determining whether the index substantially reflects the source data as of a current point-of-time such that any outage from locking access to the source data and updating the image with current changes would be less than a defined amount of time.
11. The method of claim 1 , wherein the act of making the consistent image consistent with the source data as of the subsequent point-in-time comprises:
identifying one or more changes to the source data as of the point-in-time; and
updating the consistent image to reflect the identified changes.
12. The method of claim 11 , further comprising repeating the acts of identifying changes, updating the image, and updating the index until a defined event occurs.
13. The method of claim 12 , further comprising allowing access to the index if the defined event occurs and terminating if the defined event does not occur.
14. The method of claim 12 , wherein the defined event includes a point when a number of current changes for updating the image is small enough such that any outage from locking access to the source data and updating the image with the current changes would be less than a defined amount of time.
15. The method of claim 14 , further comprising if the current number of changes for updating the image is small enough:
blocking access to the source data;
updating the image to reflect the current changes;
updating the index to reflect the updated image; and
allowing access to the updated index.
16. The method of claim 12 , wherein the defined event includes a point when a number of current changes to process is greater than a number of changes previously processed.
17. The method of claim 16 , further comprising terminating if the number of current changes to process is greater than the number previously processed.
18. A computer-readable medium having instructions to perform an index building method implementable on a programmable control device, the method comprising:
creating a snapshot image of source data at a point-in-time;
making the snapshot image consistent with the source data as of the point-in-time;
creating the index from the consistent image;
making the consistent image consistent with the source data as of a subsequent point-in-time; and
updating the index to reflect the subsequent consistent image.
19. The method of claim 18 , wherein the acts of identifying, updating and creating are repeated during a single index building operation.
20. The method of claim 18 , wherein the act of making the snapshot consistent with the source data as of the point-in-time comprises:
identifying one or more changes to the source data as of the point-in-time; and
updating the snapshot image to reflect the identified changes.
21. The method of claim 20 , wherein the act of creating the snapshot image of the source data at the point-in-time comprises creating the snapshot image in a manner that does not substantially block access to the source data.
22. The method of claim 20 , wherein the acts of identifying the one or more changes and updating the snapshot image further comprises:
identifying the one or more changes to the source data as of the point-in-time that do not exist in the image;
updating the snapshot image to reflect the identified changes; and
removing all updates from the snapshot image made after the point-in-time.
23. The method of claim 22 , wherein the acts of identifying the one or more changes and updating the snapshot image further comprises:
identifying one or more in-flight units-of-work associated with the updates to the source data as of the point-in-time; and
removing each of the updates associated with each identified in-flight unit-of-work from the snapshot image.
24. The method of claim 18 , wherein the act of making the image consistent comprises obtaining one or more log records from a database management system log.
25. The method of claim 18 , wherein the act of making the image consistent comprises asynchronously capturing one or more log records associated with the source data.
26. The method of claim 18 , wherein the act of creating the index from the updated image comprises:
extracting one or more index keys from the consistent image; and
creating the index using the extracted index keys.
27. The method of claim 18 , further comprising determining whether the index substantially reflects the source data as of a current point-of-time such that any outage from locking access to the source data and updating the image with current changes would be less than a defined amount of time.
28. The method of claim 18 , wherein the act of making the consistent image consistent with the source data as of the subsequent point-in-time comprises:
identifying one or more changes to the source data as of the point-in-time; and
updating the consistent image to reflect the identified changes.
29. The method of claim 28 , further comprising repeating the acts of identifying changes, updating the image, and updating the index until a defined event occurs.
30. The method of claim 29 , further comprising allowing access to the index if the defined event occurs and terminating if the defined event does not occur.
31. The method of claim 29 , wherein the defined event includes a point when a number of current changes for updating the image is small enough such that any outage from locking access to the source data and updating the image with the current changes would be less than a defined amount of time.
32. The method of claim 31 , further comprising if the current number of changes for updating the image is small enough:
locking access to the source data;
updating the image to reflect the current changes;
updating the index to reflect the updated image; and
allowing access to the updated index.
33. The method of claim 29 , wherein the defined event includes a point when a number of current changes to process is greater than a number of changes previously processed.
34. The method of claim 33 , further comprising terminating if the number of current changes to process is greater than the number previously processed.
35. An index building method, comprising:
creating a consistent image of source data as of a point-in-time;
creating an index from the consistent image; and
repeating acts of making the image consistent as of a subsequent point-in-time and updating the index to reflect the subsequent consistent image until a defined event occurs.
36. The method of claim 35 , wherein the act of creating the consistent image of the source data as of the point-in-time comprises creating the consistent image in a manner that does not substantially block access to the source data.
37. The method of claim 35 , wherein the act of creating the consistent image of the source data as of the point-in-time further comprises:
creating a snapshot image of the source data at the point-in-time;
identifying one or more changes to the source data as of the point-in-time that do not exist in the image;
updating the snapshot image to reflect the identified changes; and
removing all updates from the snapshot image made after the point-in-time.
38. The method of claim 37 , wherein the act of creating the consistent image of the source data as of the point-in-time further comprises:
identifying one or more in-flight units of work associated with the updates to the source data as of the point-in-time; and
removing each update associated with each identified in-flight unit of work from the snapshot image.
39. The method of claim 35 , wherein the act of creating the index from the consistent image comprises:
extracting one or more index keys from the consistent image; and
creating the index using the extracted index keys.
40. The method of claim 35 , further comprising allowing access to the index if the defined event occurs and terminating if the defined event does not occur.
41. The method of claim 35 , wherein the defined event includes a point when a number of current changes for updating the image is small enough such that any outage from locking access to the source data and updating the image with the current changes would be less than a defined amount of time.
42. The method of claim 41 , further comprising if the current number of changes for updating the image is small enough:
locking access to the source data;
updating the image to reflect the current changes;
updating the index to reflect the updated image; and
allowing access to the index.
43. The method of claim 35 , wherein the defined event includes a point when a number of current changes to process is greater than a number of changes previously processed.
44. The method of claim 43 , further comprising terminating if the number of current changes to process is greater than the number previously processed.
45. A computer-readable medium having instructions to perform an index building method implementable on a programmable control device, the method comprising:
creating a consistent image of source data as of a point-in-time;
creating an index from the consistent image; and
repeating acts of making the image consistent as of a subsequent point-in-time and updating the index to reflect the subsequent consistent image until a defined event occurs.
46. The method of claim 45 , wherein the act of creating the consistent image of the source data as of the point-in-time comprises creating the consistent image in a manner that does not substantially block access to the source data.
47. The method of claim 45 , wherein the act of creating the consistent image of the source data as of the point-in-time further comprises:
creating a snapshot image of the source data at the point-in-time;
identifying one or more changes to the source data as of the point-in-time that do not exist in the image;
updating the snapshot image to reflect the identified changes; and
removing all updates from the snapshot image made after the point-in-time.
48. The method of claim 47 , wherein the act of creating the consistent image of the source data as of the point-in-time further comprises:
identifying one or more in-flight units of work associated with the updates to the source data as of the point-in-time; and
removing each update associated with each identified in-flight unit of work from the snapshot image.
49. The method of claim 45 , wherein the act of creating the index from the consistent image comprises:
extracting one or more index keys from the consistent image; and
creating the index using the extracted index keys.
50. The method of claim 45 , further comprising allowing access to the index if the defined event occurs and terminating if the defined event does not occur.
51. The method of claim 45 , wherein the defined event includes a point when a number of current changes for updating the image is small enough such that any outage from locking access to the source data and updating the image with the current changes would be less than a defined amount of time.
52. The method of claim 51 , further comprising if the current number of changes for updating the image is small enough:
locking access to the source data;
updating the image to reflect the current changes;
updating the index to reflect the updated image; and
allowing access to the index.
53. The method of claim 45 , wherein the defined event includes a point when a number of current changes to process is greater than a number of changes previously processed.
54. The method of claim 53 , further comprising terminating if the number of current changes to process is greater than the number previously processed.