Automatic data store architecture detection
A system is configured for automatic recognition of data store architecture and tracking dynamic changes and evolution in data store architecture. The system is a complementary system, which can be added onto an existing data store system using the existing interfaces or can be integrated with a data store system. The system comprises three main components that are configured to compose an approximation of the data store architecture. The first of these components is adapted to execute an analysis of the architecture of the data store; the second of the components is adapted to collect and compile statistics from said data store; and the third of the components is adapted to compose an approximation of the architecture of said data store.
1 . A computerized method comprising:
determining a first approximation of a data store architecture of data store tables of a data store, the data store including objects, the first approximation including an estimate of attributes of the objects, the attributes including relationships between the objects;
receiving a user query;
responsive to the received user query, determining a second approximation, the second approximation including an update of the determined first approximation;
determining that the data store has changed;
determining a data store table insertion, a data store table modification, or a data store table removal detected; and
output a notification of the data store table insertion, the data store table modification, or the data store table removal.
2 . The computerized method of claim 1 , further comprising:
in response to determining that the data store has changed, analyzing an impact of the data store changes to the data store architecture based, at least in part, on a content of a query repository.
3 . An apparatus comprising:
a processor;
a data store communicatively coupled to the processor, the data store having at least one data store table; and
a memory coupled to the processor and storing instructions that, when executed by the processor, causes the apparatus to perform operations including:
determine a first approximation of a data store architecture of the data store, the first approximation including an estimate of attributes of objects of the data store, the attributes including relationships between the objects;
responsive to a received user query, determine a second approximation, the second approximation including an update of the determined first approximation;
determine that the data store has changed; and
in response to determining a data store table insertion, a data store table modification, or a data store table removal is detected, output a notification about the data store table insertion, the data store table modification, or the data store table removal.
4 . The apparatus according to claim 3 , wherein the first approximation or the second approximation includes a collection of statistics constructed from data extracted from the data store and stored as separate entities.
5 . The apparatus according to claim 3 , wherein the first approximation or the second approximation is used as an input to a data arrangement application, or wherein the first approximation or the second approximation is configured to be used by a data store administrator to change the data store architecture or another data store architecture.
6 . The apparatus according to claim 3 , wherein the operations further include:
check designated resources of a system associated with the data store, by:
tracking indicators of a performance of the system, and
determining resources of the system are available for use by the apparatus to carry out another cycle of data store architecture approximation improvement or if the system is too busy or does not have enough available memory, disc space, or processing available for use by the apparatus to carry out another cycle of data store architecture approximation improvement;
analyzing system resources limits, and
generating and provide a resources analysis report.
7 . The apparatus according to claim 3 , wherein the operations further include:
check designated resources of a system associated with the data store;
analyze data store system resources limits; and
generate and provide a resources analysis report.
8 . The apparatus according to claim 3 , wherein the apparatus is configured to work with a subset of the data store, the subset identified by volume reduction.
9 . The apparatus according to claim 3 , wherein the apparatus is configured to work with a mirrored data store, wherein the data store comprises the mirrored data store or is associated with the mirrored data store.
10 . The apparatus according to claim 3 , wherein the apparatus is configured to work with a data warehouse to improve performance by modifying an aggregation layer.
11 . The apparatus according to claim 3 , wherein the first approximation is performed independently of user queries.
12 . A system comprising:
a processor;
a data store communicatively coupled to the processor, the data store having at least one data store table; and
a memory coupled to the processor and storing instructions that, when executed by the processor, causes the system to perform operations including:
determine a first approximation of a data store architecture, the first approximation including attributes of objects of the data store, the attributes including relationships between the objects;
determine if one or more user queries are available;
execute a syntactic analysis of available user queries;
use the syntactic analysis of available user queries to generate statistics related to query content;
use the statistics related to query content to determine a second approximation of the data store, the second approximation of the data store including an update of the determined first approximation;
determine that the data store has changed based at least in part on detecting a data store table insertion, a data store table modification, or a data store table removable by automatically carrying out data store architecture analysis comprising comparing successive approximations of the architecture of the data store;
wherein the successive approximations include the first and second approximations; and
output a notification about the data store changes.
13 . The system according to claim 12 , wherein the syntactic analysis used to generate query content related statistics comprises usage information corresponding to the at least one table, filter(s) used, strength of filter(s), query complexity, user priority, query execution time, or frequency of same query template usage.
14 . The system according to claim 12 , wherein the determination as to whether one or more user queries are available comprises determining if one or more user queries are available in a queries repository, a query log, or a query stream.
15 . The system according to claim 12 , wherein the data store architecture comprises a relationship between a plurality of columns of the at least one data store table.
16 . The system according to claim 12 , wherein the operations further include:
identify columns that appear in a given query; and
determine if a first column of the identified columns and a second column of the identified columns are treated as equivalents in the given query.
17 . The system according to claim 12 , wherein the operations further include:
determine size of columns that appear in a given query; and
determine total size of the columns that appear in the given query.
18 . The system according to claim 12 , wherein the system is configured to limit syntactic query analysis to queries that do not alter the data store.
19 . The system according to claim 12 , wherein the operations further include perform syntactic query analysis on nested queries that comprise a sub-query, wherein sub-queries are processed recursively in a bottom-up way.