Method for synthetic data generation for query workloads
View Patent ↗Generation of synthetic database data includes annotated query subplans for a multiple table query workload that includes a desired cardinality for nodes (v) in the subplans. The subplans may be merged and represented by a direct acyclic graph (DAG). The maximum entropy joint probability distribution for each attribute (x) for each node (v) is determined as: p ( x ) = exp ( ∑ v w v f v ( x ) Z ) for each node ν, where w v is a weight of node v, f v is a conjunct of predicates in a subplan rooted at node v, and Z is a normalization factor. This distribution is determined such that the desired cardinality, and selectivities for each node v determined from the desired cardinality, are satisfied. The data for a plurality of tables are generated by sampling the maximum entropy joint probability distribution for a domain of attributes (x) of a plurality of tables. Data may be efficiently generated for multiple table queries and for DAGs.
1. A computer program product for synthetic data generation, the computer program product comprising a non-transitory computer readable storage medium having program code embodied therewith, the program code executable by a processor to:
receive annotated query subplans for a multiple table query workload comprising a desired cardinality for a plurality of nodes (v) in the annotated query subplans;
determine a maximum entropy joint probability distribution for each attribute (x) for each node (v) as:
p
(
x
)
=
exp
[
(
∑
v
w
v
f
v
(
x
)
Z
]
)
for each node ν, wherein
w v comprises a weight of node v,
f v comprises a conjunct of predicates in a subplan rooted at node v, and
Z comprises a normalization factor,
wherein an iterative scaling algorithm is used to determine the weight w v for each node (v), wherein the iterative scaling algorithm comprises:
repeating until convergence:
Let
p
(
x
)
=
exp
[
(
∑
v
w
v
f
v
(
x
)
Z
]
)
For each v = 1, . . . , n max :
Let
E
p
[
f
v
]
=
∑
x
p
(
x
)
f
v
(
x
)
Update
w
v
=
w
v
+
1
n
(
log
K
[
f
v
]
E
p
[
f
v
]
-
log
1
-
K
[
f
v
]
1
-
E
p
[
f
v
]
)
wherein f ν comprises a characteristic function of a conjunct of predicates at node ν, K[f ν ] comprises an observed expectation of f ν , and E p [f ν ] is an estimated expectation of f ν ,
wherein the desired cardinality, and selectivities for each node v determined from the desired cardinality, are satisfied; and
generate data for a plurality of tables by sampling the maximum entropy joint probability distribution for a domain of attributes (x) of a plurality of tables.
2. The computer program product of claim 1 , wherein the annotated query subplans are merged and represented as a directed acyclic graph (DAG), wherein the plurality of nodes (v) are nodes of the DAG.
3. The computer program product of claim 2 , wherein the plurality of tables follow a snowflake schema, wherein in determining the maximum entropy joint probability distribution for each attribute (x) for each node (v) for queries involving dimension table attributes, the program code executable by the processor further:
considers a dimension table as a fact table for a sub-dimension table;
determines a probability distribution for the dimension table; and
uses a filter intermediate result from the dimension table as a primary key input in determining a probability distribution in the sub-dimension table.
4. The computer program product of claim 1 , wherein the program code executable by the processor to sample the maximum entropy joint probability distribution for the domain of attributes (x) of the plurality of tables is performed using Gibbs sampling.
5. The computer program product of claim 1 , wherein the plurality of tables follow a snowflake schema, wherein in generating the data for the plurality of tables by sampling the maximum entropy joint probability distribution for the domain of attributes (x) of the plurality of tables, the data for any dimension tables are generated before any fact tables.
6. A system comprising:
a processor; and
a computer readable storage medium having program code embodied therewith, the program code executable by the processor to:
receive annotated query subplans for a multiple table query workload comprising a desired cardinality for a plurality of nodes (v) in the annotated query subplans;
determine a maximum entropy joint probability distribution for each attribute (x) for each node (v) as:
p
(
x
)
=
exp
[
(
∑
v
w
v
f
v
(
x
)
Z
]
)
for each node ν, wherein
w v comprises a weight of node v,
f v comprises a conjunct of predicates in a subplan rooted at node v,
Z comprises a normalization factor,
wherein an iterative scaling algorithm is used to determine the weight w v for each node (v), wherein the iterative scaling algorithm comprises:
repeating until convergence:
Let
p
(
x
)
=
exp
[
(
∑
v
w
v
f
v
(
x
)
Z
]
)
For each v = 1, . . . , n max :
Let
E
p
[
f
v
]
=
∑
x
p
(
x
)
f
v
(
x
)
Update
w
v
=
w
v
+
1
n
(
log
K
[
f
v
]
E
p
[
f
v
]
-
log
1
-
K
[
f
v
]
1
-
E
p
[
f
v
]
)
wherein f ν comprises a characteristic function of a conjunct of predicates at node ν, K[f ν ] comprises an observed expectation of f ν , and E p [f ν ] is an estimated expectation of f ν ,
wherein the desired cardinality, and selectivities for each node v determined from the desired cardinality, are satisfied; and
generate data for a plurality of tables by sampling the maximum entropy joint probability distribution for a domain of attributes (x) of a plurality of tables.
7. The system of claim 6 , wherein the annotated query subplans are merged and represented as a directed acyclic graph (DAG), wherein the plurality of nodes (v) are nodes of the DAG.
8. The system of claim 7 , wherein the plurality of tables follow a snowflake schema, wherein in determining the maximum entropy joint probability distribution for each attribute (x) for each node (v) for queries involving dimension table attributes, the program code executable by the processor further:
considers a dimension table as a fact table for a sub-dimension table;
determines a probability distribution for the dimension table; and
uses a filter intermediate result from the dimension table as a primary key input in determining a probability distribution in the sub-dimension table.
9. The system of claim 6 , wherein the program code executable by the processor to sample the maximum entropy joint probability distribution for the domain of attributes (x) of the plurality of tables is performed using Gibbs sampling.
10. The system of claim 6 , wherein the plurality of tables follow a snowflake schema, wherein in generating the data for the plurality of tables by sampling the maximum entropy joint probability distribution for the domain of attributes (x) of the plurality of tables, the data for any dimension tables are generated before any fact tables.