Systems and methods for modeling continuous stochastic processes with dynamic normalizing flows
Systems and methods for machine learning architecture for time series data prediction. The system may include a processor and a memory storing processor-executable instructions. The processor-executable instructions, when executed, may configure the processor to: obtain time series data associated with a data query; generate a predicted value based on a sampled realization of the time series data and a continuous time generative model, the continuous time generative model trained to define an invertible mapping to maximize a log-likelihood of a set of predicted values for a time range associated with the time series data; and generate a signal providing an indication of the predicted value associated with the data query.
1. A system for a machine learning architecture for time series data prediction comprising:
a processor; and
a memory coupled to the processor and storing processor-executable instructions that, when executed, configure the processor to:
obtain time series data associated with a data query;
generate a predicted value by executing a machine learning application based on a sampled realization of the time series data, the machine learning application comprising a continuous time generative model trained to define an invertible mapping to maximize a log-likelihood of a set of predicted values for a time range associated with the time series data, wherein generation of the predicted value comprises:
computing the predicted value based on a joint distribution X τ =F θ (W τ ;τ), ∀τ∈[0, T], where F θ (⋅; τ): d → d is the invertible mapping parametrized by the learnable parameters θ for every τ∈[0, T], and W τ is a d-dimensional Wiener process, such that the log-likelihood
ℒ
=
log
p
x
τ
1
,
…
,
x
τ
n
(
x
τ
1
,
…
,
x
τ
n
)
is maximized, where p X (x) represents a probability density function of x;
wherein the invertible mapping is based on solving an initial value problem defined by:
d
dt
(
h
τ
(
t
)
a
τ
(
t
)
)
=
(
f
θ
(
h
τ
(
t
)
,
a
τ
(
t
)
,
t
)
g
θ
(
a
τ
(
t
)
,
t
)
)
,
(
h
τ
(
t
0
)
a
τ
(
t
0
)
)
=
(
w
τ
τ
)
,
where h τ (t)∈ d , t∈[t 0 ,t 1 ], ƒ θ : d × ×[t 0 ,t 1 ]→ d , and g θ : ×[t o ,t 1 ]→ , and the joint distribution F θ (w τ ; τ) is defined as a solution of h τ (t); and
generate a signal providing an indication of the predicted value associated with the data query for performing a downstream task.
2. The system of claim 1 , wherein the invertible mapping is parameterized by a latent variable having an isotropic Gaussian prior distribution.
3. The system of claim 1 , wherein the predicted value represents one or more observed process data points based on a sampled realization of a Weiner process and the invertible mapping to provide a time continuous observed realization of the Weiner process.
4. The system of claim 3 , wherein the obtained time series data includes an incomplete realization of a continuous stochastic process.
5. The system of claim 4 , wherein: the joint distribution is defined as the solution of h τ (t) at
F
θ
(
w
τ
;
τ
)
:
=
h
τ
(
t
1
)
=
h
τ
(
t
0
)
+
∫
t
0
t
1
f
θ
(
h
τ
(
t
)
,
a
τ
(
t
)
,
t
)
d
t
.
6. The system of claim 1 , wherein the processor-executable instructions, when executed, configure the processor to:
obtain a training dataset including irregular time series data over time associated with an incomplete realization of a continuous stochastic process; and
generate the invertible mapping associated with the continuous time generative model based on maximizing the log-likelihood of the set of predicted values.
7. The system of claim 1 , wherein the obtained time series data includes observed process data points, wherein the predicted value represents a likelihood determination of stochastic process data points based on the observed process data points and an inverse of the invertible mapping of the continuous time generative model.
8. The system of claim 1 , wherein the obtained time series data includes unobserved process data points, and wherein the predicted value represents a conditional probability density of stochastic data points based on the unobserved process data points and an inverse of the invertible mapping of the continuous time generative model.
9. The system of claim 8 , wherein the conditional probability density provides for data point interpolation associated with a stochastic process based on a Brownian bridge.
10. The system of claim 8 , wherein the conditional probability density provides for data point extrapolation of data points associated with a stochastic process based on a multivariate Gaussian conditional probability distribution.
11. The system of claim 1 , wherein the continuous time generative model is based on an augmented neural ordinary differential equation (ANODE) including a multi-layer perceptron (MLP model) having at least 4 hidden layers.
12. A method for a machine learning architecture for time series data prediction comprising:
obtaining time series data associated with a data query;
generating a predicted value by executing a machine learning application based on a sampled realization of the time series data, the machine learning application comprising a continuous time generative model trained to define an invertible mapping to maximize a log-likelihood of a set of predicted values for a time range associated with the time series data, wherein generation of the predicted value comprises:
computing the predicted value based on a joint distribution X τ =F θ (W τ ; τ), ∀τ∈[0, T], where F θ (⋅;τ): d → d is the invertible mapping parametrized by the learnable parameters θ for every τ∈[0, T], and W τ is a d-dimensional Wiener process, such that the log-likelihood
ℒ
=
log
p
x
τ
1
,
…
,
x
τ
n
(
x
τ
1
,
…
,
x
τ
n
)
is maximized, where p X (x) represents a probability density function of x;
wherein the invertible mapping is based on solving an initial value problem defined by:
d
dt
(
h
τ
(
t
)
a
τ
(
t
)
)
=
(
f
θ
(
h
τ
(
t
)
,
a
τ
(
t
)
,
t
)
g
θ
(
a
τ
(
t
)
,
t
)
)
,
(
h
τ
(
t
0
)
a
τ
(
t
0
)
)
=
(
w
τ
τ
)
,
where h τ (t)∈ d , t∈[t 0 , t 1 ], ƒ θ : d × ×[t 0 , t 1 ]→ d , and g θ : ×[t 0 , t 1 ]→ , and the joint distribution F θ (w τ ; τ) is defined as a solution of h τ (t), and
generating a signal providing an indication of the predicted value associated with the data query for performing a downstream task.
13. The method of claim 12 , wherein the invertible mapping is parameterized by a latent variable having an isotropic Gaussian prior distribution.
14. The method of claim 12 , wherein the predicted value represents one or more observed process data points based on a sampled realization of a Weiner process and the invertible mapping to provide a time continuous observed realization of the Weiner process.
15. The method of claim 14 , wherein the obtained time series data includes an incomplete realization of a continuous stochastic process.
16. The method of claim 15 , wherein: the joint distribution is defined as the solution of h τ (t) at
t
=
t
1
:
F
θ
(
w
τ
;
τ
)
:
=
h
τ
(
t
1
)
=
h
τ
(
t
0
)
+
∫
t
0
t
1
f
θ
(
h
τ
(
t
)
,
a
τ
(
t
)
,
t
)
d
t
.
17. The method of claim 12 , comprising:
obtaining a training dataset including irregular time series data over time associated with an incomplete realization of a continuous stochastic process; and
generating the invertible mapping associated with the continuous time generative model based on maximizing the log-likelihood of the set of predicted values.
18. The method of claim 12 , wherein the obtained time series data includes observed process data points, wherein the predicted value represents a likelihood determination of stochastic process data points based on the observed process data points and an inverse of the invertible mapping of the continuous time generative model.
19. The method of claim 12 , wherein the obtained time series data includes unobserved process data points, and wherein the predicted value represents a conditional probability density of stochastic data points based on the unobserved process data points and an inverse of the invertible mapping of the continuous time generative model.
20. A non-transitory computer-readable medium having stored thereon machine interpretable instructions which, when executed by a processor, cause the processor to perform a method for a machine learning architecture for time series data prediction, the method comprising:
obtaining time series data associated with a data query;
generating a predicted value by executing a machine learning application based on a sampled realization of the time series data, the machine learning application comprising a continuous time generative model trained to define an invertible mapping to maximize a log-likelihood of a set of predicted values for a time range associated with the time series data, wherein generation of the predicted value comprises;
computing the predicted value based on a joint distribution X τ =F θ (W τ ; τ), ∀τ∈[0, T], where F θ (ω;τ): d → d is the invertible mapping parametrized by the learnable parameters θ for every τ∈[0, T], and W τ is a d-dimensional Wiener process, such that the log-likelihood
ℒ
=
log
p
x
τ
1
,
…
,
x
τ
n
(
x
τ
1
,
…
,
x
τ
n
)
is maximized, where p X (x) represents a probability density function of x;
wherein the invertible mapping is based on solving an initial value problem defined by:
d
dt
(
h
τ
(
t
)
a
τ
(
t
)
)
=
(
f
θ
(
h
τ
(
t
)
,
a
τ
(
t
)
,
t
)
g
θ
(
a
τ
(
t
)
,
t
)
)
,
(
h
τ
(
t
0
)
a
τ
(
t
0
)
)
=
(
w
τ
τ
)
,
where h τ (t)∈ d , t∈[t 0 , t 1 ], ƒ θ : d ×R×[t 0 , t 1 ]→ d , and g θ : ×[t 0 , t 1 ]→ , and the joint distribution F θ (w τ ; τ) is defined as a solution of h τ (t); and
generating a signal providing an indication of the predicted value associated with the data query for performing a downstream task.