Neural ODE for irregularly-sampled time series
A computer-implemented method for updating hidden states in a recurrent neural network (RNN) to predict future data from multivariate time-series data with irregular time intervals is provided including inputting, for each of time steps at observations, observation data at a current time step in the multivariate time-series data to the RNN, for each of the time steps: subdividing a time interval between a previous time step and the current time step by a predetermined number, for each of subdivided time steps calculating a first element of the hidden state at a current subdivided time step using ODE-RNNs, and calculating a second element of the hidden state at the current subdivided time step using the last updated hidden state and a hidden state at the previous time step so that the last updated hidden state is decayed to be close to the hidden state at the previous time step.
1 . A computer-implemented method for updating hidden states in a recurrent neural network (RNN) to predict future data from multivariate time-series data with irregular time intervals, the method comprising:
controlling an amount of information that the RNN updates to a next hidden state for irregularly sampled time series data on a basis of elapsed time and capturing discrete dynamics of hidden states by inputting, for each time step at observations, observation data at a current time step in the multivariate time-series data to the RNN;
for each of the time steps:
subdividing a time interval between a previous time step and the current time step by a predetermined number, wherein subdivided time intervals between adjacent time steps are equal;
for each of subdivided time steps:
calculating a first element of the hidden state at a current subdivided time step using neural ordinary differential equations (Neural ODEs) based on a last updated hidden state and a time interval between a previous subdivided time step and the current subdivided time step;
calculating a second element of the hidden state at the current subdivided time step using the last updated hidden state and a hidden state at the previous time step so that the last updated hidden state is decayed to be close to the hidden state at the previous time step based on a time interval between the current subdivided time step and the previous time step; and
updating the hidden state at the current subdivided time step to a weighted mean of the first element and second element.
2 . The computer-implemented method of claim 1 , wherein, when updating hidden states at all the subdivided time steps is completed, generating the hidden state at the current time step using a recurrent layer of the RNN based on the observation data at the current time step and the last updated hidden state during the time interval between the previous time step and the current time step.
3 . The computer-implemented method of claim 1 , wherein the first element is a gate mechanism and the second element is a decay mechanism.
4 . The computer-implemented method of claim 3 , wherein the gate mechanism controls how much information from values generated with neural ODEs is carried over to next hidden state for each update of the neural ODEs.
5 . The computer-implemented method of claim 3 , wherein the decay mechanism applies a decay to the hidden state updated with neural ODEs.
6 . The computer-implemented method of claim 5 , wherein the hidden state updated with neural ODEs are close to hidden states last updated with RNNs based on elapsed time from a last observation.
7 . The computer-implemented method of claim 2 , wherein the generated hidden state is given as:
h t i +1 =RNNCell( h t i +ΔN s ′,x i+1 )
where RNNCell is a recurrent layer of the RNN, h′ t i +ΔN s is a hidden state before a next update, and x i+1 is an observation at time step i+ 1.
8 . A computer program product for updating hidden states in a recurrent neural network (RNN) to predict future data from multivariate time-series data with irregular time intervals, the computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a computer to cause the computer to:
control an amount of information that the RNN updates to a next hidden state for irregularly sampled time series data on a basis of elapsed time and capturing discrete dynamics of hidden states by causing the computer to input, for each time step at observations, observation data at a current time step in the multivariate time-series data to the RNN;
for each of the time steps:
subdivide a time interval between a previous time step and the current time step by a predetermined number, wherein subdivided time intervals between adjacent time steps are equal;
for each of subdivided time steps:
calculate a first element of the hidden state at a current subdivided time step using neural ordinary differential equations (Neural ODEs) based on a last updated hidden state and a time interval between a previous subdivided time step and the current subdivided time step;
calculate a second element of the hidden state at the current subdivided time step using the last updated hidden state and a hidden state at the previous time step so that the last updated hidden state is decayed to be close to the hidden state at the previous time step based on a time interval between the current subdivided time step and the previous time step; and
update the hidden state at the current subdivided time step to a weighted mean of the first element and second element.
9 . The computer program product of claim 8 , wherein, when updating hidden states at all the subdivided time steps is completed, generating the hidden state at the current time step using a recurrent layer of the RNN based on the observation data at the current time step and the last updated hidden state during the time interval between the previous time step and the current time step.
10 . The computer program product of claim 8 , wherein the first element is a gate mechanism and the second element is a decay mechanism.
11 . The computer program product of claim 9 , wherein the gate mechanism controls how much information from values generated with neural ODEs is carried over to next hidden state for each update of the neural ODEs.
12 . The computer program product of claim 9 , wherein the decay mechanism applies a decay to the hidden state updated with neural ODEs.
13 . The computer program product of claim 12 , wherein the hidden state updated with neural ODEs are close to hidden states last updated with RNNs based on elapsed time from a last observation.
14 . The computer program product of claim 9 , wherein the generated hidden state is given as:
h t i +1 =RNNCell( h′ t i +ΔN s ,x i+1 )
where RNNCell is a recurrent layer of the RNN, h′ t i +ΔN s is a hidden state before a next update, and x i+1 is an observation at time, step i+1.
15 . A system for updating hidden states in a recurrent neural network (RNN) to predict future data from multivariate time-series data with irregular time intervals, the system comprising:
a memory; and
one or more processors in communication with the memory configured to:
control an amount of information that the RNN updates to a next hidden state for irregularly sampled time series data on as basis of elapsed time and capturing discrete dynamics of hidden states by causing the one or more processors to input, for each time step at observations, observation data at a current time step in the multivariate time-series data to the RNN;
for each of the time steps:
subdivide a time interval between a previous time step and the current time step by a predetermined number, wherein subdivided time intervals between adjacent time steps are equal;
for each of subdivided time steps:
calculate a first element of the hidden state at a current subdivided time step using neural ordinary differential equations (Neural ODEs) based on a last updated hidden state and a time interval between a previous subdivided time step and the current subdivided time step;
calculate a second element of the hidden state at the current subdivided time step using the last updated hidden state and a hidden state at the previous time step so that the last updated hidden state is decayed to be close to the hidden state at the previous time step based on a time interval between the current subdivided time step and the previous time step; and
update the hidden state at the current subdivided time step to a weighted mean of the first element and second element.
16 . The system of claim 15 , wherein, when updating hidden states at all the subdivided time steps is completed, generating the hidden state at the current time step using a recurrent layer of the RNN based on the observation data at the current time step and the last updated hidden state during the time interval between the previous time step and the current time step.
17 . The system of claim 15 , wherein the first element is a gate mechanism and the second element is a decay mechanism.
18 . The system of claim 16 , wherein the gate mechanism controls how much information from values generated with neural ODEs is carried over to next hidden state for each update of the neural ODEs.
19 . The system of claim 16 , wherein the decay mechanism applies a decay to the hidden state updated with neural ODEs.
20 . The system of claim 19 , wherein the hidden state updated with neural ODEs are close to hidden states last updated with RNNs based on elapsed time from a last observation.