Systems and methods for training matrix-based differentiable programs
Methods and apparatus for training a matrix-based differentiable program using a photonics-based processor. The matrix-based differentiable program includes at least one matrix-valued variable associated with a matrix of values in a Euclidean vector space. The method comprises configuring components of the photonics-based processor to represent the matrix of values as an angular representation, processing, using the components of the photonics-based processor, training data to compute an error vector, determining in parallel, at least some gradients of parameters of the angular representation, wherein the determining is based on the error vector and a current input training vector, and updating the matrix of values by updating the angular representation based on the determined gradients.
1. A method for training a matrix-based differentiable program using a photonics-based processor, the matrix-based differentiable program including at least one matrix-valued variable associated with a matrix of values in a Euclidean vector space, the method comprising:
configuring components of the photonics-based processor to represent the matrix of values as an angular representation;
processing, using the components of the photonics-based processor, training data to compute an error vector;
determining in parallel, at least some gradients of parameters of the angular representation, wherein the determining is based on the error vector and a current input training vector; and
updating the matrix of values by updating the angular representation based on the determined gradients;
wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
using singular value decomposition to transform the matrix of values into a first unitary matrix, a second unitary matrix and a diagonal matrix of signed or normalized singular values;
decomposing the first unitary matrix into a first set of unitary transfer matrices;
decomposing the second unitary matrix into a second set of unitary transfer matrices; configuring a first set of components of the photonics-based processor based on the first
set of unitary transfer matrices;
configuring a second set of components of the photonics-based processor based on the diagonal matrix of singular values; and
configuring a third set of components of the photonics-based processor based on the second set of unitary transfer matrices.
2. The method of claim 1 , wherein the first set of unitary transfer matrices includes a plurality of columns, and wherein determining in parallel, at least some of the gradients of parameters of the angular representation comprises determining in parallel, gradients for multiple columns of the plurality of columns.
3. The method of claim 1 , wherein processing the training data to compute an error vector comprises:
encoding the input training vector using photonic signals;
providing the photonic signals as input to the photonics-based processor to generate an output vector;
decoding the output vector;
determining, based on the decoded output vector, a value of a loss function for the input training vector; and
computing the error vector based on aggregating the values of the loss function determined for a set of input training vectors in the training data.
4. The method of claim 1 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix, gradients of the parameters for each column of the decomposed second unitary matrix, and gradients of the parameters for the singular values.
5. The method of claim 4 , wherein determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix comprises:
(a) computing a block diagonal derivative matrix containing derivatives with respect to each parameter in a selected column k of the first set of unitary transfer matrices;
(b) computing a product of the error vector with all unitary transfer matrices in the decomposed first unitary matrix between the selected column k and an output column of the first set unitary transfer matrices;
(c) computing a product of an input training data vector with all unitary transfer matrices in the decomposed first unitary matrix from an input column of the first set of unitary transfer matrices up to and including the selected column k of the first set of unitary transfer matrices;
(d) multiplying the result of act (c) by the block diagonal derivative matrix computed in act (a); and
computing inner products between successive pairs of elements from the output of act (b) and the output of act (d).
6. The method of claim 1 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises backpropagating the error vector through the angular representation of the matrix of values.
7. The method of claim 1 , wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
partitioning the matrix of values into two or more matrices having smaller dimensions;
and configuring components of the photonics-based processor to represent the values in one of the two or more matrices having smaller dimensions.
8. The method of claim 1 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components using a plurality of iterations, and
wherein for some iterations of the plurality of iterations only the parameters for the second set of components is updated.
9. The method of claim 1 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components, and
wherein a first learning rate for updating the first and third sets of components is different than a second learning rate for updating the second set of components.
10. A non-transitory computer readable medium encoded with a plurality of instructions that, when executed by at least one photonics-based processor perform a method for training a latent variable graphical model, the latent variable graphical model including at least one matrix-valued latent variable associated with a matrix of values in a Euclidean vector space, the method comprising:
configuring components of the photonics-based processor to represent the matrix of values as an angular representation;
processing, using the components of the photonics-based processor, training data to compute an error vector;
determining in parallel, at least some gradients of parameters of the angular representation, wherein the determining is based on the error vector and a current input training vector; and
updating the matrix of values by updating the angular representation based on the determined gradients;
wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
using singular value decomposition to transform the matrix of values into a first unitary matrix, a second unitary matrix and a diagonal matrix of signed or normalized singular values;
decomposing the first unitary matrix into a first set of unitary transfer matrices;
decomposing the second unitary matrix into a second set of unitary transfer matrices; configuring a first set of components of the photonics-based processor based on the first
set of unitary transfer matrices;
configuring a second set of components of the photonics-based processor based on the diagonal matrix of singular values; and
configuring a third set of components of the photonics-based processor based on the second set of unitary transfer matrices.
11. The non-transitory computer-readable medium of claim 10 , wherein the first set of unitary transfer matrices includes a plurality of columns, and wherein determining in parallel, at least some of the gradients of parameters of the angular representation comprises determining in parallel, gradients for multiple columns of the plurality of columns.
12. The non-transitory computer-readable medium of claim 10 , wherein processing the training data to compute an error vector comprises:
encoding the input training vector using photonic signals;
providing the photonic signals as input to the photonics-based processor to generate an output vector;
decoding the output vector;
determining, based on the decoded output vector, a value of a loss function for the input training vector; and
computing the error vector based on aggregating the values of the loss function determined for a set of input training vectors in the training data.
13. The non-transitory computer-readable medium of claim 10 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix, gradients of the parameters for each column of the decomposed second unitary matrix, and gradients of the parameters for the singular values.
14. The non-transitory computer-readable medium of claim 13 , wherein determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix and the decomposed second unitary matrix comprises:
(a) computing a block diagonal derivative matrix containing derivatives with respect to each parameter in a selected column k of the first set of unitary transfer matrices;
(b) computing a product of the error vector with all unitary transfer matrices in the decomposed first unitary matrix between selected column k of the first set of unitary transfer matrices and an output column of the first set unitary transfer matrices;
(c) computing a product of an input training data vector with all unitary transfer matrices in the decomposed first unitary matrix from an input column of the first set of unitary transfer matrices up to and including the selected column k of the first set of unitary transfer matrices;
(d) multiplying the result of act (c) by the block diagonal derivative matrix computed in act (a); and
computing inner products between successive pairs of elements from the output of act (b) and the output of act (d).
15. The non-transitory computer-readable medium of claim 10 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises backpropagating the error vector through the angular representation of the matrix of values.
16. The non-transitory computer-readable medium of claim 10 , wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
partitioning the matrix of values into two or more matrices having smaller dimensions;
and configuring components of the photonics-based processor to represent the values in one of the two or more matrices having smaller dimensions.
17. The non-transitory computer-readable medium of claim 10 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components using a plurality of iterations, and
wherein for some iterations of the plurality of iterations only the parameters for the second set of components is updated.
18. The non-transitory computer-readable medium of claim 10 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components, and
wherein a first learning rate for updating the first and third sets of components is different than a second learning rate for updating the second set of components.
19. A photonics-based processing system, comprising: a photonics processor; and
a non-transitory computer readable medium encoded with a plurality of instructions that, when executed by the photonics processor perform a method for training a latent variable graphical model, the latent variable graphical model including at least one matrix-valued latent variable associated with a matrix of values in a Euclidean vector space, the method comprising:
configuring components of the photonics processor to represent the matrix of values as an angular representation;
processing, using the components of the photonics processor, training data to compute an error vector;
determining in parallel, at least some gradients of parameters of the angular representation, wherein the determining is based on the error vector and a current input training vector; and
updating the matrix of values by updating the angular representation based on the determined gradients;
wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
using singular value decomposition to transform the matrix of values into a first unitary matrix, a second unitary matrix and a diagonal matrix of signed or normalized singular values;
decomposing the first unitary matrix into a first set of unitary transfer matrices;
decomposing the second unitary matrix into a second set of unitary transfer matrices; configuring a first set of components of the photonics-based processor based on the first
set of unitary transfer matrices;
configuring a second set of components of the photonics-based processor based on the diagonal matrix of singular values; and
configuring a third set of components of the photonics-based processor based on the second set of unitary transfer matrices.
20. The photonics-based processing system of claim 19 , wherein the first set of unitary transfer matrices includes a plurality of columns, and wherein determining in parallel, at least some of the gradients of parameters of the angular representation comprises determining in parallel, gradients for multiple columns of the plurality of columns.
21. The photonics-based processing system of claim 19 , wherein processing the training data to compute an error vector comprises:
encoding the input training vector using photonic signals;
providing the photonic signals as input to the photonics-based processor to generate an output vector;
decoding the output vector;
determining, based on the decoded output vector, a value of a loss function for the input training vector; and
computing the error vector based on aggregating the values of the loss function determined for a set of input training vectors in the training data.
22. The photonics-based processing system of claim 20 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix, gradients of the parameters for each column of the decomposed second unitary matrix, and gradients of the parameters for the singular values.
23. The photonics-based processing system of claim 22 , wherein determining in parallel, gradients of the parameters for each column of the decomposed first unitary matrix and the decomposed second unitary matrix comprises:
(a) computing a block diagonal derivative matrix containing derivatives with respect to each parameter in a selected column k of the first set of unitary transfer matrices;
(b) computing a product of the error vector with all unitary transfer matrices in the decomposed first unitary matrix between selected column k of the first set of unitary transfer matrices and an output column of the first set unitary transfer matrices;
(c) computing a product of an input training data vector with all unitary transfer matrices in the decomposed first unitary matrix from an input column of the first set of unitary transfer matrices up to and including the selected column k of the first set of unitary transfer matrices;
(d) multiplying the result of act (c) by the block diagonal derivative matrix computed in act (a); and
computing inner products between successive pairs of elements from the output of act (b) and the output of act (d).
24. The photonics-based processing system of claim 19 , wherein determining in parallel, at least some gradients of parameters of the angular representation comprises backpropagating the error vector through the angular representation of the matrix of values.
25. The photonics-based processing system of claim 19 , wherein configuring components of the photonics-based processor to represent the matrix of values as an angular representation comprises:
partitioning the matrix of values into two or more matrices having smaller dimensions; and
configuring components of the photonics-based processor to represent the values in one of the two or more matrices having smaller dimensions.
26. The photonics-based processing system of claim 19 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components using a plurality of iterations, and
wherein for some iterations of the plurality of iterations only the parameters for the second set of components is updated.
27. The photonics-based processing system of claim 19 , wherein updating the angular representation comprises updating the parameters for the first set of components, the second set of components and the third set of components, and
wherein a first learning rate for updating the first and third sets of components is different than a second learning rate for updating the second set of components.