Shuffling-type gradient method for training machine learning models with big data

1. A computer-implemented method for training of machine learning models with a shuffling-type gradient operation, the computer-implemented method comprising:

uniformly randomly distributing data samples or coordinate updates of a training data;

calculating the learning rates for a no-shuffling scheme and a shuffling scheme;

performing a combined operation of the no-shuffling scheme and the shuffling scheme of the training data using a stochastic gradient descent (SGD) algorithm;

switching to performing only the shuffling scheme from the no-shuffling scheme based on one or more predetermined criterion; and

training the machine learning models with the training data based on the combined no-shuffling scheme and shuffling scheme.

2. The computer-implemented method of claim 1 , wherein the data samples or coordinate updates of the training data comprise big data.

3. The computer-implemented method of claim 2 , wherein the SGD algorithm used for performing the combined operation of the no-shuffling scheme and the shuffling scheme comprises at least one of Adagrad, Adam, RMSProp, or SGD Momentum algorithms as the no-shuffling scheme.

4. The computer-implemented method of claim 2 , wherein:

the one or more predetermined criterion comprise a machine learning rate; and

the switching to performing only the shuffling scheme occurs according to the learning rate.

5. The computer-implemented method of claim 2 , wherein:

the one or more predetermined criterion comprise a learning rate; and

the switching to performing only the shuffling scheme occurs according to the learning rate.

6. The computer-implemented method of claim 2 , wherein the predetermined criterion comprises learning rate rules, and the method further comprises providing one or more learning rate rules that determine one or more of a predetermined theoretical complexity and a practical performance of the SGD algorithm.

7. The computer-implemented method of claim 6 , wherein the learning rate rules include:

diminishing a learning rate scheme and obtaining a highest convergence rate result in a non-convex supervised learning model.

8. The computer-implemented method of claim 2 , wherein the machine learning models are trained for a deep neural network (DNN) application.

9. The computer-implemented method of claim 2 , wherein the machine learning models are trained for one or more of supervised learning operations.

10. A computing device including a configurable machine learning framework for training of machine learning models with a shuffling-type gradient operation of big data, the machine learning framework comprising:

a processor;

a memory coupled to the processor, the memory storing instructions to cause the processor to perform acts comprising:

generate a machine learning framework including a combined operation of a no-shuffling scheme and a shuffling scheme using a gradient descent (SGD) algorithm, the machine learning framework further configured for receiving a predetermined criterion for switching to performing only the shuffling scheme from the no-shuffling scheme, and training the machine learning models based on the combined operation of the no-shuffling scheme and the shuffling scheme.

11. The computing device of claim 10 , further comprising receiving a user-selection of the predetermined criterion for switching to performing only the shuffling scheme from the no-shuffling scheme.

12. The computing device of claim 10 , further comprising configuring the machine learning framework to perform the combined operation of the shuffling scheme and the no-shuffling scheme from one or more SGD algorithms comprising Adagrad, Adam, RMSProp, or SGD Momentum as the no-shuffling scheme.

13. The computing device of claim 10 , wherein the predetermined criterion for switching to performing only the shuffling scheme from the no-shuffling scheme comprises a learning rate.

14. The computing device of claim 10 , wherein the predetermined criterion for switching to performing only the shuffling scheme from the no-shuffling scheme comprises a learning rate.

15. The computing device of claim 10 , wherein the machine learning framework is configured to prompt for at least one of an error rate or a learning rate as the predetermined criterion for switching to performing only the shuffling scheme from the no-shuffling scheme.

16. A computing device configured for performing a shuffling-type gradient for training machine learning models with big data using a stochastic gradient descent (SGD), the computer-implemented method comprising:

a processor;

a memory coupled to the processor, the memory storing instructions to cause the processor to perform acts comprising:

uniformly randomly distributing data samples or coordinate updates;

calculating the learning rates for a no-shuffling scheme and a shuffling scheme;

performing a combined operation of the no-shuffling scheme and the shuffling scheme of the data samples or coordinate updates;

switching to performing only the shuffling scheme from the no-shuffling scheme based on a predetermined criterion; and

training the learning models with the data samples or coordinate updates based on performing the combined no-shuffling scheme and the shuffling scheme.

17. The computing device of claim 16 , wherein switching to performing only the shuffling scheme occurs according to a learning rate of the machine learning algorithm.

18. The computing device of claim 16 , wherein

switching to performing only the shuffling scheme occurs according to an error learning rate of the machine learning.

19. A computer-implemented method of determining a shuffling-type gradient method for training a machine learning model with big data using a stochastic gradient descent (SGD) method, the method comprising:

uniformly randomly distributing data samples or coordinate updates of a training data;

calculating learning rates for a no-shuffling scheme and a shuffling scheme;

combining a no-shuffling scheme and a shuffling scheme into a framework, and

operating a diminishing learning rate scheme and obtaining a highest known convergence rate result for randomly distributed samples in a non-convex supervised learning model.

20. The computer-implemented method of claim 19 , wherein the machine learning models are trained for a deep neural network (DNN) application.

21. The computer-implemented method of claim 19 , wherein the machine learning models are trained for supervised learning operations.

22. A non-transitory computer-readable storage medium tangibly embodying a computer-readable program code having computer-readable instructions that, when executed, causes a computer device to perform a shuffling-type gradient method for training a machine learning model with big data using a stochastic gradient descent (SGD), the shuffling-type gradient method comprising:

uniformly randomly distributing data samples or coordinate updates:

calculating learning rates for a no-shuffling scheme and a shuffling scheme;

performing a combined operation of the no-shuffling scheme and the shuffling scheme of the data samples or coordinate updates;

switching to performing operation of only the shuffling scheme from the no-shuffling scheme based on one or more predetermined criterion; and

training the learning models with the data samples or coordinate updates based on performing the combined no-shuffling scheme and the shuffling scheme.

23. The non-transitory computer-readable storage medium of claim 22 , wherein the predetermined criterion comprises learning rate rules, and the shuffling-type gradient method further comprises providing one or more learning rate rules that determine one or more of a predetermined theoretical complexity and a practical performance of the SGD algorithm.

24. The non-transitory computer-readable storage medium method of claim 23 , wherein providing the learning rate rules includes operating a diminishing learning rate scheme and obtaining a highest convergence rate result in a non-convex supervised learning model.

25. The non-transitory computer-readable storage medium according to claim 22 , wherein the machine learning models are trained for one or more of supervised learning operations.