Systems and methods for network optimization in a distributed big data environment
Systems and methods for network optimization in a distributed big data environment are provided. According to an aspect of the invention, a processor performs an optimization method by dividing a data set into a plurality of partitions. For each of the partitions, the processor generates a mathematical representation of a model by associating input data with elements of the model, wherein the mathematical representation includes an objective and at least one constraint. The processor forms a master objective by combining the objectives for the partitions, and forms a set of master constraints by combining the constraints for the partitions. The processor then generates an optimized solution based on the master objective and the master constraints.
1. A processor executable method of determining an optimized distribution of resources across products stored at a plurality of memories via a solution logic of the processor, the method comprising:
dividing, by data partitioning logic of the processor, a data set describing the resources at the plurality of memories into a plurality of partitions, wherein the data set is too large to fit within a local memory of the processor that is separate from the plurality of memories;
for each of the partitions of the plurality of memories, generating, by a mathematical representation generation logic of the processor, a linear programming file (LP file) that is a mathematical representation of a linear programming model by associating input data with elements of the model, wherein the mathematical representation includes an objective and at least one constraint;
generating, by a combination logic of the processor, a single master LP file to be stored in the local memory of the processor, wherein the single master LP file is a master mathematical representation of a master model, wherein the generating is via combining the LP files of each partition, the combining including:
forming a master objective by combining the objectives for the LP files; and
forming a set of master constraints by combining the constraints for the LP files;
storing the single master LP file in the local memory of the processor that is separate from the plurality of memories in place of the data set that is too large to fit within the local memory of the processor; and
generating, by the solution logic of the processor, an optimized solution to the single master LP file, stored in the local memory of the processor, based on the master objective and the master constraints, the optimized solution being a terminal solution for the optimized distribution of the resources described by the data set.
2. The method according to claim 1 , wherein the data set is divided into the partitions on a market-by-market basis.
3. The method according to claim 1 , wherein each of the partitions is unique.
4. The method according to claim 1 , wherein the master objective and the master constraints are formed by incrementally aggregating the objectives and the constraints for the partitions.
5. The method according to claim 1 , wherein the master constraints are formed by aggregating constraints with the same name.
6. The method according to claim 1 , wherein each of the mathematical representations includes a plurality of constraints that together form a matrix.
7. The method according to claim 1 , wherein the optimized solution is generated by linear programming.
8. The method according to claim 1 , wherein generating the optimized solution comprises:
determining whether a solution exists based on the master objective and the master constraints; and
if the solution does not exist, defining slacks that relax the master constraints, and generating the optimized solution based on the master objective and the master constraints as relaxed by the slacks.
9. The method according to claim 8 , wherein the optimized solution is generated by minimizing or maximizing the master objective while minimizing a sum of the slacks.
10. The method according to claim 1 , wherein the optimized solution maximizes or minimizes a target metric that is defined by the model.
11. The method according to claim 10 , wherein the optimized solution maximizes revenue.
12. The method according to claim 1 , wherein the optimized solution indicates a value for each of a plurality of decision variables defined by the model.
13. The method according to claim 12 , wherein each of the decision variables indicates a number of units of a product to sell.
14. An optimization system to optimize distribution of resources across products at a plurality of memories, the system comprising:
a local memory; and
a processor coupled to the memory, the processor comprising:
data partitioning logic that divides a data set describing the resources at the plurality of memories that are separate from the local memory into a plurality of partitions, wherein the data set is too large to fit within the local memory;
mathematical representation generation logic that generates, for each of the partitions of the plurality of memories, a linear programming file (LP file) that is a mathematical representation of a linear programming model by associating input data with elements of the model, wherein the mathematical representation includes an objective and at least one constraint;
combination logic that generates a single master LP file to be stored in the local memory, wherein the single master LP file is a master mathematical representation of a master model, wherein the generating is via combining the LP files of each partition, the combining including: forming a master objective by combining the objectives for the LP files, and that forms a set of master constraints by combining the constraints for the LP files; and
solution logic that generates an optimized solution to the single master LP file, stored in the local memory that is separate from the plurality of memories in place of the data set that is too large to fit within the local memory, based on the master objective and the master constraints, the optimized solution being a terminal solution for the optimized distribution of the resources described by the data set.
15. A non-transitory computer-readable medium comprising computer instructions executable by a processor to cause the processor to perform a method of determining an optimized distribution of resources across products at a plurality of memories via a program of the processor, the method comprising:
dividing a data set describing the resources at the plurality of memories into a plurality of partitions, wherein the data set is too large to fit within a local memory of the processor that is separate from the plurality of memories;
for each of the partitions of the plurality of memories, generating a linear programming file (LP file) that is a mathematical representation of a linear programming model by associating input data with elements of the model, wherein the mathematical representation includes an objective and at least one constraint;
generating, a single master LP file that is a master mathematical representation of a master model to be stored in the local memory of the processor, via combining the LP files of each partition, the combining including:
forming a master objective by combining the objectives for the partitions; and
forming a set of master constraints by combining the constraints for the partitions;
storing the single master LP file in the local memory of the processor that is separate from the plurality of memories in place of the data set that is too large to fit within the local memory of the processor; and
generating an optimized solution to the single master LP file stored in the local memory of the processor, via running the program thereon, based on the master objective and the master constraints, the optimized solution being a terminal solution for the optimized distribution of the resources described by the data set.