Scalable simulation system with scalable data propagation
Methods, systems, computer-readable media, and apparatuses for grouping bridges in a simulation are presented. In some examples, grouping bridges may result in more efficient usage of data connections in a simulation and less duplicative data being sent during the simulation. The simulation may be performed by receiving an indication of a streaming query for each worker of a plurality of workers in a worker layer. A plurality of bridges in a bridge layer may be run, and the plurality of bridges may be configured to facilitate data communications between the plurality of workers in the worker layer and one or more databases in a database layer. Each worker of the plurality of workers may be assigned to a different bridge of the plurality of bridges. Based on the streaming query for each worker, the plurality of bridges may be grouped into different groups of bridges.
1. One or more non-transitory computer readable media storing computer executable instructions that, when executed, cause a system to perform a computer simulation by:
receiving an indication of a streaming query for each worker of a plurality of workers in a worker layer of the computer simulation;
running, on a plurality of machines, a plurality of bridges in a bridge layer of the computer simulation, wherein the plurality of bridges are configured to facilitate data communications between the plurality of workers in the worker layer and one or more databases in a database layer of the computer simulation;
assigning each worker of the plurality of workers to a different bridge of the plurality of bridges; and
based on the streaming query for each worker, grouping the plurality of bridges into different groups of bridges, wherein each group of bridges is running on a different machine of the plurality of machines.
2. The one or more non-transitory computer readable media of claim 1 , wherein the plurality of bridges comprise a first plurality of bridges, the one or more non-transitory computer readable media storing computer executable instructions that, when executed, cause the system to perform the computer simulation by:
running, on a second plurality of machines, a second plurality of bridges, wherein the first plurality of bridges are configured to facilitate data communications between the plurality of workers in the worker layer and the second plurality of bridges, and wherein the second plurality of bridges are configured to facilitate data communications between the first plurality of bridges and the one or more databases.
3. The one or more non-transitory computer readable media of claim 1 , wherein a first worker, of the plurality of workers, comprises an unmanaged worker, and wherein a second worker, of the plurality of workers, comprises a managed worker.
4. The one or more non-transitory computer readable media of claim 1 , wherein the one or more databases comprise a plurality of database shards, wherein each database shard is configured to store data from distinct data domains.
5. The one or more non-transitory computer readable media of claim 1 , wherein grouping the plurality of bridges into different groups of bridges is based on an amount of data corresponding to an overlapping portion of streaming queries in each of the different groups of bridges.
6. The one or more non-transitory computer readable media of claim 1 , storing computer executable instructions that, when executed, cause the system to perform the computer simulation by:
running, on a machine of the plurality of machines, a database client, wherein each bridge running on the machine is connected to the database client, and wherein the database client is configured to subscribe to a streaming query for receiving, from the one or more databases, streaming data corresponding to the streaming query.
7. The one or more non-transitory computer readable media of claim 6 , storing computer executable instructions that, when executed, cause the system to perform the computer simulation by:
combining streaming queries on the machine into a combined streaming query, wherein the database client is configured to subscribe to the combined streaming query for receiving, from the one or more databases, streaming data corresponding to the combined streaming query.
8. The one or more non-transitory computer readable media of claim 6 , wherein the database client is configured to divide the streaming data corresponding to the streaming query into a plurality of portions of streaming data, and wherein the database client is configured to send, to each bridge running on the machine, one of the plurality of portions of the streaming data.
9. The one or more non-transitory computer readable media of claim 1 , storing computer executable instructions that, when executed, cause the system to perform the computer simulation by:
determining to migrate, to a first machine of the plurality of machines, a first bridge of the plurality of bridges and running on a second machine of the plurality of machines, wherein a first worker is assigned to the first bridge;
running, on the first machine of the plurality of machines, a second bridge; and
assigning, to the second bridge running on the first machine, the first worker.
10. The one or more non-transitory computer readable media of claim 9 , wherein determining to migrate the first bridge is based on an amount of data corresponding to an overlapping portion of a streaming query for the first worker with streaming queries for workers assigned to bridges running on the first machine.
11. The one or more non-transitory computer readable media of claim 1 , wherein the plurality of machines comprise a plurality of virtual machines.
12. A method comprising:
receiving, by a computing device, an indication of a streaming query for each worker of a plurality of workers in a worker layer of a computer simulation;
running, on a plurality of machines, a plurality of bridges in a bridge layer of the computer simulation, wherein the plurality of bridges are configured to facilitate data communications between the plurality of workers in the worker layer and one or more databases in a database layer of the computer simulation;
assigning, by the computing device, each worker of the plurality of workers to a different bridge of the plurality of bridges; and
based on the streaming query for each worker, grouping, by the computing device, the plurality of bridges into different groups of bridges, wherein each group of bridges is running on a different machine of the plurality of machines.
13. The method of claim 12 , wherein the plurality of bridges comprise a first plurality of bridges, the method further comprising:
running, on a second plurality of machines, a second plurality of bridges, wherein the first plurality of bridges are configured to facilitate data communications between the plurality of workers in the worker layer and the second plurality of bridges, and wherein the second plurality of bridges are configured to facilitate data communications between the first plurality of bridges and the one or more databases.
14. The method of claim 12 , wherein a first worker, of the plurality of workers, comprises an unmanaged worker, and wherein a second worker, of the plurality of workers, comprises a managed worker.
15. The method of claim 12 , wherein the one or more databases comprise a plurality of database shards, wherein each database shard is configured to store data from distinct data domains.
16. The method of claim 12 , wherein grouping the plurality of bridges into different groups of bridges is based on an amount of data corresponding to an overlapping portion of streaming queries in each of the different groups of bridges.
17. An apparatus comprising:
one or more computer processor controlling some operations of the apparatus; and
memory storing computer-executable instructions that, when executed by the one or more computer processor, cause the apparatus to:
receive an indication of a streaming query for each worker of a plurality of workers in a worker layer of a computer simulation;
run, on a plurality of machines, a plurality of bridges in a bridge layer of the computer simulation, wherein the plurality of bridges are configured to facilitate data communications between the plurality of workers in the worker layer and one or more databases in a database layer of the computer simulation;
assign each worker of the plurality of workers to a different bridge of the plurality of bridges; and
based on the streaming query for each worker, group the plurality of bridges into different groups of bridges, wherein each group of bridges is running on a different machine of the plurality of machines.
18. The apparatus of claim 17 , wherein the memory stores computer-executable instructions that, when executed by the one or more computer processor, cause the apparatus to:
run, on a machine of the plurality of machines, a database client, wherein each bridge running on the machine is connected to the database client, and wherein the database client is configured to subscribe to a streaming query for receiving, from the one or more databases, streaming data corresponding to the streaming query.
19. The apparatus of claim 18 , wherein the memory stores computer-executable instructions that, when executed by the one or more computer processor, cause the apparatus to:
combine streaming queries on the machine into a combined streaming query, wherein the database client is configured to subscribe to the combined streaming query for receiving, from the one or more databases, streaming data corresponding to the combined streaming query.
20. The apparatus of claim 18 , wherein the database client is configured to divide the streaming data corresponding to the streaming query into a plurality of portions of streaming data, and wherein the database client is configured to send, to each bridge running on the machine, one of the plurality of portions of the streaming data.