Coordination at a machine for construction objectives
A vehicle moves through a construction environment and performs one or more construction actions in the environment. A master coordinator associated with the construction environment updates a global state of the construction environment based on the performed construction actions. The master coordinator may determine a plan of construction actions for machines in the environment using a generative language model and send instruction to perform the construction actions or capture data from an associated viewpoint based on the global state to the machines. Each machine may have a local coordinator that determines its local state based on sensor data and may alter actions within alteration constraints provided by the master coordinator. The master coordinator receives local states from the machines over time, updates the global state accordingly, and sends instructions to the one or more machines based on the updated global state.
1 . A method for performing actions in a construction zone, the method comprising:
applying a local coordinator model to a construction zone plan, the local coordinator model executing locally on a machine in the construction zone and configured to:
determine one or more actions for the machine to perform in the construction zone to implement the construction zone plan;
cause the machine to capture, using one or more sensors on the machine, one or more sensor measurements describing the one or more actions and the construction zone;
generate, based on the one or more sensor measurements, a data structure that represents a current local state of the machine performing the one or more actions; and
transmit the data structure to an overall coordinator model executing on a network system;
in response to transmitting the data structure to the overall coordinator model, receiving, at the local coordinator model, instructions from the overall coordinator model to implement a modified construction zone plan different from the construction zone plan;
applying the local coordination model to the modified construction zone plan, the local coordination model configured to:
determine one or more actions for the machine to perform in the construction zone to implement the modified construction zone plan; and
generate machine instructions that, when executed by the machine, implement the modified construction zone plan; and
executing the generated machine instructions to implement the modified construction zone plan.
2 . The method of claim 1 , wherein the construction plan includes one or more actions, each action associated with a sequence of sub-actions that, when performed by the machine in an order indicated in the sequence, cause the machine to perform the respective action.
3 . The method of claim 2 , further comprising;
generating, for each sub-action in each sequence associate with an action, a set of instructions for performing the sub-actions; and
causing the machine to execute the generated sets of instructions in the order indicated by the sequence.
4 . The method of claim 1 , wherein each action is associated with a set of alteration constraints indicative of modifications permitted for performance of the action by the machine, each alteration constraints associated with a set of conditions, the method further comprising:
in response to detecting that the set of conditions of one of the alteration constraints has occurred, providing, to the local coordinator model, the instructions associated with the respective action and the alteration constraints, wherein the local coordinator model determines one or more alterations for the action within the alteration constraints; and
receiving, from the local coordinator model, machine instructions that, when executed by the machine, implement the altered action.
5 . The method of claim 4 , wherein the altered action is associated with a new location, speed, time of performance, or ordering from the respective action.
6 . The method of claim 1 , further comprising:
providing the modified construction zone plan to the local coordinator model with one or more project resource parameters, the project resource parameters including one or more of the following:
available construction materials, available construction machines, design documents, schedules of construction machines, schedules of construction workers, specifications for each available machine, operating conditions required for each available machine, and power requirements for each available machine.
7 . The method of claim 6 , further comprising:
configuring the local coordinator model to interact with a set of third models each associated with a project resource parameter, wherein each of the set of third models has access to historical knowledge related to its respective project resource parameter.
8 . The method of claim 1 , wherein each local state reflects a current position of a respective machine in the construction zone, current conditions of the construction zone based on data captured by the respective machines, and positions of one or more machines or operators within a threshold range of the respective machine in the construction zone.
9 . A non-transitory computer-readable storage medium storing instructions that, when executed, cause a processor to perform steps comprising:
applying a local coordinator model to a construction zone plan, the local coordinator model executing locally on a machine in the construction zone and configured to:
determine one or more actions for the machine to perform in the construction zone to implement the construction zone plan;
cause the machine to capture, using one or more sensors on the machine, one or more sensor measurements describing the one or more actions and the construction zone;
generate, based on the one or more sensor measurements, a data structure that represents a current local state of the machine performing the one or more actions; and
transmit the data structure to an overall coordinator model executing on a network system;
in response to transmitting the data structure to the overall coordinator model, receiving, at the local coordinator model, instructions from the overall coordinator model to implement a modified construction zone plan different from the construction zone plan;
applying the local coordination model to the modified construction zone plan, the local coordination model configured to:
determine one or more actions for the machine to perform in the construction zone to implement the modified construction zone plan; and
generate machine instructions that, when executed by the machine, implement the modified construction zone plan; and
executing the generated machine instructions to implement the modified construction zone plan.
10 . The non-transitory computer-readable storage medium of claim 9 , wherein the construction plan includes one or more actions, each action associated with a sequence of sub-actions that, when performed by the machine in an order indicated in the sequence, cause the machine to perform the respective action.
11 . The non-transitory computer-readable storage medium of claim 10 , the steps further comprising;
generating, for each sub-action in each sequence associate with an action, a set of instructions for performing the sub-actions; and
causing the machine to execute the generated sets of instructions in the order indicated by the sequence.
12 . The non-transitory computer-readable storage medium of claim 9 , wherein each action is associated with a set of alteration constraints indicative of modifications permitted for performance of the action by the machine, each alteration constraints associated with a set of conditions, the steps further comprising:
in response to detecting that the set of conditions of one of the alteration constraints has occurred, providing, to the local coordinator model, the instructions associated with the respective action and the alteration constraints, wherein the local coordinator model determines one or more alterations for the action within the alteration constraints; and
receiving, from the local coordinator model, machine instructions that, when executed by the machine, implement the altered action.
13 . The non-transitory computer-readable storage medium of claim 12 , wherein the altered action is associated with a new location, speed, time of performance, or ordering from the respective action.
14 . The non-transitory computer-readable storage medium of claim 9 , further comprising:
providing the modified construction zone plan to the local coordinator model with one or more project resource parameters, the project resource parameters including one or more of the following:
available construction materials, available construction machines, design documents, schedules of construction machines, schedules of construction workers, specifications for each available machine, operating conditions required for each available machine, and power requirements for each available machine.
15 . The non-transitory computer-readable storage medium of claim 14 , further comprising:
configuring the local coordinator model to interact with a set of third models each associated with a project resource parameter, wherein each of the set of third models has access to historical knowledge related to its respective project resource parameter.
16 . The non-transitory computer-readable storage medium of claim 9 , wherein each local state reflects a current position of a respective machine in the construction zone, current conditions of the construction zone based on data captured by the respective machines, and positions of one or more machines or operators within a threshold range of the respective machine in the construction zone.
17 . A system comprising:
a processor; and
a non-transitory computer-readable storage medium storing instructions that, when executed, cause the processor to perform steps comprising:
applying a local coordinator model to a construction zone plan, the local coordinator model executing locally on a machine in the construction zone and configured to:
determine one or more actions for the machine to perform in the construction zone to implement the construction zone plan;
cause the machine to capture, using one or more sensors on the machine, one or more sensor measurements describing the one or more actions and the construction zone;
generate, based on the one or more sensor measurements, a data structure that represents a current local state of the machine performing the one or more actions; and
transmit the data structure to an overall coordinator model executing on a network system;
in response to transmitting the data structure to the overall coordinator model, receiving, at the local coordinator model, instructions from the overall coordinator model to implement a modified construction zone plan different from the construction zone plan;
applying the local coordination model to the modified construction zone plan, the local coordination model configured to:
determine one or more actions for the machine to perform in the construction zone to implement the modified construction zone plan; and
generate machine instructions that, when executed by the machine, implement the modified construction zone plan; and
executing the generated machine instructions to implement the modified construction zone plan.
18 . The system of claim 17 , wherein the construction plan includes one or more actions, each action associated with a sequence of sub-actions that, when performed by the machine in an order indicated in the sequence, cause the machine to perform the respective action.
19 . The system of claim 18 , the steps further comprising;
generating, for each sub-action in each sequence associate with an action, a set of instructions for performing the sub-actions; and
causing the machine to execute the generated sets of instructions in the order indicated by the sequence.
20 . The system of claim 17 , wherein each action is associated with a set of alteration constraints indicative of modifications permitted for performance of the action by the machine, each alteration constraints associated with a set of conditions, the steps further comprising:
in response to detecting that the set of conditions of one of the alteration constraints has occurred, providing, to the local coordinator model, the instructions associated with the respective action and the alteration constraints, wherein the local coordinator model determines one or more alterations for the action within the alteration constraints; and
receiving, from the local coordinator model, machine instructions that, when executed by the machine, implement the altered action.