Endpoint command invocation system using worker selection

1. A system, comprising:

one or more hardware processors with associated memory that implement an endpoint command invocation system (“ECIS”), configured to execute an invocation manager (IM) and a plurality of invocation workers (IWs), wherein

the IM is configured to:

receive an invocation of a command that indicates a plurality of command recipients,

store information regarding the command in a data store,

use a global version of a connection map to determine one or more of the IWs that have at least one connection to at least one of the command recipients, and

dispatch the command to the one or more IWs; and

each of the one or more IWs is configured to:

receive the dispatched command from the IM,

determine whether a command recipient is online or offline,

when the command recipient is online:

use a local version of the connection map to determine a connection to the command recipient, and

send the dispatched command to the command recipient over the connection, and

when the command recipient is offline:

send the dispatched command to a dispatch queue corresponding to the command recipient, wherein the dispatch queue stores the dispatched command as a pending command.

2. The system of claim 1 , wherein

the ECIS is implemented as part of a service that collects data about machines in a computer network, and

the command recipients are agents executing on the machines.

3. The system of claim 2 , wherein

the service is a machine assessment service configured to assess the collected data for security vulnerabilities in the machines or the computer network.

4. The system of claim 2 , wherein

at least some of the machines are virtual machine instances.

5. The system of claim 1 , wherein

the connection is a transport layer security (TLS) connection established over a public network.

6. The system of claim 1 , wherein at least one IW is configured to:

receive a request for a command from the command recipient,

determine that the dispatch queue for the command recipient is storing one or more pending commands, and

send the one or more pending commands to the command recipient.

7. The system of claim 5 , wherein at least one IW is configured to, in response to the request from the command recipient:

update the local version of the connection map to include a new connection to the command recipient.

8. The system of claim 6 , wherein at least one IW is configured to:

update the global version of the connection map to include the new connection to the command recipient.

9. The system of claim 1 , wherein the IM is configured to:

store command information about the command in a data store, including a status of the command,

set the status to a created status when the command information is stored, and

set the status to a dispatched status when the command is sent to the first command recipient.

10. The system of claim 1 , wherein at least one IW is configured to:

determine that a write to the dispatch queue has failed, and

send the write to a backup dispatch queue.

11. The system of claim 1 , wherein

the local version of the connection map is stored in a distributed in-memory key-value database.

12. The system of claim 1 , wherein

at least one IW is configured to receive long poll requests from command recipients to continuously poll for commands.

13. The system of claim 2 , wherein

a long poll request specifies a time-to-live parameter indicating a timeout of the long poll request.

14. A method, comprising:

performing, by an endpoint command invocation system (“ECIS”) implemented by one or more computer systems:

executing an invocation manager (IM), wherein the execution comprises:

receiving an invocation of a command that indicates a plurality of command recipients;

storing information regarding the command in a data store;

using a global version of a connection map to determine one or more of invocation workers (IWs) of the ECIS that have at least one connection to at least one of the command recipients; and

dispatching the command to the one or more IWs; and

executing the one or more IWs, wherein the execution comprises:

receiving the dispatched command from the IM;

determining whether a command recipient is online or offline;

when the command recipient is online:

using a local version of the connection map to determine a connection to the command recipient; and

sending the dispatched command to the command recipient over the connection; and

when the command recipient is offline:

sending the dispatched command to a dispatch queue corresponding to the command recipient, wherein the dispatch queue stores the dispatched command as a pending command.

15. The method of claim 14 , wherein

the ECIS is implemented as part of a service that collects data about machines in a computer network, and

the command recipients are agents executing on the machines.

16. The method of claim 15 , wherein

the service is a machine assessment service configured to assess the collected data for security vulnerabilities in the machines or the computer network.

17. The method of claim 14 , wherein

the connection is a transport layer security (TLS) connection established over a public network.

18. The method of claim 14 , further comprising at least one IW:

receiving a request for a command from the command recipient;

determining that the dispatch queue for the command recipient is storing one or more pending commands; and

sending the one or more pending commands to the command recipient.

19. The method of claim 18 , further comprising at least one IW:

in response to the request from the command recipient:

updating the local version of the connection map to include a new connection to the command recipient; and

updating the global version of the connection map to include the new connection to the command recipient.

20. The method of claim 14 , further comprising at least one IW:

determining that a write to the dispatch queue has failed; and

sending the write to a backup dispatch queue.