Secure hybrid data transfer through connection and request direction decoupling

1 . A method comprising:

receiving, from a source controller by a source server at a source of a distributed streaming platform, a request to initiate a reverse connection to a destination server at a destination of the distributed streaming platform;

in response to receiving the request, establishing, by the source server, a connection to the destination server, the connection being used to transfer data from the source to the destination; and

reversing the connection, the reversing comprising:

transmitting, by the source server, a reverse connection request to the destination server after establishing the connection;

responsive to the reverse connection request, causing the destination server to remove the connection from a network server at the destination that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server and to add the connection to a network client at the destination that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client;

receiving, by the source server, a response to the reverse connection request from the destination server; and

responsive to receiving the response to the reverse connection request, removing, by the source server, the connection from a network client at the source that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client at the source and adding the connection to a network server at the source that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server at the source, the reversing enabling a client in the destination to send requests to the source server for the data on the reversed connection.

2 . The method of claim 1 , wherein:

the source server comprises a source broker in a source cluster on-premises;

the destination server comprises a destination broker in a destination cluster in a cloud; and

the establishing the connection and triggering the reversing is performed by the source broker.

3 . The method of claim 2 , further comprising:

authenticating the source broker to the destination broker using credentials of security mechanisms supported in the destination cluster, wherein the reverse connection request to reverse the established connection is authorized by the destination server based on an authenticated destination service identity associated with the connection or the reverse connection request from the source server.

4 . The method of claim 1 , further comprising:

after establishing the connection and before reversing the connection, negotiating an application programming interface (API) version to be used in subsequent requests.

5 . The method of claim 1 , wherein:

the causing the destination server to remove the connection from the network server at the destination and to add the connection to the network client at the destination comprises causing the destination server to remove a server-side channel on which the reverse connection request was received from the network server at the destination and to add a client-side channel to the network client at the destination; and

the removing the connection from the network client at the source and adding the connection to the network server at the source comprises removing a client-side channel that was used to send the reverse connection request from a network client at the source and adding a server-side channel to a network server at the source.

6 . The method of claim 1 , further comprising:

establishing, by the source controller at the source, a persistent reverse connection to a destination controller at the destination on which a destination client can request a connection from any source broker to any destination broker.

7 . The method of claim 1 , further comprising:

establishing a source service identity associated with the request based on connection or request credentials being authenticated by the source server using security mechanisms supported in a source cluster; and

authorizing, by the source server, the request to initiate the reverse connection based on the source service identity associated with the request.

8 . The method of claim 7 , further comprising:

after reversing the connection, associating, by the source server, the source service identity to a server-side of the reversed connection, wherein further requests from the destination are securely authorized using the source service identity.

9 . The method of claim 1 , further comprising:

after reversing the connection, receiving, by the source server, a request for data from the client at the destination; and

in response to receiving the request for data, transmitting the data from a topic at the source to a topic at the destination.

10 . A system comprising:

one or more hardware processors; and

a memory storing instructions that, when executed by the one or more hardware processors, cause the one or more hardware processors to perform operations comprising:

receiving, from a source controller by a source server at a source of a distributed streaming platform, a request to initiate a reverse connection to a destination server at a destination of the distributed streaming platform;

in response to receiving the request, establishing, by the source server, a connection to the destination server, the connection being used to transfer data from the source to the destination; and

reversing the connection, the reversing comprising:

transmitting, by the source server, a reverse connection request to the destination server after establishing the connection:

responsive to the reverse connection request, causing the destination server to remove the connection from a network server at the destination that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server and to add the connection to a network client at the destination that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client;

receiving, by the source server, a response to the reverse connection request from the destination server; and

responsive to receiving the response to the reverse connection request, removing, by the source server, the connection from a network client at the source that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client at the source and adding the connection to a network server at the source that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server at the source, the reversing enabling a client in the destination to send requests to the source server for the data on the reversed connection.

11 . The system of claim 10 , wherein:

the source server comprises a source broker in a source cluster on-premises;

the destination server comprises a destination broker in a destination cluster in a cloud; and

the establishing the connection and triggering the reversing is performed by the source broker.

12 . The system of claim 10 , wherein the operations further comprise:

after establishing the connection and before reversing the connection, negotiating an application programming interface (API) version to be used in subsequent requests.

13 . The system of claim 10 , wherein:

the causing the destination server to remove the connection from the network server at the destination and to add the connection to the network client at the destination in response to receiving the reverse connection request comprises causing the destination server to remove a server-side channel on which the reverse connection request was received from the network server at the destination and to add a client-side channel to the network client at the destination; and

the removing the connection from the network client at the source and adding the connection to the network server in the source comprises removing a client-side channel that was used to send the reverse connection request from a network client at the source and adding a server-side channel to a network server at the source.

14 . The system of claim 10 , wherein the operations further comprise:

after reversing the connection, receiving, by the source server, a request for data from the client at the destination; and

in response to receiving the request for data, transmitting the data from a topic at the source to a topic at the destination.

15 . A machine-storage medium comprising instructions which, when executed by one or more hardware processors of a machine, cause the machine to perform operations comprising:

receiving, from a source controller by a source server at a source of a distributed streaming platform, a request to initiate a reverse connection to a destination server at a destination of the distributed streaming platform;

in response to receiving the request, establishing, by the source server, a connection to the destination server, the connection being used to transfer data from the source to the destination; and

reversing the connection, the reversing comprising:

transmitting, by the source server, a reverse connection request to the destination server after establishing the connection;

responsive to the reverse connection request, causing the destination server to remove the connection from a network server at the destination that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server and to add the connection to a network client at the destination that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client;

receiving, by the source server, the response to the reverse connection request from the destination server; and

responsive to receiving the response to the reverse connection request, removing, by the source server, the connection from a network client at the source that establishes connections and processes traffic to and from brokers, with a state of each connection stored in a channel managed by a selector of the network client at the source and adding the connection to a network server at the source that accepts connections on one or more listeners and allocates each connection to a processor from its pool of processors, with a state of each connection stored in a channel managed by a selector associated with an assigned processor of the network server at the source, the reversing enabling a client in the destination to send requests to the source server for the data on the reversed connection.

16 . The machine-storage medium of claim 15 , wherein:

the source server comprises a source broker in a source cluster on-premises;

the destination server comprises a destination broker in a destination cluster in the cloud; and

the establishing the connection and triggering the reversing is performed by the source broker.

17 . The machine-storage medium of claim 15 , wherein the operations further comprise:

after establishing the connection and before reversing the connection, negotiating an application programming interface (API) version to be used in subsequent requests.

18 . The machine-storage medium of claim 15 , wherein:

the causing the destination server to remove the connection from the network server at the destination and to add the connection to the network client at the destination comprises causing the destination server to remove a server-side channel on which the reverse connection request was received from the network server at the destination and to add a client-side channel to the network client at the destination; and

the removing the connection from the network client at the source and adding the connection to the network server at the source comprises removing a client-side channel that was used to send the reverse connection request from a network client at the source and adding a server-side channel to a network server at the source.

19 . The machine-storage medium of claim 15 , wherein the operations further comprise:

establishing, by the source controller at the source, a persistent reverse connection to a destination controller at the destination on which a destination client can request a connection from any source broker to any destination broker.

20 . The machine-storage medium of claim 15 , wherein the operations further comprise:

after reversing the connection, receiving, by the source server, a request for data from the client at the destination; and

in response to receiving the request for data, transmitting the data from a topic at the source to a topic at the destination.