Communication protocol for a distributed event streaming platform
Systems and methods are described for passing event messages between components of a computing cluster. An example method includes identifying, by an event sender, a receiving component to receive a record, determining, by a processing device, whether the receiving component is a protocol aware component, and upon determining that the receiving component is the protocol aware component, sending the record to the receiving component using a first protocol. Sending the record to the receiving component using the first protocol bypasses a dispatch service of the event sender.
1. A method comprising:
identifying, by an event sender, a receiving component to receive a record;
determining, by a processing device, whether the receiving component is a protocol aware component; and
upon determining that the receiving component is the protocol aware component, sending the record to the receiving component using a first protocol, wherein sending the record to the receiving component using the first protocol bypasses a dispatch service of the event sender and bypasses an ingress service of the receiving component.
2. The method of claim 1 , wherein the record is a first record and the receiving component is a first receiving component, and
wherein the method further comprises:
identifying a second receiving component to receive a second record; and
upon determining that the second receiving component is not the protocol aware component, sending the second record to the second receiving component using a second protocol different from the first protocol.
3. The method of claim 2 , wherein the ingress service of the receiving component is a Hypertext Transfer Protocol (HTTP) ingress service, and wherein sending the second record to the second receiving component using the second protocol comprises resolving a Uniform Resource Locator (URL) of the HTTP ingress service.
4. The method of claim 1 , wherein sending the record to the receiving component using the first protocol comprises resolving a Uniform Resource Locator (URL) of a target topic of the receiving component and storing the record to the target topic.
5. The method of claim 1 , further comprising:
after sending the record to the receiving component using the first protocol, determining that the record was not received successfully by the receiving component; and
responsive to determining that the record was not received successfully by the receiving component, sending the record to the receiving component using a second protocol different from the first protocol.
6. The method of claim 1 , wherein the protocol aware component comprises a component compatible with a binary Kafka protocol.
7. A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
identify, by an event sender, a receiving component to receive a record;
determine whether the receiving component is a protocol aware component; and
upon determining that the receiving component is the protocol aware component, send the record to the receiving component using a first protocol, wherein sending the record to the receiving component using the first protocol bypasses a dispatch service of the event sender and bypasses an ingress service of the receiving component.
8. The system of claim 7 , wherein the record is a first record and the receiving component is a first receiving component, and
wherein the processing device is further to:
identify a second receiving component to receive a second record; and
upon determining that the second receiving component is not the protocol aware component, send the second record to the second receiving component using a second protocol different from the first protocol.
9. The system of claim 8 , wherein the ingress service of the receiving component is a Hypertext Transfer Protocol (HTTP) ingress service, and to send the second record to the second receiving component using the second protocol, the processing device is to resolve a Uniform Resource Locator (URL) of the HTTP ingress service.
10. The system of claim 7 , wherein, to send the record to the receiving component using the first protocol, the processing device is to resolve a Uniform Resource Locator (URL) of a target topic of the receiving component and store the record to the target topic.
11. The system of claim 7 , wherein the processing device is further to:
after sending the record to the receiving component using the first protocol, determine that the record was not received successfully by the receiving component; and
responsive to determining that the record was not received successfully by the receiving component, send the record to the receiving component using a second protocol different from the first protocol.
12. The system of claim 7 , wherein the protocol aware component comprises a component compatible with a binary Kafka protocol.
13. A non-transitory computer-readable medium, having instructions stored thereon which, when executed by a processing device, cause the processing device to:
identify, by an event sender, a receiving component to receive a record;
determine, by the processing device, whether the receiving component is a protocol aware component; and
upon determining that the receiving component is the protocol aware component, send the record to the receiving component using a first protocol, wherein to send the record to the receiving component using the first protocol bypasses a dispatch service of the event sender and bypasses an ingress service of the receiving component.
14. The non-transitory computer-readable medium of claim 13 , wherein the record is a first record and the receiving component is a first receiving component, and
wherein the processing device is further to:
identify a second receiving component to receive a second record; and
upon determining that the second receiving component is not the protocol aware component, send the second record to the second receiving component using a second protocol different from the first protocol.
15. The non-transitory computer-readable medium of claim 14 , wherein the ingress service of the receiving component is a Hypertext Transfer Protocol (HTTP) ingress service, and to send the second record to the second receiving component using the second protocol, the processing device is to resolve a Uniform Resource Locator (URL) of the HTTP ingress service deployed by the second receiving component.
16. The non-transitory computer-readable medium of claim 13 , wherein, to send the record to the receiving component using the first protocol, the processing device is to resolve a Uniform Resource Locator (URL) of a target topic of the receiving component and store the record to the target topic.
17. The non-transitory computer-readable medium of claim 13 , wherein the processing device is further to:
after sending the record to the receiving component using the first protocol, determine that the record was not received successfully by the receiving component; and
responsive to determining that the record was not received successfully by the receiving component, send the record to the receiving component using a second protocol different from the first protocol.