Communication protocol for Knative Eventing's Kafka components
Systems and methods are described for passing event messages between components of a computing cluster. An example method includes obtaining, at an event sender, a record that contains an event, wherein the event sender is a data storage component of an event streaming platform. The method also includes identifying a receiving component that is designated as a receiver for the record, and determining, by a processing device, whether the receiving component is a native protocol aware component, wherein the native protocol is native to the event streaming platform. Upon determining that the receiving component is a native protocol aware component, the record is sent to the receiving component using the native protocol. Upon determining that the receiving component is not a native protocol aware component, the record is sent to the receiving component using Hyper Text Transfer Protocol (HTTP).
1. A method comprising:
obtaining, at an event sender, a record that contains an event, wherein the event sender comprises a data storage component of an event streaming platform;
identifying a receiving component that is designated as a receiver for the record;
determining, by a processing device, whether the receiving component is a native protocol aware component, wherein the native protocol is native to the event streaming platform;
upon determining that the receiving component is a native protocol aware component, sending the record to the receiving component using the native protocol; and
upon determining that the receiving component is not a native protocol aware component, sending the record to the receiving component using Hyper Text Transfer Protocol (HTTP);
wherein sending the record to the receiving component using the native protocol comprises bypassing an HTTP dispatch service deployed by the event sender.
2. The method of claim 1 , wherein sending the record to the receiving component using HTTP comprises converting the record to a CloudEvent message.
3. The method of claim 1 , wherein sending the record to the receiving component using HTTP comprises resolving a Uniform Resource Locator (URL) of an HTTP ingress service deployed by the receiving component.
4. The method of claim 1 , wherein sending the record to the receiving component using the native protocol comprises resolving a Uniform Resource Locator (URL) of a target topic of the receiving component and storing the record to the target topic directly.
5. The method of claim 1 , wherein sending the record to the receiving component using the native protocol comprises bypassing an HTTP ingress service deployed in the receiving component.
6. The method of claim 1 , comprising:
after sending the record to the receiving component using the native protocol, waiting to receive an indication that the record was received by the receiving component; and
in response to determining that the indication has not been received, retry sending the record to the receiving component using the native protocol.
7. The method of claim 1 , comprising:
after a specified number of re-tries, sending the record to the receiving component using HTTP.
8. The method of claim 1 , wherein the event sender is a Knative Kafka source, a Knative Kafka broker, or a Knative Kafka channel.
9. A system comprising:
a memory; and
a processing device operatively coupled to the memory, the processing device to:
obtain, at the processing device, a record that contains an event;
identify a receiving component that is designated as a receiver for the record;
determine whether the receiving component is a native protocol aware component, wherein the native protocol is native to an event streaming platform;
when determined that the receiving component is a native protocol aware component, send the record to the receiving component using the native protocol; and
when determined that the receiving component is not a native protocol aware component, send the record to the receiving component using Hyper Text Transfer Protocol (HTTP);
wherein to send the record to the receiving component using the native protocol comprises to bypass an HTTP dispatch service deployed by the event sender.
10. The system of claim 9 , wherein to send the record to the receiving component using HTTP comprises to convert the record to a CloudEvent message.
11. The system of claim 9 , wherein to send the record to the receiving component using HTTP comprises to resolve a Uniform Resource Locator (URL) of an HTTP ingress service deployed by the receiving component.
12. The system of claim 9 , wherein to send the record to the receiving component using the native protocol comprises to resolve a Uniform Resource Locator (URL) of a target topic of the receiving component and to store the record to the target topic directly.
13. The system of claim 9 , wherein to send the record to the receiving component using the native protocol comprises to bypass an HTTP ingress service deployed in the receiving component.
14. The system of claim 9 , wherein the processing device is to:
receive an indication from the receiving component to indicate that the record was received by the receiving component; and
if the indication is not received after a specified number of re-tries in the native protocol, send the record to the receiving component using HTTP.
15. The system of claim 9 , wherein the processing device includes a Knative Kafka source, a Knative Kafka broker, or a Knative Kafka channel.
16. A non-transitory computer-readable medium, having instructions stored thereon which, when executed by a processing device, cause the processing device to:
obtain, at the processing device, a record that contains an event;
identify a receiving component that is designated as a receiver for the record;
determine, by the processing device, whether the receiving component is a native protocol aware component;
when determined that the receiving component is a native protocol aware component, send the record to the receiving component using the native protocol; and
when determined that the receiving component is not a native protocol aware component, send the record to the receiving component using Hyper Text Transfer Protocol (HTTP);
wherein to send the record to the receiving component using the native protocol comprises to bypass an HTTP dispatch service deployed by the event sender.
17. The non-transitory computer-readable medium of claim 16 , wherein to send the record to the receiving component using HTTP comprises to convert the record to a CloudEvent message and resolve a Uniform Resource Locator (URL) of a Kubernetes HTTP ingress service deployed by the receiving component.
18. The non-transitory computer-readable medium of claim 16 , wherein to send the record to the receiving component using the native protocol comprises to resolve a Uniform Resource Locator (URL) of a target topic of the receiving component and to store the record to the target topic directly.
19. The non-transitory computer-readable medium of claim 16 , wherein to send the record to the receiving component using the native protocol comprises to bypass an HTTP ingress service deployed in the receiving component.
20. The non-transitory computer-readable medium of claim 16 , having instructions stored thereon which, when executed by a processing device, cause the processing device to:
receive an indication from the receiving component to indicate that the record was received by the receiving component; and
in response to determining that the indication was not received after a specified number of re-tries in the native protocol, send the record to the receiving component using HTTP.