Apparatus and method for low-latency message request/response processing
An apparatus and method for low-latency real-time request processing. For example, one implementation comprises a front end and a plurality of micro-services each utilizing a different type of hardware and/or software tailored to perform a specialized type of processing. In one implementation, the front end receives a request message (e.g., an HTTP request) containing program code/data in its header and forwards the request to certain micro-services in accordance with the type of processing required and/or other variables such as load on the micro-services. A central storage may be shared by the front end and the micro-services to exchange data and/or the data may be stored in the intermediate messages passed between the front end and micro-services.
1. A method comprising:
receiving an application-layer request message from a client electronic device, the application-layer request message including program data;
transmitting an initial message to a first micro-service of a series of micro-services associated with the program data, wherein the series of micro-services are selected from a plurality of micro-services, each micro-service in the plurality of micro-services utilizes different hardware or software for processing, and the first micro-service provides a first type of processing;
processing the program data at the first micro-service using the first type of processing to generate an intermediate result;
providing the intermediate result to a second micro-service in the series of micro-services based on a determination of a second type of processing required by the program data;
generating responsive to receipt of a final result from a last micro-service in the series of micro-services, an application-layer response message using the final result; and
transmitting the application-layer response message to the client electronic device.
2. The method of claim 1 , wherein the first micro-service comprises a first type of hardware accelerator to perform the first type of processing on the program data.
3. The method of claim 1 further comprising:
storing the intermediate result in a data storage service coupled to the plurality of micro-services; and
transmitting to the second micro-service an intermediate message with a link that points to the intermediate result in the data storage service.
4. The method of claim 1 , further comprising:
identifying a current load on each of the plurality of micro-services; and
selecting the first micro-service based on the current load on the first micro-service relative to a current load on another micro-service in the plurality of micro-services, wherein the another micro-service is implemented in an electronic device with a same type of hardware or software to perform a same type of processing as the first micro-service.
5. The method of claim 1 , wherein each of the plurality of micro-services are to:
perform network-related operations in accordance with the program data; and
perform application-specific operations in accordance with the program data or an intermediate result.
6. The method of claim 5 , wherein the network-related operations comprise evaluating quality-of-service or latency requirements specified in the program data.
7. The method of claim 6 , wherein the program data is to specify a maximum latency which the client electronic device will accept,
wherein each micro-service in the plurality of micro-services are to evaluate the program data to determine whether a latency in processing the application-layer request message will rise above a maximum latency and, if so, to transmit a message informing the client electronic device.
8. The method of claim 5 , wherein the application-specific operations comprise image processing operations,
wherein the first micro-service is to decode an image to generate a decoded image and the second micro-service in the series of micro-services is to perform modifications to the decoded image to generate a modified image, and
wherein the intermediate result of the first micro-service comprises the decoded image used by the second micro-service to generate an intermediate result comprising the modified image.
9. The method of claim 1 , further comprising:
updating the application-layer request message with identification codes uniquely identifying the series of micro-services.
10. A non-transitory machine-readable medium having instructions stored thereon which, when executed by one or more machines, causes the one or more machines to:
receive an application-layer request message from a client electronic device, the application-layer request message including program data;
transmit an initial message to a first micro-service of a series of micro-services associated with the program data, wherein the series of micro-services are selected from a plurality of micro-services, each micro-service in the plurality of micro-services utilizes different hardware or software for processing, and the first micro-service provides a first type of processing;
process the program data at the first micro-service using the first type of processing to generate an intermediate result;
provide the intermediate result to a second micro-service in the series of micro-services based on a determination of a second type of processing required by the program data;
generate responsive to receipt of a final result from a last micro-service in the series of micro-services, an application-layer response message using the final result; and
transmit the application-layer response message to the client electronic device.
11. The non-transitory machine-readable medium of claim 10 , wherein the first micro-service comprises a first type of hardware accelerator to perform the first type of processing on the program data.
12. The non-transitory machine-readable medium of claim 10 , wherein the instructions further cause the one or more machines to:
store the intermediate result in a data storage service coupled to the plurality of micro-services; and
transmit to the second micro-service an intermediate message with a link that points to the intermediate result in the data storage service.
13. The non-transitory machine-readable medium of claim 10 , wherein the instructions further cause the one or more machines to:
identify a current load on each of the plurality of micro-services; and
select the first micro-service based on the current load on the first micro-service relative to a current load on another micro-service in the plurality of micro-services, wherein the another micro-service is implemented in an electronic device with a same type of hardware or software to perform a same type of processing as the first micro-service.
14. The non-transitory machine-readable medium of claim 10 , wherein each of the plurality of micro-services are to:
perform network-related operations in accordance with the program data; and
perform application-specific operations in accordance with the program data or an intermediate result.
15. The non-transitory machine-readable medium of claim 14 , wherein the network-related operations comprise evaluating quality-of-service or latency requirements specified in the program data.
16. The non-transitory machine-readable medium of claim 15 , wherein the program data is to specify a maximum latency which the client electronic device will accept,
wherein each micro-service in the plurality of micro-services are to evaluate the program data to determine whether a latency in processing the application-layer request message will rise above a maximum latency and, if so, to transmit a message informing the client electronic device.
17. The non-transitory machine-readable medium of claim 14 , wherein the application-specific operations comprise image processing operations,
wherein the first micro-service is to decode an image to generate a decoded image and the second micro-service in the series of micro-services is to perform modifications to the decoded image to generate a modified image, and
wherein the intermediate result of the first micro-service comprises the decoded image used by the second micro-service to generate an intermediate result comprising the modified image.
18. The non-transitory machine-readable medium of claim 1 , wherein the instructions further cause the one or more machines to:
update the application-layer request message with identification codes uniquely identifying the series of micro-services.
19. An electronic device, comprising:
a set of memory components; and
a processor coupled to the set of memory components, wherein the set of memory components store instructions which, when executed by the processor, cause the electronic device to:
receive an application-layer request message from a client electronic device, the application-layer request message including program data;
transmit an initial message to a first micro-service of a series of micro-services associated with the program data, wherein the series of micro-services are selected from a plurality of micro-services, each micro-service in the plurality of micro-services utilizes different hardware or software for processing, and the first micro-service provides a first type of processing;
process the program data at the first micro-service using the first type of processing to generate an intermediate result;
provide the intermediate result to a second micro-service in the series of micro-services based on a determination of a second type of processing required by the program data;
generate responsive to receipt of a final result from a last micro-service in the series of micro-services, an application-layer response message using the final result; and
transmit the application-layer response message to the client electronic device.
20. The system of claim 19 , wherein the instructions when executed by the processor further cause the electronic device to:
store the intermediate result in a data storage service coupled to the plurality of micro-services; and
transmit to the second micro-service an intermediate message with a link that points to the intermediate result in the data storage service.