Message consumer orchestration framework
In a decoupled messaging system, a producer device sends produced messages to a message fabric. The message fabric either stores the received message in designated queues, from which consumer devices may retrieve the messages, or broadcasts the messages to consumer devices that subscribe to topics assigned to the messages. A message consumption orchestrator service gathers capabilities information for the message producer, the message fabric, and the message consumer, and configures resources allocated for the message consumer to more fully optimize message throughput. Further, the message consumption orchestrator identifies patterns in changes of the rate of produced messages, i.e., through analysis of historical data and/or received information. The message consumption orchestrator configures resources for the message consumer to account for predicted need based on the identified patterns.
1. A method comprising:
using a message consumer executing on a particular device to pull, from a message fabric, messages that were produced by a message producer;
gathering capabilities information for at least one of:
the message producer,
the message fabric, or
the message consumer;
based, at least in part, on the capabilities information, automatically executing at least one consumer clone, of the message consumer, on one or more devices, other than the particular device;
wherein, prior to executing the at least one consumer clone, the one or more devices are not being used to pull messages from the message fabric for the message consumer;
wherein automatically executing the at least one consumer clone, of the message consumer, on one or more devices, other than the particular device is further based, at least in part, on a target pull rate for the message consumer pulling messages from the message fabric;
wherein the method is performed by one or more computing devices.
2. The method of claim 1 , further comprising setting the target pull rate by:
determining whether a particular queue within the message fabric has more than a threshold number of messages stored therein;
in response to determining that the particular queue within the message fabric has more than the threshold number of messages stored therein, setting the target pull rate to be faster than a rate that the message producer is currently sending messages to the particular queue.
3. A method comprising:
based on historical information, identifying a pattern in timing of changes in a rate of message flow between a message producer and a message fabric;
based, at least in part, on the pattern, predicting a future change to a flow of messages from the message producer through the message fabric to a message consumer;
in response to predicting the future change to the flow of messages, automatically scheduling a future adjustment of consumer resources allocated to the message consumer;
wherein the future adjustment is scheduled to occur at a scheduled adjustment time;
in response to reaching the scheduled adjustment time, and based, at least in part, on the predicted change, automatically adjusting consumer resources allocated to the message consumer;
receiving at least a portion of the historical information from a third party information source;
wherein the historical information comprises information identifying two or more critical dates of a certain type;
wherein a first critical date of the two or more critical dates occurred in the past;
wherein predicting the future change to the flow of messages from the message producer through the message fabric to the message consumer comprises:
determining that, on the first critical date, the flow of messages had a particular peak rate, and based, at least in part, on the type of the first critical date being the same as the type of a second critical date, of the two or more critical dates, predicting that the second critical date of the two or more critical dates is the date of the future change to the flow of messages;
wherein the method is performed by one or more computing devices.
4. The method of claim 3 , further comprising gathering at least a portion of the historical information from one or more of the message producer, the message fabric, or the message consumer.
5. One or more non-transitory computer readable media storing one or more sequences of instructions which, when executed by one or more processors, cause:
using a message consumer executing on a particular device to pull, from a message fabric, messages that were produced by a message producer;
gathering capabilities information for at least one of:
the message producer,
the message fabric, or
the message consumer;
based, at least in part, on the capabilities information, automatically executing at least one consumer clone, of the message consumer, on one or more devices, other than the particular device;
wherein, prior to executing the at least one consumer clone, the one or more devices are not being used to pull messages from the message fabric for the message consumer;
wherein automatically executing the at least one consumer clone, of the message consumer, on one or more devices, other than the particular device is further based, at least in part, on a target pull rate for the message consumer pulling messages from the message fabric.
6. The one or more non-transitory computer readable media of claim 5 , wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, cause setting the target pull rate by:
determining whether a particular queue within the message fabric has more than a threshold number of messages stored therein;
in response to determining that the particular queue within the message fabric has more than the threshold number of messages stored therein, setting the target pull rate to be faster than a rate that the message producer is currently sending messages to the particular queue.
7. One or more non-transitory computer readable media storing one or more sequences of instructions which, when executed by one or more processors, cause:
based on historical information, identifying a pattern in timing of changes in a rate of message flow between a message producer and a message fabric;
based, at least in part, on the pattern, predicting a future change to a flow of messages from the message producer through the message fabric to a message consumer;
in response to predicting the future change to the flow of messages, automatically scheduling a future adjustment of consumer resources allocated to the message consumer;
wherein the future adjustment is scheduled to occur at a scheduled adjustment time;
in response to reaching the scheduled adjustment time, and based, at least in part, on the predicted change, automatically adjusting consumer resources allocated to the message consumer;
receiving at least a portion of the historical information from a third party information source;
wherein the historical information comprises information identifying two or more critical dates of a certain type;
wherein a first critical date of the two or more critical dates occurred in the past;
wherein predicting the future change to the flow of messages from the message producer through the message fabric to the message consumer comprises:
determining that, on the first critical date, the flow of messages had a particular peak rate, and
based, at least in part, on the type of the first critical date being the same as the type of a second critical date, of the two or more critical dates, predicting that the second critical date of the two or more critical dates is the date of the future change to the flow of messages.
8. The one or more non-transitory computer readable media of claim 7 , wherein the one or more sequences of instructions further comprise instructions which, when executed by one or more processors, cause gathering at least a portion of the historical information from one or more of the message producer, the message fabric, or the message consumer.