Processing system with interspersed processors with multi-layer interconnection

1. An apparatus, comprising:

a plurality of processors; and

a plurality of configurable communication elements coupled to the plurality of processors in an interspersed arrangement, wherein a particular configurable communication element of the plurality of configurable communication elements is coupled to at least one processor of the plurality of processors, and wherein the particular configurable communication element is configured to:

receive one or more messages from the at least one processor; and

forward the one or more messages to a different configurable communication element of the plurality of configurable communication elements via one or more other configurable communication elements of the plurality of configurable communication elements; and

wherein a given one of the one or more other configurable communication elements is configured to receive at least part of the one or more messages.

2. The apparatus of claim 1 , wherein to receive the at least part of the one or more messages, the given one of the one or more other configurable communication elements is further configured to store the at least part of the one or more messages in a memory.

3. The apparatus of claim 2 , wherein the given one of the one or more other configurable communication elements is further configured to, in response to storing the at least part of the one or more messages in the memory, remove the at least part of the one or more messages before forwarding a remainder of the one or more messages to another one of the one or more other configurable communication elements.

4. The apparatus of claim 1 , wherein the one or more messages includes a request for access to a particular resource, and wherein the particular configurable communication element of the plurality of configurable communication elements is further configured to arbitrate between multiple requests for access to the particular resource.

5. The apparatus of claim 1 , wherein the particular configurable communication element is further configured to forward the one or more messages to the different configurable communication element based upon an amount of message traffic to the different configurable communication element.

6. The apparatus of claim 1 , wherein the particular configurable communication element is further configured to forward the one or more messages to the different configurable communication element using configuration information received via a secondary interconnection network.

7. A method of operating a multiprocessor system, comprising:

receiving, by a particular configurable communication element of a plurality of configurable communication elements, a message from a particular processor of a plurality of processors, wherein the plurality of configurable communication elements is coupled to the plurality of processors in an interspersed arrangement; and

forwarding the message to a different configurable communication element of the plurality of configurable communication elements via one or more other configurable communication elements of the plurality of configurable communication elements; and

receiving, by a given one of the one or more other configurable communication elements, at least a portion of the message.

8. The method of claim 7 , wherein receiving, by the given one of the one or more configurable communication elements, the at least a portion of the message includes storing the at least a portion of the message in a memory.

9. The method of claim 8 , further comprising, removing, by the given one of the one or more configurable communication elements, the at least a portion of the message from the message, in response to the storing the at least a portion of the message, prior to forwarding a remaining portion of the message to another one of the one or more other configurable communication elements.

10. The method of claim 7 , further comprising, forwarding the message to the different configurable communication element based upon an amount of message traffic to the different configurable communication element.

11. The method of claim 10 , further comprising, monitoring the amount of message traffic by the particular configurable communication element.

12. The method of claim 7 , further comprising, forwarding, by the particular configurable communication element, the message to the different configurable communication element based on configuration information received via a secondary interconnection network.

13. The method of claim 12 , further comprising, storing at least a portion of the configuration information in a register included in the particular configurable communication element.

14. The method of claim 7 , further comprising, forwarding, by the particular configurable communication element, the message to the different configurable communication element based upon header information include in the message.

15. A system, comprising:

a plurality of processors,

a plurality of memories interspersed among the plurality of processors, wherein each memory is coupled to a respective local subset of the plurality of processors wherein each processor included in a particular local subset is physically adjacent to a particular memory coupled to the particular local subset; and

a plurality of routing engines, wherein each routing engine of the plurality of routing engines is coupled to a respective one of the plurality of memories, a first subset of a plurality of links, and a second subset of the plurality of links;

wherein each link of the first subset of the plurality of links is coupled to a respective one of a first subset of the plurality of routing engines;

wherein each link of the second subset of the plurality of links is coupled to a respective one of a second subset of the plurality of routing engines;

wherein a particular routing engine included in the plurality of routing engines is coupled to a first subset of the plurality of processors, and to a second subset of the plurality of processors that is different from the first subset, and wherein a particular processor included in the second subset of the plurality of processor is not physically adjacent to the particular routing engine; and

wherein the particular routing engine is configured to:

receive one or more messages from a particular processor included in the first or second subset of the plurality of processors; and

forward the one or more messages to a different routing engine of the plurality of routing engines via one or more other routing engines of the plurality of routing engines; and

wherein a given one of the one or more other routing engines is configured to receive at least part of the one or more messages.

16. The system of claim 15 , wherein to receive the at least part of the one or more messages, the given one of the one or more other routing engines is further configured to store the at least part of the one or more messages in a memory.

17. The system of claim 16 , wherein the given one of the one or more other routing engines is further configured to, in response to storing the at least part of the one or more messages in the memory, remove the at least part of the one or more messages before forwarding a remainder of the one or more messages to another one of the one or more other routing engines.

18. The system of claim 15 , wherein the one or more messages includes a request for access to a particular resource, and wherein the particular routing engine of the plurality of routing engines is further configured to arbitrate between multiple requests for access to the particular resource.

19. The system of claim 15 , wherein particular routing engine is further configured to forward the one or more messages to the different routing engine based upon an amount of message traffic to the different routing engine.

20. The system of claim 15 , wherein the particular routing engine is further configured to forward the one or more messages to the different routing engine using configuration information received via a secondary interconnection network.