Dynamic migration of point-of-coherency and point-of-serialization in NUMA coherent interconnects
A system for dynamically controlling point-of-coherency or a point-of-serialization of shared data includes a plurality of processing engines grouped into a plurality of separate clusters and a shared communications path communicatively connecting each of the plurality of clusters to one another. Each respective cluster includes memory shared by the processing engines of the respective cluster, each unit of data in the memory being assigned to a single owner cluster responsible for maintaining an authoritative copy and a single manager cluster permanently responsible for assigning the owner cluster responsibility. Each respective cluster also includes a controller configured to receive data requests, track each of a manager status and an ownership status of the respective cluster, and control ownership status changes with respect to respective units of data based at least in part on the tracked ownership and manager statuses of the respective cluster.
1 . A multi-processing system comprising:
a plurality of separate clusters, each respective cluster including a respective plurality of processing engines and a respective memory shared by the processing engines of the respective cluster; and
a shared communications path communicatively connecting each of the plurality of clusters to one another, wherein for each unit of data stored in the memory of the plurality of clusters, a single manager cluster is permanently assigned as a manager responsible for assigning responsibility for maintaining an authoritative copy of the unit of data, and at any given time a single owner cluster is dynamically assigned the responsibility for maintaining the authoritative copy of the unit of data by the manager cluster.
2 . The multi-processing system of claim 1 , wherein the plurality of separate clusters are non-overlapping.
3 . The multi-processing system of claim 2 , wherein the respective memory shared by the processing engines of the respective cluster comprises cache lines, and wherein each respective cluster is configured to store data within the cache lines of the respective cluster.
4 . The multi-processing system of claim 3 , wherein the cache lines of the respective cluster are configured to communicate with cache lines of other clusters over the shared communications path.
5 . The multi-processing system of claim 1 , wherein each respective cluster further is programmed according to control logic to determine owner clusters, and is configured to determine the owner clusters for each unit of data originating at the respective cluster.
6 . The multi-processing system of claim 5 , wherein the control logic defines state information of the clusters, wherein the state information includes:
a first state indicating that a designated cluster is the manager and the owner cluster;
a second state indicating that the designated cluster is the manager and not the owner cluster;
a third state indicating that the designated cluster is not the manager and is the owner cluster; and
a fourth state indicating that the designated cluster is not the manager and is not the owner cluster.
7 . The multi-processing system of claim 6 , wherein the respective cluster is configured to communicate over the shared communications path only when the state information of the unit of data at the respective cluster is one of the second state or the fourth state.
8 . The multi-processing system of claim 6 , wherein the control logic includes a default setting for each unit of data originating at the respective cluster, wherein the default setting is the first state.
9 . The multi-processing system of claim 8 , wherein the control logic includes a rule for resetting the state of the unit of data to the first state in response to the unit of data being stored at two or more separate clusters.
10 . The multi-processing system of claim 8 , wherein the control logic includes a rule for determining the state of the unit of data to be the fourth state in response to the respective memory of the respective cluster lacking an indication of the state of the unit of data.
11 . The multi-processing system of claim 1 , wherein the plurality of clusters are configured to communicate at least one of a coherency check or a serialization operation over the shared communications path.
12 . The multi-processing system of claim 1 , wherein the respective memory includes one or more dynamic random access memory (DRAM) chips.
13 . The multi-processing system of claim 1 , wherein the respective memory further includes a second respective memory assigned to specific processing engines of the respective cluster, wherein the second respective memory is separate from the respective memory shared by the processing engines of the respective cluster.
14 . A method comprising:
receiving, at a respective cluster of a multi-processing system including a plurality of separate clusters communicatively connected over a shared communications path, a unit of data for storage at the respective cluster;
assigning, by the respective cluster, the unit of data to a cache line of a respective memory shared by a plurality of processing engines of the respective cluster; and
setting, by the respective cluster, each of a manager status and an ownership status of the cluster with respect to the unit of data assigned to the cache line, wherein setting the manager status and the ownership status is performed according to control logic, and wherein the control logic is configured to permanently assign one cluster of the multi-processing system as a manager cluster responsible for assigning responsibility to an owner cluster for maintaining an authoritative copy of the unit of data, and dynamically assign only one cluster of the multi-processing system as the owner cluster with the responsibility for maintaining the authoritative copy of the unit of data.
15 . The method of claim 14 , wherein the control logic defines state information of the clusters, wherein the state information includes:
a first state indicating that a designated cluster is the manager cluster and the owner cluster;
a second state indicating that the designated cluster is the manager cluster and not the owner cluster;
a third state indicating that the designated cluster is not the manager cluster and is the owner cluster; and
a fourth state indicating that the designated cluster is not the manager cluster and is not the owner cluster.
16 . The method of claim 15 , further comprising sending, by the respective cluster, a communication to ensure serialization of a data request with respect to the unit of data only when the state information of the unit of data at the respective cluster is one of the second state or the fourth state.
17 . The method of claim 16 , wherein the communication is sent to the manager cluster.
18 . The method of claim 15 , further comprising applying a default setting for each unit of data originating at the respective cluster, wherein the default setting is the first state.
19 . The method of claim 18 , further comprising resetting the state of the unit of data to the first state in response to the unit of data being stored at two or more separate clusters.
20 . The method of claim 15 , further comprising determining the state of the unit of data to be the fourth state in response to the respective memory of the respective cluster lacking an indication of the state of the unit of data.