Reestablishing synchronization in a memory system

1. A system for reestablishing synchronization across multiple channels in a memory system within a computer, the system comprising:

a plurality of channels within the computer each providing communication with a memory buffer chip and a plurality of memory devices; and

a memory control unit coupled to the plurality of channels within the computer, the memory control unit configured to perform a method comprising:

receiving an out-of-synchronization indication associated with at least one of the channels;

performing, by the memory control unit, a first stage of a three stage process of reestablishing synchronization comprising:

selectively stopping new traffic on the plurality of channels;

waiting for a first time period to expire;

resuming traffic on the plurality of channels based on the first time period expiring;

verifying that synchronization is reestablished for a second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the first stage of reestablishing synchronization for a number of times before performing a second stage of the three stage process of reestablishing synchronization;

performing, by the memory control unit, the second stage of reestablishing synchronization comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

resuming traffic on the plurality of channels based on the first time period expiring; and

verifying that synchronization is reestablished for the second time period; and

performing a third stage of the three stage process of reestablishing synchronization based on determining that a memory buffer chip out-of-sync condition exists, the third stage comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

waiting for a write reorder queue empty status indicator from the memory buffer chips before sending a synchronization command;

sending the synchronization command to the memory buffer chips on each of the channels;

waiting for a third time period to expire;

verifying that a replay did not occur during the third time period, wherein the replay comprises a recovery retransmission sequence from a replay buffer that causes a faulty channel to go out of synchronization with non-faulty instances of the channels;

waiting a fourth time period before resuming traffic on the plurality of channels;

resuming traffic on the plurality of channels;

verifying that synchronization is reestablished for the second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the third stage of reestablishing synchronization for a number of times before declaring a failure.

2. The system of claim 1 , wherein performing the second stage of reestablishing synchronization further comprises:

waiting for the write reorder queue empty status indicator from the memory buffer chips;

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress;

based on determining that synchronization is not reestablished for the second time period, determining whether the memory buffer chip out-of-sync condition exists; and

based on determining that the memory buffer chip out-of-sync condition does not exist, repeating the second stage of reestablishing synchronization for a number of times before advancing to the third stage of reestablishing synchronization or declaring a failure.

3. The system of claim 1 , wherein performing the first stage of reestablishing synchronization further comprises:

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress.

4. The system of claim 1 , wherein selectively stopping new traffic on the plurality of channels further comprises:

stopping the new traffic on the plurality of channels unless a skip counter is enabled and not exceeded.

5. A computer implemented method for reestablishing synchronization across multiple channels in a memory system within a computer, the method comprising:

receiving an out-of-synchronization indication associated with at least one of a plurality of channels in the memory system within the computer, wherein the plurality of channels within the computer each provide communication with a memory buffer chip and a plurality of memory devices;

performing, by a memory control unit in communication with the channels within the computer, a first stage of a three stage process of reestablishing synchronization comprising:

selectively stopping new traffic on the plurality of channels;

waiting for a first time period to expire;

resuming traffic on the plurality of channels based on the first time period expiring;

verifying that synchronization is reestablished for a second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the first stage of reestablishing synchronization for a number of times before performing a second stage of the three stage process of reestablishing synchronization;

performing, by the memory control unit, the second stage of reestablishing synchronization comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

resuming traffic on the plurality of channels based on the first time period expiring; and

verifying that synchronization is reestablished for the second time period; and

performing a third stage of the three stage process of reestablishing synchronization based on determining that a memory buffer chip out-of-sync condition exists, the third stage comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

waiting for a write reorder queue empty status indicator from the memory buffer chips before sending a synchronization command;

sending the synchronization command to the memory buffer chips on each of the channels;

waiting for a third time period to expire;

verifying that a replay did not occur during the third time period, wherein the replay comprises a recovery retransmission sequence from a replay buffer that causes a faulty channel to go out of synchronization with non-faulty instances of the channels;

waiting a fourth time period before resuming traffic on the plurality of channels;

resuming traffic on the plurality of channels;

verifying that synchronization is reestablished for the second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the third stage of reestablishing synchronization for a number of times before declaring a failure.

6. The method of claim 5 , wherein performing the second stage of reestablishing synchronization further comprises:

waiting for the write reorder queue empty status indicator from the memory buffer chips;

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress;

based on determining that synchronization is not reestablished for the second time period, determining whether the memory buffer chip out-of-sync condition exists; and

based on determining that the memory buffer chip out-of-sync condition does not exist, repeating the second stage of reestablishing synchronization for a number of times before advancing to the third stage of reestablishing synchronization or declaring a failure.

7. The method of claim 5 , wherein performing the first stage of reestablishing synchronization further comprises:

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress.

8. A computer program product for reestablishing synchronization across multiple channels in a memory system within a computer, the computer program product comprising:

a non-transitory machine readable storage medium readable by a processing circuit and storing instructions for execution by the processing circuit for performing a method comprising:

receiving an out-of-synchronization indication associated with at least one of a plurality of channels in the memory system within the computer, wherein the plurality of channels within the computer each provide communication with a memory buffer chip and a plurality of memory devices:

performing, by a memory control unit in communication with the channels within the computer, a first stage of a three stage process of reestablishing synchronization comprising:

selectively stopping new traffic on the plurality of channels; waiting for a first time period to expire;

resuming traffic on the plurality of channels based on the first time period expiring;

verifying that synchronization is reestablished for a second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the first stage of reestablishing synchronization for a number of times before performing a second stage of the three stage process of reestablishing synchronization;

performing, by the memory control unit, the second stage of reestablishing synchronization comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

resuming traffic on the plurality of channels based on the first time period expiring; and

verifying that synchronization is reestablished for the second time period; and

performing a third stage of the three stage process of reestablishing synchronization based on determining that a memory buffer chip out-of-sync condition exists, the third stage comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

waiting for a write reorder queue empty status indicator from the memory buffer chips before sending a synchronization command;

sending the synchronization command to the memory buffer chips on each of the channels;

waiting for a third time period to expire;

verifying that a replay did not occur during the third time period, wherein the replay comprises a recovery retransmission sequence from a replay buffer that causes a faulty channel to go out of synchronization with non-faulty instances of the channels;

waiting a fourth time period before resuming traffic on the plurality of channels;

resuming traffic on the plurality of channels;

verifying that synchronization is reestablished for the second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the third stage of reestablishing synchronization for a number of times before declaring a failure.

9. The computer program product of claim 8 , wherein performing the second stage of reestablishing synchronization further comprises:

waiting for the write reorder queue empty status indicator from the memory buffer chips;

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress;

based on determining that synchronization is not reestablished for the second time period, determining whether the memory buffer chip out-of-sync condition exists; and

based on determining that the memory buffer chip out-of-sync condition does not exist, repeating the second stage of reestablishing synchronization for a number of times before advancing to the third stage of reestablishing synchronization or declaring a failure.

10. The computer program product of claim 8 , wherein performing the first stage of reestablishing synchronization further comprises:

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress.