Reestablishing synchronization in a memory system

1. A computer-implemented method for reestablishing synchronization across multiple channels in a memory system within a computer, the method comprising:

receiving an out-of-synchronization indication associated with at least one of a plurality of channels in the memory system within the computer, wherein the plurality of channels within the computer each provide communication with a memory buffer chip and a plurality of memory devices;

performing, by a memory control unit in communication with the channels within the computer, a first stage of reestablishing synchronization comprising:

selectively stopping new traffic on the plurality of channels unless a skip counter is enabled and not exceeded;

decrementing a first time period while waiting for the first time period to expire based on determining that a replay is not in progress;

resuming traffic on the plurality of channels based on the first time period expiring;

verifying that synchronization is reestablished for a second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the first stage of reestablishing synchronization for a number of times before performing a second stage of reestablishing synchronization;

performing, by the memory control unit, the second stage of reestablishing synchronization comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

waiting for a write reorder queue empty status indicator from the memory buffer chips;

decrementing the first time period while waiting for the first time period to expire based on determining that the replay is not in progress, wherein the replay comprises a recovery retransmission sequence from a replay buffer that causes a faulty channel to go out of synchronization with non-faulty instances of the channels;

resuming traffic on the plurality of channels based on the first time period expiring;

verifying that synchronization is reestablished for the second time period;

based on determining that synchronization is not reestablished for the second time period, determining whether a memory buffer chip out-of-sync condition exists; and

based on determining that the memory buffer chip out-of-sync condition does not exist, repeating the second stage of reestablishing synchronization for a number of times before advancing to a third stage of reestablishing synchronization or declaring a failure; and

performing the third stage of reestablishing synchronization based on determining that the memory buffer chip out-of-sync condition exists, the third stage comprising:

stopping new traffic on the plurality of channels;

waiting for outstanding traffic on the plurality of channels to complete;

waiting for the write reorder queue empty status indicator from the memory buffer chips before sending a synchronization command;

sending the synchronization command to the memory buffer chips on each of the channels;

waiting for a third time period to expire;

verifying that the replay did not occur during the third time period;

waiting a fourth time period before resuming traffic on the plurality of channels;

resuming traffic on the plurality of channels;

verifying that synchronization is reestablished for the second time period; and

based on determining that synchronization is not reestablished for the second time period, repeating the third stage of reestablishing synchronization for a number of times before declaring the failure.