Granted Patent B2

US 11,875,198 · App. 17/208,295 · Granted Jan 16, 2024

Inventors: Vladimir Shveidel (Pardes-Hana, IL); Lior Kamran (Rishon LeZion, IL)

Assignee: EMC IP Holding Company LLC

G06F9/522G06F9/30087G06F9/3877G06F9/4881G06F9/5038

Abstract

Claims (59)

1. An apparatus comprising:

at least one processing device comprising a processor coupled to a memory;

said at least one processing device being configured:

to establish at least first and second groups of synchronization objects in a storage system based at least in part on synchronization object type, the first group of synchronization objects being established for a first synchronization object type, the second group of synchronization objects being established for a second synchronization object type different than the first synchronization object type, and the first and second groups of synchronization objects being associated with respective ones of a first synchronization object type queue and a second synchronization object type queue, with each such synchronization object type queue being implemented at least in part utilizing the memory; and

for each of the first and second groups of synchronization objects:

to insert entries into a corresponding one of the synchronization object type queues for respective synchronization objects of a corresponding one of the first and second groups of synchronization objects;

to execute on the processor an associated one of a plurality of monitor threads, the associated monitor thread being for the corresponding group, the associated monitor thread being configured to scan the entries of the corresponding synchronization object type queue for synchronization object issue detection, in accordance with one or more designated detection conditions, wherein different detection conditions are designated for use with different ones of the first and second groups of synchronization objects; and

responsive to at least one of the scanned entries meeting the one or more designated detection conditions, to take at least one automated action for its associated synchronization object;

wherein the associated monitor thread is configured to operate in accordance with a series of monitoring cycles and wherein one or more synchronization objects added to the corresponding group in a given one of the monitoring cycles have corresponding entries inserted into an additional queue, and further wherein the entries inserted into the additional queue are subsequently merged into the synchronization object type queue for the corresponding group; and

wherein taking at least one automated action comprises executing a callback function specified for the corresponding group, the callback function being specified for the corresponding group in conjunction with establishing the corresponding group for the synchronization object type.

2. The apparatus of claim 1 wherein said at least one processing device comprises at least a subset of a plurality of processing cores of the storage system.

3. The apparatus of claim 1 wherein said at least one processing device executes threads of one or more applications utilizing at least one of a plurality of processing cores of the storage system, the threads of the one or more applications comprising different sub-threads of a first thread of a first application executing on a particular one of the plurality of processing cores of the storage system.

4. The apparatus of claim 3 wherein the first application comprises a block-storage application of the storage system and the first thread comprises a truck thread that when executing on the particular processing core is configured to utilize at least a portion of available processing resources of that processing core.

5. The apparatus of claim 3 wherein each of the sub-threads of the first thread is in one of multiple designated states at a particular point in time, including a running state, a ready state and a suspended state.

6. The apparatus of claim 5 wherein different ones of the sub-threads that are in the suspended state are enqueued in respective different ones of a plurality of thread queues in order to wait for access to respective corresponding synchronization objects associated with resources of the particular processing core.

7. The apparatus of claim 1 wherein the synchronization objects comprise one or more of:

respective distinct locks in the storage system; and

at least one of a semaphore, a timer, a barrier, a memory pool and a thread pool.

8. The apparatus of claim 1 wherein establishing at least first and second groups of synchronization objects in a storage system based at least in part on synchronization object type comprises:

receiving registration information for a particular synchronization object type via an application programming interface; and

establishing a group for the particular synchronization object type based at least in part on the registration information.

9. The apparatus of claim 1 wherein the entries in the additional queue for the one or more synchronization objects added to the group of synchronization objects in the given monitoring cycle are subsequently merged into the synchronization object type queue for the corresponding group in conjunction with completion of the given monitoring cycle and are monitored by the associated monitor thread via their entries in the synchronization object type queue in a next monitoring cycle.

10. The apparatus of claim 1 wherein one or more synchronization objects to be deleted from the corresponding group of synchronization objects in a given one of the monitoring cycles have their respective entries marked as deleted in the synchronization object type queue in the given monitoring cycle and any entries so marked are removed from the synchronization object type queue by the associated monitor thread in conjunction with the scan of the entries of the synchronization object type queue.

11. The apparatus of claim 1 wherein taking at least one automated action further comprises at least one of:

generating an alert to at least one other thread;

performing one or more diagnostic operations; and

storing identifying information of at least one other thread holding the corresponding synchronization object.

12. A computer program product comprising a non-transitory processor-readable storage medium having stored therein program code of one or more software programs, wherein the program code when executed by at least one processing device comprising a processor coupled to a memory, causes said at least one processing device:

to establish at least first and second groups of synchronization objects in a storage system based at least in part on synchronization object type, the first group of synchronization objects being established for a first synchronization object type, the second group of synchronization objects being established for a second synchronization object type different than the first synchronization object type, and the first and second groups of synchronization objects being associated with respective ones of a first synchronization object type queue and a second synchronization object type queue, with each such synchronization object type queue being implemented at least in part utilizing the memory; and

for each of the first and second groups of synchronization objects:

to insert entries into a corresponding one of the synchronization object type queues for respective synchronization objects of a corresponding one of the first and second groups of synchronization objects;

to execute on the processor an associated one of a plurality of monitor threads, the associated monitor thread being for the corresponding group, the associated monitor thread being configured to scan the entries of the corresponding synchronization object type queue for synchronization object issue detection, in accordance with one or more designated detection conditions, wherein different detection conditions are designated for use with different ones of the first and second groups of synchronization objects; and

responsive to at least one of the scanned entries meeting the one or more designated detection conditions, to take at least one automated action for its associated synchronization object;

wherein the associated monitor thread is configured to operate in accordance with a series of monitoring cycles and wherein one or more synchronization objects added to the corresponding group in a given one of the monitoring cycles have corresponding entries inserted into an additional queue, and further wherein the entries inserted into the additional queue are subsequently merged into the synchronization object type queue for the corresponding group; and

wherein taking at least one automated action comprises executing a callback function specified for the corresponding group, the callback function being specified for the corresponding group in conjunction with establishing the corresponding group for the synchronization object type.

13. The computer program product of claim 12 wherein the synchronization objects comprise one or more of:

respective distinct locks in the storage system; and

at least one of a semaphore, a timer, a barrier, a memory pool and a thread pool.

14. A method performed by at least one processing device comprising a processor coupled to a memory, the method comprising:

establishing at least first and second groups of synchronization objects in a storage system based at least in part on synchronization object type, the first group of synchronization objects being established for a first synchronization object type, the second group of synchronization objects being established for a second synchronization object type different than the first synchronization object type, and the first and second groups of synchronization objects being associated with respective ones of a first synchronization object type queue and a second synchronization object type queue, with each such synchronization object type queue being implemented at least in part utilizing the memory; and

for each of the first and second groups of synchronization objects:

inserting entries into a corresponding one of the synchronization object type queues for respective synchronization objects of a corresponding one of the first and second groups of synchronization objects;

executing on the processor an associated one of a plurality of monitor threads, the associated monitor thread being for the corresponding group, the associated monitor thread being configured to scan the entries of the corresponding synchronization object type queue for synchronization object issue detection, in accordance with one or more designated detection conditions, wherein different detection conditions are designated for use with different ones of the first and second groups of synchronization objects; and

responsive to at least one of the scanned entries meeting the one or more designated detection conditions, taking at least one automated action for its associated synchronization object;

wherein the associated monitor thread is configured to operate in accordance with a series of monitoring cycles and wherein one or more synchronization objects added to the corresponding group in a given one of the monitoring cycles have corresponding entries inserted into an additional queue, and further wherein the entries inserted into the additional queue are subsequently merged into the synchronization object type queue for the corresponding group; and

wherein taking at least one automated action comprises executing a callback function specified for the corresponding group, the callback function being specified for the corresponding group in conjunction with establishing the corresponding group for the synchronization object type.

15. The method of claim 14 wherein the synchronization objects comprise one or more of:

respective distinct locks in the storage system; and

at least one of a semaphore, a timer, a barrier, a memory pool and a thread pool.

16. The method of claim 14 wherein said at least one processing device executes threads of one or more applications utilizing at least one of a plurality of processing cores of the storage system, the threads of the one or more applications comprising different sub-threads of a first thread of a first application executing on a particular one of the plurality of processing cores of the storage system.

17. The method of claim 14 wherein establishing at least first and second groups of synchronization objects in a storage system based at least in part on synchronization object type comprises:

receiving registration information for a particular synchronization object type via an application programming interface; and

establishing a group for the particular synchronization object type based at least in part on the registration information.

18. The method of claim 14 wherein the entries in the additional queue for the one or more synchronization objects added to the group of synchronization objects in the given monitoring cycle are subsequently merged into the synchronization object type queue for the corresponding group in conjunction with completion of the given monitoring cycle and are monitored by the associated monitor thread via their entries in the synchronization object type queue in a next monitoring cycle.

19. The method of claim 14 wherein one or more synchronization objects to be deleted from the corresponding group of synchronization objects in a given one of the monitoring cycles have their respective entries marked as deleted in the synchronization object type queue in the given monitoring cycle and any entries so marked are removed from the synchronization object type queue by the associated monitor thread in conjunction with the scan of the entries of the synchronization object type queue.

20. The method of claim 14 wherein taking at least one automated action further comprises at least one of:

generating an alert to at least one other thread;

performing one or more diagnostic operations; and

storing identifying information of at least one other thread holding the corresponding synchronization object.

Assignments (10)