IP Library › Granted Patent US 9,317,521
Granted Patent B1
US 9,317,521 · App. 13/730,058 · Granted Apr 19, 2016

Cloud object store for checkpoints of high performance computing applications using decoupling middleware

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,317,521
App. No.
13/730,058
Granted
Apr 19, 2016
Kind
B1
Abstract

Cloud object storage is enabled for checkpoints of high performance computing applications using a middleware process. A plurality of files, such as checkpoint files, generated by a plurality of processes in a parallel computing system are stored by obtaining said plurality of files from said parallel computing system; converting said plurality of files to objects using a log structured file system middleware process; and providing said objects for storage in a cloud object storage system. The plurality of processes may run, for example, on a plurality of compute nodes. The log structured file system middleware process may be embodied, for example, as a Parallel Log-Structured File System (PLFS). The log structured file system middleware process optionally executes on a burst buffer node.

Claims (38)

1. A method for storing a plurality of files generated by a plurality of processes in a parallel computing system, comprising the steps of:

obtaining said plurality of files from said parallel computing system;

converting said plurality of files to objects using a log structured file system middleware process; and

providing said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is an input-output (I/O) conversion middleware process between said parallel computing system and said cloud object storage system.

2. The method of claim 1 , wherein said plurality of processes are running on a plurality of compute nodes.

3. The method of claim 1 , wherein said plurality of files comprise checkpoint files generated by said plurality of compute nodes.

4. The method of claim 1 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).

5. The method of claim 1 , wherein said log structured file system middleware process executes on a burst buffer node.

6. The method of claim 1 , wherein said step of providing said objects for storage in said cloud object storage system further comprises the step of providing said objects to said cloud storage API of said cloud object storage system.

7. The method of claim 6 , wherein said log structured file system middleware process comprises said software interface module for providing said objects for storage to said cloud storage API.

8. The method of claim 7 , wherein said software interface module embeds one or more directory paths into a name of each of said objects provided to said cloud storage API.

9. The method of claim 7 , wherein said log structured file system middleware process comprises a software module for generating a namespace of said plurality of files.

10. The method of claim 1 , wherein said step of converting said plurality of files to objects further comprises converting said plurality of files to a plurality of one or more of get operations and put operations on said set of objects, wherein a put operation passes data for an entire object, and a get operation retrieves data for an entire object.

11. The method of claim 1 , wherein said log structured file system middleware process processes partial file input/output operations by buffering entire objects in one or more buffers and applying one or more of partial read and write commands using said one or more buffers.

12. A tangible machine-readable recordable storage medium for storing a plurality of files generated by a plurality of processes in a parallel computing system, wherein one or more software programs when executed by one or more processing devices implement the following steps:

obtaining said plurality of files from said parallel computing system;

converting said plurality of files to objects using a log structured file system middleware process; and

providing said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is an input-output (I/O) conversion middleware process between said parallel computing system and said cloud object storage system.

13. An apparatus for storing a plurality of files generated by a plurality of processes in a parallel computing system, comprising:

a memory; and

at least one hardware device operatively coupled to the memory and configured to:

obtain said plurality of files;

convert said plurality of files to objects using a log structured file system middleware process; and

provide said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is an input-output (I/O) conversion middleware process between said parallel computing system and said cloud object storage system.

14. The apparatus of claim 13 , wherein said plurality of processes are running on a plurality of compute nodes.

15. The apparatus of claim 13 , wherein said plurality of files comprise checkpoint files generated by said plurality of compute nodes.

16. The apparatus of claim 13 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).

17. The apparatus of claim 13 , wherein said log structured file system middleware process executes on a burst buffer node.

18. The apparatus of claim 13 , wherein said objects are provided for storage in said cloud object storage system by providing said objects to said cloud storage API of said cloud object storage system.

19. The apparatus of claim 13 , wherein said plurality of files are converted to objects by converting said plurality of files to a plurality of one or more of get operations and put operations on said set of objects, wherein a put operation passes data for an entire object, and a get operation retrieves data for an entire object.

20. The apparatus of claim 13 , wherein said log structured file system middleware process processes partial file input/output operations by buffering entire objects in one or more buffers and applying one or more of partial read and write commands using said one or more buffers.

21. A data storage system for storing a plurality of files generated by a plurality of processes in a parallel computing system, comprising:

a processing unit for obtaining said plurality of files, converting said plurality of files to objects using a log structured file system middleware process and providing said objects for storage in a cloud object storage system using a software interface module of said log structured file system middleware process, wherein said software interface module communicates with a cloud storage application programming interface (API) of said cloud object storage system, wherein said log structured file system middleware process is an input-output (I/O) conversion middleware process between said parallel computing system and said cloud object storage system; and

said cloud object storage system for storing said objects.

22. The data storage system of claim 21 , wherein said plurality of processes are running on a plurality of compute nodes.

23. The data storage system of claim 21 , wherein said plurality of files comprise checkpoint files generated by said plurality of compute nodes.

24. The data storage system of claim 21 , wherein said log structured file system middleware process comprises a Parallel Log-Structured File System (PLFS).

25. The data storage system of claim 21 , wherein said log structured file system middleware process executes on a burst buffer node.

Assignments (12)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 8, 2018
From: LOS ALAMOS NATIONAL SECURITY, LLC
To: TRIAD NATIONAL SECURITY, LLC
Reel/Frame 047485/0323 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2016
From: EMC CORPORATION
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 040203/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
CONFIRMATORY LICENSE Recorded Mar 6, 2014
From: LOS ALAMOS NATIONAL SECURITY
To: U.S. DEPARTMENT OF ENERGY
Reel/Frame 032361/0408 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 25, 2013
From: GRIDER, GARY A.
To: LOS ALAMOS NATIONAL SECURITY, LLC
Reel/Frame 031670/0939 →