IP Library › Granted Patent US 9,087,075
Granted Patent B1
US 9,087,075 · App. 13/536,331 · Granted Jul 21, 2015

Storing files in a parallel computing system using list-based index to identify replica files

View Patent ↗
Loading inventors, assignments & file history…
Monitor This Case
Get email alerts when status or documents change.
Order Certified Copies
Most orders are placed with the USPTO same day — all within 24 business hours.
Order via The Patent Place →
Pre-filled with this patent's details
Quick Facts
Patent No.
US 9,087,075
App. No.
13/536,331
Granted
Jul 21, 2015
Kind
B1
Abstract

Improved techniques are provided for storing files in a parallel computing system using a list-based index to identify file replicas. A file and at least one replica of the file are stored in one or more storage nodes of the parallel computing system. An index for the file comprises at least one list comprising a pointer to a storage location of the file and a storage location of the at least one replica of the file. The file comprises one or more of a complete file and one or more sub-files. The index may also comprise a checksum value for one or more of the file and the replica(s) of the file. The checksum value can be evaluated to validate the file and/or the file replica(s). A query can be processed using the list.

Claims (38)

1. A method performed by a middleware process for storing at least one sub-file generated by a distributed application in a parallel computing system, wherein said sub-file comprises a portion of a larger file, said method comprising the steps of:

generating at least one replica of said sub-file;

storing said sub-file and said at least one replica of said sub-file in one or more storage nodes of said parallel computing system; and

updating an index for said sub-file using said middleware process, said index comprising a length of said sub-file, a first list comprising a pointer to a storage location of said sub-file and a pointer to a storage location of said at least one replica of said sub-file and a second list comprising a first offset for said sub-file within a first physical file storing said file and a second offset for said at least one replica of said sub-file within a second physical file storing at least one replica of said file.

2. The method of claim 1 , wherein said pointer to a storage location of said sub-file comprises a pointer to said first physical file storing said sub-file and wherein said pointer to said storage location of said at least one replica of said sub-file comprises a pointer to said second physical file storing said at least one replica of said sub-file.

3. The method of claim 2 , wherein said index further comprises said second list comprising said first offset for said file within said first physical file and said second offset for said at least one replica of said file within said second physical file.

4. The method of claim 1 , wherein said index further comprises a logical offset for said sub-file.

5. The method of claim 1 , wherein said index further comprises a checksum value for one or more of said sub-file and said at least one replica of said sub-file.

6. The method of claim 5 , further comprising the step of evaluating said checksum value to validate one or more of said sub-file and said at least one replica of said sub-file.

7. The method of claim 1 , further comprising the step of reloading one or more of said sub-file and said at least one replica of said sub-file using one or more of said first list and said second list.

8. The method of claim 1 , further comprising the step of delivering one of said sub-file and said at least one replica of said sub-file based on a determination of a nearest copy from one or more of said first list and said second list.

9. The method of claim 1 , wherein a replication strategy can be separately specified for each of a plurality of said sub-files.

10. The method of claim 1 , further comprising the step of processing a query using one or more of said first list and said second list.

11. The method of claim 1 , wherein said one or more of said storage nodes reside on one or more tiers of a multi-tier storage system.

12. An apparatus for storing at least one sub-file generated by a distributed application in a parallel computing system, wherein said sub-file comprises a portion of a larger file, said apparatus comprising:

a memory; and

at least one hardware device operatively coupled to the memory and configured to implement a middleware process that performs the following steps:

generating at least one replica of said sub-file;

storing said sub-file and said at least one replica of said sub-file in one or more storage nodes of said parallel computing system; and

updating an index for said sub-file using said middleware process, said index comprising a length of said sub-file, a first list comprising a pointer to a storage location of said sub-file and a pointer to a storage location of said at least one replica of said sub-file and a second list comprising a first offset for said sub-file within a first physical file storing said file and a second offset for said at least one replica of said sub-file within a second physical file storing at least one replica of said file.

13. The apparatus of claim 12 , wherein said pointer to a storage location of said sub-file comprises a pointer to said first physical file storing said sub-file and wherein said pointer to said storage location of said at least one replica of said sub-file comprises a pointer to said second physical file storing said at least one replica of said sub-file.

14. The apparatus of claim 13 , wherein said index further comprises said second list comprising said first offset for said file within said first physical file and said second offset for said at least one replica of said file within said second physical file.

15. The apparatus of claim 12 , wherein said index further comprises a logical offset for said sub-file.

16. The apparatus of claim 12 , wherein said index further comprises a checksum value for one or more of said sub-file and said at least one replica of said sub-file.

17. The apparatus of claim 16 , wherein said at least one hardware device is further configured to evaluate said checksum value to validate one or more of said sub-file and said at least one replica of said sub-file.

18. The apparatus of claim 12 , wherein said at least one hardware device is further configured to reload one or more of said sub-file and said at least one replica of said sub-file using one or more of said first list and said second list.

19. The apparatus of claim 12 , wherein said at least one hardware device is further configured to deliver one of said sub-file and said at least one replica of said sub-file based on a determination of a nearest copy from one or more of said first list and said second list.

20. The apparatus of claim 12 , wherein a replication strategy can be separately specified for each of a plurality of said sub-files.

21. The apparatus of claim 12 , wherein said at least one hardware device is further configured to process a query using one or more of said first list and said second list.

22. The apparatus of claim 12 , wherein said one or more of said storage nodes reside on one or more tiers of a multi-tier storage system.

23. A data storage system for storing at least one sub-file generated by a distributed application in a parallel computing system, wherein said sub-file comprises a portion of a larger file, said data storage system comprising:

a hardware processing unit for implementing a middleware process that performs the following steps:

generating at least one replica of said sub-file; and

updating an index for said sub-file, said index comprising a length of said sub-file, a first list comprising a pointer to a storage location of said sub-file and a pointer to a storage location of said at least one replica of said sub-file and a second list comprising a first offset for said sub-file within a first physical file storing said file and a second offset for said at least one replica of said sub-file within a second physical file storing at least one replica of said file; and

a storage medium for storing said sub-file and said at least one replica of said sub-file in one or more storage nodes of said parallel computing system.

24. The data storage system of claim 23 , wherein said pointer to a storage location of said sub-file comprises a pointer to said first physical file storing said sub-file and wherein said pointer to said storage location of said at least one replica of said sub-file comprises a pointer to said second physical file storing said at least one replica of said sub-file and wherein said index further comprises said second list comprising said first offset for said file within said first physical file and said second offset for said at least one replica of said file within said second physical file.

25. The data storage system of claim 23 , wherein said index further comprises a checksum value for one or more of said sub-file and said at least one replica of said sub-file.

26. The data storage system of claim 23 , wherein said processing unit is further configured to evaluate said checksum value to validate one or more of said sub-file and said at least one replica of said sub-file.

Assignments (14)
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (053546/0001) Recorded Jun 23, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC IP HOLDING COMPANY LLC
Reel/Frame 071642/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (045455/0001) Recorded May 20, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061753/0001 →
RELEASE OF SECURITY INTEREST IN PATENTS PREVIOUSLY RECORDED AT REEL/FRAME (040136/0001) Recorded Apr 26, 2022
From: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
To: DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO ASAP SOFTWARE EXPRESS, INC.); DELL MARKETING L.P. (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO CREDANT TECHNOLOGIES, INC.); DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL PRODUCTS L.P.; DELL MARKETING CORPORATION (SUCCESSOR-IN-INTEREST TO FORCE10 NETWORKS, INC. AND WYSE TECHNOLOGY L.L.C.); EMC CORPORATION (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MAGINATICS LLC); EMC IP HOLDING COMPANY LLC (ON BEHALF OF ITSELF AND AS SUCCESSOR-IN-INTEREST TO MOZY, INC.); SCALEIO LLC
Reel/Frame 061324/0001 →
RELEASE OF SECURITY INTEREST Recorded Nov 3, 2021
From: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH
To: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL, L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; WYSE TECHNOLOGY L.L.C.
Reel/Frame 058216/0001 →
SECURITY AGREEMENT Recorded Apr 22, 2020
From: CREDANT TECHNOLOGIES INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 053546/0001 →
SECURITY AGREEMENT Recorded Mar 21, 2019
From: CREDANT TECHNOLOGIES, INC.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL USA L.P.; EMC CORPORATION; FORCE10 NETWORKS, INC.; WYSE TECHNOLOGY L.L.C.; EMC IP HOLDING COMPANY LLC
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A.
Reel/Frame 049452/0223 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Nov 7, 2018
From: LOS ALAMOS NATIONAL SECURITY, LLC
To: TRIAD NATIONAL SECURITY, LLC
Reel/Frame 048007/0874 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 29, 2016
From: EMC CORPORATION
To: EMC IP HOLDING COMPANY LLC
Reel/Frame 040203/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: CREDIT SUISSE AG, CAYMAN ISLANDS BRANCH, AS COLLATERAL AGENT
Reel/Frame 040134/0001 →
SECURITY AGREEMENT Recorded Sep 21, 2016
From: ASAP SOFTWARE EXPRESS, INC.; AVENTAIL LLC; CREDANT TECHNOLOGIES, INC.; DELL USA L.P.; DELL INTERNATIONAL L.L.C.; DELL MARKETING L.P.; DELL PRODUCTS L.P.; DELL SOFTWARE INC.; DELL SYSTEMS CORPORATION; EMC CORPORATION; EMC IP HOLDING COMPANY LLC; FORCE10 NETWORKS, INC.; MAGINATICS LLC; MOZY, INC.; SCALEIO LLC; SPANNING CLOUD APPS LLC; WYSE TECHNOLOGY L.L.C.
To: THE BANK OF NEW YORK MELLON TRUST COMPANY, N.A., AS NOTES COLLATERAL AGENT
Reel/Frame 040136/0001 →
CONFIRMATORY LICENSE Recorded Mar 6, 2014
From: LOS ALAMOS NATIONAL SECURITY
To: U.S. DEPARTMENT OF ENERGY
Reel/Frame 032362/0148 →
CORRECTIVE ASSIGNMENT TO CORRECT THE REMOVE ASSIGNOR'S NAME PREVIOUSLY RECORDED ON REEL 029013 FRAME 0685. ASSIGNOR(S) HEREBY CONFIRMS THE ASSIGNMENT. Recorded Aug 15, 2013
From: FAIBISH, SORIN; BENT, JOHN M.; TZELNI, PERCY; ZHANG, ZHENHUA
To: EMC CORPORATION
Reel/Frame 031016/0973 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Apr 26, 2013
From: GRIDER, GARY
To: LOS ALAMOS NATIONAL SECURITY, LLC
Reel/Frame 030294/0525 →
ASSIGNMENT OF ASSIGNOR'S INTEREST Recorded Sep 24, 2012
From: FAIBISH, SORIN; BENT, JOHN M.; TZELNIC, PERCY; ZHANG, ZHENHUA; GRIDER, GARY
To: EMC CORPORATION
Reel/Frame 029013/0685 →