Vector processor storage
A method comprising: receiving, at a vector processor, a request to store data; performing, by the vector processor, one or more transforms on the data; and directly instructing, by the vector processor, one or more storage device to store the data; wherein performing one or more transforms on the data comprises: erasure encoding the data to generate n data fragments configured such that any k of the data fragments are usable to regenerate the data, where k is less than n; and wherein directly instructing one or more storage device to store the data comprises: directly instructing the one or more storage devices to store the plurality of data fragments.
1. A method comprising:
receiving, at a vector processor, a request to store data;
performing, by the vector processor, one or more transforms on the data; and
instructing, by the vector processor, one or more storage devices to store the data;
wherein performing one or more transforms on the data comprises:
erasure encoding the data to generate n data fragments configured such that any m data fragments are recoverable using k data fragments, where n=m+k; and
wherein instructing one or more storage devices to store the data comprises:
instructing the one or more storage devices to store the plurality of data fragments.
2. The method of claim 1 , wherein performing, by the vector processor, one or more transforms on the data consists of:
performing, by the vector processor, one or more invertible transforms on the data.
3. The method of claim 1 , wherein:
performing one or more transforms on the data further comprises:
calculating a cryptographic hash for the data; and
instructing one or more storage device to store the data comprises:
instructing one or more storage devices to store the cryptographic hash for the data.
4. The method of claim 1 , wherein performing one or more transforms on the data comprises:
logging a storage event associated with the request to store data.
5. The method of claim 1 , wherein performing one or more transforms on the data comprises:
encrypting and/or compressing the data.
6. The method of claim 1 , wherein performing one or more transforms on the data comprises:
identifying one or more plugins configured to be used for the one or more transforms.
7. The method of claim 1 , wherein storing the data fragments at one or more storage devices comprises:
instructing, by the vector processor, a storage device to store at least k of the data fragments, where k is the minimum number of data fragments that can be used to regenerate the data.
8. The method of claim 1 , wherein receiving, at a vector processor, a request to store data comprises:
executing a write instruction in a kernel being executed by the vector processor.
9. The method of claim 1 , further comprising, before directly instructing one or more storage device to store the data:
storing, by the vector processor, the data in non-volatile memory.
10. The method of claim 1 , wherein instructing one or more storage device to store the data comprises:
instructing a plurality of storage device in parallel to store the data.
11. A method comprising:
receiving, at a vector processor, a request to retrieve data;
instructing one or more storage devices to retrieve a plurality of data fragments associated with the data from the one or more storage devices; and
performing, by the vector processor, one or more transforms on the data fragments to obtain the data;
wherein the plurality of data fragments comprises at least k data fragments among n erasure encoded data fragments, where any m data fragments are recoverable using k data fragments, and n=m+k; and
wherein performing one or more transforms on the data fragments to obtain the data comprises:
erasure decoding at least k data fragments to generate the data.
12. The method of claim 11 , wherein performing one or more transforms on the data comprises:
logging a retrieval event associated with the request to retrieve data.
13. The method of claim 11 , wherein performing one or more transforms on the data fragments comprises one or more of:
decrypting and/or decompressing the data; or
identifying one or more plugins configured to be used for the one or more transforms.
14. The method of claim 11 , wherein directly instructing one or more storage devices to retrieve a plurality of data fragments associated with the data from the one or more storage devices comprises:
sending, by the vector processor to the one or more storage devices, a request for a data fragment associated with the data; and
receiving, at the vector processor, at least one data fragment associated with the data in response to one or more of the requests.
15. The method of claim 14 , wherein the request for a data fragment comprises a memory address, and receiving at least one data fragment comprises:
reading a memory associated with the memory address; and
when the data stored in the memory can be used to generate the data, determining that the at least one data fragment has been received.
16. The method of claim 11 , wherein receiving a request to retrieve data comprises:
executing a read instruction in a kernel being executed by the vector processor.
17. A system comprising:
one or more vector processors; and
a memory;
wherein the memory comprises instructions which, when executed by the one or more vector processors, configure the one or more vector processors to perform the method of claim 1 .
18. One or more non-transitory computer readable media comprising instructions which, when executed by one or more vector processors, cause the one or more vector processors to perform the method of claim 1 .
19. A system comprising:
one or more vector processors; and
a memory;
wherein the memory comprises instructions which, when executed by the one or more vector processors, configure the one or more vector processors to perform the method of claim 11 .
20. One or more non-transitory computer readable media comprising instructions which, when executed by one or more vector processors, cause the one or more vector processors to perform the method of claim 11 .