Vector processor storage
A method comprising: receiving, at a vector processor, a request to store data; performing, by the vector processor, one or more transforms on the data; and directly instructing, by the vector processor, one or more storage device to store the data; wherein performing one or more transforms on the data comprises: erasure encoding the data to generate n data fragments configured such that any k of the data fragments are usable to regenerate the data, where k is less than n; and wherein directly instructing one or more storage device to store the data comprises: directly instructing the one or more storage devices to store the plurality of data fragments.
1. A method comprising:
receiving, at a vector processor, a request to store data;
performing, by the vector processor, one or more transforms on the data, wherein performing one or more transforms on the data comprises erasure encoding the data to generate n data fragments configured such that any k of the data fragments are usable to regenerate the data, where k is less than n;
instructing, by the vector processor, one or more storage devices to store the data, wherein instructing one or more storage devices to store the data comprises instructing the one or more storage devices to store the data without the mediation of a central processing unit (CPU); and
receiving, by the vector processor, an acknowledgment from the one or more storage devices that m of the data fragments have been stored, in which k≤m<n.
2. The method of claim 1 , wherein performing, by the vector processor, one or more transforms on the data comprises performing, by the vector processor, one or more invertible transforms on the data.
3. The method of claim 1 , wherein:
performing one or more transforms on the data further comprises calculating a cryptographic hash for the data; and
instructing one or more storage device to store the data comprises instructing the one or more storage devices to store the cryptographic hash for the data.
4. The method of claim 1 , wherein performing one or more transforms on the data comprises logging a storage event associated with the request to store data.
5. The method of claim 1 , wherein performing one or more transforms on the data comprises encrypting and/or compressing the data.
6. The method of claim 1 , wherein performing one or more transforms on the data comprises identifying one or more plugins configured to be used for the one or more transforms.
7. The method of claim 1 , wherein receiving, at a vector processor, a request to store data comprises executing a write instruction in a kernel being executed by the vector processor.
8. The method of claim 1 , further comprising, before instructing one or more storage device to store the data, storing, by the vector processor, the data in non-volatile memory.
9. The method of claim 1 , wherein instructing one or more storage device to store the data comprises instructing a plurality of the one or more storage devices to store the data in parallel.
10. A system comprising:
one or more vector processors; and
a memory comprising instructions which, when executed by the one or more vector processors, configure the one or more vector processors to perform the method of claim 1 .
11. One or more non-transitory computer readable media comprising instructions which, when executed by one or more vector processors, cause the one or more vector processors to perform the method of claim 1 .
12. A method comprising:
receiving, at a first vector processor, a request to retrieve data;
broadcasting, by the first vector processor, instructions to retrieve a plurality of data fragments associated with the data, wherein the instructions are received by a second vector processor and one or more storage devices in communication with the first vector processor;
in response to receipt of the instructions from the first vector processor, broadcasting, by the second vector processor, the instructions to retrieve the plurality of data fragments associated with the data, wherein the instructions are received by one or more storage devices in communication with the second vector processor;
receiving, by the first vector processor, the data fragments from the one or more storage devices in communication with the first and second vector processors; and
performing, by the first vector processor, one or more transforms on the data fragments to obtain the data.
13. The method of claim 12 , wherein performing one or more transforms on the data comprises logging a retrieval event associated with the request to retrieve data.
14. The method of claim 12 , wherein performing one or more transforms on the data fragments comprises one or more of:
decrypting and/or decompressing the data; or
identifying one or more plugins configured to be used for the one or more transforms.
15. The method of claim 12 , wherein receiving a request to retrieve data comprises executing a read instruction in a kernel being executed by the first vector processor.
16. A system comprising:
a first and a second vector processor; and
a memory comprising instructions which, when executed by the first and second vector processors, configure the first and second vector processors to perform the method of claim 12 .
17. One or more non-transitory computer readable media comprising instructions which, when executed by a first and a second vector processor, cause the first and second vector processors to perform the method of claim 12 .