System and method for data storage, transfer, synchronization, and security using automated model monitoring and training
A system and method for data storage, transfer, synchronization, and security using automated system efficacy monitoring and model training, wherein statistical analyses of test datasets are used to determine if the probability distribution of two datasets are within a pre-determined range, and responsive to that determination new encoding and decoding algorithms may be retrained in order to produce new data chunklets. The new data chunklets may then be processed and assigned new codewords which are compiled into an updated codebook which may be distributed back to encoding and decoding systems and devices.
1. A system for storing, retrieving, and transmitting data in a highly compact format, comprising:
a computing device comprising a processor, a memory, and a first plurality of programming instructions;
a monitor database stored in the memory of the computing device, wherein the monitor database comprises a previous training dataset, a first measured probability distribution associated with the previous training dataset, performance metrics, and model predictions; and
a codebook training module comprising a second plurality of programming instructions stored in the memory and operable on the processor, wherein the second plurality of programming instructions, when operating on the processor, cause the processor to:
receive data;
format the received data into a test dataset;
retrieve the first measured probability distribution associated with the previous training dataset from the monitor database;
use one or more algorithms to measure a second probability distribution of the test dataset;
compare the first and second measured probability distributions to compute the difference in distribution statistics between the test dataset and the previous training dataset;
check if the difference in distributions exceeds a pre-determined difference threshold;
use the test dataset to retrain encoding and decoding algorithms;
utilize the retrained algorithms to create new data chunklets;
create new codeword for each new data chunklet;
store each new data chunklet and its associated new codeword in an updated codebook; and
send the updated codebook to a plurality of encoding and decoding machines.
2. The system of claim 1 , wherein the computing device is a cloud-based computing device.
3. The system of claim 1 , further comprising a network device manager comprising a third plurality of programming instructions stored in the memory and operable on the processor, wherein the third plurality of programming instructions, when operating on the processor, cause the processor to:
receive device data from at least one of a plurality of network connected devices;
store received device data in a storage device operating on the memory;
analyze device data to monitor network connected device resource consumption and time periods of device downtime; and
forward device data to a codebook update engine.
4. The system of claim 3 , further comprising a codebook update engine comprising a network device manager comprising a fourth plurality of programming instructions stored in the memory and operable on the processor, wherein the fourth plurality of programming instructions, when operating on the processor, cause the processor to:
receive updated codebooks;
store updated codebooks in a cache;
receive device data from the network device manager; and
publish updated codebooks to network connected devices associated with the received device data.
5. A method for storing, retrieving, and transmitting data in a highly compact format, comprising the steps of:
receiving data;
formatting the received data into a test dataset;
retrieving a first measured probability distribution associated with a previous training dataset from a monitor database;
using one or more algorithms to measure a second probability distribution of the test dataset;
comparing the first and second measured probability distributions to compute the difference in distribution statistics between the test dataset and the previous training dataset;
checking if the difference in distributions exceeds a pre-determined difference threshold;
using the test dataset to retrain encoding and decoding algorithms;
utilizing the retrained algorithms to create new data chunklets;
creating new codeword for each new data chunklet;
storing each new data chunklet and its associated new codeword in an updated codebook; and
sending the updated codebook to a plurality of encoding and decoding machines.
6. The method of claim 5 , wherein the computing device is a cloud-based computing device.
7. The method of claim 5 , further comprising the steps of:
receiving device data from at least one of a plurality of network connected devices;
storing received device data in a storage device operating on the memory;
analyzing device data to monitor network connected device resource consumption and time periods of device downtime; and
forwarding device data to a codebook update engine.
8. The method of claim 7 , further comprising the steps of:
receiving updated codebooks;
storing updated codebooks in a cache;
receiving device data from the network device manager; and
publishing updated codebooks to network connected devices associated with the received device data.