Configurable data generator
View Patent ↗Embodiments associated with configurable, repeatable, data generation are described. One example method includes manipulating a redundancy parameter that controls data redundancy in binary large objects (BLOBs) to be included in a generated data set. The redundancy parameters may control variations in repeatable variable length sequences included in BLOBs. The example method also includes manipulating a parameter(s) that controls custom designed sequences included in BLOBs. With the redundancy and custom designed sequences described, the example method then generates BLOBs based, at least in part, on the redundancy parameters and the custom-designed sequences. BLOBs may include byte sequences repeated at different frequencies and configurable user-designed sequences. Manipulating the redundancy parameter, manipulating the custom-designed sequences, generating the BLOBs, and providing the BLOBS may be performed by separate processes acting in parallel.
1. A non-transitory computer-readable medium storing computer-executable instructions that when executed by a computer cause the computer to perform a method, the method comprising:
manipulating one or more redundancy parameters that control redundancy in data to be generated, where manipulating the one or more redundancy parameters includes manipulating a degree of internal redundancy for a subset of the data, a degree of external redundancy between subsets of the data, a frequency with which internal redundancy is to vary, or a frequency with which external redundancy is to vary;
manipulating one or more parameters that control custom-designed sequences to be included in the data, where manipulating the one or more parameters that control custom-designed sequences includes manipulating a sequence length distribution, where manipulating the sequence length distribution follows a kurtosis rule, where the kurtosis rule defines the sequence length distribution to follow a geometric frequency distribution;
generating the data based, at least in part, on the one or more redundancy parameters, where the data includes one or more variable custom-designed sequences, and where the data comprises one or more binary large objects exhibiting byte-sequence variability with binary large object dispersion,
where the data include redundant spans that are specified as random seed generated variable length patterns from within a constrained number-space, and where the redundant spans are controlled, at least in part, by the redundancy parameters; and
providing the data from the computer to a data de-duplicator, where manipulating the one or more redundancy parameters, manipulating the one or more parameters that control custom-designed sequences, generating the data, and providing the data are performed at least partially in parallel on the computer.
2. The non-transitory computer-readable medium of claim 1 ,
where the method is performed in a multi-threaded system,
where manipulating the one or more redundancy parameters and manipulating the one or more parameters that control custom-designed sequences are performed by one or more populator threads,
where generating the data is performed by one or more writer threads, and
where the one or more populator threads and the one or more writer threads are configured to operate in parallel in the multi-threaded system.
3. The non-transitory computer-readable medium of claim 2 , the method comprising:
controlling the number of populator threads used to perform the method, and
controlling the number of writer threads used to perform the method.
4. The non-transitory computer-readable medium of claim 2 , where the one or more populator threads share a common data generator, and where the common data generator is one of, a random number generator, a pseudo-random number generator, a checksum generator, and a hash generator.
5. The non-transitory computer-readable medium of claim 4 , the method comprising:
stopping data generation by the common data generator;
storing a state to which the common data generator advanced while generating test data, and
restarting data generation by the common data generator at a point controlled, at least in part, by the state.
6. The non-transitory computer-readable medium of claim 1 , the method comprising:
selectively controlling the rate at which the data is provided to the data de-duplicator, where the rate is configurable up to a rate that will saturate the data de-duplicator.
7. The non-transitory computer-readable medium of claim 1 , where providing the data from the computer to the data de-duplicator includes temporarily storing a portion of the data in a memory-based buffer in the computer.