Systems and Methods for Construction and Implementation of Efficient Feature Extractors
Systems and methods are disclosed for constructing and implementing feature extractors for inference on structured inputs. A computing system determines feature directions from a dataset, quantizes them into a restricted set of value representations, and iteratively updates the dataset. The iterative process removes contributions corresponding to the quantized feature directions while retaining the residual quantization error, thereby generating additional quantized feature directions. In some embodiments, shared arithmetic structures are constructed across multiple feature extractors by identifying channel tuples having common or proportional coefficients and creating merged channels. The resulting feature extractors may be synthesized into hardware representations, including FPGA configurations and ASIC implementations, and may generate Boolean, multi-bit, or integer-valued feature outputs.
1 . A computer-implemented method for constructing a plurality of quantized feature extractors for synthesis into hardware, the method comprising:
determining, from a dataset, a feature direction, wherein the feature direction is a vector or a tensor and comprises at least two components;
quantizing the feature direction to produce a quantized feature direction;
modifying the dataset by removing therefrom a contribution corresponding to the quantized feature direction, rather than removing therefrom a contribution corresponding to the feature direction prior to quantization, thereby producing a modified dataset that retains residual error introduced by the quantization;
iteratively repeating the determining, quantizing, and modifying to produce at least one further quantized feature direction, each further quantized feature direction being determined from a corresponding modified dataset from which contributions of previously produced quantized feature directions have been removed; and
outputting data representing the quantized feature direction and the at least one further quantized feature direction for synthesis into hardware.
2 . The method of claim 1 , further comprising converting at least one quantized feature direction produced by the method into a combinational-logic netlist for implementation as static logic on a field-programmable gate array or an application-specific integrated circuit.
3 . The method of claim 2 , wherein input dimensions corresponding to zero-valued entries of the at least one quantized feature direction are omitted from the combinational-logic netlist, and wherein entries whose magnitude is a power of two are implemented via bit shifting.
4 . The method of claim 1 , wherein the dataset comprises a dataset of feature extractor inputs constructed by extracting, from a dataset of interest, a collection of input vectors, each input vector corresponding to a local region, a patch, or a subset of features of the dataset of interest,
wherein determining the feature direction comprises applying a decomposition that identifies one or more of:
a direction of greatest variance of the dataset of feature extractor inputs;
a direction associated with a greatest singular value of the dataset of feature extractor inputs; and
a direction associated with a greatest eigenvalue of the dataset of feature extractor inputs.
5 . The method of claim 1 , wherein modifying the dataset comprises, for each data point in the dataset, computing an activation by an inner product of the data point with the quantized feature direction and subtracting from the data point an outer product of the activation with the quantized feature direction, thereby removing from the dataset a projection onto a subspace defined by the quantized feature direction rather than a subspace defined by the feature direction prior to quantization.
6 . The method of claim 1 , wherein the quantized feature direction defines a subspace different from a subspace defined by the feature direction prior to quantization, and wherein the modified dataset retains components that would have been removed had the feature direction prior to quantization been used, such that a subsequent feature direction determined from the modified dataset compensates, at least in part, for quantization-induced subspace deviation of the quantized feature direction.
7 . The method of claim 1 , wherein the determining, quantizing, and modifying are performed for at least four iterations to produce at least four quantized feature directions.
8 . The method of claim 1 , wherein modifying the dataset further comprises augmenting at least one data point in the modified dataset with at least one additional element representing an output activation of a previously identified quantized feature extractor.
9 . The method of claim 1 , wherein the plurality of quantized feature extractors are for use in a hardware implementation of an inference system processing real-valued or integer-valued inputs.
10 . The method of claim 1 , wherein the dataset comprises a dataset of feature extractor inputs constructed by extracting, from a dataset of interest, a collection of input vectors, each input vector corresponding to a local region, a patch, or a subset of features of the dataset of interest, and wherein the method further comprises, for at least one quantized feature direction produced by the method:
computing an activation distribution by applying the at least one quantized feature direction to the dataset of feature extractor inputs;
determining a plurality of thresholds from the activation distribution, including a plurality of quantile or quantile-based thresholds; and
defining a feature extractor configured to produce a multi-bit output by comparing an activation produced from an input using the at least one quantized feature direction against the plurality of quantile or quantile-based thresholds.
11 . The method of claim 1 , further comprising constructing a second level of feature extractors by:
applying a first level of quantized feature extractors to data derived from the dataset to produce intermediate feature representations;
constructing a second dataset from the intermediate feature representations; and
performing the determining, quantizing, and modifying on the second dataset to obtain a second plurality of quantized feature directions,
wherein precision of the intermediate feature representations is reduced before constructing the second dataset by omitting least significant bits or by linear or non-linear quantization of the intermediate feature representations.
12 . The method of claim 1 , wherein the quantized feature direction has entries selected from a set {−2, −1, 0, 1, 2}.
13 . The method of claim 1 , wherein quantizing the feature direction comprises applying a rounding function biased toward values associated with lower hardware implementation cost, including a bias toward zero.
14 . The method of claim 1 , wherein determining the feature direction comprises extracting a trained weight vector from a layer of a neural network that has been trained on the dataset or on a related dataset.
15 . The method of claim 1 , further comprising:
identifying, across the plurality of quantized feature extractors, a multi-element tuple of input signal dimensions for which two or more of the quantized feature extractors assign identical or proportional nonzero coefficient values;
creating a merged channel representing a sum or weighted sum of the input signal dimensions of the multi-element tuple;
replacing, in the two or more quantized feature extractors, references to the multi-element tuple with a reference to the merged channel and adjusting one or more coefficients to account for the replacement; and
outputting data representing the quantized feature extractors and the merged channel for synthesis into shared hardware in which the merged channel corresponds to a shared arithmetic unit routed to the two or more quantized feature extractors that reference the merged channel, thereby reducing a total number of arithmetic operations relative to independent implementation of the plurality of feature extractors.
16 . An application-specific integrated circuit (ASIC), comprising:
input circuitry configured to receive an inference input and to form a feature extractor input vector therefrom;
feature extraction circuitry configured to compute one or more activation values according to a quantized feature direction and at least one further quantized feature direction embodied in the ASIC; and
feature output circuitry configured to generate one or more feature outputs from the one or more activation values, wherein the ASIC is manufactured based on data representing the quantized feature direction and the at least one further quantized feature direction, the data being produced by a process comprising:
determining, from a dataset, a feature direction, wherein the feature direction is a vector or a tensor and comprises at least two components;
quantizing the feature direction to produce a quantized feature direction;
modifying the dataset by removing therefrom a contribution corresponding to the quantized feature direction, rather than removing therefrom a contribution corresponding to the feature direction prior to quantization, thereby producing a modified dataset that retains residual error introduced by the quantization;
iteratively repeating the determining, quantizing, and modifying to produce at least one further quantized feature direction, each further quantized feature direction being determined from a corresponding modified dataset from which contributions of previously produced quantized feature directions have been removed; and
outputting data representing the quantized feature direction and the at least one further quantized feature direction for synthesis into hardware.
17 . A computer-implemented method for constructing a shared hardware implementation of a plurality of feature extractors, each feature extractor defined by coefficients over a common set of input signal dimensions or over respective overlapping sets of input signal dimensions, the method comprising:
identifying, across the plurality of feature extractors, a selected multi-element tuple of input signal dimensions that satisfies a sharing criterion for a plurality of the feature extractors, the sharing criterion comprising at least one criterion selected from a set of criteria that includes: (i) respective feature extractors assigning identical nonzero coefficient values to the input signal dimensions of the multi-element tuple, and (ii) respective feature extractors assigning proportional nonzero coefficient values to the input signal dimensions of the selected multi-element tuple;
creating a merged channel representing a sum or weighted sum of the input signal dimensions of the selected multi-element tuple;
modifying, for at least some of the feature extractors that satisfy the sharing criterion for the selected multi-element tuple, the at least some feature extractors by replacing references to the selected multi-element tuple with a reference to the merged channel and adjusting one or more coefficients to account for the replacement;
iteratively repeating the identifying, creating, and modifying using a modified set of feature extractors, including previously created merged channels as candidates for further merging; and
outputting data representing the modified set of feature extractors and the merged channels for synthesis into shared hardware,
wherein each merged channel corresponds to a shared arithmetic unit whose output is routed to two or more feature extractors that reference the merged channel, thereby reducing a total number of arithmetic operations relative to independent implementation of the plurality of feature extractors.
18 . The method of claim 17 , wherein outputting data representing the modified set of feature extractors and the merged channels comprises outputting a combinational-logic netlist or configuration data for implementation on a field-programmable gate array or in an application-specific integrated circuit.
19 . The method of claim 17 , wherein the input signal dimensions comprise any combination of one or more of: input channels; channels of a multi-channel input signal; spatial positions within a kernel, patch, receptive field, or window; entries of a feature extractor input vector; feature indices; and temporal positions.
20 . The method of claim 17 , wherein the selected multi-element tuple comprises a selected pair of input signal dimensions, wherein identifying the selected pair comprises evaluating coefficient values assigned by the plurality of feature extractors to pairs of input signal dimensions and selecting the selected pair together with a matching nonzero coefficient value for which a count of feature extractors assigning the matching nonzero coefficient value to both input signal dimensions of the selected pair is maximized under constraints including hardware efficiency constraints, and
wherein modifying the at least some feature extractors that assign the matching nonzero coefficient value to both input signal dimensions of the selected pair comprises setting coefficient entries for the selected pair to zero and introducing a nonzero coefficient entry for the merged channel, thereby reducing a number of nonzero coefficient entries in the feature extractor.