Generating output examples using bit blocks

1. A method comprising:

receiving a request to generate an output example of a particular type having T N-bit samples, wherein N and T are respective integers greater than one, and wherein each of the N*T bits in the output example has a respective value;

accessing dependency data, wherein the dependency data:

partitions the N*T bits in the output example into a plurality of blocks of bits, each block of bits comprising a respective plurality of bits from the output example,

for each of a plurality of the blocks, defines a respective dependency for the block that identifies one or more other blocks in the plurality of blocks on which the respective values of the bits in the block depend; and

wherein the dependency data arranges the blocks into a matrix structure having a plurality of rows and a plurality of columns, and wherein, for each particular block of the plurality of blocks, the dependency identifies that the particular block is dependent on any block which is before the particular block in the row of the particular block, and is not dependent on any block that is more than H columns after the particular block in a row that is before the row of the particular block, wherein H is a fixed positive integer; and

generating the output example by, at each of a plurality of generation time steps:

identifying one or more current blocks for the generation time step, wherein each current block is a block for which the respective values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and

generating, using one or more autoregressive neural networks, the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block.

2. The method of claim 1 , wherein, for generation time steps at which a plurality of current blocks are identified, the current blocks are generated in parallel.

3. The method of claim 1 , wherein, for a given block in row i and column j, the dependency identifies that the given block depends only on the respective values of bits in:

blocks 1 through j−1 in row i and, for each row i-k below row i, blocks 1 through (j−1+kH), wherein k is an integer.

4. The method of claim 3 , wherein identifying one or more current blocks for the generation time step comprises, determining, for each row of the plurality of rows, whether there is a block in the row for which all of the other blocks identified in the dependency for the block have already been generated.

5. The method of claim 4 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each row in which a current block is identified:

generating a network input that conditions an autoregressive neural network that is specific to the row on the respective values of the bits in the blocks identified in the dependency for the current block in the row;

processing the network input using the autoregressive neural network that is specific to the row, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.

6. The method of claim 4 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each current block:

generating a network input that conditions an autoregressive neural network on the respective values of the bits in the blocks identified in the dependency for the current block;

processing the network input using an instance of the autoregressive neural network, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.

7. The method of claim 1 , wherein each block also depends on a context input, and wherein generating the respective values of the bits in the current blocks is also conditioned on the context input.

8. A system comprising one or more computers and one or more storage devices storing instructions that when executed by the one or more computers cause the one or more computers to perform operations comprising:

receiving a request to generate an output example of a particular type having T N-bit samples, wherein N and T are respective integers greater than one, and wherein each of the N*T bits in the output example has a respective value;

accessing dependency data, wherein the dependency data:

partitions the N*T bits in the output example into a plurality of blocks of bits, each block of bits comprising a respective plurality of bits from the output example,

for each of a plurality of the blocks, defines a respective dependency for the block that identifies one or more other blocks in the plurality of blocks on which the respective values of the bits in the block depend; and

wherein the dependency data arranges the blocks into a matrix structure having a plurality of rows and a plurality of columns, and wherein, for each particular block of the plurality of blocks, the dependency identifies that the particular block is dependent on any block which is before the particular block in the row of the particular block, and is not dependent on any block that is more than H columns after the particular block in a row that is before the row of the particular block, wherein H is a fixed positive integer; and

generating the output example by, at each of a plurality of generation time steps:

identifying one or more current blocks for the generation time step, wherein each current block is a block for which the respective values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and

generating, using one or more autoregressive neural networks, the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block.

9. The system of claim 8 , wherein, for generation time steps at which a plurality of current blocks are identified, the current blocks are generated in parallel.

10. The system of claim 8 , wherein, for a given block in row i and column j, the dependency identifies that the given block depends only on the respective values of bits in:

blocks 1 through j−1 in row i and, for each row i-k below row i, blocks 1 through (j−1+kH), wherein k is an integer.

11. The system of claim 10 , wherein identifying one or more current blocks for the generation time step comprises, determining, for each row of the plurality of rows, whether there is a block in the row for which all of the other blocks identified in the dependency for the block have already been generated.

12. The system of claim 11 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each row in which a current block is identified:

generating a network input that conditions an autoregressive neural network that is specific to the row on the respective values of the bits in the blocks identified in the dependency for the current block in the row;

processing the network input using the autoregressive neural network that is specific to the row, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.

13. The system of claim 11 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each current block:

generating a network input that conditions an autoregressive neural network on the respective values of the bits in the blocks identified in the dependency for the current block;

processing the network input using an instance of the autoregressive neural network, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.

14. The system of claim 8 , wherein each block also depends on a context input, and wherein generating the respective values of the bits in the current blocks is also conditioned on the context input.

15. One or more non-transitory computer storage media storing instructions that when implemented by one or more computers cause the one or more computers to perform operations comprising:

receiving a request to generate an output example of a particular type having T N-bit samples, wherein N and T are respective integers greater than one, and wherein each of the N*T bits in the output example has a respective value;

accessing dependency data, wherein the dependency data:

partitions the N*T bits in the output example into a plurality of blocks of bits, each block of bits comprising a respective plurality of bits from the output example,

for each of a plurality of the blocks, defines a respective dependency for the block that identifies one or more other blocks in the plurality of blocks on which the respective values of the bits in the block depend; and

wherein the dependency data arranges the blocks into a matrix structure having a plurality of rows and a plurality of columns, and wherein, for each particular block of the plurality of blocks, the dependency identifies that the particular block is dependent on any block which is before the particular block in the row of the particular block, and is not dependent on any block that is more than H columns after the particular block in a row that is before the row of the particular block, wherein H is a fixed positive integer; and

generating the output example by, at each of a plurality of generation time steps:

identifying one or more current blocks for the generation time step, wherein each current block is a block for which the respective values of the bits in all of the other blocks identified in the dependency for the block have already been generated; and

generating, using one or more autoregressive neural networks, the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block.

16. The computer storage media of claim 15 , wherein, for generation time steps at which a plurality of current blocks are identified, the current blocks are generated in parallel.

17. The computer storage media of claim 15 , wherein, for a given block in row i and column j, the dependency identifies that the given block depends only on the respective values of bits in:

blocks 1 through j−1 in row i and, for each row i-k below row i, blocks 1 through (j−1+kH), wherein k is an integer.

18. The computer storage media of claim 17 , wherein identifying one or more current blocks for the generation time step comprises, determining, for each row of the plurality of rows, whether there is a block in the row for which all of the other blocks identified in the dependency for the block have already been generated.

19. The computer storage media of claim 18 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each row in which a current block is identified:

generating a network input that conditions an autoregressive neural network that is specific to the row on the respective values of the bits in the blocks identified in the dependency for the current block in the row;

processing the network input using the autoregressive neural network that is specific to the row, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.

20. The computer storage media of claim 18 , wherein generating the respective values of the bits in the current blocks for the generation time step conditioned on, for each current block, the already generated respective values of the bits in the other blocks identified in the dependency for the current block comprises, for each current block:

generating a network input that conditions an autoregressive neural network on the respective values of the bits in the blocks identified in the dependency for the current block;

processing the network input using an instance of the autoregressive neural network, wherein the autoregressive neural network is configured to process the network input to generate a likelihood distribution over possible respective value combinations, each possible respective value combination including a respective bit respective value for each of the bits in the current block;

selecting a respective value combination from the possible respective value combinations in accordance with the likelihood distribution, and

assigning the respective bit respective values in the selected respective value combination to the corresponding bits in the current block.