Data pipeline for cameras with multiple color channels
Provided are methods for processing raw data for sensors with multiple color channels, which can include receiving raw image data associated with an image sensor, wherein the raw image data includes a plurality of groups of pixels including a first group of pixels associated with a first filter and a second group of pixels associated with a second filter. The methods may also include generating, from the raw image data, a plurality of wavelength-based data groups including a first wavelength-based data group and a second wavelength-based data group. The first wavelength-based data group may include first pixels corresponding to the first group of pixels and the second wavelength-based data group may include second pixels corresponding to the second group of pixels. The methods may also include encoding the first wavelength-based data group and the second wavelength-based data group. Systems and computer program products are also provided.
1 . A method, comprising:
receiving, by at least one processor, raw image data associated with an image sensor, wherein the raw image data comprises a plurality of groups of pixels, wherein a first group of pixels of the plurality of groups of pixels comprises pixels associated with a first filter, a second group of pixels of the plurality of groups of pixels comprises pixels associated with a second filter, and a third group of pixels of the plurality of groups of pixels comprises pixels associated with the second filter;
generating, from the raw image data, a plurality of wavelength-based data groups comprising a first wavelength-based data group and a second wavelength-based data group, the first wavelength-based data group comprising first pixels corresponding to the first group of pixels and the second wavelength-based data group comprising second pixels corresponding to the second group of pixels, wherein generating the second wavelength-based data group comprises determining a value for the second pixels of the second wavelength-based data group based on one of the second group of pixels or the third group of pixels;
encoding, with the at least one processor, the first wavelength-based data group to generate an encoded first wavelength-based data group;
encoding, with the at least one processor, the second wavelength-based data group to generate an encoded second wavelength-based data group;
determining a comparative image dataset based on differences between pixels of the second group of pixels and pixels of the third group of pixels; and
encoding the comparative image dataset to generate an encoded comparative image dataset,
wherein encoding the first wavelength-based data group is independent of encoding of other wavelength-based data groups of the plurality of wavelength-based data groups.
2 . The method of claim 1 , wherein a third group of pixels of the plurality of groups of pixels comprises pixels associated with a third filter, the method further comprising:
generating, from the raw image data, a third wavelength-based data group, the third wavelength-based data group comprising third pixels corresponding to the third group of pixels; and
encoding, with the at least one processor, the third wavelength-based data group to generate an encoded third wavelength-based data group.
3 . The method of claim 2 , wherein the third filter is an infrared filter.
4 . The method of claim 2 , wherein encoding the first wavelength-based data group and encoding the second wavelength-based data group comprises encoding the first wavelength-based data group and encoding the second wavelength-based data group using a first encoder, and encoding the third wavelength-based data group comprises encoding the third wavelength-based data group using a second encoder different form the first encoder.
5 . The method of claim 1 , wherein the second filter is one of green, yellow or clear.
6 . The method of claim 1 , wherein generating the plurality of wavelength-based data groups comprises calculating a value for the second pixels of the second wavelength-based data group based on the second group of pixels and the third group of pixels.
7 . The method of claim 6 , wherein calculating the value for the second pixels of the second wavelength-based data group comprises calculating an average between at least one pixel of the second group of pixels and at least one pixel of the third group of pixels.
8 . The method of claim 1 , wherein encoding the first wavelength-based data group and encoding the second wavelength-based data group comprises encoding the first wavelength-based data group and encoding the second wavelength-based data group using a first encoder and encoding the comparative image dataset comprises encoding the comparative image dataset using a second encoder different from the first encoder.
9 . The method of claims 1 , wherein a fourth group of pixels of the plurality of groups of pixels comprises pixels associated with a third filter, the method further comprising:
generating, from the raw image data, a third wavelength-based data group, the third wavelength-based data group comprising third pixels corresponding to the fourth group of pixels; and
encoding the third wavelength-based data group to generate a third wavelength-based data group.
10 . The method of claim 9 , wherein the first filter is red, the second filter is green, and the third filter is blue.
11 . The method of claim 10 , wherein the second group of pixels comprises green pixels on a same row as the first group of pixels, and the third group of pixels comprises green pixels on a same row as the fourth group of pixels.
12 . The method of claim 9 , wherein the first filter is red, the second filter is yellow, and the third filter is blue.
13 . The method of claim 9 , wherein the first filter is red, the second filter is clear, and the third filter is blue.
14 . The method of claim 9 , wherein encoding the first wavelength-based data group and encoding the second wavelength-based data group comprises encoding the first wavelength-based data group and encoding the second wavelength-based data group using a first encoder and encoding the third wavelength-based data group comprises encoding the third wavelength-based data group using a second encoder different from the first encoder.
15 . The method of claim 9 , wherein a fifth group of pixels of the plurality of groups of pixels comprises pixels associated with a fourth filter, the method further comprising:
generating, from the raw image data, a fourth wavelength-based data group, the fourth wavelength-based data group comprising fourth pixels corresponding to the fifth group of pixels; and
encoding the fourth wavelength-based data group to generate an encoded fourth wavelength-based data group.
16 . The method of claim 9 , wherein the third filter is an infrared filter.
17 . The method of claim 1 , wherein encoding the first wavelength-based data group and encoding the second wavelength-based data group comprises:
converting values of the first pixels and the second pixels from a first format to a second format;
encoding the converted values of the first pixels to generate the encoded first wavelength-based data group; and
encoding the converted values of the second pixels to generate the encoded second wavelength-based data group.
18 . The method of claim 1 , wherein each of the first pixels of the first wavelength-based data group correspond to a first color and each of the second pixels of the second wavelength-based data group correspond to a second color.
19 . The method of claim 1 , wherein generating the first wavelength-based data group and the second wavelength-based data group comprises assigning the first group of pixels to the first wavelength-based data group and the second group of pixels to the second wavelength-based data group.
20 . The method of claim 1 , wherein a height of the first wavelength-based data group and a height of the second wavelength-based data group is half a height of the raw image data, and wherein a width of the first wavelength-based data group and a width of the second wavelength-based data group is half a width of the raw image data.
21 . A system, comprising:
a data store storing computer-executable instructions; and
a processor configured to:
receive raw image data associated with an image sensor, wherein the raw image data comprises a plurality of groups of pixels, wherein a first group of pixels of the plurality of groups of pixels comprises pixels associated with a first filter, a second group of pixels of the plurality of groups of pixels comprises pixels associated with a second filter, and a third group of pixels of the plurality of groups of pixels comprises pixels associated with the second filter;
generate, from the raw image data, a plurality of wavelength-based data groups comprising a first wavelength-based data group and a second wavelength-based data group, the first wavelength-based data group comprising first pixels corresponding to the first group of pixels and the second wavelength-based data group comprising second pixels corresponding to the second group of pixels, wherein generating the second wavelength-based data group comprises determining a value for the second pixels of the second wavelength-based data group based on one of the second group of pixels or the third group of pixels;
encode the first wavelength-based data group to generate an encoded first wavelength-based data group;
encode the second wavelength-based data group to generate an encoded second wavelength-based data group;
determine a comparative image dataset based on differences between pixels of the second group of pixels and pixels of the third group of pixels; and
encode the comparative image dataset to generate an encoded comparative image dataset,
wherein encoding the first wavelength-based data group is independent of encoding of other wavelength-based data groups of the plurality of wavelength-based data groups.
22 . Non-transitory computer-readable media comprising computer-executable instructions that, when executed by a computing system, causes the computing system to:
receive raw image data associated with an image sensor, wherein the raw image data comprises a plurality of groups of pixels, wherein a first group of pixels of the plurality of groups of pixels comprises pixels associated with a first filter, a second group of pixels of the plurality of groups of pixels comprises pixels associated with a second filter, and a third group of pixels of the plurality of groups of pixels comprises pixels associated with the second filter;
generate, from the raw image data, a plurality of wavelength-based data groups comprising a first wavelength-based data group and a second wavelength-based data group, the first wavelength-based data group comprising first pixels corresponding to the first group of pixels and the second wavelength-based data group comprising second pixels corresponding to the second group of pixels, wherein generating the second wavelength-based data group comprises determining a value for the second pixels of the second wavelength-based data group based on one of the second group of pixels or the third group of pixels;
encode the first wavelength-based data group to generate an encoded first wavelength-based data group;
encode the second wavelength-based data group to generate an encoded second wavelength-based data group;
determine a comparative image dataset based on differences between pixels of the second group of pixels and pixels of the third group of pixels; and
encode the comparative image dataset to generate an encoded comparative image dataset,
wherein encoding the first wavelength-based data group is independent of encoding of other wavelength-based data groups of the plurality of wavelength-based data groups.