Granted Patent B1

US 10,664,963 · App. 16/032,879 · Granted May 26, 2020

Inventors: Francesco Rossi (Sunnyvale, CA); Xiaohuan C. Wang (Los Altos, CA); Bartlomiej W. Rymkowski (San Jose, CA); Xiaojin Shi (Cupertino, CA); Marco Zuliani (San Jose, CA); Alexey Marinichev (Mountain View, CA)

Assignee: Apple Inc.

G06T5/50G06N3/0454G06N3/08G06T2207/20221

Abstract

Claims (75)

1. An image processing method, comprising:

obtaining a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtaining a first set of device performance parameters for a first device at a first time;

obtaining a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

selecting a first one of the neural networks from the first set based, at least in part, on the: first resolution, the first capture mode, and the first set of device performance parameters;

applying the selected first neural network to the first target image to create a stylized version of the first target image; and

storing the stylized version of the first target image in a memory.

2. The method of claim 1 , further comprising:

obtaining a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

selecting a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

applying the selected second neural network to the second target image to create a stylized version of the second target image; and

storing the stylized version of the second target image in the memory.

3. The method of claim 1 , further comprising:

obtaining a second set of device performance parameters for the first device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtaining a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

selecting a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

applying the selected second neural network to the second target image to create a stylized version of the second target image; and

storing the stylized version of the second target image in the memory.

4. The method of claim 1 , wherein:

the first set of device performance parameters comprises a measurement of at least one of the following: a device power state; a device processing load state; a device battery state; and a device thermal state.

5. The method of claim 1 , wherein:

selecting a first one of the neural networks comprises consulting a pre-stored look-up table on the first device.

6. The method of claim 1 , wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

7. The method of claim 1 , further comprising:

modifying one or more parameters of the selected first neural network before applying the selected first neural network to the first target image.

8. A non-transitory program storage device comprising instructions stored thereon to cause one or more processors to:

obtain a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtain a first set of device performance parameters for a first device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

select a first one of the neural networks from the first set based, at least in part, on the: first resolution, the first capture mode, and the first set of device performance parameters;

apply the selected first neural network to the first target image to create a stylized version of the first target image; and

store the stylized version of the first target image in a memory.

9. The non-transitory program storage device of claim 8 , further comprising instructions stored thereon to cause one or more processors to:

obtain a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

10. The non-transitory program storage device of claim 8 , further comprising instructions stored thereon to cause one or more processors to:

obtain a second set of device performance parameters for the first device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtain a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

11. The non-transitory program storage device of claim 8 , wherein the first set of device performance parameters comprises a measurement of at least one of the following: a device power state; a device processing load state; a device battery state; and a device thermal state.

12. The non-transitory program storage device of claim 8 , wherein:

the instructions to select a first one of the neural networks further comprise instructions to consult a pre-stored look-up table on the first device.

13. The non-transitory program storage device of claim 8 , wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

14. A device, comprising:

an image sensor;

a display screen;

a memory communicatively coupled to the image sensor;

one or more processors operatively coupled to the image sensor and the memory configured to execute instructions causing the one or more processors to:

obtain a first set of neural networks for applying a first artistic style, wherein each neural network in the first set comprises a plurality of layers, and wherein each neural network in the first set is configured to operate under a set of device performance parameters;

obtain a first set of device performance parameters for the device at a first time;

obtain a first target image captured at a second time that is later than the first time, wherein the first target image has a first resolution, and wherein the first target image was captured in a first capture mode;

select a first one of the neural networks from the first set based, at least in part, on the: first resolution, the first capture mode, and the first set of device performance parameters;

apply the selected first neural network to the first target image to create a stylized version of the first target image; and

store the stylized version of the first target image in the memory.

15. The device of claim 14 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a second target image captured at a third time that is later than the second time, wherein the second target image has a second resolution, wherein the second resolution is different than the first resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the first set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

16. The device of claim 14 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

obtain a second set of device performance parameters for the device at a third time that is later than the second time, wherein the second set of device performance parameters are different than the first set of device performance parameters;

obtain a second target image captured at a fourth time that is later than the third time, wherein the second target image has a second resolution, and wherein the second target image was captured in a second capture mode;

select a second one of the neural networks from the first set based, at least in part, on the: second resolution, the second capture mode, and the second set of device performance parameters, wherein the second neural network is different from the first neural network;

apply the selected second neural network to the second target image to create a stylized version of the second target image; and

store the stylized version of the second target image in the memory.

17. The device of claim 14 , wherein the first set of device performance parameters comprises a measurement of at least one of the following: a device power state; a device processing load state; a device battery state; and a device thermal state.

18. The device of claim 14 , wherein:

the instructions to select a first one of the neural networks further comprise instructions to consult a pre-stored look-up table on the device.

19. The device of claim 14 , wherein each neural network in the first set is configured to operate under a different set of device performance parameters.

20. The device of claim 14 , wherein the one or more processors are further configured to execute instructions causing the one or more processors to:

modify one or more parameters of the selected first neural network before applying the selected first neural network to the first target image.

Assignments (1)

Continuity (1)

Cited By (9)

US 12,293,142 US 12,313,800 US 12,373,916 US 12,400,291 US 12,423,588 US 12,437,186 US 12,461,711 US 12,499,356 US 12,548,113