Characterizing vision systems

1. A computer-implemented method for characterizing a vision system, the method comprising:

providing a condition space comprising a plurality of dimensions, each dimension representing a respective condition axis, wherein each point in the condition space specifies a set of conditions under which a vision system may operate;

providing an image;

sampling the condition space according to a pseudo-random sequence to determine a plurality of test conditions usable to characterize the vision system, wherein each test condition corresponds to a respective set of conditions;

generating a plurality of test images corresponding to the plurality of test conditions based on the image;

performing a vision inspection on each of the plurality of test images to generate respective test results;

analyzing the test results to determine conditions under which the vision system operates correctly; and

storing the test results.

2. The method of claim 1 , wherein the condition axes of the condition space comprise axes related to one or more of:

image object geometry;

camera geometry;

lighting;

focus blur;

motion blur; or

noise.

3. The method of claim 2 , wherein the condition axes related to image object geometry comprise axes for one or more of:

object scale;

object rotation; or

object offset.

4. The method of claim 2 , wherein the condition axes related to camera geometry comprise axes for one or more of:

camera position;

look at position;

camera pan; or

camera rotation.

5. The method of claim 2 , wherein the condition axes related to lighting comprise axes for one or more of:

brightness;

contrast; or

gamma.

6. The method of claim 2 , wherein the condition axes related to motion blur comprise axes for one or more of:

length of blur; or

angle of blur.

7. The method of claim 2 , wherein the condition axes related to noise comprise axes for one or more of:

noise type, comprising: uniform, Gaussian, and/or impulse;

standard deviation; or

amount of noise.

8. The method of claim 1 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

generating the plurality of test images by applying image processing functions to the image that simulate the plurality of test conditions.

9. The method of claim 8 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

receiving user input to a graphical user interface specifying one or more of:

test conditions to simulate;

number of samples of the condition space to use; or

range values for one or more of the condition axes.

10. The method of claim 1 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image;

d) if the vision inspection fails, performing a) - c) for one or more condition points in a specified neighborhood of the condition point to determine a point for which the vision inspection passes.

11. The method of claim 1 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image;

d) if the vision inspection fails,

e) modifying parameters of the vision inspection;

f) performing a vision inspection on the test image using the modified parameters; and

g) performing e)-f) in an iterative manner until the vision inspection passes.

12. The method of claim 1 , wherein said analyzing comprises determining one or more of:

fraction of passes of the vision inspection within the condition space; or

a largest convex hull in the condition space specifying conditions under which vision inspections by the vision system will pass within a specified tolerance.

13. The method of claim 1 , wherein the pseudo-random sequence comprises a low-discrepancy sequence.

14. A non-transitory computer accessible memory medium that stores program instructions for characterizing a vision system, wherein the program instructions are executable by a processor to:

provide a condition space comprising a plurality of dimensions, each dimension representing a respective condition axis, wherein each point in the condition space specifies a set of conditions

under which a vision system may operate;

provide an image;

sample the condition space according to a pseudo-random sequence to determine a plurality of test conditions usable to characterize the vision system, wherein each test condition corresponds to a respective set of conditions;

generate a plurality of test images corresponding to the plurality of test conditions based on the image;

perform a vision inspection on each of the plurality of test images to generate respective test results; analyze the test results to determine conditions under which the vision system operates correctly.

15. A computer accessible memory medium that stores program instructions for characterizing a vision system, wherein the program instructions are executable by a processor to:

provide a condition space comprising a plurality of dimensions, each dimension representing a respective condition axis, wherein each point in the condition space specifies a set of conditions under which a vision system may operate;

provide an image;

sample the condition space according to a pseudo-random sequence to determine a plurality of test conditions usable to characterize the vision system, wherein each test condition corresponds to a respective set of conditions;

generate a plurality of test images corresponding to the plurality of test conditions based on the image;

perform a vision inspection on each of the plurality of test images to generate respective test results;

analyze the test results to determine conditions under which the vision system operates correctly.

16. The method of claim 15 , wherein the condition axes of the condition space comprise axes related to one or more of:

image object geometry;

camera geometry;

lighting;

focus blur;

motion blur; or

noise.

17. The method of claim 16 , wherein the condition axes related to image object geometry comprise axes for one or more of:

object scale;

object rotation; or

object offset.

18. The method of claim 17 , wherein the condition axes related to camera geometry comprise axes for one or more of:

camera position;

look at position;

camera pan; or

camera rotation.

19. The method of claim 16 , wherein the condition axes related to lighting comprise axes for one or more of:

brightness;

contrast; or

gamma.

20. The method of claim 16 , wherein the condition axes related to motion blur comprise axes for one or more of:

length of blur; or

angle of blur.

21. The method of claim 16 , wherein the condition axes related to noise comprise axes for one or more of:

noise type, comprising: uniform, Gaussian, and/or impulse;

standard deviation; or

amount of noise.

22. The method of claim 15 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

generating the plurality of test images by applying image processing functions to the image that simulate the plurality of test conditions.

23. The method of claim 8 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

receiving user input to a graphical user interface specifying one or more of:

test conditions to simulate;

number of samples of the condition space to use; or range values for one or more of the condition axes.

24. The method of claim 15 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image;

d) if the vision inspection fails, performing a) - c) for one or more condition points in a specified neighborhood of the condition point to determine a point for which the vision inspection passes.

25. The method of claim 15 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image; d) if the vision inspection fails,

e) modifying parameters of the vision inspection;

f) performing a vision inspection on the test image using the modified parameters; and

g) performing e)-f) in an iterative manner until the vision inspection passes.

26. The method of claim 15 , wherein said analyzing comprises determining one or more of:

fraction of passes of the vision inspection within the condition space; or

a largest convex hull in the condition space specifying conditions under which vision inspections by the vision system will pass within a specified tolerance.

27. The method of claim 15 , wherein said sampling comprises:

sampling the condition space according to a pseudo-random sequence.

28. The method of claim 27 , wherein the pseudo-random sequence comprises a low-discrepancy sequence.

29. A non-transitory computer accessible memory medium that stores program instructions for characterizing a vision system, wherein the program instructions are executable by a processor to:

provide a condition space comprising three or more dimensions, each dimension representing a respective condition axis, wherein each point in the condition space specifies a set of conditions under which a vision system may operate;

provide an image;

sample the condition space to determine a plurality of test conditions usable to characterize the vision system, wherein each test condition corresponds to a respective set of conditions;

generate a plurality of test images corresponding to the plurality of test conditions based on the image;

perform a vision inspection on each of the plurality of test images to generate respective test results;

analyze the test results to determine conditions under which the vision system operates correctly; and

store the test results.

30. The non-transitory computer accessible memory medium of claim 29 , wherein said sampling comprises:

sampling the condition space according to a pseudo-random sequence.

31. The non-transitory computer accessible memory medium of claim 30 , wherein the pseudo-random sequence comprises a low-discrepancy sequence.

32. The non-transitory computer accessible memory medium of claim 29 , wherein the condition axes of the condition space comprise axes related to one or more of:

image object geometry;

camera geometry;

lighting;

focus blur;

motion blur; or

noise.

33. The non-transitory computer accessible memory medium of claim 29 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

generating the plurality of test images by applying image processing functions to the image that simulate the plurality of test conditions.

34. The non-transitory computer accessible memory medium of claim 33 , wherein said generating a plurality of test images corresponding to the plurality of test conditions based on the image comprises:

receiving user input to a graphical user interface specifying one or more of:

test conditions to simulate;

number of samples of the condition space to use; or

range values for one or more of the condition taxes.

35. The non-transitory computer accessible memory medium of claim 29 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image;

d) if the vision inspection fails, performing a)-c) for one or more condition points in a specified neighborhood of the condition point to determine a point for which the vision inspection passes.

36. The non-transitory computer accessible memory medium of claim 29 , wherein said sampling the condition space, said generating the plurality of test images, and said performing a vision inspection on each of the plurality of test images comprises:

for each of a plurality of iterations:

a) sampling the condition space to generate a condition point;

b) generating a test image based on the condition point;

c) performing a vision inspection on the test image;

d) if the vision inspection fails,

e) modifying parameters of the vision inspection;

f) performing a vision inspection on the test image using the modified parameters; and

g) performing e)-f) in an iterative manner until the vision inspection passes.

37. The non-transitory computer accessible memory medium of claim 29 , wherein said analyzing comprises determining one or more of:

fraction of passes of the vision inspection within the condition space; or

a largest convex hull in the condition space specifying conditions under which vision inspections by the vision system will pass within a specified tolerance.