Image pickup apparatus, information processing apparatus, display apparatus, information processing system, image data sending method, image displaying method, and computer program

1. An image pickup apparatus, comprising:

a camera configured to produce data of a plurality of kinds of images from an image frame obtained by picking up an image of an object as a moving picture for each of pixel strings which configure a row; and

an image sending unit configured to extract a pixel string in a region requested from a host terminal from within the data of each of the plurality of kinds of images and connect the pixel strings to each other for each unit number of pixels for connection determined on a basis of a given rule to produce a stream and then transmit the stream to the host terminal;

wherein the image sending unit switchably determines whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image.

2. The image pickup apparatus according to claim 1 ,

wherein the image sending unit sets the unit number of pixels for connection for an image, to which a fixed number is to be applied, to a number of pixels for one row in the requested region or a number of pixels obtained by equally dividing one row into a given number but determines the unit number of pixels for connection of an image, to which a variable value is to be applied, such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value.

3. The image pickup apparatus according to claim 1 ,

wherein, before the camera starts production of a pixel string of a row of an image to which a fixed number is to be applied, the image sending unit ends the connection of an extracted pixel string of a preceding row of the image to the stream.

4. The image pickup apparatus according to claim 1 ,

wherein, after the connection of the pixel strings for one frame of an image, to which a fixed number is to be applied, to the stream ends, the image sending unit connects, to a remaining pixel string of an image to which a variable number is to be applied, a pixel string of an image, to which a fixed number is to be applied, of a next image frame.

5. The image pickup apparatus according to claim 1 ,

wherein the camera produces a pair of data of the plurality of kinds of images from stereo images picked up by two cameras which pick up images of an object from left and right points of view spaced by a known distance from each other; and

the image sending unit connects, with regard to an image to which a fixed number is to be applied, next to a pixel string of the unit for connection of one of the pair of images, a pixel string of the unit for connection of an other one of the pair of images at a corresponding position, but starts, with regard to an image to which a variable number is to be applied, after the connection of all of pixel strings of one of the pair of images to the stream ends, connection of a pixel string of the other one of the pair of images.

6. An information processing apparatus comprising:

an image processing unit configured to produce data of a plurality of kinds of images to be displayed in one screen;

an image extraction unit configured to connect pixel strings, which configure a row of the data of the plurality of kinds of images, to each other for each unit number of pixels for connection determined on a basis of a given rule to produce a stream; and

a communication unit configured to transmit the stream to a display apparatus;

wherein the image extraction unit switchably determines whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image,

wherein the image extraction unit sets the unit number of pixels for connection for an image, to which a fixed number is to be applied, to a number of pixels for one row of the image or a number of pixels obtained by equally dividing one row into a given number but determines the unit number of pixels for connection of an image, to which a variable value is to be applied, such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value.

7. The information processing apparatus according to claim 6 ,

wherein the image processing unit produces a pair of data of the plurality of kinds of images for regions obtained by dividing a screen of a head-mounted display unit, which is the display apparatus, into left and right portions; and

the image extraction unit connects, with regard to an image to which a fixed number is to be applied, next to a pixel string of the unit for connection of one of the pair of images, a pixel string of the unit for connection of an other one of the pair of images at a corresponding position, but starts, with regard to an image to which a variable number is to be applied, after the connection of all of pixel strings of one of the pair of images to the stream ends, connection of a pixel string of the other one of the pair of images.

8. A display apparatus comprising:

a communication unit configured to receive, from a host terminal, a stream which is configured by connecting pixel strings which configure a row of data of a plurality of kinds of images to be displayed in one screen for each unit number of pixels determined on a basis of a predetermined rule and includes data of a first kind image wherein a pixel number for one row or a pixel number obtained by equally dividing one row into a predetermined number is used as a unit for connection and data of a second kind image wherein a number of pixels with which a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value is used as a unit for connection;

a frame memory configured to store pixel strings of the second kind image included in the received stream irrespective of the unit for connection; and

a display image production unit configured to produce, every time a pixel string of the unit for connection of the first kind image included in the stream is received, pixel strings included in one row of a display screen from the pixel string and read out a pixel string included in a corresponding row of the second kind image from the frame memory and then output the produced pixel strings and the read out pixel string to positions at which the pixel strings are to be displayed in accordance with outputting scanning of the row of the display screen.

9. The display apparatus according to claim 8 ,

wherein the stream includes, as the first kind image, data of an image of a first resolution which represents an entire display image and includes, as the second type image, data of an image of a second resolution which represents a region of part of the display image and is 4n times the first resolution, n being a natural number; and

the display image production unit enlarges the image of the first resolution to 4 n times and superposes the enlarged image on the image of the second resolution in a region represented by the image of the second resolution and then outputs the resulting image thereby to output one image which has different levels of detail in different regions thereof.

10. The display apparatus according to claim 9 , further comprising

a gazing point detection unit configured to detect a gazing point of a user on a displayed image and notify the host terminal of the gazing point;

wherein the image of the second resolution is an image in a region determined by the host terminal so as to include the gazing point in the display image, and

the display image production unit outputs an image in which a region represented by a high level of detail moves in response to a movement of the gazing point.

11. An information processing system comprising:

an image pickup apparatus configured to pick up an image of an object as a moving picture;

a host terminal; and

a display apparatus configured to display a moving picture on a basis of an image frame obtained from the image picked up by the image pickup apparatus;

the image pickup apparatus including

a camera configured to produce data of a plurality of kinds of images from the image frame for each of pixel strings which configure a row, and

an image sending unit configured to extract a pixel string in a region requested from the host terminal from within the data of the plurality of kinds of images and produce a stream configured by connecting the pixel strings for each number of pixels determined on a basis of a predetermined rule and then transmit the stream to the host terminal;

the host terminal including

an image processing unit configured to classify a stream transmitted thereto from the image pickup apparatus for each kind of an image and perform a predetermined process to produce data of a plurality of kinds of images to be used for display,

an image extraction unit configured to produce a stream configured by connecting pixel strings, which configure a row of the data of the plurality of kinds of images, for each pixel number determined on a basis of a predetermined rule, and

a communication unit configured to transmit the stream to the display apparatus;

both of the image sending unit of the image pickup apparatus and the image extraction unit of the host terminal switchably determining whether the unit number of pixels for connection is to be set to a fixed pixel number equal to a number of pixels for one row in a region of an object of transmission or a number of pixels obtained by equally dividing one row into a given number or to a variable pixel number such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value in response to the kind of each image;

the display apparatus including

a frame memory configured to store a pixel string of an image of the variable pixel number included in the received stream irrespective of the unit for connection, and

a display image production unit configured to produce, every time a pixel string of the fixed pixel number included in the stream is received, pixel strings included in one row of a display screen from the received pixel string and read out a pixel string included in a corresponding row from within the image of the variable pixel number from the frame memory and then output the produced pixel strings and the read out pixel string to a position at which the pixel strings are to be displayed in accordance with outputting scanning of the row of the display screen.

12. The information processing system according to claim 11 ,

wherein the display apparatus further includes a gazing point detection unit configured to detect a gazing point of a user on a displayed image thereon and notify the host terminal of information of position coordinates;

the communication unit of the host terminal requests the image pickup apparatus for transmission of a high-resolution image in a region determined so as to include the gazing point in the image frame and a low-resolution image representative of an entire region of the image frame;

both of the image sending unit of the image pickup apparatus and the image extraction unit of the host terminal produce the stream such that the high-resolution image has the variable pixel number while the low-resolution image has the fixed pixel number; and

the display image production unit of the display apparatus superposes the high-resolution image on an image obtained by enlarging the low-resolution image to output an image in which a region represented by a high level of detail moves in response to a movement of the gazing point.

13. The information processing system according to claim 12 ,

wherein the image sending unit of the image pickup apparatus changes over a kind of an image to be included in the stream in response to a speed of the movement of the gazing point.

14. An image data sending method performed by an image pickup apparatus, comprising:

producing data of a plurality of kinds of images from an image frame obtained by picking up an image of an object as a moving picture for each of pixel strings which configure a row and storing the produced data into a memory;

extracting a pixel string in a region requested from a host terminal from within the data of the plurality of kinds of images stored in the memory and producing a stream configured by connecting the pixel strings for each number of pixels determined on a basis of a given rule; and

transmitting the produced stream to the host terminal;

the producing switchably determining whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image,

wherein the extracting sets the unit number of pixels for connection for an image, to which a fixed number is to be applied, to a number of pixels for one row of the image or a number of pixels obtained by equally dividing one row into a given number but determines the unit number of pixels for connection of an image, to which a variable value is to be applied, such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value.

15. An image data sending method performed by an information processing apparatus, comprising:

producing data of a plurality of kinds of images to be displayed in one screen and storing the produced data into a memory;

producing a stream which is configured by connecting pixel strings which are stored in the memory and configure a row of the data of the plurality of kinds of images for each unit number of pixels determined on a basis of a given rule; and

transmitting the stream to a display apparatus;

wherein the producing switchably determines whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image,

wherein, after the connection of the pixel strings for one frame of an image, to which a variable number is to be applied, to the stream ends,

and a pixel string of an image, to which a variable number is to be applied, of a next image frame is connected to a remaining pixel string of an image to which a fixed number is to be applied.

16. An image displaying method performed by a display apparatus, comprising:

receiving, from a host terminal, a stream which is configured by connecting pixel strings which configure a row of data of a plurality of kinds of images to be displayed in one screen for each unit number of pixels determined on a basis of a predetermined rule and includes data of a first kind image wherein a pixel number for one row or a pixel number obtained by equally dividing one row into a predetermined number is used as a unit for connection and data of a second kind image wherein a number of pixels with which a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value is used as a unit for connection;

storing pixel strings of the second kind image included in the received stream irrespective of the unit for connection in a frame memory; and

producing, every time a pixel string of the unit for connection of the first kind image included in the stream is received, pixel strings included in one row of a display screen from the pixel string, reading out a pixel string included in a corresponding row of the second kind image from the frame memory and then outputting the produced pixel strings and the read out pixel string to positions at which the pixel strings are to be displayed in accordance with outputting scanning of the row of the display screen.

17. A non-transitory computer readable medium having stored there on a computer program for a computer, the computer program causing the computer to execute the steps of:

producing data of a plurality of kinds of images from an image frame obtained by picking up an image of an object as a moving picture for each of pixel strings which configure a row;

extracting a pixel string in a region requested from a host terminal from within the data of each of the plurality of kinds of images and connecting the pixel strings to each other for each unit number of pixels for connection determined on a basis of a given rule to produce a stream; and

transmitting the stream to the host terminal;

wherein the producing switchably determines whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image,

wherein the extracting sets the unit number of pixels for connection for an image, to which a fixed number is to be applied, to a number of pixels for one row of the image or a number of pixels obtained by equally dividing one row into a given number but determines the unit number of pixels for connection of an image, to which a variable value is to be applied, such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value.

18. A non-transitory computer readable medium having stored there on a computer program for a computer, the computer program causing the computer to execute the steps of:

producing data of a plurality of kinds of images to be displayed in one screen;

connecting pixel strings, which configure a row of the data of the plurality of kinds of images, to each other for each unit number of pixels for connection determined on a basis of a given rule to produce a stream; and

transmitting the stream to a display apparatus;

wherein the producing switchably determines whether the unit number of pixels for connection is to be set to a fixed value or a variable value in response to the kind of each image,

wherein the connecting sets the unit number of pixels for connection for an image, to which a fixed number is to be applied, to a number of pixels for one row of the image or a number of pixels obtained by equally dividing one row into a given number but determines the unit number of pixels for connection of an image, to which a variable value is to be applied, such that a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value.

19. A non-transitory computer readable medium having stored there on a computer program for a computer, the computer program causing the computer to execute the steps of:

receiving, from a host terminal, a stream which is configured by connecting pixel strings which configure a row of data of a plurality of kinds of images to be displayed in one screen for each unit number of pixels determined on a basis of a predetermined rule and includes data of a first kind image wherein a pixel number for one row or a pixel number obtained by equally dividing one row into a predetermined number is used as a unit for connection and data of a second kind image wherein a number of pixels with which a length of the stream when the connection of the plurality of kinds of images makes one round becomes equal to a given value is used as a unit for connection;

storing pixel strings of the second kind image included in the received stream irrespective of the unit for connection in a frame memory; and

producing, every time a pixel string of the unit for connection of the first kind image included in the stream is received, pixel strings included in one row of a display screen from the pixel string and reading out a pixel string included in a corresponding row of the second kind image from the frame memory and then outputting the produced pixel strings and the read out pixel string to positions at which the pixel strings are to be displayed in accordance with outputting scanning of the row of the display screen.