System for recognizing user hand gesture and providing virtual reality content based on deep learning using transfer learning

1. A system of recognizing a user hand gesture and providing virtual reality content based on deep learning using transfer learning, the system comprising:

a head mounted display (HMD) configured to display, to a user, virtual reality content based on a hand gesture;

a hand gesture sensor configured to acquire hand data of the user; and

a controller device configured to receive the hand data and convert the received hand data into hand gesture data, manage a hand gesture recognizer on the basis of deep learning, and perform content execution management,

wherein the controller device compares a similarity between the hand gesture data and existing hand gesture datasets to select a candidate dataset to be used for the transfer learning and uses a similarity comparison analysis result value of the candidate dataset to construct a new hand gesture dataset.

2. The system of claim 1 , wherein the hand gesture sensor transmits the hand data, which is acquired using at least one of a camera disposed on a front surface of the HMD, a data glove worn by the user, a thimble worn by the user, or a separately disposed camera, to the controller device.

3. The system of claim 1 , wherein the controller device transmits and receives the hand data and haptic data to and from the hand gesture sensor.

4. The system of claim 1 , wherein the controller device performs accumulation and interpolation on a different piece of the hand data to convert the different piece of the hand data into the hand gesture data.

5. The system of claim 1 , wherein the controller device outputs a recognition result according to a presence or absence of the hand gesture recognizer that derives a hand gesture recognition result using the hand gesture data.

6. The system of claim 5 , wherein the controller device, in response to a result output of absence of the hand gesture recognizer with respect to the hand gesture data, executes content for generating a deep learning-based user hand gesture recognizer using transfer learning.

7. The system of claim 6 , wherein the controller device further performs learning on the deep learning-based hand gesture recognizer or selects a candidate user hand gesture recognizer and constructs a partial user hand gesture dataset to be used for transfer learning.

8. The system of claim 5 , wherein the controller device, in response to output of the hand gesture recognition result for the hand gesture data, executes hand gesture recognition-based user content.

9. The system of claim 5 , wherein the controller device uses latest user hand data of the hand gesture data to render a hand in the content.

10. A method of recognizing a user hand gesture and providing virtual reality content based on deep learning using transfer learning, the method comprising the steps of:

(a) receiving hand data and converting the received hand data into hand gesture data;

(b) identifying whether a suitable hand gesture classifier exists using the hand gesture data;

(c) when it is identified in the step (b) that the suitable hand gesture classifier does not exist, executing content for generating a deep learning-based hand gesture recognizer using transfer learning, and when it is identified in the step (b) that the suitable hand gesture classifier exists, executing hand gesture recognition-based user content; and

(d) comparing a similarity between the hand gesture data and existing hand gesture datasets to select a candidate dataset to be used for the transfer learning and uses a similarity comparison analysis result value of the candidate dataset to construct a new hand gesture dataset.

11. An apparatus for recognizing a user hand gesture and providing virtual reality content based on deep learning using transfer learning, the apparatus comprising:

an inputter configured to receive hand data;

a memory in which a program for providing virtual reality content using the hand data is stored; and

a processor configured to execute the program,

wherein the processor receives the hand data and converts the received hand data into hand gesture data, manages a hand gesture recognizer on the basis of deep learning, and performs content execution management,

wherein the processor compares a similarity between the hand gesture data an existing hand gesture datasets to select a candidate dataset to be used for the transfer learning and uses a similarity comparison analysis result value of the candidate dataset to construct a new hand gesture dataset.

12. The apparatus of claim 11 , wherein the hand data is acquired using at least one of a camera disposed on a front surface of a head mounted display (HMD), a data glove worn by a user, a thimble worn by a user, or a separately disposed camera.

13. The apparatus of claim 11 , wherein the processor performs accumulation and interpolation on a different piece of the hand data to convert the different piece of the hand data into the hand gesture data.

14. The apparatus of claim 11 , wherein the processor outputs a recognition result according to a presence or absence of the hand gesture recognizer that derives a hand gesture recognition result using the hand gesture data.

15. The apparatus of claim 14 , wherein the processor, in response to a result output of absence of the hand gesture recognizer with respect to the hand gesture data, executes content for generating a deep learning-based user hand gesture recognizer using transfer learning.

16. The apparatus of claim 15 , wherein the processor further performs learning on the deep learning-based hand gesture recognizer or selects a candidate user hand gesture recognizer and construct a partial user hand gesture dataset to be used for transfer learning.

17. The apparatus of claim 14 , wherein the processor, in response to output of the hand gesture recognition result for the hand gesture data, executes hand gesture recognition-based user content.