Method and device for providing recipe video
A device and method for providing a recipe video are provided, the method including obtaining text feature information corresponding to each step of a cooking recipe, the cooking recipe comprising a plurality of steps corresponding to a plurality of coking actions; obtaining image feature information corresponding to each frame of a plurality of frames constituting a cooking video; matching each step of the cooking recipe with a respective section of the cooking video based on a correlation between the obtained text feature information and the obtained image feature information obtained using a machine learned matching model, wherein the respective section of the cooking video corresponds to a respective step among the plurality of steps; and generating the recipe video by using the respective section of the cooking video matched with each step of the cooking recipe.
1 . A method of providing a recipe video, the method comprising:
obtaining text feature information corresponding to each step of a cooking recipe, the cooking recipe comprising a plurality of steps corresponding to a plurality of coking actions;
obtaining image feature information corresponding to each frame of a plurality of frames constituting a cooking video;
matching each step of the cooking recipe with a respective section of the cooking video based on a correlation between the obtained text feature information and the obtained image feature information obtained using a machine learned matching model, wherein the respective section of the cooking video corresponds to a respective step among the plurality of steps; and
generating the recipe video by using the respective section of the cooking video matched with each step of the cooking recipe.
2 . The method of claim 1 , wherein
the obtaining of the image feature information comprises:
obtaining information about a target of interest comprising a section of interest and a region of interest;
determining, among the plurality of frames, a frame corresponding to the section of interest and a patch within the frame corresponding to the region of interest according to the information about the target of interest; and
obtaining the image feature information corresponding to the determined frame and the patch within the frame from the cooking video.
3 . The method of claim 2 , wherein the obtaining of the information about the target of interest comprises obtaining information about the section of interest based on an operational situation of equipment in a room and information about the region of interest based on a location of the equipment in the room.
4 . The method of claim 2 , wherein the obtaining of the information about the target of interest comprises obtaining information about the section of interest and obtaining information about the region of interest based on tracking of a cooking action and equipment in a room.
5 . The method of claim 2 , wherein the obtaining of the information about the target of interest comprises obtaining information about at least one video of interest among a plurality of cooking videos, and obtaining information about the section of interest and the region of interest for the at least one video of interest.
6 . The method of claim 1 , wherein
the generating of the recipe video comprises:
selecting, based on saliency according to a predetermined criterion, at least one key frame from among frames constituting the respective section of the cooking video matched with each step of the cooking recipe; and
generating the recipe video comprising the selected at least one key frame.
7 . The method of claim 6 , wherein
the selecting of the at least one key frame comprises:
dividing, based on similarity, the frames constituting the respective section of the cooking video matched with each step, into clusters; and
determining the at least one key frame that satisfies a predetermined condition for each cluster of frames, based on saliency estimated according to at least one of a result of scoring an image quality, a result of scoring a degree to which a main action is performed, or a result of scoring a degree of cooking progress.
8 . The method of claim 1 , wherein the machine learned matching model receives the obtained text feature information and the obtained image feature information as inputs and outputs the correlation between the obtained image feature information and the obtained text feature information.
9 . The method of claim 8 , wherein the machine learned matching model is trained based on a dataset including training text feature information and training image feature information to calculate the correlation between the obtained text feature information and the obtained image feature information through a matrix multiplication between the obtained text feature information and the obtained image feature information.
10 . A non-transitory computer-readable medium storing instructions that, when executed by at least one processor of a recipe video device, cause the at least one processor to:
obtain text feature information corresponding to each step of a cooking recipe, the cooking recipe comprising a plurality of steps corresponding to a plurality of coking actions;
obtain image feature information corresponding to each frame of a plurality of frames constituting a cooking video;
match each step of the cooking recipe with a respective section of the cooking video based on a correlation between the obtained text feature information and the obtained image feature information obtained using a machine learned matching model, wherein the respective section of the cooking video corresponds to a respective step among the plurality of steps; and
generate the recipe video by using the respective section of the cooking video matched with each step of the cooking recipe.
11 . A device for providing a recipe video, the device comprising:
a memory storing one or more instructions; and
a processor configured to execute the one or more instructions to:
obtain text feature information corresponding to each step of a cooking recipe, the cooking recipe comprising a plurality of steps corresponding to a plurality of coking actions,
obtain image feature information corresponding to each frame of a plurality of frames constituting a cooking video,
match each step of the cooking recipe with a respective section of the cooking video based on a correlation between the obtained text feature information and the obtained image feature information obtained using a machine learned matching model, wherein the respective section of the cooking video corresponds to a respective step among the plurality of steps, and
generate the recipe video by using the respective section of the cooking video matched with each step of the cooking recipe.
12 . The device of claim 11 , wherein the processor is further configured to execute the one or more instructions to:
obtain information about a target of interest comprising a section of interest and a region of interest,
determine, among the plurality of frames, a frame corresponding to the section of interest and a patch within the frame corresponding to the region of interest according to the information about the target of interest, and
obtain the image feature information corresponding to the determined frame and the patch within the frame from the cooking video.
13 . The device of claim 12 , wherein the processor is further configured to execute the one or more instructions to obtain information about the section of interest based on an operational situation of equipment in a room and information about the region of interest based on a location of the equipment in the room.
14 . The device of claim 12 , wherein the processor is further configured to execute the one or more instructions to obtain information about the section of interest and obtain information about the region of interest based on tracking of a cooking action and equipment in a room.
15 . The device of claim 12 , wherein the processor is further configured to execute the one or more instructions to obtain information about at least one video of interest among a plurality of cooking videos, and obtain information about the section of interest and the region of interest for the at least one video of interest.
16 . The device of claim 11 , wherein the processor is further configured to execute the one or more instructions to:
select, based on saliency according to a predetermined criterion, at least one key frame from among frames constituting the respective section of the cooking video matched with each step of the cooking recipe, and
generate the recipe video comprising the selected at least one key frame.
17 . The device of claim 16 , wherein the processor is further configured to execute the one or more instructions to:
divide, based on similarity, the frames constituting the respective section of the cooking video matched with each step, into clusters, and
determine the at least one key frame that satisfies a predetermined condition for each cluster of frames, based on saliency estimated according to at least one of a result of scoring an image quality, a result of scoring a degree to which a main action is performed, or a result of scoring a degree of cooking progress.
18 . The device of claim 11 , wherein the machine learned matching model receives the obtained text feature information and the obtained image feature information as inputs and outputs the correlation between the obtained image feature information and the obtained text feature information.
19 . The device of claim 18 , wherein the machine learned matching model is trained based on a dataset including training text feature information and training image feature information to calculate the correlation between the obtained text feature information and the obtained image feature information through a matrix multiplication between the obtained text feature information and the obtained image feature information.
20 . The device of claim 19 , further comprising
a communication interface,
wherein the processor is further configured to execute the one or more instructions to,
via the communication interface, receive the cooking video and transmit the generated recipe video to an external device.