Display device
The present disclosure discloses a display device comprising: a communication unit; a display unit; a microphone; and a control unit which acquires screen image data by capturing a content provider screen including at least one piece of content outputted through the display unit upon receiving a wake-up word, inputs the screen image data to a screen recognition model corresponding to the content provider to detect at least one content region, outputs a labeling icon corresponding to each content region by using coordinate information of the content region corresponding to each of the at least one content regions detected by the screen recognition model, and when a user utterance corresponding to any one of the labeling icons is received, executes the content corresponding to the labeling icon.
1 . A display device comprising:
a display unit configured to output a screen provided by a content provider;
a microphone; and
a controller configured to obtain screen image data by capturing a content provider screen including at least one content output through the display unit upon receiving a wake-up word, detect at least one content area by inputting the screen image data to a screen recognition model corresponding to the content provider, output a labelling icon corresponding to each of the at least one content area using coordinate information corresponding to each of the at least one content area detected by the screen recognition model, and execute content corresponding to the labelling icon upon receiving user utterance corresponding to any one of the labelling icon.
2 . The display device of claim 1 , wherein the coordinate information of the content area includes a content click coordinate and a content labelling coordinate.
3 . The display device of claim 2 , wherein the content click coordinate is a center coordinate, and the content labelling coordinate is an upper left coordinate of the content area.
4 . The display device of claim 1 , wherein, based on the screen image data being input, the screen recognition model lists up a contour candidate of the screen image data, aggregates the candidates to detect a content area, and detects a content area based on the detected region.
5 . The display device of claim 4 , wherein the controller separates the screen image data into R, G, and B channels and inputs each of separated channel images to the screen recognition model, and
the screen recognition model detects the content area of each of the R, G, and B channels, and aggregates content areas detected for the respective channels to detect the content area.
6 . The display device of claim 1 , wherein the screen recognition model includes an artificial neural network, and based on the screen image data being input, the screen recognition model is learned to output the coordinate information of the content area located in the screen.
7 . The display device of claim 1 , wherein the screen recognition model includes a first screen recognition model and a second screen recognition model, and
based on the screen image data being input, the first screen recognition model lists up a contour candidate of the screen image data and aggregates the candidates to detect a first content area,
the second screen recognition model includes an artificial neural network, and based on the screen image data being input, the screen recognition model detects a second content area located in the screen, and
the controller aggregates the first content area and the second content area to detect a content area.
8 . The display device of claim 1 , wherein the labelling icon includes at least one of object information displayed in the content area, text information displayed around the content area, and placement information.
9 . A method of operating a display device, the method comprising:
obtaining screen image data by capturing a content provider screen including at least one content output through a display unit upon receiving a wake-up word;
inputting the screen image data to a screen recognition model corresponding to the content provider;
detecting at least one content area using the screen recognition model;
obtaining coordinate information of a content area of the detected content area;
outputting a labelling icon corresponding to each of the at least one content area using the coordinate information of the content area corresponding to each of the at least one content area detected by the screen recognition model; and
upon receiving user utterance corresponding to any one of the labelling icon, executing content corresponding to the labelling icon.
10 . The method of claim 9 , wherein the coordinate information of the content area includes a content click coordinate and a content labelling coordinate.
11 . The method of claim 10 , wherein the content click coordinate is a center coordinate, and the content labelling coordinate is an upper left coordinate of the content area.
12 . The method of claim 9 , wherein the detecting of the at least one content area using the screen recognition model includes:
based on the screen image data being input, listing up a contour candidate of the screen image data, aggregating the candidates to detect a content area, and detecting a content area based on the detected region, by the screen recognition model.
13 . The method of claim 12 , wherein the detecting of the at least one content area using the screen recognition model includes:
separating the screen image data into R, G, and B channels and inputting each of separated channel images to the screen recognition model; and
detecting the content area of each of the R, G, and B channels, and aggregating content areas detected for the respective channels to detect the content area.
14 . The method of claim 9 , wherein the screen recognition model includes an artificial neural network, and based on the screen image data being input, the screen recognition model is learned to output the coordinate information of the content area located in the screen.
15 . The method of claim 9 , wherein the screen recognition model includes a first screen recognition model and a second screen recognition model,
the detecting of the at least one content area using the screen recognition model includes:
based on the screen image data being input, listing up a contour candidate of the screen image data and aggregating the candidates to detect a first content area, by the first screen recognition model;
based on the screen image data being input, the screen recognition model detects a second content area located in the screen, the second screen recognition model including an artificial neural network, and
aggregating the content area and the content area to detect a content area.
16 . The method of claim 9 , wherein the labelling icon includes at least one of object information displayed in the content area, text information displayed around the content area, and placement information.