Method, device, and program for providing matching information through analysis of sound information

1 . A method performed by a computing device, a user terminal, and a sensor, comprising:

obtaining, by the user terminal, sound information;

generating, by the user terminal, a spectrogram or a mel-spectrogram of the sound information by analyzing of the sound information;

identifying, by the sensor that is located in a living space of a user, whether the user is located in a specific space that is the living space of the user, by recognizing radio waves transmitted from a radio-frequency identification (RFID) module of the user terminal;

identifying, by the computing device, whether the user is located in the specific space, based on sensing information received from the sensor;

in response to an identification that the user is located in the specific space, obtaining, by the computing device, the spectrogram or the mel-spectrogram of the sound information from the user terminal;

obtaining, by the computing device, user feature information, based on the spectrogram or the mel-spectrogram of the sound information; and

providing, by the computing device, matching information corresponding to the user feature information, to one or more of the user terminals,

wherein the matching information comprises user-specific advertisement information,

wherein the providing of the matching information comprises:

generating an environmental feature table, based on one or more pieces of user feature information each corresponding to one of one or more pieces of the spectrogram or the mel-spectrogram of the sound information, that has been continuously obtained in the living space of the user in each time zone at intervals of a predetermined time period; and

providing the matching information, based on the environmental feature table,

wherein the environmental feature table comprises information on statistics of each piece of user feature information obtained at the intervals of the predetermined time period, and

wherein the providing of the matching information further comprises:

identifying a first point in time for providing the matching information, based on the environmental feature table, wherein the first point in time is a time point when a specific activity is performed in each time zone, and the matching information includes the user-specific advertisement information related to the specific activity; and

providing the matching information at the first point in time.

2 . The method of claim 1 , wherein the obtaining of the user feature information comprises:

identifying the user or an object through an analysis of the sound information; and

generating the user feature information, based on the identified user or object.

3 . The method of claim 1 , wherein the obtaining of the user feature information comprises:

generating activity time information related to a period of time during which the user does activities in the specific space through an analysis of the sound information; and

generating the user feature information, based on the activity time information.

4 . The method of claim 1 , wherein the providing of the matching information comprises:

providing the matching information, based on points in time, when a plurality of pieces of user feature information corresponding to a plurality of pieces of sound information obtained for a predetermined time are obtained and a frequency of the plurality of pieces of user feature information.

5 . The method of claim 1 , wherein the obtaining of the sound information comprises:

preprocessing the obtained sound information; and

identifying sound feature information corresponding to the preprocessed sound information, and

wherein the sound feature information comprises:

first feature information indicating whether the sound information is related to at least one of linguistic sound or non-linguistic sound; and

second feature information related to object classification.

6 . The method of claim 5 , wherein the obtaining of the user feature information comprises:

obtaining the user feature information, based on the sound feature information corresponding to the sound information, and

wherein the obtaining of the user feature information further comprises at least one of:

obtaining the user feature information corresponding to the sound information by inputting the sound information into a first sound model, when the sound feature information includes the first feature information indicating that the sound information is related to linguistic sound; and

obtaining the user feature information corresponding to the sound information by inputting the sound information into a second sound model, when the sound feature information includes the first feature information indicating that the sound information is related to non-linguistic sound.

7 . The method of claim 6 , wherein the first sound model comprises:

a neural network model trained to analyze sound information related to linguistic sound and identify at least one of text, a subject, or an emotion related to the sound information,

wherein the second sound model comprises:

a neural network model trained to analyze sound information related to non-linguistic sound and obtain object identification information or object state information related to the sound information, and

wherein the user feature information comprises:

at least one of first user feature information related to at least one of the text, the subject, or the emotion related to the sound information or second user feature information related to the object identification information or the object state information related to the sound information.

8 . The method of claim 7 , wherein the providing of the matching information comprises:

when the obtained user feature information comprises the first user feature information and the second user feature information, obtaining correlation information related to a correlation between the first user feature information and the second user feature information;

updating the matching information, based on the correlation information; and

providing the updated matching information.

9 . A non-transitory computer-readable recording medium on which program for executing a method of providing matching information through sound information analysis in conjunction with a computing device, a user terminal, and a sensor is recorded, wherein the method comprises:

obtaining, by the user terminal, sound information;

generating, by the user terminal, a spectrogram or a mel-spectrogram of the sound information by analyzing of the sound information;

identifying, by the sensor that is located in a living space of a user, whether the user is located in a specific space that is the living space of the user, by recognizing radio waves transmitted from a radio-frequency identification (RFID) module of the user terminal;

identifying, by the computing device, whether the user is located in the specific space, based on sensing information received from the sensor;

in response to an identification that the user is located in the specific space, obtaining, by the computing device, the spectrogram or the mel-spectrogram of the sound information from the user terminal;

obtaining, by the computing device, user feature information, based on the spectrogram or the mel-spectrogram of the sound information; and

providing, by the computing device, matching information corresponding to the user feature information, to one or more of the user terminals,

wherein the matching information comprises user-specific advertisement information,

wherein the providing of the matching information comprises:

generating an environmental feature table, based on one or more pieces of user feature information each corresponding to one of one or more pieces of the spectrogram or the mel-spectrogram of the sound information, that has been continuously obtained in the living space of the user in each time zone at intervals of a predetermined time period; and

providing the matching information, based on the environmental feature table,

wherein the environmental feature table comprises information on statistics of each piece of user feature information obtained at the intervals of the predetermined time period, and

wherein the providing of the matching information further comprises:

identifying a first point in time for providing the matching information, based on the environmental feature table, wherein the first point in time is a time point when a specific activity is performed in each time zone, and the matching information includes the user-specific advertisement information related to the specific activity; and

providing the matching information at the first point in time.