Scene selection method, scene selection apparatus and program
Provided is a technique for accurately estimating a scene even when the number of input signals increases. A scene estimation method includes: when S is the number of scenes and M is the number of input acoustic signals, an acoustic signal encoding step of generating, by a scene estimation device, an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M); and a scene selection step of selecting, by the scene estimation device, a scene from which M input acoustic signals are acquired from among S scenes, using the integrated acoustic feature amount.
1 . A scene selection method comprising:
when S is a number of scenes and M is a number of input acoustic signals,
an acoustic signal encoding step of generating, by a scene selection device, an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M), wherein the generating the integrated acoustic feature amount further comprises: p 2 an m-th acoustic encoding step (m=1, . . . , M) of generating an m-th acoustic feature amount from the m-th input acoustic signal,
an m-th conditional acoustic encoding step (m=1, . . . , M) of generating an m-th conditional acoustic feature amount from the m-th acoustic feature amount and the m-th input acoustic signal acquisition position, a dimension of the m-th conditional acoustic feature amount being smaller than a dimension of the m-th acoustic feature amount, thereby removing a feature that is redundant according to a position of acquiring the m-th input acoustic signal, and
an integrated acoustic encoding step of generating the integrated acoustic feature amount from the m-th conditional acoustic feature amount (m=1, . . . , M); and
a scene selection step of selecting, by the scene selection device, a scene from which M input acoustic signals are acquired from among S scenes, using the integrated acoustic feature amount, wherein
the m-th acoustic encoding step (m=1, . . . , M) is performed by a multi-layer Convolutional Neural Network (CNN) and the input to the multi-layer CNN is a logarithmic mel spectrogram obtained by applying a short-time Fourier transform (STFT) to the m-th input acoustic signal and by further applying a mel filter bank,
the m-th conditional acoustic encoding step (m=1, . . . , M) is performed by a first neural network comprising a linear layer and the input to the first neural network is a first vector obtained by combining the m-th acoustic feature amount and the m-th input acoustic signal acquisition position,
the integrated acoustic encoding step is performed by a second neural network comprising a linear layer and the input to the second neural network is a second vector obtained by combining the m-th conditional acoustic feature amount (m=1, . . . , M), and
the scene selection step is performed by a third neural network comprising a linear layer and a Softmax layer.
2 . A scene selection method comprising:
when S is a number of scenes, M is a number of input acoustic signals, and N is a number of input video signals,
an acoustic signal encoding step of generating, by a scene selection device, an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M), wherein the generating the integrated acoustic feature amount further comprises:
an m-th acoustic encoding step (m=1, . . . , M) of generating an m-th acoustic feature amount from the m-th input acoustic signal;
an m-th conditional acoustic encoding step (m=1, . . . , M) of generating an m-th conditional acoustic feature amount from the m-th acoustic feature amount and the m-th input acoustic signal acquisition position, a dimension of the m-th conditional acoustic feature amount being smaller than a dimension of the m-th acoustic feature amount, thereby removing a feature that is redundant according to a position of acquiring the m-th input acoustic signal; and
an integrated acoustic encoding step of generating the integrated acoustic feature amount from them-th conditional acoustic feature amount (m=1, . . . , M);
a video signal encoding step of generating, by the scene selection device, an integrated video feature amount from an n-th input video signal (n=1, . . . , N) and a position where the n-th input video signal is acquired (hereinafter referred to as an n-th input video signal acquisition position) (n=1, . . . , N)
wherein the generating the integrated video feature amount further comprises:
an n-th video encoding step (n=1, . . . , N) of generating an n-th video feature amount from then-th input video signal;
an n-th conditional video encoding step (n=1, . . . , N) of generating an n-th conditional video feature amount from the n-th video feature amount and the n-th input video signal acquisition position, a dimension of the n-th conditional video feature amount being smaller than a dimension of the n-th video feature amount, thereby removing a feature that is redundant according to a position of acquiring the n-th input video signal; and
an integrated video encoding step of generating the integrated video feature amount from the n-th conditional video feature amount (n=1, . . . , N); and
a scene selection step of selecting, by the scene selection device, a scene from which M input acoustic signals and N input video signals are acquired from among S scenes, using the integrated acoustic feature amount and the integrated video feature amount, wherein
the m-th acoustic encoding step (m=1, . . . , M) is performed by a multi-layer Convolutional Neural Network (CNN) and the input to the multi-layer CNN is a logarithmic mel spectrogram obtained by applying a short-time Fourier transform (STFT) to the m-th input acoustic signal and by further applying a mel filter bank,
the m-th conditional acoustic encoding step (m=1, . . . , M) is performed by a first neural network comprising a linear layer and the input to the first neural network is a first vector obtained by combining the m-th acoustic feature amount and the m-th input acoustic signal acquisition position,
the integrated acoustic encoding step is performed by a second neural network comprising a linear layer and the input to the second neural network is a second vector obtained by combining the m-th conditional acoustic feature amount (m=1, . . . , M),
the n-th video encoding step (n=1, . . . , N) is performed by ResNet,
the n-th conditional video encoding step (n=1, . . . , N) is performed by a third neural network comprising a linear layer and the input to the third neural network is a third vector obtained by combining the n-th video feature amount and the n-th input video signal acquisition position,
the integrated video encoding step is performed by a fourth neural network comprising a linear layer and the input to the fourth neural network is a fourth vector obtained by combining the n-th conditional video feature amount (n=1, . . . , N), and
the scene selection step is performed by a fifth neural network comprising a linear layer and a Softmax layer and the input to the fifth neural network is a fifth vector obtained by combining the integrated acoustic feature amount and the integrated video feature amount.
3 . A scene selection device comprising:
when S is a number of scenes and M is a number of input acoustic signals,
an acoustic signal encoding circuitry configured to generate an integrated acoustic feature amount from an m-th input acoustic signal (m=1, . . . , M) and a position where the m-th input acoustic signal is acquired (hereinafter referred to as an m-th input acoustic signal acquisition position) (m=1, . . . , M), wherein the acoustic signal encoding circuitry further comprises:
an m-th acoustic encoding circuitry (m=1, . . . , M) configured to generate an m-th acoustic feature amount from the m-th input acoustic signal,
an m-th conditional acoustic encoding circuitry (m=1, . . . , M) configured to generate an m-th conditional acoustic feature amount from the m-th acoustic feature amount and the m-th input acoustic signal acquisition position, a dimension of the m-th conditional acoustic feature amount being smaller than a dimension of the m-th acoustic feature amount, thereby removing a feature that is redundant according to a position of acquiring the m-th input acoustic signal, and
an integrated acoustic encoding circuitry configured to generate the integrated acoustic feature amount from the m-th conditional acoustic feature amount (m=1, . . . , M); and
a scene selection circuitry configured to select a scene from which M input acoustic signals are acquired from among S scenes, using the integrated acoustic feature amount, wherein
the m-th acoustic encoding circuitry (m=1, . . . , M) is configured using a multi-layer Convolutional Neural Network (CNN) and the input to the multi-layer CNN is a logarithmic mel spectrogram obtained by applying a short-time Fourier transform (STFT) to the m-th input acoustic signal and by further applying a mel filter bank,
the m-th conditional acoustic encoding circuitry (m=1, . . . , M) is configured using a first neural network comprising a linear layer and the input to the first neural network is a first vector obtained by combining the m-th acoustic feature amount and the m-th input acoustic signal acquisition position,
the integrated acoustic encoding circuitry is configured using a second neural network comprising a linear layer and the input to the second neural network is a second vector obtained by combining the m-th conditional acoustic feature amount (m=1, . . . , M), and
the scene selection circuitry is configured using a third neural network comprising a linear layer and a Softmax layer.
4 . A non-transitory computer-readable recording medium recording a program for causing a computer to execute the scene selection method according to claim 1 .
5 . A non-transitory computer-readable recording medium recording a program for causing a computer to execute the scene selection method according to claim 2 .