Audio object clustering by utilizing temporal variations of audio objects
Embodiments of the present invention relate to audio object clustering by utilizing temporal variation of audio objects. There is provided a method of estimating temporal variation of an audio object for use in audio object clustering. The method comprises obtaining at least one segment of an audio track associated with the audio object, the at least one segment containing the audio object; estimating variation of the audio object over a time duration of the at least one segment based on at least one property of the audio object and adjusting, at least partially based on the estimated variation of the audio object, a contribution of the audio object to the determination of a centroid in the audio object clustering. Corresponding system and computer program product are disclosed.
1. A method for utilizing temporal variation of an audio object in audio object clustering, the method comprising:
determining a plurality of centroids for a plurality of audio object clusters, wherein the plurality of audio object clusters includes a plurality of audio objects, wherein determining the plurality of centroids includes, for each audio object of the plurality of audio objects:
obtaining at least one segment of an audio track associated with the audio object, the at least one segment containing the audio object;
estimating variation of the audio object over a time duration of the at least one segment based on at least one property of the audio object; and
adjusting, at least partially based on the estimated variation, a contribution of the audio object to determination of a centroid in the audio object clustering,
wherein:
the contribution of the audio object is determined at least partially based on estimation of perceptual importance of the audio object, and adjusting the contribution comprises applying to the perceptual importance of the audio object a gain which decreases as the estimated variation increases; and/or
adjusting the contribution of the audio object comprises excluding, at least partially based on a determination that the estimated variation is greater than a predefined variation threshold, the audio object from the determination of the centroid in the audio object clustering; and
allocating each audio object of the plurality of audio objects to one of the plurality of audio object clusters according to a closest centroid of the plurality of centroids.
2. The method according to claim 1 , wherein obtaining the at least one segment of the audio track comprises segmenting the audio track based on at least one of:
consistency of a feature of the audio object;
a perceptual property of the audio object that indicates a level of perception of the audio object; and
a predefined time window.
3. The method according to claim 2 , wherein the perceptual property of the audio object comprises at least one of:
loudness of the audio object;
energy of the audio object; and
perceptual importance of the audio object.
4. The method according to claim 1 , wherein the at least one property of the audio object includes a perceptual property of the audio object that indicates a level of perception of the audio object, and wherein estimating the variation of the audio object comprises:
estimating discontinuity of the perceptual property over the time duration of the at least one segment.
5. The method according to claim 4 , wherein estimating the discontinuity of the perceptual property comprises estimating at least one of:
a dynamic range of the perceptual property over the time duration;
a transition frequency of the perceptual property over the time duration; and
a high-order statistics of the perceptual property over the time duration.
6. The method according to claim 1 , wherein estimating the variation of the audio object comprises:
estimating a spatial velocity of the audio object over the time duration of the at least one segment.
7. The method according to claim 1 , wherein adjusting the contribution of the audio object comprises:
adjusting, at least partially based on the estimated variation, probability that the audio object is selected as the centroid in the audio object clustering.
8. The method according to claim 1 , wherein the excluding of the audio object is further based on a set of constraints, the set of constraints including at least one of:
the audio object is excluded if at least one audio object within a predefined proximity of the audio object is not excluded from the determination of the centroid; and
the audio object is excluded if the audio object has been excluded from the determination of the centroid in a previous frame of the at least one segment.
9. The method according to claim 1 , further comprising:
determining complexity of a scene associated with the audio object,
wherein the contribution of the audio object is adjusted based on the estimated variation of the audio object and the determined complexity of the scene.
10. The method according to claim 9 , wherein the complexity of the scene is determined based on at least one of:
the number of audio objects in the scene;
the number of output clusters; and
a distribution of audio objects in the scene.
11. A system for utilizing temporal variation of an audio object in audio object clustering, the system comprising:
a determining unit configured to determine a plurality of centroids for a plurality of audio object clusters, wherein the plurality of audio object clusters includes a plurality of audio objects, wherein the determining unit includes:
a segment obtaining unit configured to obtain at least one segment of an audio track associated with each audio object of the plurality of audio objects, the at least one segment containing the audio object;
a variation estimating unit configured to estimate variation of the audio object over a time duration of the at least one segment based on at least one property of the audio object; and
a penalizing unit configured to adjust, at least partially based on the estimated variation, a contribution of the audio object to determination of a centroid in the audio object clustering,
wherein:
the system further comprises a comparing unit configured to compare the estimated variation to a predefined variation threshold, and the penalizing unit comprises a soft penalizing unit configured to apply to the perceptual importance of the audio object a gain which decreases as the estimated variation increases; and/or
the contribution of the audio object is determined at least partially based on estimation of perceptual importance of the audio object, and the penalizing unit comprises a hard penalizing unit configured to exclude, at least partially based on a determination by the comparing unit that the estimated variation is greater than the predefined variation threshold, the audio object from the determination of the centroid in the audio object clustering; and
an allocating unit configured to allocate each audio object of the plurality of audio objects to one of the plurality of audio object clusters according to a closest centroid of the plurality of centroids.
12. The system according to claim 11 , wherein the segment obtaining unit comprises a segmentation unit configured to segment the audio track based on at least one of:
consistency of a feature of the audio object;
a perceptual property of the audio object that indicates a level of perception of the audio object; and
a predefined time window.
13. The system according to claim 12 , wherein the perceptual property of the audio object comprises at least one of:
loudness of the audio object;
energy of the audio object; and
perceptual importance of the audio object.
14. The system according to claim 11 , wherein the at least one property of the audio object includes a perceptual property of the audio object that indicates a level of perception of the audio object, and wherein the variation estimating unit comprises:
a discontinuity estimating unit configured to estimate discontinuity of the perceptual property over the time duration of the at least one segment.
15. The system according to claim 14 , wherein the discontinuity estimating unit is configured to estimate at least one of:
a dynamic range of the perceptual property over the time duration;
a transition frequency of the perceptual property over the time duration; and
a high-order statistics of the perceptual property over the time duration.
16. The system according to claim 11 , wherein the variation estimating unit comprises:
a velocity estimating unit configured to estimate a spatial velocity of the audio object over the time duration of the at least one segment.
17. The system according to claim 11 , wherein the penalizing unit is configured to:
adjust, at least partially based on the estimated variation of the audio object, probability that the audio object is selected as the centroid in the audio object clustering.
18. The system according to claim 17 , wherein the excluding of the audio object is further based on a set of constraints, the set of constraints including at least one of:
the audio object is excluded if at least one audio object within a predefined proximity of the audio object is not excluded from the determination of the centroid; and
the audio object is excluded if the audio object that has been excluded from the determination of the centroid in a previous frame of the at least one segment.
19. The system according to claim 11 , further comprising:
a scene complexity determining unit configured to determine complexity of a scene associated with the audio object,
wherein the penalizing unit is configured to adjust the contribution of the audio object based on the estimated variation of the audio object and the determined complexity of the scene.
20. The system according to claim 19 , wherein the scene complexity determining unit is configured to determine the complexity of the scene based on at least one of:
the number of audio objects in the scene;
the number of output clusters; and
a distribution of audio objects in the scene.
21. A computer program product for utilizing temporal variation of an audio object in audio object clustering, the computer program product being tangibly stored on a non-transient computer-readable medium and comprising machine executable instructions which, when executed, cause the machine to perform steps of the method according to claim 1 .