Smart streaming of data between external systems and service providers in an on-demand environment
In accordance with embodiments, there are provided mechanisms and methods for facilitating smart streaming of data between external systems and service providers in an on-demand services environment according to one embodiment. In one embodiment and by way of example, a method comprises splitting, by a streaming server computing device (“streaming device”), a file including messages associated with multiple tenants in a multitenant environment based on growth of the file over a time-period. The method may further include grouping, by the streaming device, the messages into multiple files based on intelligence fields associated with one or more of the messages and the tenants, where each file of the multiple files corresponds to a tenant of the multiple tenants. The method may further include uploading, by the streaming device, the multiple files to a database device associated with a service provider and coupled to the streaming server device.
1. A method comprising:
monitoring, by a streaming device, a file over a time-period for detecting growth of the file over the time-period, the file having messages corresponding to tenants in a multi-tenant environment;
splitting, by the streaming device, the file into multiple files based on the growth of the file, wherein the growth represents increase in contents of the messages over the time-period such that the file is split into the multiple files when the growth has reached a growth threshold;
grouping, by the streaming device, the messages into the multiple files based on intelligence fields associated with one or more of the messages and the tenants, wherein each file of the multiple files corresponds to a tenant;
eliminating one or more content duplications from the one or more messages of the multiple files by (1) removing the one or more content duplications or (2) merging the one or more messages with one or more identical messages without the one or more content duplications;
uploading, by the streaming device, the multiple files to a database; and
communicating, by the streaming device, the multiple files to one or more computing devices via the database.
2. The method of claim 1 , wherein the intelligence fields are computed by an external system and communicated on to the service provider using a streaming data pipeline as facilitated by the streaming server device.
3. The method of claim 1 , further comprising evaluating, by the streaming device, the multiple files for detecting the one or more duplications in the one or more messages of the multiple files.
4. The method of claim 1 , further comprising placing, streaming device, the multiple files in multiple folders prior to uploading to the database device, wherein each of the multiple folders corresponds to a tenant of the multiple tenants.
5. The method of claim 1 , wherein uploading is facilitated through one or more calls comprising Simple Object Access Protocol (SOAP) API calls such that the one or more calls are within a plurality of calls allocated to the multiple tenants.
6. A database system comprising:
a streaming device including a processing device coupled to a storage device, the processing device to execute instructions to facilitate operations comprising:
monitoring a file over a time-period for detecting growth of the file over the time-period, the file having messages corresponding to tenants in a multi-tenant environment;
splitting the file into multiple files based on the growth of the file, wherein the growth represents increase in contents of the messages over the time-period such that the file is split into the multiple files when the growth has reached a growth threshold;
grouping the messages into the multiple files based on intelligence fields associated with one or more of the messages and the tenants, wherein each file of the multiple files corresponds to a tenant;
eliminating one or more content duplications from the one or more messages of the multiple files by (1) removing the one or more content duplications or (2) merging the one or more messages with one or more identical messages without the one or more content duplications;
uploading the multiple files to a database; and
communicating the multiple files to one or more computing devices via the database.
7. The system of claim 6 , wherein the intelligence fields are computed by an external system and communicated on to the service provider using a streaming data pipeline as facilitated by the streaming server device.
8. The system of claim 6 , wherein the operations further comprise evaluating, by the streaming device, the multiple files for detecting the one or more duplications in the one or more messages of the multiple files.
9. The system of claim 6 , wherein the operations further comprise placing the multiple files in multiple folders prior to uploading to the database device, wherein each of the multiple folders corresponds to a tenant of the multiple tenants.
10. The system of claim 6 , wherein uploading is facilitated through one or more calls comprising Simple Object Access Protocol (SOAP) API calls such that the one or more calls are within a plurality of calls allocated to the multiple tenants.
11. A non-transitory machine-readable medium having stored thereon instructions which, when executed by a processing device, cause the processing device to perform operations comprising:
monitoring a file over a time-period for detecting growth of the file over the time-period, the file having messages corresponding to tenants in a multi-tenant environment;
splitting the file into multiple files based on the growth of the file, wherein the growth represents increase in contents of the messages over the time-period such that the file is split into the multiple files when the growth has reached a growth threshold;
grouping the messages into the multiple files based on intelligence fields associated with one or more of the messages and the tenants, wherein each file of the multiple files corresponds to a tenant;
eliminating one or more content duplications from the one or more messages of the multiple files by (1) removing the one or more content duplications or (2) merging the one or more messages with one or more identical messages without the one or more content duplications;
uploading the multiple files to a database; and
communicating the multiple files to one or more computing devices via the database.
12. The non-transitory machine-readable medium of claim 11 , wherein the intelligence fields are computed by an external system and communicated on to the service provider using a streaming data pipeline as facilitated by the streaming server device.
13. The non-transitory machine-readable medium of claim 11 , wherein the operations further comprise evaluating, by the streaming device, the multiple files for detecting the one or more duplications in the one or more messages of the multiple files.
14. The non-transitory machine-readable medium of claim 11 , wherein the operations further comprise placing the multiple files in multiple folders prior to uploading to the database device, wherein each of the multiple folders corresponds to a tenant of the multiple tenants.
15. The non-transitory machine-readable medium of claim 11 , wherein uploading is facilitated through one or more calls comprising Simple Object Access Protocol (SOAP) API calls such that the one or more calls are within a plurality of calls allocated to the multiple tenants.