Real-time ad hoc querying of data records
Methods and apparatus are provided for real-time ad hoc querying of data records, such as call detail records. An exemplary method comprises obtaining data records, wherein each data records comprises a plurality of fields and wherein one or more fields are used to aggregate a history of the data records; storing the data records in a consolidation queue (and optionally, a waiting queue), based on predefined indexing criteria; storing the data records from the consolidation queues in a historical database based on one or more predefined criteria related to a queue length and/or a time interval; creating a pre-processed index that maps indexed information for a given entity to a portion of the historical database; and, in response to a user query regarding attributes of the given entity: retrieving the data records related to the given entity from the historical database based on the pre-processed index for the given entity; storing the retrieved data records of the given entity into an in-memory database; and processing the retrieved data records of the given entity to answer the one or more user queries.
1. A method, comprising the steps of:
obtaining a plurality of data records, wherein each of said data records comprise a plurality of fields and wherein one or more of said fields are used to aggregate a history of at least a portion of said data records;
storing each of said plurality of data records in one or more of a plurality of consolidation queues, based on predefined indexing criteria;
storing at least a portion of said plurality of data records from one or more of said consolidation queues in a historical database based on one or more predefined criteria related to one or more of a queue length and a time interval;
creating a pre-processed index that maps indexed information for a given entity to a portion of said historical database; and
in response to one or more user queries regarding one or more attributes of said given entity, performing the following steps:
retrieving at least a portion of the data records related to the given entity from the historical database based on the pre-processed index for the given entity;
storing the retrieved data records of the given entity into an in-memory database; and
processing said retrieved data records of the given entity to answer said one or more user queries.
2. The method of claim 1 , wherein said plurality of data records comprise call detail records and wherein said consolidation queues comprise a call source queue and a call destination queue for each of a plurality of users of a telecommunication service.
3. The method of claim 2 , wherein said one or more user queries comprise one or more of a query from a law enforcement agency and a customer profiling query.
4. The method of claim 1 , wherein said given entity comprises one or more of a user of a telecommunication service, an item of telecommunication equipment and a geographic area.
5. The method of claim 1 , wherein said storing in said plurality of consolidation queues further comprises storing said plurality of data records in a waiting queue to accommodate bursts of said data records.
6. The method of claim 1 , further comprising the step of updating said pre-processed index performed substantially simultaneously with a consolidation of one or more of said consolidation queues.
7. The method of claim 1 , further comprising the step of deleting data from said historical database based on predefined maintenance criteria.
8. The method of claim 1 , wherein said historical database comprises pre-aggregated data, aggregated for each of a plurality of entities based on said one or more of said fields.
9. The method of claim 1 , wherein said predefined indexing criteria comprises indexing by one or more fields of said data records that are targets of said one or more user queries.
10. A computer program product, comprising a tangible machine-readable storage medium having encoded therein executable code of one or more software programs, wherein the one or more software programs when executed by at least one processing device perform the following steps:
obtaining a plurality of data records, wherein each of said data records comprise a plurality of fields and wherein one or more of said fields are used to aggregate a history of at least a portion of said data records;
storing each of said plurality of data records in one or more of a plurality of consolidation queues, based on predefined indexing criteria;
storing at least a portion of said plurality of data records from one or more of said consolidation queues in a historical database based on one or more predefined criteria related to one or more of a queue length and a time interval;
creating a pre-processed index that maps indexed information for a given entity to a portion of said historical database; and
in response to one or more user queries regarding one or more attributes of said given entity, performing the following steps:
retrieving at least a portion of the data records related to the given entity from the historical database based on the pre-processed index for the given entity;
storing the retrieved data records of the given entity into an in-memory database; and
processing said retrieved data records of the given entity to answer said one or more user queries.
11. The computer program product of claim 10 , wherein said plurality of data records comprise call detail records and wherein said consolidation queues comprise a call source queue and a call destination queue for each of a plurality of users of a telecommunication service.
12. The computer program product of claim 10 , wherein said storing in said plurality of consolidation queues further comprises storing said plurality of data records in a waiting queue to accommodate bursts of said data records.
13. The computer program product of claim 10 , wherein said historical database comprises pre-aggregated data, aggregated for each of a plurality of entities based on said one or more of said fields.
14. The computer program product of claim 10 , wherein said predefined indexing criteria comprises indexing by one or more fields of said data records that are targets of said one or more user queries.
15. A system, comprising:
a memory; and
at least one processing device, coupled to the memory, operative to implement the following steps:
obtaining a plurality of data records, wherein each of said data records comprise a plurality of fields and wherein one or more of said fields are used to aggregate a history of at least a portion of said data records;
storing each of said plurality of data records in one or more of a plurality of consolidation queues, based on predefined indexing criteria;
storing at least a portion of said plurality of data records from one or more of said consolidation queues in a historical database based on one or more predefined criteria related to one or more of a queue length and a time interval;
creating a pre-processed index that maps indexed information for a given entity to a portion of said historical database; and
in response to one or more user queries regarding one or more attributes of said given entity, performing the following steps:
retrieving at least a portion of the data records related to the given entity from the historical database based on the pre-processed index for the given entity;
storing the retrieved data records of the given entity into an in-memory database; and
processing said retrieved data records of the given entity to answer said one or more user queries.
16. The system of claim 15 , wherein said plurality of data records comprise call detail records and wherein said consolidation queues comprise a call source queue and a call destination queue for each of a plurality of users of a telecommunication service.
17. The system of claim 15 , wherein said storing in said plurality of consolidation queues further comprises storing said plurality of data records in a waiting queue to accommodate bursts of said data records.
18. The system of claim 15 , further comprising the step of deleting data from said historical database based on predefined maintenance criteria.
19. The system of claim 15 , wherein said historical database comprises pre-aggregated data, aggregated for each of a plurality of entities based on said one or more of said fields.
20. The system of claim 15 , wherein said predefined indexing criteria comprises indexing by one or more fields of said data records that are targets of said one or more user queries.