Method and software for obtaining answers to complex questions based on information retrieved from big data systems
A method is provided for enabling a software user to obtain answers based on information retrieved from Big Data systems to complex questions, which comprises the steps of: providing a plurality of queries associated with different query families, the different query families comprise at least one family of simple queries and at least one query family of complex query family, and each of the families is associated with the following characterizing elements: information sources, entity type for which one or more features would be synthesized, attributes to be used as filters and attributes for calculating the new features; retrieving data from the respective information sources; processing the retrieved data to enable evaluating results of the queries associated with the at least one simple query families; calculating solutions to all queries and synthesizing features characterizing the answers, based on the calculated solutions.
1. A method comprising:
providing a simple query family that groups a plurality of first simple queries, the simple query family defined by:
an information source for the first simple queries,
an entity type for which features should be generated,
first attributes that are available as filters, and
second attributes that are usable for calculating new features;
providing a complex query family that groups a plurality of first complex queries, the complex query family being dependent on the simple query family and the complex query family defined by:
the information source for the first simple queries,
the entity type for which the features should be generated, and
third attributes that are available as filters that, at least in part, include results of one of the first simple queries;
performing the one of the first simple queries on the information source, by:
identifying the simple query family,
determining the entity type defined for the simple query family,
preprocessing data in the information source defined for the simple query family to construct a list of data retrieved from the information source according to the entity type, and
evaluating in one batch each first simple query in the simple query family including the one of the first simple queries, using the list of data;
obtaining results of the performing of the one of the first simple queries;
using the results to construct one of the third attributes that is available as a filter for the complex query family;
performing one of the first complex queries using the one of the third attributes.
2. The method of claim 1 , further comprising a step of combining queries associated with a same query family.
3. The method of claim 1 , wherein in a case that at least one query that belongs to a family of queries has not yet been solved, the method further comprises the steps of:
(i) generating a family of queries F that depends only on a family of queries comprising queries that have already been solved;
(ii) if F is not an empty group,
a. Selecting a query q that belongs to family F of queries;
b. Deleting query q from family F; and
c. Solving query q;
(iii) repeating steps a to c, until family F does not comprise any still unanswered queries.
4. The method of claim 1 , wherein the information source is one or more Big Data systems storing data that relates to activities of a plurality of users at the Internet.
5. The method of claim 1 , wherein the information source is one or more Big Data systems storing data that relates to details of voice calls or text messages associated with a plurality of users.
6. The method of claim 1 , wherein the features are used in a process selected from among machine-learning, data mining, and artificial intelligence.
7. The method of claim 1 , wherein the information source stores one of: social media data, communications data, consumption data, customer related data, and any combination thereof.
8. A non-transitory computer-readable storage media storing one or more sequences of instructions which when executed cause one or more processors to:
provide a simple query family that groups a plurality of first simple queries, the simple query family defined by:
an information source for the first simple queries,
an entity type for which features should be generated,
first attributes that are available as filters, and
second attributes that are usable for calculating new features;
provide a complex query family that groups a plurality of first complex queries, the complex query family being dependent on the simple query family and the complex query family defined by:
the information source for the first simple queries,
the entity type for which the features should be generated, and
third attributes that are available as filters that, at least in part, include results of one of the first simple queries;
perform the one of the first simple queries on the information source, by:
identifying the simple query family,
determining the entity type defined for the simple query family,
preprocessing data in the information source defined for the simple query family to construct a list of data retrieved from the information source according to the entity type, and
evaluating in one batch each first simple query in the simple query family including the one of the first simple queries, using the list of data;
obtain results of the performing of the one of the first simple queries;
use the results to construct one of the third attributes that is available as a filter for the complex query family;
perform one of the first complex queries using the one of the third attributes.
9. The non-transitory computer-readable storage media of claim 8 , wherein the information source is one or more Big Data systems comprises data that relates to activities of a plurality of users at the Internet and/or data that relates to details of voice calls or text messages associated with a plurality of users.
10. The non-transitory computer-readable storage media of claim 8 , wherein the one or more sequences of instructions further comprise an instruction to combine queries associated with a same query family.
11. The non-transitory computer-readable storage media of claim 8 , wherein the information source is one or more Big Data systems storing data that relates to activities of a plurality of users at the Internet.
12. The non-transitory computer-readable storage media of claim 8 , wherein the information source is one or more Big Data systems storing data that relates to details of voice calls or text messages associated with a plurality of users.
13. The non-transitory computer-readable storage media of claim 8 , wherein the features are adapted to be used in a process selected from among machine-learning, data mining, and artificial intelligence.
14. The non-transitory computer-readable storage media of claim 8 , wherein the information source stores one of: social media data, communications data, consumption data, customer related data, and any combination thereof.
15. The method of claim 1 , wherein:
the information source stores call detail records (CDRs),
the entity type for which the features should be generated is a calling number of a calling subscriber,
the first attributes that are available as filters include day of the week that a call took place, and
the second attributes that are usable for calculating new features include a duration of a telephone call.
16. The method of claim 1 , further comprising:
providing an additional simple query family that groups a plurality of second simple queries;
wherein the third attributes that are available as filters for the complex query family further include results of one of the second simple queries; and
wherein the one of the first complex queries is further performed using a second one of the third attributes that is constructed from results of the one of the second simple queries.