Generating derived dimensions to modify data collection schemas without additional data ingestion
The present disclosure relates to systems, non-transitory computer-readable media, and methods for generating derived dimensions within ingested data collections. In particular, in one or more embodiments, the disclosed systems generate derived dimensions that modify or correct one or more organizational schemas of an ingested data collection when querying the ingested data collection. For example, the disclosed systems receive a user-defined definition, and generates a fallback expression based on the user-defined definition and aligned to the schema of a particular ingested data collection. In at least one embodiment, the disclosed systems generate the derived dimension by identifying and modifying one or more existing dimensions in the schema based on the fallback expression when reading data from the ingested data collection.
1 . A computer-implemented method comprising:
accessing, by a digital data analytics management system, an ingested data collection categorized into a plurality of dimensions, the plurality of dimensions being set prior to ingestion of the ingested data collection in accordance with a data organization schema, wherein the data organization schema specifies how to categorize raw data into the plurality of dimensions via ingestion of the raw data and wherein each dimension from the plurality of dimensions comprises a data category;
generating, utilizing at least one dimension of the plurality of dimensions, a derived dimension that includes a new, post-ingestion dimension applied to the ingested data collection to change a reading of the ingested data collection by changing how the plurality of dimensions are applied to the ingested data collection in generating one or more reports without changing the ingested data collection or the data organization schema by:
determining at least one dimension of the plurality of dimensions that corresponds with a user-defined definition for the derived dimension; and
modifying the at least one dimension of the plurality of dimensions based on the user-defined definition without an additional data ingestion when retrieving data from the ingested data collection based on a query that invokes the derived dimension; and
generating a report for the ingested data collection that reflects a changed reading of the ingested data collection in accordance with the derived dimension.
2 . The computer-implemented method as recited in claim 1 , wherein modifying the at least one dimension of the plurality of dimensions based on the user-defined definition comprises reading the data out of the ingested data collection to change a categorization of the data in the report compared to a categorization of the data in the ingested data collection.
3 . The computer-implemented method as recited in claim 2 , further comprising:
receiving an additional query that does not invoke the derived dimension; and
reading the data out of the ingested data collection without changing the categorization of the data in the ingested data collection.
4 . The computer-implemented method as recited in claim 1 , further comprising receiving the user-defined definition for the derived dimension by receiving a user- defined conditional fallback expression indicating how to modify the at least one dimension of the plurality of dimensions.
5 . The computer-implemented method as recited in claim 1 , further comprising modifying the at least one dimension of the plurality of dimensions based on the user-defined definition by reading the at least one dimension and an additional dimension from the ingested data collection into the derived dimension stored in a meta-field thereby merging the at least one dimension and the additional dimension into the derived dimension.
6 . The computer-implemented method as recited in claim 1 , further comprising modifying the at least one dimension of the plurality of dimensions based on the user-defined definition by removing the at least one dimension from the plurality of dimensions when reading the data from the ingested data collection in response to the query.
7 . The computer-implemented method as recited in claim 1 , further comprising modifying the at least one dimension of the plurality of dimensions based on the user-defined definition by renaming the at least one dimension when reading or extracting the at least one dimension from the ingested data collection in response to the query.
8 . The computer-implemented method as recited in claim 1 , wherein generating the derived dimension is done at query-time and is performed in seconds.
9 . A non-transitory computer-readable storage medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations comprising:
accessing, within a digital data analytics management system, an ingested data collection categorized into a plurality of dimensions, the plurality of dimensions being set prior to ingestion of data into the ingested data collection in accordance with a data organization schema, wherein the data organization schema specifies how to categorize raw data into the plurality of dimensions via ingestion of the raw data and wherein each dimension from the plurality of dimensions comprises a data category;
receiving a query associated with a derived dimension that includes a new, post-ingestion dimension applied to the ingested data collection to change a reading of the ingested data collection by changing how the plurality of dimensions are applied to the ingested data collection in generating one or more reports without changing the ingested data collection or the data organization schema;
generating, based on the query associated with the derived dimension, the derived dimension utilizing at least one dimension of the plurality of dimensions by:
determining at least one dimension of the plurality of dimensions that corresponds with a user-defined definition for the derived dimension; and
based on determining that a condition specified by the user-defined definition is satisfied by the at least one dimension, reading the at least one dimension of the plurality of dimensions from the ingested data collection based on the user-defined definition without an additional ingestion of the data; and
generating, in response to determining results of the query, a report comprising the results of the query reflecting a changed reading of the ingested data collection in accordance with the derived dimension.
10 . The non-transitory computer-readable storage medium as recited in claim 9 , wherein reading the at least one dimension of the plurality of dimensions from the ingested data collection based on the user-defined definition comprises assigning data associated with the at least one dimension to an additional dimension within the ingested data collection when retrieving the additional dimension in response to the query.
11 . The non-transitory computer-readable storage medium as recited in claim 9 , wherein reading the at least one dimension of the plurality of dimensions from the ingested data collection based on the user-defined definition comprises earmarking the at least one dimension to be skipped when responding to the query.
12 . The non-transitory computer-readable storage medium as recited in claim 9 , wherein reading the at least one dimension of the plurality of dimensions from the ingested data collection based on the user-defined definition comprises renaming a column when retrieving the column from the ingested data collection.
13 . The non-transitory computer-readable storage medium as recited in claim 12 , wherein generating, based on the query associated with the derived dimension, the derived dimension comprises generating the derived dimension without modifying a categorization of the ingested data collection.
14 . The non-transitory computer-readable storage medium as recited in claim 9 , wherein the operations further comprise accessing the ingested data collection in response to data being ingested according to a preconfigured data organization schema by a digital data analytics engine.
15 . A system comprising:
at least one computer memory device comprising:
a data organization schema configured according to a plurality of dimensions, and
a plurality of user-defined definitions for generating derived dimensions; and
one or more servers configured to cause the system to:
ingest data from a digital data analytics engine into the plurality of dimensions within an ingested data collection according to the data organization schema, wherein the data organization schema specifies how to categorize the data into the plurality of dimensions via ingestion of the data and wherein each dimension from the plurality of dimensions comprises a data category;
generate, utilizing a dimension of the plurality of dimensions, a derived dimension that includes a new, post-ingestion dimension applied to the ingested data collection to change a reading of the ingested data collection by changing how the plurality of dimensions are applied to the ingested data collection in generating one or more reports without changing the ingested data collection or the data organization schema by:
determining the dimension of the plurality of dimensions corresponds with a user-defined definition for the derived dimension; and
modifying the dimension of the plurality of dimensions based on the user-defined definition without an additional ingestion of the data when retrieving the data from the ingested data collection based on a query that invokes the derived dimension; and
generate a report for the ingested data collection that reflects a changed reading of the ingested data collection according to the derived dimension in connection with remaining dimensions of the plurality of dimensions.
16 . The system as recited in claim 15 , wherein the one or more servers are further configured to cause the system to generate, based on the user-defined definition, a fallback expression aligned with the data organization schema of the ingested data collection.
17 . The system as recited in claim 16 , wherein modifying the dimension of the plurality of dimensions based on the user-defined definition comprises extracting the dimension and modifying the dimension according to the fallback expression.
18 . The system as recited in claim 17 , wherein the one or more servers are further configured to cause the system to store the derived dimension in a meta-field that is separate from the ingested data collection.
19 . The system as recited in claim 17 , wherein the one or more servers are further configured to cause the system to ingest data from the digital data analytics engine by:
receiving a copy of the data from a user interaction platform; and
populating the plurality of dimensions with the copied data based on one or more rules associated with each dimension according to the data organization schema, wherein the data organization schema comprises digital rules for ingesting raw data into one or more of the plurality of dimensions.
20 . The system as recited in claim 19 , wherein modifying the dimension of the plurality of dimensions based on the user-defined definition comprises merging, copying, adding, or deleting columns when extracting data from the ingested data collection without affecting original data in the digital data analytics engine.