Virtual column construction method based on data lake and data query method

1 . A data query method based on a data lake, comprising:

acquiring a first data query request, wherein the first data query request is used for requesting to perform a data query for a target virtual column;

replacing a column name of the target virtual column in the first data query request with an expression corresponding to the target virtual column to obtain a second data query request; and

performing data query processing according to the second data query request, wherein the target virtual column is constructed by using following virtual column construction operations based on the data lake:

determining an expression to be used from at least one statement to be analyzed, after acquiring the at least one statement to be analyzed;

determining virtual column construction description information corresponding to the expression to be used, wherein the virtual column construction description information comprises the expression to be used and a column name corresponding to the expression to be used;

generating a virtual column construction request, according to the virtual column construction description information corresponding to the expression to be used; and

constructing a virtual column corresponding to the expression to be used, according to the virtual column construction request, and

wherein the at least one statement to be analyzed comprises a plurality of statements to be analyzed, and

determining the expression to be used from the at least one statement to be analyzed comprises: if an occurrence frequency of a first expression in a plurality of expressions to be analyzed corresponding to the plurality of statements to be analyzed is higher than a preset frequency threshold, determining the first expression as the expression to be used.

2 . The method according to claim 1 , wherein a number of the plurality of statements to be analyzed is N; and

determining the expression to be used from the at least one statement to be analyzed comprises:

determining an n-th expression to be analyzed from an n-th statement to be analyzed, wherein n is a positive integer, n≤N, and N is a positive integer;

performing statistical analysis processing on N expressions to be analyzed to obtain an expression statistical result; and

if the expression statistical result indicates that the occurrence frequency of the first expression in the N expressions to be analyzed is higher than the preset frequency threshold, determining the first expression as the expression to be used.

3 . The method according to claim 2 , wherein determining the n-th expression to be analyzed from the n-th statement to be analyzed comprises:

performing syntax conversion processing on the n-th statement to be analyzed to obtain a syntax conversion result; and

extracting the n-th expression to be analyzed from the syntax conversion result.

4 . The method according to claim 2 , wherein the method further comprises:

performing syntax tree construction processing on the n-th expression to be analyzed to obtain a syntax tree of the n-th expression to be analyzed; and

performing the statistical analysis processing on the N expressions to be analyzed to obtain the expression statistical comprises:

performing the statistical analysis processing on syntax trees of the N expressions to be analyzed to obtain the expression statistical result.

5 . The method according to claim 1 , wherein

a process of determining the column name corresponding to the expression to be used comprises:

if at least one statement to be referred exists in the at least one statement to be analyzed, determining the column name corresponding to the expression to be used from the at least one statement to be referred, wherein a preset semantic identity condition is met between an expression carried by the at least one statement to be referred and the expression to be used, and the at least one statement to be referred comprises a column name corresponding to the expression carried by the at least one statement to be referred.

6 . The method according to claim 5 , wherein the at least one statement to be referred comprises a plurality of statements to be referred, and determining the column name corresponding to the expression to be used from the at least one statement to be referred comprises:

performing statistical analysis processing on column names corresponding to expressions carried by the plurality of statements to be referred to obtain a column name statistical result; and

determining the column name corresponding to the expression to be used according to the column name statistical result.

7 . The method according to claim 6 , wherein the column names corresponding to the expressions carried by the plurality of statements to be referred comprise a target column name; and

determining the column name corresponding to the expression to be used according to the column name statistical result comprises:

if the column name statistical result indicates that an occurrence frequency of the target column name meets a preset frequency condition, determining the target column name as the column name corresponding to the expression to be used.

8 . The method according to claim 5 , wherein the method further comprises:

if the at least one statement to be referred does not exist in the at least one statement to be analyzed, determining a second expression from at least one expression to be matched which is preset, according to similarity representation data between the at least one expression to be matched and the expression to be used, wherein similarity representation data between the second expression and the expression to be used meets a preset similarity condition;

searching for a column name corresponding to the second expression from a pre-constructed mapping relation, wherein the mapping relation is used for recording a correspondence between respective expressions to be matched and column names corresponding thereto; and

determining the column name corresponding to the expression to be used according to the column name corresponding to the second expression.

9 . The method according to claim 8 , wherein a number of the expressions to be matched is M; and

a process of determining similarity representation data between an m-th expression to be matched and the expression to be used comprises:

determining a field name vector and a keyword vector of the expression to be used; and

determining the similarity representation data between the m-th expression to be matched and the expression to be used, according to a similarity between the field name vector of the expression to be used and a field name vector of the m-th expression to be matched and a similarity between the keyword vector of the expression to be used and a keyword vector of the m-th expression to be matched, wherein m is a positive integer, m≤M, and M is a positive integer.

10 . The method according to claim 9 , wherein at least one of following is implemented:

a process of determining the field name vector of the expression to be used comprises:

performing field name extraction processing on the expression to be used to obtain a field name extraction result; and performing vectorizing processing on the field name extraction result to obtain the field name vector of the expression to be used; or

a process of determining the keyword vector of the expression to be used comprises:

performing keyword extraction processing on the expression to be used to obtain a keyword extraction result; and performing vectorizing processing on the keyword extraction result to obtain the keyword vector of the expression to be used.

11 . The method according to claim 1 , wherein the virtual column construction description information further comprises a data type corresponding to the expression to be used; and

the data type is determined according to a data type corresponding to a column name carried by the expression to be used.

12 . An electronic device, comprising: a processor and a memory,

wherein the memory is configured to store instructions or a computer program; and

the processor is configured to execute the instructions or the computer program in the memory, such that the electronic device performs following data query operations based on a data lake:

acquiring a first data query request, wherein the first data query request is used for requesting to perform a data query for a target virtual column;

replacing a column name of the target virtual column in the first data query request with an expression corresponding to the target virtual column to obtain a second data query request; and

performing data query processing according to the second data query request, wherein the target virtual column is constructed by using following virtual column construction operations based on the data lake:

determining an expression to be used from at least one statement to be analyzed, after acquiring the at least one statement to be analyzed;

determining virtual column construction description information corresponding to the expression to be used, wherein the virtual column construction description information comprises the expression to be used and a column name corresponding to the expression to be used;

generating a virtual column construction request, according to the virtual column construction description information corresponding to the expression to be used; and

constructing a virtual column corresponding to the expression to be used, according to the virtual column construction request, and

wherein the at least one statement to be analyzed comprises a plurality of statements to be analyzed, and

determining the expression to be used from the at least one statement to be analyzed comprises: if an occurrence frequency of a first expression in a plurality of expressions to be analyzed corresponding to the plurality of statements to be analyzed is higher than a preset frequency threshold, determining the first expression as the expression to be used.

13 . The electronic device according to claim 12 , wherein a number of the plurality of statements to be analyzed is N; and

determining the expression to be used from the at least one statement to be analyzed comprises:

determining an n-th expression to be analyzed from an n-th statement to be analyzed, wherein n is a positive integer, n≤N, and N is a positive integer;

performing statistical analysis processing on N expressions to be analyzed to obtain an expression statistical result; and

if the expression statistical result indicates that the occurrence frequency of the first expression in the N expressions to be analyzed is higher than the preset frequency threshold, determining the first expression as the expression to be used.

14 . The electronic device according to claim 13 , wherein determining the n-th expression to be analyzed from the n-th statement to be analyzed comprises:

performing syntax conversion processing on the n-th statement to be analyzed to obtain a syntax conversion result; and

extracting the n-th expression to be analyzed from the syntax conversion result.

15 . The electronic device according to claim 13 , wherein the instructions are executed or the computer program is executed such that the electronic device further performs a following operation:

performing syntax tree construction processing on the n-th expression to be analyzed to obtain a syntax tree of the n-th expression to be analyzed; and

performing the statistical analysis processing on the N expressions to be analyzed to obtain the expression statistical result comprises:

performing the statistical analysis processing on syntax trees of the N expressions to be analyzed to obtain the expression statistical result.

16 . A non-transitory computer-readable storage medium having therein stored instructions or a computer program which, when run on a device, cause(s) the device to perform following data query operations based on a data lake:

acquiring a first data query request, wherein the first data query request is used for requesting to perform a data query for a target virtual column;

replacing a column name of the target virtual column in the first data query request with an expression corresponding to the target virtual column to obtain a second data query request; and

performing data query processing according to the second data query request, wherein the target virtual column is constructed by using following virtual column construction operations based on the data lake:

determining an expression to be used from at least one statement to be analyzed, after acquiring the at least one statement to be analyzed;

determining virtual column construction description information corresponding to the expression to be used, wherein the virtual column construction description information comprises the expression to be used and a column name corresponding to the expression to be used;

generating a virtual column construction request, according to the virtual column construction description information corresponding to the expression to be used; and

constructing a virtual column corresponding to the expression to be used, according to the virtual column construction request, and

wherein the at least one statement to be analyzed comprises a plurality of statements to be analyzed, and

determining the expression to be used from the at least one statement to be analyzed comprises: if an occurrence frequency of a first expression in a plurality of expressions to be analyzed corresponding to the plurality of statements to be analyzed is higher than a preset frequency threshold, determining the first expression as the expression to be used.

17 . The non-transitory computer-readable storage medium according to claim 16 , wherein a number of the plurality of statements to be analyzed is N; and

determining the expression to be used from the at least one statement to be analyzed comprises:

determining an n-th expression to be analyzed from an n-th statement to be analyzed, wherein n is a positive integer, n≤N, and N is a positive integer;

performing statistical analysis processing on N expressions to be analyzed to obtain an expression statistical result; and

if the expression statistical result indicates that the occurrence frequency of the first expression in the N expressions to be analyzed is higher than the preset frequency threshold, determining the first expression as the expression to be used.

18 . The non-transitory computer-readable storage medium according to claim 17 , wherein determining the n-th expression to be analyzed from the n-th statement to be analyzed comprises:

performing syntax conversion processing on the n-th statement to be analyzed to obtain a syntax conversion result; and

extracting the n-th expression to be analyzed from the syntax conversion result.

19 . The non-transitory computer-readable storage medium according to claim 17 , wherein the instructions or the computer program which, when run on the device, cause(s) the device to further perform a following operation:

performing syntax tree construction processing on the n-th expression to be analyzed to obtain a syntax tree of the n-th expression to be analyzed; and

performing the statistical analysis processing on the N expressions to be analyzed to obtain the expression statistical result comprises:

performing the statistical analysis processing on syntax trees of the N expressions to be analyzed to obtain the expression statistical result.

20 . The non-transitory computer-readable storage medium according to claim 16 , wherein a process of determining the column name corresponding to the expression to be used comprises:

if at least one statement to be referred exists in the at least one statement to be analyzed, determining the column name corresponding to the expression to be used from the at least one statement to be referred, wherein a preset semantic identity condition is met between an expression carried by the statement to be referred and the expression to be used, and the statement to be referred comprises a column name corresponding to the expression carried by the statement to be referred.