Measuring the logging quality of a computer program
Techniques are described for measuring or quantifying the logging behavior in the source code of a computer program. In particular, the techniques select a method identified as exhibiting the ideal logging behavior in a computer program and then compute the overall logging quality score for the entire computer program based on the deviation in logging behaviors between the selected method and all other methods in the source code of the project. This overall logging quality score can be compared to various benchmarks of existing projects with high logging quality. If the software logging quality is found to be low, various steps can be taken by the developers to improve the logging before the software release.
1. A method for quantifying a quality of software logging, said method comprising:
under control of one or more computer systems configured with executable instructions,
determining a logging behavior value for each method of a plurality of methods in source code of a software program, the logging behavior value quantifying a logging behavior of said each method;
selecting a first method from the plurality of methods that is identified as exhibiting an ideal logging behavior;
determining, for each method other than the first method, a root mean square deviation (RMSD) between the logging behavior value of said each method and the logging behavior value of the first method; and
computing an overall log quality score for the software program based at least in part on the RMSD of said each method in the plurality of methods.
2. The method of claim 1 , wherein determining the logging behavior value further comprises:
computing the logging behavior value of a method based on one or more of:
a number of non-error log statements per executable lines of code in the method;
a number of error or fatal level log statements per error condition; or
a verbosity level ratio of debug/trace level log statements to info/warn level log statements.
3. The method of claim 2 , wherein the logging behavior value is computed according to:
Log B ( m i )= a×n li +β×n ei +γ×r vi
wherein:
n li represents the number of non-error log statements per executable lines of code in the method;
n ei represents the number of error or fatal level log statements per error condition in the method;
r vi represents the verbosity level ratio of the method;
α, β, and γ each represent a user-configurable weight;
m i represents the method for which the logging behavior is being evaluated; and
Log B(m i ) represents the logging behavior value of the method.
4. The method of claim 1 , wherein the overall log quality score for the software program is computed according to:
Log
Q
=
∑
t
=
1
n
(
L
o
^
g
B
-
Log
B
(
m
i
)
)
2
n
wherein:
LôgB =Log B (arg max m i LogRank( m i ))
wherein:
n represents the total number of methods in the software program;
arg max m i LogRank(m i ) represents the selected first method identified as exhibiting the ideal logging behavior; and
Log Q represents the overall log quality score for the software program.
5. The method of claim 1 , wherein selecting the first method further comprises:
computing a log rank value of said each method in the plurality of methods based on historical log modification data associated with the method and based on bug priority data; and
ranking the plurality of methods according to the log rank value of said each method, wherein the first method contains the highest log rank value.
6. The method of claim 1 , further comprising:
comparing the overall log quality score for the software program against a benchmark to determine whether the overall log quality score is satisfactory.
7. The method of claim 1 , further comprising:
storing the plurality of methods in a static call graph, wherein each node of the static call graph represents an individual method of the plurality of methods and wherein each edge in the static call graph represents a function call between methods.
8. A computing system, comprising:
at least one processor; and
memory including instructions that, when executed by the at least one processor, cause the computing system to:
determine a logging behavior value for each method of a plurality of methods in source code of a software program, the logging behavior value quantifying a logging behavior of said each method;
select a first method from the plurality of methods that is identified as exhibiting an ideal logging behavior;
determine, for each method other than the first method, a root mean square deviation (RMSD) between the logging behavior value of said each method and the logging behavior value of the first method; and
compute an overall log quality score for the software program based at least in part on the RMSD of said each method in the plurality of methods.
9. The computing system of claim 8 , wherein determining the logging behavior value further comprises:
computing the logging behavior value of a method based on one or more of:
a number of non-error log statements per executable lines of code in the method;
a number of error or fatal level log statements per error condition; or
a verbosity level ratio of debug/trace level log statements to info/warn level log statements.
10. The computing system of claim 9 , wherein the logging behavior value is computed according to:
Log B ( m i )= a×n li +β×n ei +r×r vi
wherein:
n li represents the number of non-error log statements per executable lines of code in the method;
n ei represents the number of error or fatal level log statements per error condition in the method;
r vi represents the verbosity level ratio of the method;
α, β, and γ each represent a user-configurable weight;
m i represents the method for which the logging behavior is being evaluated; and
Log B(m i ) represents the logging behavior value of the method.
11. The computing system of claim 8 , wherein the overall log quality score for the software program is computed according to:
Log
Q
=
∑
t
=
1
n
(
L
o
^
g
B
-
Log
B
(
m
i
)
)
2
n
wherein:
LôgB =Log B (arg max m i LogRank( m i ))
wherein:
n represents the total number of methods in the software program;
arg max m i LogRank(m i ) represents the selected first method identified as exhibiting the ideal logging behavior; and
Log Q represents the overall log quality score for the software program.
12. The computing system of claim 8 , wherein selecting the first method further comprises:
computing a log rank value of said each method in the plurality of methods based on historical log modification data associated with the method and based on bug priority data; and
ranking the plurality of methods according to the log rank value of said each method, wherein the first method contains the highest log rank value.
13. The computing system of claim 8 , wherein the memory further comprises instructions that, when executed by the at least one processor, cause the computing system to:
compare the overall log quality score for the software program against a benchmark to determine whether the overall log quality score is satisfactory.
14. The computing system of claim 8 , wherein the memory further comprises instructions that, when executed by the at least one processor, cause the computing system to:
store the plurality of methods in a static call graph, wherein each node of the static call graph represents an individual method of the plurality of methods and wherein each edge in the static call graph represents a function call between methods.
15. A non-transitory computer readable storage medium containing one or more sequences of instructions, the instructions when executed by one or more processors causing the one or more processors to execute a set of operations comprising:
determining a logging behavior value for each method of a plurality of methods in source code of a software program, the logging behavior value quantifying a logging behavior of said each method;
selecting a first method from the plurality of methods that is identified as exhibiting an ideal logging behavior;
determining, for each method other than the first method, a root mean square deviation (RMSD) between the logging behavior value of said each method and the logging behavior value of the first method; and
computing an overall log quality score for the software program based at least in part on the RMSD of said each method in the plurality of methods.
16. The non-transitory computer readable storage medium of claim 15 , wherein determining the logging behavior value further comprises:
computing the logging behavior value of a method based on one or more of:
a number of non-error log statements per executable lines of code in the method;
a number of error or fatal level log statements per error condition; or
a verbosity level ratio of debug/trace level log statements to info/warn level log statements.
17. The non-transitory computer readable storage medium of claim 16 , wherein the logging behavior value is computed according to:
Log B ( m i )= a×n li +β×n ei +γ×r vi
wherein:
n li represents the number of non-error log statements per executable lines of code in the method;
n ei represents the number of error or fatal level log statements per error condition in the method;
r vi represents the verbosity level ratio of the method;
α, β, and γ each represent a user-configurable weight;
m i represents the method for which the logging behavior is being evaluated; and
Log B(m i ) represents the logging behavior value of the method.
18. The non-transitory computer readable storage medium of claim 15 , wherein the overall log quality score for the software program is computed according to:
Log
Q
=
∑
t
=
1
n
(
L
o
^
g
B
-
Log
B
(
m
i
)
)
2
n
wherein:
LôgB =Log B (arg max m i LogRank( m i ))
wherein:
n represents the total number of methods in the software program;
arg max m i LogRank(m i ) represents the selected first method identified as exhibiting the ideal logging behavior; and
Log Q represents the overall log quality score for the software program.
19. The non-transitory computer readable storage medium of claim 15 , wherein selecting the first method further comprises:
computing a log rank value of said each method in the plurality of methods based on historical log modification data associated with the method and based on bug priority data; and
ranking the plurality of methods according to the log rank value of said each method, wherein the first method contains the highest log rank value.
20. The non-transitory computer readable storage medium of claim 15 , further comprising instructions when executed by one or more processors, cause the one or more processors to execute the operation of:
comparing the overall log quality score for the software program against a benchmark to determine whether the overall log quality score is satisfactory.