Measuring the free energy in an evaluation
Provided are processes of balancing between exploration and optimization with knowledge discovery processes applied to unstructured data with tight interrogation budgets. A process may determine an alignment score of each entity participating in an evaluation of a feature in a knowledge discovery process based on feedback received from the respective entities for the feature. Feedback of an entity that is mapped in the PGN may be processed to determine an alignment score of the entity for the feature, e.g., based on how the entity scored a feature. A plurality of different distributions indicative of alignment scores may be processed for display to visually indicate to a user the alignment of participating entities in their evaluations of the features.
1. A computer-implemented method comprising:
obtaining, with a computer system, a probabilistic graphical model based on a time series data set of feedback received from respective entities, among a plurality of entities, for a plurality of features corresponding to an evaluation;
determining, with the computer system, for each entity, a score indicative of feedback received from the entity for each feature to obtain a set of scores for respective features of the plurality of features, each score being based on the probabilistic graphical model;
determining, with the computer system, for each feature, a respective distribution based on the set of scores obtained for the respective feature to form a set of distributions; and
generating, with the computer system, for display, a plot of the set of distributions, wherein statistical dispersion of the distributions corresponds to free energy of assessments of the plurality of entities of the evaluation.
2. The method of claim 1 , wherein:
the generated plot of the set of distributions is configured to cause the display of visual elements indicative of, for each of the distributions in the set of distributions, an amplitude of the respective distributions, a measure of variance of the respective distributions, and a measure of central tendency of the respective distributions.
3. The method of claim 1 , wherein a measure of central tendency of the distributions in the set of distributions corresponds to relevance of respective features attributed by the plurality of entities to the evaluation.
4. The method of claim 1 , wherein the generated plot is configured to cause the display of multiple, overlapping distributions on a two-dimensional cartesian plane with each of the distributions being represented by respective areas of varying color with at least some transparency, such that colors of overlapping areas are combined.
5. The method of claim 1 , comprising:
sending the generated plot of the distribution in the form of browser-executable instructions and data to a client computing device upon which the generated plot is to be displayed, wherein the browser-executable instructions comprise event handlers mapped to graphical elements in a legend of themes for which group alignment is to be displayed in the generated plot, the event handlers causing a visual weight of corresponding data in the plot to change responsive to selection of corresponding graphical elements.
6. The method of claim 1 , comprising:
obtaining the probabilistic graphical model by executing an iterative process by which the plurality of entities engage in structured deliberation about the evaluation.
7. The method of claim 1 , wherein the probabilistic graphical model models reasons of the entities for results of the evaluation.
8. The method of claim 1 , wherein the evaluation implements a linear decision model of a decision to be made collectively by the plurality of entities.
9. The method of claim 1 , comprising:
obtaining the feedback with a structured deliberation process integrated with team collaboration software; and
causing the team collaboration software to display the plot of the set of distributions.
10. The method of claim 1 , wherein the probabilistic graphical model comprises a Bayesian belief network.
11. The method of claim 1 , wherein the probabilistic graphical model comprises a probabilistic graphical network.
12. The method of claim 11 , wherein nodes of the probabilistic graphical network correspond to responses in the feedback from the plurality of entities and edges correspond to associations between the responses.
13. The method of claim 11 , wherein nodes of the probabilistic graphical network correspond to themes in the feedback from the plurality of entities and edges correspond to associations between the themes.
14. The method of claim 1 , comprising:
obtaining the feedback with steps for scaling A/B testing.
15. The method of claim 1 , comprising:
obtaining the feedback with steps for sampling a semantic space.
16. The method of claim 1 , comprising:
steps for determining measurements based on distributions determined based on the probabilistic graphical model.
17. The method of claim 1 , wherein obtaining the probabilistic graphical model comprises steps for determining the probabilistic graphical model.
18. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a computer system, effectuate operations comprising:
obtaining, with a computer system, a probabilistic graphical model based on a time series data set of feedback received from respective entities, among a plurality of entities, for a plurality of features corresponding to an evaluation;
determining, with the computer system, for each entity, a score indicative of feedback received from the entity for each feature to obtain a set of scores for respective features of the plurality of features, each score being based on the probabilistic graphical model;
determining, with the computer system, for each feature, a respective distribution based on the set of scores obtained for the respective feature to form a set of distributions; and
generating, with the computer system, for display, a plot of the set of distributions, wherein statistical dispersion of the distributions corresponds to free energy of assessments of the plurality of entities of the evaluation.
19. The medium of claim 18 , wherein:
the generated plot of the set of distributions is configured to cause the display of visual elements indicative of, for each of the distributions in the set of distributions, an amplitude of the respective distributions, a measure of variance of the respective distributions, and the measure of central tendency of the respective distributions.
20. The medium of claim 18 , wherein a measure of central tendency of the distributions in the set of distributions corresponds to relevance of respective features attributed by the plurality of entities to the evaluation.
21. The medium of claim 18 , wherein the generated plot is configured to cause the display of multiple, overlapping distributions on a two-dimensional cartesian plane with each of the distributions being represented by respective areas of varying color with at least some transparency, such that colors of overlapping areas are combined.
22. The medium of claim 18 , comprising:
sending the generated plot of the distribution in the form of browser-executable instructions and data to a client computing device upon which the generated plot is to be displayed, wherein the browser-executable instructions comprise event handlers mapped to graphical elements in a legend of themes for which group alignment is to be displayed in the generated plot, the event handlers causing a visual weight of corresponding data in the plot to change responsive to selection of corresponding graphical elements.
23. The medium of claim 18 , comprising:
obtaining the probabilistic graphical model by executing an iterative process by which the plurality of entities engage in structured deliberation about the evaluation.
24. The medium of claim 18 , wherein the probabilistic graphical model models reasons of the entities for results of the evaluation.
25. The medium of claim 18 , wherein the evaluation implements a linear decision model of a decision to be made collectively by the plurality of entities.
26. The medium of claim 18 , comprising:
obtaining the feedback with a structured deliberation process integrated with team collaboration software; and
causing the team collaboration software to display the plot of the set of distributions.
27. The medium of claim 18 , wherein the probabilistic graphical model comprises a Bayesian belief network.
28. The medium of claim 18 , wherein the probabilistic graphical model comprises a probabilistic graphical network.
29. The medium of claim 27 , wherein nodes of the probabilistic graphical network correspond to responses in the feedback from the plurality of entities and edges correspond to associations between the responses.
30. The medium of claim 27 , wherein nodes of the probabilistic graphical network correspond to themes in the feedback from the plurality of entities and edges correspond to associations between the themes.
31. The medium of claim 18 , comprising:
obtaining the feedback with steps for scaling AB testing.
32. The medium of claim 18 , comprising:
obtaining the feedback with steps for sampling a semantic space.
33. The medium of claim 18 , comprising:
steps for determining measurements based on distributions determined based on the probabilistic graphical model.
34. The medium of claim 18 , wherein obtaining the probabilistic graphical model comprises steps for determining the probabilistic graphical model.
35. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a computer system, effectuate operations comprising:
obtaining, with a computer system, a probabilistic graphical model based on a time series data set of feedback received from respective entities, among a plurality of entities, for a plurality of features corresponding to an evaluation;
determining, with the computer system, for each entity, a score indicative of feedback received from the entity for each feature to obtain a set of scores for respective features of the plurality of features, each score being based on the probabilistic graphical model;
determining, with the computer system, for each feature, a respective distribution based on the set of scores obtained for the respective feature to form a set of distributions; and
generating, with the computer system, for display, a plot of the set of distributions, wherein a measure of central tendency of the distributions in the set of distributions corresponds to relevance of respective features attributed by the plurality of entities to the evaluation.
36. The medium of claim 35 , wherein:
the generated plot of the set of distributions is configured to cause the display of visual elements indicative of, for each of the distributions in the set of distributions, an amplitude of the respective distributions, a measure of variance of the respective distributions, and the measure of central tendency of the respective distributions.
37. The medium of claim 35 , wherein statistical dispersion of the distributions in the set of distributions corresponds to consensus in the feedback received from respective entities.
38. The medium of claim 35 , wherein the generated plot is configured to cause the display of multiple, overlapping distributions on a two-dimensional cartesian plane with each of the distributions being represented by respective areas of varying color with at least some transparency, such that colors of overlapping areas are combined.
39. The medium of claim 35 , comprising:
sending the generated plot of the distribution in the form of browser-executable instructions and data to a client computing device upon which the generated plot is to be displayed, wherein the browser-executable instructions comprise event handlers mapped to graphical elements in a legend of themes for which group alignment is to be displayed in the generated plot, the event handlers causing a visual weight of corresponding data in the plot to change responsive to selection of corresponding graphical elements.
40. The medium of claim 35 , comprising:
obtaining the probabilistic graphical model by executing an iterative process by which the plurality of entities engage in structured deliberation about the evaluation.
41. The medium of claim 35 , wherein the probabilistic graphical model models reasons of the entities for results of the evaluation.
42. The medium of claim 35 , wherein the evaluation implements a linear decision model of a decision to be made collectively by the plurality of entities.
43. The medium of claim 35 , wherein the probabilistic graphical model comprises a Bayesian belief network.
44. The medium of claim 35 , wherein the probabilistic graphical model comprises a probabilistic graphical network.
45. The medium of claim 44 , wherein nodes of the probabilistic graphical network correspond to responses in the feedback from the plurality of entities and edges correspond to associations between the responses.
46. The medium of claim 44 , wherein nodes of the probabilistic graphical network correspond to themes in the feedback from the plurality of entities and edges correspond to associations between the themes.
47. The medium of claim 35 , comprising:
obtaining the feedback with steps for scaling AB testing.
48. The medium of claim 35 , comprising:
obtaining the feedback with steps for sampling a semantic space.
49. The medium of claim 35 , comprising:
steps for determining measurements based on distributions determined based on the probabilistic graphical model.
50. The medium of claim 35 , wherein obtaining the probabilistic graphical model comprises steps for determining the probabilistic graphical model.
51. A tangible, non-transitory, machine-readable medium storing instructions that, when executed by a computer system, effectuate operations comprising:
obtaining, with a computer system, a probabilistic graphical model based on a time series data set of feedback received from respective entities, among a plurality of entities, for a plurality of features corresponding to an evaluation, the feedback being obtained with a structured deliberation process integrated with team collaboration software;
determining, with the computer system, for each entity, a score indicative of feedback received from the entity for each feature to obtain a set of scores for respective features of the plurality of features, each score being based on the probabilistic graphical model;
determining, with the computer system, for each feature, a respective distribution based on the set of scores obtained for the respective feature to form a set of distributions; and
generating, with the computer system, for display, a plot of the set of distributions and causing the team collaboration software to display the plot.
52. The medium of claim 51 , wherein:
the generated plot of the set of distributions is configured to cause the display of visual elements indicative of, for each of the distributions in the set of distributions, an amplitude of the respective distributions, a measure of variance of the respective distributions, and a measure of central tendency of the respective distributions.
53. The medium of claim 51 , wherein the generated plot is configured to cause the display of multiple, overlapping distributions on a two-dimensional cartesian plane with each of the distributions being represented by respective areas of varying color with at least some transparency, such that colors of overlapping areas are combined.
54. The medium of claim 51 , comprising:
obtaining the probabilistic graphical model by executing an iterative process by which the plurality of entities engage in structured deliberation about the evaluation.
55. The medium of claim 51 , wherein the probabilistic graphical model models reasons of the entities for results of the evaluation.
56. The medium of claim 51 , wherein the evaluation implements a linear decision model of a decision to be made collectively by the plurality of entities.
57. The medium of claim 51 , wherein the probabilistic graphical model comprises a Bayesian belief network.
58. The medium of claim 51 , wherein the probabilistic graphical model comprises a probabilistic graphical network.
59. The medium of claim 58 , wherein nodes of the probabilistic graphical network correspond to responses in the feedback from the plurality of entities and edges correspond to associations between the responses.
60. The medium of claim 58 , wherein nodes of the probabilistic graphical network correspond to themes in the feedback from the plurality of entities and edges correspond to associations between the themes.
61. The medium of claim 51 , comprising:
obtaining the feedback with steps for scaling A/B testing.
62. The medium of claim 51 , comprising:
obtaining the feedback with steps for sampling a semantic space.
63. The medium of claim 51 , comprising:
steps for determining measurements based on distributions determined based on the probabilistic graphical model.
64. The medium of claim 51 , wherein obtaining the probabilistic graphical model comprises steps for determining the probabilistic graphical model.