Trackable reasoning and analysis for crowdsourcing and evaluation

1. A computer-implemented method to structure an analytical workflow that improves reasoning based on a problem context and demonstrated abilities of each individual user, the method comprising:

displaying a reasoning problem to an analyst;

receiving input from the analyst to identify a reasoning problem type of the reasoning problem;

providing a recommended analytic approach for the reasoning problem type to the analyst;

assisting the analyst in analyzing and evaluating one or more information sources relevant to the reasoning problem;

guiding the analyst through a structured technique (ST) to support reasoning of the analyst in formulation of a solution to the reasoning problem; and

generating a report that includes the analyst's solution to the reasoning problem based on input from the analyst.

2. The method of claim 1 , further comprising one of the following prior to displaying the recommended analytic approach for the reasoning problem type to the analyst:

automatically generating the recommended analytic approach based on the reasoning problem type; or

receiving the recommended analytic approach from a plurality of analysts.

3. The method of claim 1 , further comprising influencing cognitive behaviors of the analyst to reduce a number of trivial decisions and/or logical pitfalls of the analyst and/or to mitigate common cognitive errors in rational problem analysis by the analyst by at least one of:

facilitating stigmergic communication among a plurality of analysts that includes the analyst;

nudging the analyst to at least one of: consider additional information, complete a second ST, or provide feedback that relates to the analyst's reasoning; and

providing the analyst with one or more annotation tools to annotate the one or more information sources in the analysis and evaluation of the one or more information sources.

4. The method of claim 3 , wherein facilitating stigmergic communication comprises:

aggregating prior evaluation input of the one or more information sources from a plurality of analysts; and

displaying to the analyst the prior evaluation input from the plurality of analysts in context with the one or more information sources such that the analyst's reasoning builds on the prior evaluation input.

5. The method of claim 3 , wherein facilitating stigmergic communication comprises:

aggregating prior ST input received from a plurality of analysts guided previously through the ST for the same reasoning problem; and

displaying to the analyst the prior ST input from the plurality of analysts such that the analyst's completion of the ST builds on the prior ST input.

6. The method of claim 1 , wherein assisting the analyst in analyzing and evaluating the one or more information sources comprises providing the analyst with one or more annotation tools to annotate the one or more information sources, the one or more annotation tools configured to:

highlight a portion of the one or more information sources;

tag an item described in the one or more information sources as an actor, event, fact, or assumption; tag an item described in the one or more information sources with a custom tag created by the analyst;

store an evaluation by the analyst of a relevance and credibility of the one or more information sources; and

store a comment of the analyst that connects an opinion of the analyst to a portion of the one or more information sources.

7. The method of claim 1 , wherein guiding the analyst through the ST includes at least one of:

guiding the analyst through generation of multiple hypotheses;

guiding the analyst through a structured debate in which the analyst is guided to generate first arguments in support of a hypothesis supported by the one or more information sources and to generate second arguments against the hypothesis;

guiding the analyst through consideration of pros and cons of a hypothesis supported by the one or more information sources;

guiding the analyst through an Analysis of Competing Hypotheses (ACH) ST in which the analyst is guided to generate and analyze multiple different hypotheses; and

guiding the analyst through a Key Assumptions Check (KAC) ST in which the analyst is guided to critically evaluate and examine assumptions underlying a judgment.

8. The method of claim 1 , further comprising displaying a report checklist to the analyst that identifies at least one of features of the analyst's reasoning to be made explicit for an external audience to understand the analyst's solution to the reasoning problem; justifications for the analyst's solution; alternative hypotheses considered by the analyst; and assumptions made by the analyst that could change the analyst's solution if the assumptions are incorrect.

9. The method of claim 1 , further comprising:

displaying a given one of the one or more information sources in a first portion of a user interface;

simultaneously displaying at least some annotations of the analyst in a second portion of the same user interface.

10. The method of claim 9 , wherein each annotation is associated with a corresponding portion of the one or more information sources, the method further comprising:

aggregating annotations of the analyst in the second portion of the user interface; and

in response to selection of a given annotation from the second portion of the user interface, automatically displaying the corresponding portion of the one or more information sources in the first portion of the user interface.

11. The method of claim 1 , wherein:

providing the recommended analytic approach comprises providing a recommended ST to the analyst; and

guiding the analyst through the ST comprises guiding the analyst through the recommended ST.

12. The method of claim 1 , wherein generating the report that includes the analyst's solution to the reasoning problem based on input from the analyst comprises providing a report template to the analyst, the report template including a plurality of sections each configured to receive input from the analyst.

13. The method of claim 12 , further comprising receiving input from the analyst in each of the plurality of sections of the report template, including receiving at least one of the following from the analyst:

a final judgment, conclusion, or recommendation of the analyst;

a justification of the final judgment, conclusion, or recommendation of the analyst;

a discussion of alternative hypotheses; and

a final probability assessment of the final judgment.

14. A computer-implemented method to structure an analytical workflow that improves reasoning based on a problem context and demonstrated abilities of each individual user, the method comprising:

displaying a reasoning problem to an analyst;

receiving input from the analyst to identify a reasoning problem type of the reasoning problem;

providing a recommended analytic approach for the reasoning problem type to the analyst; and

facilitating stigmergic communication among a plurality of analysts that includes the analyst while at least one of:

assisting the analyst in analyzing and evaluating one or more information sources relevant to the reasoning problem;

guiding the analyst through a structured technique (ST) to support reasoning of the analyst in formulation of a solution to the reasoning problem; and

generating a report that includes a solution to the reasoning problem based on input from one or more of the plurality of analysts.

15. The method of claim 14 , wherein facilitating the stigmergic communication comprises at least one of:

aggregating prior evaluation input of the one or more information sources from the plurality of analysts and displaying to the analyst the prior evaluation input from the plurality of analysts in context with the one or more information sources such that the analyst's reasoning builds on the prior evaluation input; and

aggregating prior ST input received from the plurality of analysts guided previously through the ST for the same reasoning problem and displaying to the analyst the prior ST input from the plurality of analysts such that the analyst's completion of the ST builds on the prior ST input.

16. The method of claim 14 , wherein facilitating the stigmergic communication comprises at least one of:

aggregating evaluation input of the one or more information sources from the plurality of analysts in real time and displaying the aggregated evaluation input to each of the plurality of analysts in real time;

aggregating ST input from the plurality of analysts in real time while the plurality of analysts are guided through the ST and displaying the aggregated ST input to each of the plurality of analysts in real time.

17. The method of claim 14 , wherein generating the report that includes the solution to the reasoning problem based on input from one or more of the plurality of analysts comprises one of:

generating the report that includes the analyst's solution to the reasoning problem based on input from the analyst; and

generating a single report for the plurality of analysts that includes a crowd-based solution to the reasoning problem based on input from the one or more of the plurality of analysts.

18. The method of claim 14 , further comprising nudging the analyst to at least one of: consider additional information, complete a second ST, or provide feedback that relates to the analyst's reasoning.

19. The method of claim 14 , wherein guiding the analyst through the ST includes at least one of:

guiding the analyst through generation of multiple hypotheses;

guiding the analyst through a structured debate in which the analyst is guided to generate first arguments in support of a hypothesis supported by the one or more information sources and to generate second arguments against the hypothesis;

guiding the analyst through consideration of pros and cons of a hypothesis supported by the one or more information sources;

guiding the analyst through an Analysis of Competing Hypotheses (ACH) ST in which the analyst is guided to generate and analyze multiple different hypotheses; and

guiding the analyst through a Key Assumptions Check (KAC) ST in which the analyst is guided to critically evaluate and examine assumptions underlying a judgment.

20. The method of claim 14 , wherein:

providing the recommended analytic approach comprises providing a recommended ST to the analyst; and

guiding the analyst through the ST comprises guiding the analyst through the recommended ST.