To collect and analyze data well, start with the decision or question you need to answer, then choose evidence and collection methods that fit it. Decide how you will analyze the material before collecting it, check its quality and ethical implications, and interpret results within the limits of the sample and study design. No single method is best for every question.
Start with the question, not the method
Write down what you need to know and who will use the answer. A question such as “How many people completed the program?” calls for countable evidence. “How did participants experience the program?” calls for accounts, observations, or other contextual evidence. Some projects need both.
| # | Preview | Product | Price | |
|---|---|---|---|---|
| 1 |
|
The Art of Statistics: How to Learn from Data | $13.50 | Buy on Amazon |
| 2 |
|
Introduction to Statistics and Data Analysis | $53.98 | Buy on Amazon |
| 3 |
|
Storytelling with Data: A Data Visualization Guide for Business Professionals | $15.74 | Buy on Amazon |
| 4 |
|
Qualitative Data Analysis: A Methods Sourcebook | $109.99 | Buy on Amazon |
As an Amazon Associate I earn from qualifying purchases.
Turn the question into an evidence plan: specify what you will observe or measure, which people, records, or events provide it, and when you need it. The CDC’s guidance on gathering credible evidence recommends planning data sources, measures, indicators, and expectations for evidence that can support the evaluation.
Identify the kind of data you need
Quantitative data
Quantitative data are numerical values or measurements. They are suited to questions about counts, frequency, differences, change, and relationships. Common analyses include frequency distributions, charts, descriptive statistics, and comparisons or relationship analyses that match the study design.
#1 Best Overall
Qualitative data
Qualitative data include spoken or written accounts, observations, documents, and audio or video. They help explore meaning, experience, context, and how or why something happened. Researchers may code material to develop themes, or use approaches such as discourse, document, or multimodal analysis. The approach should fit the question rather than be chosen just because the material is textual.
Mixed methods
Mixed methods deliberately combines quantitative and qualitative collection and analysis within one study. It can pair a measure of what happened with evidence that helps explain how or why. The strands need a planned relationship: for example, one may help interpret the other, or each may answer a different part of the question. This design takes additional integration work, expertise, time, and resources; it is useful when both kinds of evidence are needed, not as a default.
Primary and secondary data describe source history
Primary data are collected for the current study. Secondary data were gathered earlier, often for a different purpose. This distinction is separate from whether data are quantitative or qualitative: either kind can come from primary or secondary sources. Existing program records, census or population information, prior surveys, and other datasets may provide context or spare people unnecessary new data collection. First check why the data were collected, what they measure, and whether their coverage and quality suit the current question. See the Australian Institute of Family Studies guide to survey data for evaluation and the CDC evidence guidance.
Free tools Windows power users keep installed
One-click scans. No signup required.
Rank #2
Choose a collection method that produces usable evidence
Methods are not interchangeable: each produces a different kind of evidence, with different strengths and constraints.
| Need | Likely method | Strength | Constraint |
|---|---|---|---|
| Comparable answers across many people or over time | Structured survey or questionnaire | Standardized responses make breadth and comparison possible. | Question wording and fixed response choices can limit context or introduce bias. |
| Detailed accounts of experience, motivation, or emotion | Individual or group interviews | Follow-up questions can produce depth and detail. | Collection and analysis take time; reduced anonymity may affect responses. |
| Behavior in its setting | Observation | Captures behavior and context rather than relying only on self-report. | Requires care with ethics, sampling, and observer consistency. |
| Information already held in documents or datasets | Record review or secondary-data analysis | Can reduce new collection and provide context. | The original purpose and data quality may not fit the present question. |
| Performance against a defined standard | Test or assessment | Produces a measure tied to specified criteria. | The test must validly represent the performance or ability of interest. |
| A defined physical or biological measure | Physiological assessment or biological sample | Can provide evidence about specified physical measures. | Requires appropriate procedures, expertise, consent, and safeguards. |
These are examples, not an exhaustive list. The U.S. Office of Research Integrity’s overview of information-collection methods discusses surveys, interviews, observation, tests, physiological measures, record review, and biological samples.
Compare plausible options against the question, required breadth or depth, comparability, staff expertise, time, cost, ethics, validity, reliability, and whether results need to apply beyond the observed cases. Choose the simplest design that can reliably answer the question.
Rank #3
- Wiley
- Language: english
- Book - storytelling with data: a data visualization guide for business professionals
Plan the analysis before collecting
For each item of evidence, decide how it will be recorded, represented, and used to answer the question. This prevents collecting material that cannot support the intended conclusion. The Open University’s research-methodology guidance emphasizes aligning methodology with the research question.
Recommended Free Tools
- For numerical data: decide how values will be checked, summarized, visualized, and compared. Begin by examining distributions and charts; then use comparisons or relationship analyses appropriate to the design.
- For qualitative material: choose an interpretive approach suited to the question, such as thematic coding, discourse analysis, document analysis, or multimodal analysis. Decide how material will be organized and how interpretations will be grounded in the evidence.
- For mixed methods: specify what each strand contributes and how findings will be brought together. Collecting both kinds of data without a plan for integration does not, by itself, make the study informative.
Protect data quality and participants
Trustworthy findings depend on how data are collected, not only on how they are analyzed. The U.S. Office of Research Integrity says: “No matter what kind of information is collected in a research study or how it is collected, it is extremely important to carry out the collection of the information with precision (i.e., reliability), accuracy (i.e., validity), and minimal error.”
- Validity: does the measure or method capture what you intend to study?
- Reliability: could the result be reproduced under the stated procedure?
- Consistency: if multiple people collect data, do they use the same definitions and recording rules?
- Ethics and burden: are collection, sensitivity, consent, and participant effort appropriate to the question and setting?
- Feasibility: do you have the time, skill, and resources to collect and analyze the material as planned?
Make procedures precise enough that measurements and observations are recorded consistently, and account for missing or weak evidence when interpreting results.
Rank #4
Interpret findings within the limits of the design
Report what the evidence supports, who or what was observed, the context, and limitations that affect the conclusion. A numerical summary alone does not establish causation or show that a sample represents a wider population. Qualitative depth can explain experiences and context, but does not by itself establish how common an experience is across a population. The method, sampling, and evidence determine what can reasonably be claimed.
Mixed-methods work can make the relationship between measures and lived experience clearer, but findings remain specific to the study. For example, an Office for Health Improvement and Disparities mixed-methods study example reports that Naughton and colleagues’ 2016 smoking-cessation app study found geolocation accurate in 97% of smoking reports, while participants under-reported smoking on at least 56% of days. Those figures describe that particular study; they are not general benchmarks for data collection or self-report.
When reporting, distinguish observations from interpretations, explain how evidence was gathered and analyzed, and identify where the data are insufficient to answer part of the question.
Quick Recap
Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.
