Research Reliability Mistakes: Where Studies Slip and How to Fix It

Published 9 May 2025 · Updated 9 Oct 2026
Most research reliability problems start with inconsistent procedures, vague measures or raters who were never briefed together. This guide follows a study from design to write-up, shows where each slip happens and works through one agreement calculation by hand, so you can fix the weak spots in your own method.
Get a mentor to check your method section
Where research reliability breaks down
Every score mixes what you want to measure with noise. Reliability falls as the noise grows. Noise comes from the people measured, the setting, the instrument and whoever records the result. The table shows where it creeps in.
| Stage | Typical slip | Warning sign |
|---|---|---|
| Design | Vague question or undefined terms | Two readers define the key term differently |
| Collection | Procedure changes between sessions | Early and late data look different for no clear reason |
| Coding | One person codes everything alone | You cannot say how often a second coder would agree |
| Entry | Typing and copy-paste slips | Impossible values, such as an age of 250 |
| Reporting | One result treated as settled | No mention of repeat checks or limits |
Reliability mistakes in research
Open the stage that matches where your project is now.
Before you collect: an instrument nobody tried
A questionnaire that confuses two people in a trial will confuse two hundred in the study. Ambiguous wording is a known source of error, and clear wording helps. Give your draft to a few people outside the project and ask them to explain each item back to you.
Before you collect: terms with no written definition
If 'engagement' or 'success' is left open, each collector decides alone. Write a one-line definition and one example for every key term, and keep them in a short codebook that all collectors use.
During collection: changing the procedure halfway
Reading the instructions differently on day three, or swapping the room or the order of tasks, adds differences that have nothing to do with your participants. If you must change something, log the date and the reason.
During collection: collectors who were never briefed together
Three people interviewing in three styles produce three kinds of data. Hold one briefing, practise on the same sample case and agree what to do when a participant goes off topic.
During coding: a single coder and no check
Without a second coder you cannot show that your categories hold up. Have someone code a sample independently, compare the results, then settle the rules before coding the rest.
During analysis: keeping steps only in your head
Menu-driven tools may not record the clicks behind a table or graph. Save the script, or write out every step, so you or a reader can rebuild each result.
After analysis: treating one run as proof
One study is one run. In a project that tried to repeat 100 psychology studies, far fewer repeats than originals gave significant results. Report your result as evidence, and say what a repeat would need.
Types of reliability
Name the type you checked. 'Reliable' on its own tells a marker little.
| Type | Question it answers | Mistake that breaks it | Quick check |
|---|---|---|---|
| Test-retest | Same result on a second occasion? | Gap too short or too long | Retest a small group |
| Inter-rater | Do raters agree? | Vague categories | Two coders, one sample |
| Parallel forms | Do two versions match? | Forms of unequal difficulty | Correlate the two forms |
| Internal consistency | Do items agree with each other? | Mixed-up or unclear items | Report alpha for each scale |
These four types follow the Conjointly knowledge base. Our validity and reliability guide works one questionnaire through all of them.
Inter rater reliability
Inter rater reliability tells you whether two people, given the same material and the same rules, reach the same decision. Percent agreement is easy but crude, because some agreement happens by luck. Cohen's kappa adjusts for that. The counts below are invented.
| 50 survey comments | Rater 2: positive | Rater 2: negative |
|---|---|---|
| Rater 1: positive | 22 | 4 |
| Rater 1: negative | 6 | 18 |
Worked example: Cohen's kappa by hand
Use the counts above. Follow three steps.
- Observed agreement: (22 + 18) / 50 = 0.80
- Chance agreement: (26/50 × 28/50) + (24/50 × 22/50) = 0.2912 + 0.2112 = 0.5024
- Kappa: (0.80 − 0.5024) / (1 − 0.5024) = 0.60
An 80% match falls to 0.60 once luck is removed. On the Landis and Koch bands that is moderate, but those bands are not universally accepted and kappa shifts with how common each code is. Report the kappa, the sample size and what you did about the disagreements.
How to improve reliability
Work through this list before the main round of data collection.
- Write the protocol so someone else could run it from the page
- Trial the instrument and rewrite any item that needs explaining
- Brief every collector together, on the same example
- Double-code a sample and meet to settle every disagreement
- Keep a near-miss log: note slips that were caught, as well as ones that were not
- Enter data twice, or check it against the originals, and run range checks
- Run analysis from a saved script or a written list of steps
Reliability in qualitative work
Interviews and observations cannot be repeated exactly, so many qualitative researchers write about trustworthiness instead. Show an audit trail of your decisions, ask a second person to code part of the data and say how your own background may shape your reading. Stating this in the method is more convincing than claiming the data are objective.
Reporting your reliability checks
- Name the type of reliability you tested
- Give the statistic and how many cases it came from
- Say who did the checking and how disagreements were settled
- Put what you could not check in your limitations
Mistakes in how students write about reliability
- Using 'reliable' and 'valid' as if they mean the same thing
- Quoting a published scale's reliability instead of testing it in your own sample
- Giving percent agreement alone when one code is far more common than the other
- Deleting items to lift a score without saying why
- Leaving sampling out of the discussion: see sampling in research
Need feedback on your method section?
Our mentors read your methodology draft, point out where reliability is thin and suggest what to add. You write the final version. Start with how it works, or read ordinal vs interval data before you choose a statistic.
Related guides
- Validity vs reliability in research
- How to write a research question
- Likert scale in research: best practice
- Ethical considerations in research
Sources
- Conjointly (William Trochim), Types of reliability
- Wikipedia, Cohen's kappa
- Wikipedia, Reliability (statistics))
- Association for Psychological Science, Report demonstrates need for improved reproducibility in psychological science
- Association for Psychological Science, Insights from high-risk fields can help minimize mistakes in the lab
- Editage, 6 common research errors and how to avoid them
Frequently asked questions
What are common reliability mistakes in research?
Unpiloted instruments, undefined terms, changing procedures mid-study, untrained collectors, single coders and typing errors. Each adds noise that has nothing to do with your participants.
How can you improve reliability in research?
Write a clear protocol, trial the instrument, brief all collectors together, double-code a sample and check data entry. Then report each check and its result.
What are the types of reliability?
Test-retest, inter-rater, parallel forms and internal consistency. Each answers a different question, so name the one you tested.
What is inter rater reliability?
It is how far two or more raters agree on the same material. Cohen's kappa is a common statistic for two raters because it corrects for chance agreement.
Is a reliable result always valid?
No. A measure can be consistent and still miss what you meant to measure. Our validity vs reliability guide explains the gap.
Do I need to report reliability in a qualitative study?
Not usually as a statistic. Show how you kept the study consistent: an audit trail, a second coder and a note on your own influence. Check your supervisor's guidance.