Skip to content
+44 7386783749

Research Reliability Mistakes: Where Studies Slip and How to Fix It

Researcher checking data consistency on a spreadsheet

Published 9 May 2025 · Updated 9 Oct 2026

Most research reliability problems start with inconsistent procedures, vague measures or raters who were never briefed together. This guide follows a study from design to write-up, shows where each slip happens and works through one agreement calculation by hand, so you can fix the weak spots in your own method.

Get a mentor to check your method section

Where research reliability breaks down

Every score mixes what you want to measure with noise. Reliability falls as the noise grows. Noise comes from the people measured, the setting, the instrument and whoever records the result. The table shows where it creeps in.

StageTypical slipWarning sign
DesignVague question or undefined termsTwo readers define the key term differently
CollectionProcedure changes between sessionsEarly and late data look different for no clear reason
CodingOne person codes everything aloneYou cannot say how often a second coder would agree
EntryTyping and copy-paste slipsImpossible values, such as an age of 250
ReportingOne result treated as settledNo mention of repeat checks or limits

Reliability mistakes in research

Open the stage that matches where your project is now.

Before you collect: an instrument nobody tried

A questionnaire that confuses two people in a trial will confuse two hundred in the study. Ambiguous wording is a known source of error, and clear wording helps. Give your draft to a few people outside the project and ask them to explain each item back to you.

Before you collect: terms with no written definition

If 'engagement' or 'success' is left open, each collector decides alone. Write a one-line definition and one example for every key term, and keep them in a short codebook that all collectors use.

During collection: changing the procedure halfway

Reading the instructions differently on day three, or swapping the room or the order of tasks, adds differences that have nothing to do with your participants. If you must change something, log the date and the reason.

During collection: collectors who were never briefed together

Three people interviewing in three styles produce three kinds of data. Hold one briefing, practise on the same sample case and agree what to do when a participant goes off topic.

During coding: a single coder and no check

Without a second coder you cannot show that your categories hold up. Have someone code a sample independently, compare the results, then settle the rules before coding the rest.

During analysis: keeping steps only in your head

Menu-driven tools may not record the clicks behind a table or graph. Save the script, or write out every step, so you or a reader can rebuild each result.

After analysis: treating one run as proof

One study is one run. In a project that tried to repeat 100 psychology studies, far fewer repeats than originals gave significant results. Report your result as evidence, and say what a repeat would need.

Types of reliability

Name the type you checked. 'Reliable' on its own tells a marker little.

TypeQuestion it answersMistake that breaks itQuick check
Test-retestSame result on a second occasion?Gap too short or too longRetest a small group
Inter-raterDo raters agree?Vague categoriesTwo coders, one sample
Parallel formsDo two versions match?Forms of unequal difficultyCorrelate the two forms
Internal consistencyDo items agree with each other?Mixed-up or unclear itemsReport alpha for each scale

These four types follow the Conjointly knowledge base. Our validity and reliability guide works one questionnaire through all of them.

Inter rater reliability

Inter rater reliability tells you whether two people, given the same material and the same rules, reach the same decision. Percent agreement is easy but crude, because some agreement happens by luck. Cohen's kappa adjusts for that. The counts below are invented.

50 survey commentsRater 2: positiveRater 2: negative
Rater 1: positive224
Rater 1: negative618

Worked example: Cohen's kappa by hand

Use the counts above. Follow three steps.

  • Observed agreement: (22 + 18) / 50 = 0.80
  • Chance agreement: (26/50 × 28/50) + (24/50 × 22/50) = 0.2912 + 0.2112 = 0.5024
  • Kappa: (0.80 − 0.5024) / (1 − 0.5024) = 0.60

An 80% match falls to 0.60 once luck is removed. On the Landis and Koch bands that is moderate, but those bands are not universally accepted and kappa shifts with how common each code is. Report the kappa, the sample size and what you did about the disagreements.

How to improve reliability

Work through this list before the main round of data collection.

  • Write the protocol so someone else could run it from the page
  • Trial the instrument and rewrite any item that needs explaining
  • Brief every collector together, on the same example
  • Double-code a sample and meet to settle every disagreement
  • Keep a near-miss log: note slips that were caught, as well as ones that were not
  • Enter data twice, or check it against the originals, and run range checks
  • Run analysis from a saved script or a written list of steps

Reliability in qualitative work

Interviews and observations cannot be repeated exactly, so many qualitative researchers write about trustworthiness instead. Show an audit trail of your decisions, ask a second person to code part of the data and say how your own background may shape your reading. Stating this in the method is more convincing than claiming the data are objective.

Reporting your reliability checks

  • Name the type of reliability you tested
  • Give the statistic and how many cases it came from
  • Say who did the checking and how disagreements were settled
  • Put what you could not check in your limitations

Mistakes in how students write about reliability

  • Using 'reliable' and 'valid' as if they mean the same thing
  • Quoting a published scale's reliability instead of testing it in your own sample
  • Giving percent agreement alone when one code is far more common than the other
  • Deleting items to lift a score without saying why
  • Leaving sampling out of the discussion: see sampling in research

Need feedback on your method section?

Our mentors read your methodology draft, point out where reliability is thin and suggest what to add. You write the final version. Start with how it works, or read ordinal vs interval data before you choose a statistic.

Related guides

Sources

Frequently asked questions

What are common reliability mistakes in research?

Unpiloted instruments, undefined terms, changing procedures mid-study, untrained collectors, single coders and typing errors. Each adds noise that has nothing to do with your participants.

How can you improve reliability in research?

Write a clear protocol, trial the instrument, brief all collectors together, double-code a sample and check data entry. Then report each check and its result.

What are the types of reliability?

Test-retest, inter-rater, parallel forms and internal consistency. Each answers a different question, so name the one you tested.

What is inter rater reliability?

It is how far two or more raters agree on the same material. Cohen's kappa is a common statistic for two raters because it corrects for chance agreement.

Is a reliable result always valid?

No. A measure can be consistent and still miss what you meant to measure. Our validity vs reliability guide explains the gap.

Do I need to report reliability in a qualitative study?

Not usually as a statistic. Show how you kept the study consistent: an audit trail, a second coder and a note on your own influence. Check your supervisor's guidance.