Research protocols / Returns measurement

Protocol: Reviewer Agreement on Return Reasons

A method for testing whether two reviewers apply a versioned return-reason codebook consistently while preserving customer language.

Ecommerce Virtual Assistant Research Desk · Published 2026-10-13

Research illustration for Protocol: Reviewer Agreement on Return Reasons

Protocol status: The protocol measures coding consistency; it does not claim that one reason caused returns or that the code captures every customer concern.

Preserve the original evidence

Retain the customer’s wording and available item context before coding. One return can include fit, damage, expectation, delivery, and remorse signals; the study must state whether it permits one primary code or multiple labels.

Remove unnecessary identifiers from the analysis copy. Give each record a study key linked to controlled source evidence.

Freeze a versioned codebook

Define each code with inclusion, exclusion, and difficult examples. Record the codebook version used by each reviewer so changes are visible.

Keep missing evidence, contradictory evidence, and no-applicable-code as valid outcomes. Forcing every record into a familiar category can inflate apparent consistency.

Sample before assignment

Define the eligible population, window, exclusions, and sample selection before either reviewer assigns a code. Include ordinary and ambiguous records rather than selecting only well-documented returns.

Preserve flow counts from eligible population through final analysis. Replacements and inaccessible records should be traceable.

Measure disagreement honestly

Have reviewers work independently on an overlap sample. Report raw agreement, a suitable chance-adjusted measure when justified, disagreement pairs, and adjudicated outcomes.

Use disagreements to clarify definitions, not to retroactively train the study sample until it agrees. Test revised rules on a fresh bounded sample.

State interpretive limits

Agreement describes whether reviewers use the codebook consistently. It does not prove that the assigned reason is the only cause of a return or that changing one workflow will reduce returns.

Report product mix, channel, evidence completeness, reviewer learning, and selection constraints so readers can judge whether the protocol transfers to another operation.

Sources and boundaries

These sources provide public context for the protocol. Their inclusion does not mean they endorse this site or that the proposed study has been conducted.

  1. US Census Bureau - Quarterly Retail E-Commerce Sales
  2. FTC - Privacy and Security Guidance
  3. NIST - Privacy Framework
Discuss the workflow