Writing the Methodology Section Reviewers Can't Fault

The reproducibility crisis in science — documented in the Nature survey of 2016, which found that more than 70 percent of researchers had tried and failed to reproduce another scientist's results — is partly a data problem, partly an analysis problem, and significantly a writing problem. Research that cannot be reproduced from its reported methodology cannot be evaluated, built upon, or corrected. The methodology section is the document that makes reproducibility possible, and it is systematically underwritten in the research articles that most need it to be thorough.

Reviewers at major journals read methodology sections with a specific orientation: they are looking for the decisions embedded in the design that could explain the reported results without the proposed mechanism. Not because they assume fraud, but because their job is to evaluate whether the conclusions follow from the reported evidence — and that evaluation requires understanding whether the design can actually support those conclusions. An author who writes the methodology section to describe what was done is providing information. An author who writes it to anticipate reviewer concerns is making an argument.

The Feasibility and Reproducibility Reading

Reviewers read methodology sections simultaneously for feasibility and reproducibility, and these readings pull in different directions. The feasibility reading asks: "Could this have produced the reported results?" — which requires enough detail to evaluate the design. The reproducibility reading asks: "Could another lab do this?" — which requires enough procedural specificity to execute without contacting the authors.

Most methodology sections satisfy neither reading completely. They are written at a level of abstraction that is too high for reproducibility — omitting the specific parameters, reagent sources, software versions, or analysis decisions that another researcher would need to replicate the work — while also being insufficiently detailed about the design rationale for the feasibility reading. The result is a methodology section that describes the approach at a level too abstract for replication and too vague for design evaluation.

The appropriate level of detail is dictated by the journal's standards and the field's conventions, but within those constraints, more specificity is almost always preferable to less. A 2019 meta-analysis published in PLOS Biology examining the reporting quality of biomedical research found that studies with more complete methods reporting were more likely to be replicable and less likely to generate peer review concerns about methodology — not because completeness prevented all methodological problems, but because it allowed problems to be identified and addressed rather than remaining hidden.

Justifying Design Choices

The most common substantive weakness in peer-reviewed methodology sections is the absence of rationale for design choices. Authors describe what they did without explaining why they did it that way — why they chose cross-sectional over longitudinal design, why they selected one validated instrument over another equally validated option, why they set the sample size where they did, or why they used a particular statistical approach rather than alternatives that reviewers might prefer.

Every design choice is a decision made against a background of alternatives. The reviewer reading the methodology section knows what those alternatives are, and they are mentally evaluating whether the chosen approach is appropriate or whether the use of an alternative would have produced different results. A methodology section that states only what was done leaves the reviewer to infer the rationale — and if the inferred rationale is wrong or incomplete, the reviewer raises a concern that requires a response-to-reviewers clarification. That clarification should have been in the original manuscript.

Design choice rationale does not require lengthy justification for standard choices. Saying "we used a cross-sectional design because the primary research question concerns prevalence rather than change over time" is one sentence that eliminates a predictable reviewer concern. Saying "we used [specific instrument] because it has been validated in this population and contains the subscale necessary to measure the construct of interest, which [alternative instrument] does not include" takes two sentences and preempts a common reviewer preference for the alternative instrument.

Sample Size, Power Analysis, and Statistical Approach

Sample size justification is the most frequently flagged methodology concern in peer review across most empirical disciplines. A 2018 survey of editors and reviewers at fifteen high-impact journals across psychology, medicine, and economics found that inadequate sample size justification was among the top three reasons for rejection after peer review in all fifteen journals. Despite this, sample size sections in the majority of submitted manuscripts either omit a formal power analysis or present one retrospectively — calculated from the data that was collected rather than from an a priori effect size estimate.

A power analysis written for a non-specialist reviewer — which is what most methodology sections require — states four things: the expected effect size and its source (prior literature, pilot data, or the smallest effect size of practical interest); the chosen alpha level and its justification (typically 0.05, but with acknowledgment when a different threshold is appropriate); the desired power (typically 0.80, but with acknowledgment when a different standard is appropriate); and the resulting required sample size. When the available sample differs from the required sample — which is common in studies recruiting from finite populations — the methodology section should acknowledge the discrepancy and state what power the available sample provides for the primary analysis.

Statistical approach sections are most commonly criticized for two problems: using a statistical model that makes assumptions the data may not satisfy, without checking or reporting those assumptions; and using a statistical approach that is appropriate for the data but unusual enough that non-specialist reviewers will not be confident evaluating it without a brief explanation. Both problems are addressed by the same writing principle: state the statistical approach, identify the key assumptions required for it to produce valid results, and state how those assumptions were checked or why they are likely to hold given the study design.

The Sections Most Targeted in Peer Review

Meta-analyses of peer review across disciplines consistently identify a small set of methodology subsections that generate the majority of reviewer concerns. Understanding which sections carry the most risk helps authors allocate their methodology writing effort appropriately.

Sampling and participant selection generate concerns when the described sample does not match the population to which the conclusions are generalized. The fix is explicit specification of the target population, the sampling method, the inclusion and exclusion criteria, and the limitations those criteria impose on the generalizability of the findings.

Controls and comparison conditions generate concerns when reviewers can identify confounders that the design does not address. The fix is explicit discussion of the key potential confounders in the design and how each is addressed — through randomization, matching, statistical control, or restriction.

Blinding procedures generate concerns in any study where the researchers assessing outcomes could be influenced by knowledge of the study condition. The fix is explicit specification of who was blinded, at what stage of the study, and how blinding was maintained and verified.

Measurement and instrumentation generate concerns when the instruments used are not the most established options for the constructs being measured, or when validity evidence for the instruments in the specific study population is not presented. The fix is citation of validation studies for the instruments in populations comparable to the study sample.

Preregistration and Open Science Considerations

An increasing number of major journals now require or strongly prefer preregistered studies — studies where the hypotheses, design, and primary analyses were specified before data collection and registered in a public repository (OSF, ClinicalTrials.gov, AsPredicted). As of 2024, over 350 journals require or offer registered report formats, according to the Center for Open Science's registry of open science policies.

For authors who have preregistered their study, the methodology section should include the preregistration link and explicitly note where the reported analyses deviate from the preregistered plan. Deviations from preregistration are common and often scientifically justified — unexpected data patterns, recruitment challenges, or analysis decisions that become clear only after data collection. What matters is that deviations are disclosed and justified, not that there are no deviations.

For authors who have not preregistered and are submitting to a journal that prefers preregistered studies, acknowledging the absence of preregistration and noting the measures that reduce risk of researcher degrees of freedom — such as analysis scripts shared with the data — is more transparent than silence and reduces the severity of the reviewer concern.