Every lab professional has seen it. A new analyzer or digital morphology system gets demonstrated on a set of beautiful slides: crisp stains, perfectly spread cells, textbook blasts. Then it goes live, and on the first busy morning it meets a clotted EDTA tube, a lipemic sample, a smear with platelet clumps, and a slide stained with yesterday’s buffer. The performance you saw in the demo quietly falls apart.
AI models for clinical laboratories and pathology are vulnerable to exactly the same problem, and it usually starts long before anyone writes a line of code. It starts with how the training samples were chosen. Having worked around medical datasets for years, I’d argue that the single most underrated decision in building these tools is whether the data was collected consecutively or hand-picked, and that’s why teams doing serious clinical-grade medical data collection treat sampling design as seriously as the algorithm itself.
This article explains what consecutive sampling is, why it matters so much for lab and pathology AI, when enrichment is still appropriate, and what questions to ask before you trust an AI tool’s published accuracy.
What Is Consecutive Sampling?
Consecutive sampling means including every eligible patient or specimen that comes through a defined workflow during a set period, in the order they arrive, until you reach your target number. If your inclusion criteria are “all peripheral blood smears sent for manual differential between March 1 and June 30,” then a consecutive dataset contains all of them: the clean ones, the ugly ones, the normal ones, and the rare diagnoses, in the proportions your lab actually sees.
It’s useful to compare it with the other common approaches:
- Convenience sampling: using whatever samples are easiest to get, such as slides already scanned, cases from one enthusiastic pathologist, or an existing teaching collection.
- Case-control (or two-gate) sampling: collecting confirmed positive cases and confirmed negative cases separately, then combining them.
- Random sampling: randomly selecting from all eligible cases. Statistically sound, but often harder to execute in a busy lab than a consecutive series.
- Enriched sampling: deliberately adding extra cases of a rare condition so the model sees enough examples to learn from.
Only consecutive and random sampling give you a dataset that represents the real population of samples your lab handles. The others can be useful, but they answer a different question.
Why Does Sampling Method Matter for Lab and Pathology AI?
A machine learning model doesn’t learn “what leukemia looks like.” It learns what leukemia looks like in the specific samples it was shown. If those samples were selected because they were clear, classic, and well prepared, the model learns a cleaner world than the one your bench actually operates in.
This is a well-known problem in diagnostic research called spectrum bias. When studies compare obvious cases against obviously healthy controls, the test looks better than it really is, because the hard, borderline cases in the middle are missing. A classic 1999 analysis in JAMA by Lijmer and colleagues found that diagnostic accuracy studies using a case-control design reported markedly inflated accuracy, roughly a threefold overestimate in the diagnostic odds ratio, compared with studies of a clinical patient series.
This is why the methodological community treats sampling design as a core quality question. The STARD 2015 reporting guidelines for diagnostic accuracy studies, whose authors include laboratory medicine leaders, require researchers to state whether participants formed a consecutive, random, or convenience series. The QUADAS-2 risk-of-bias tool used in systematic reviews opens with two questions: was a consecutive or random sample enrolled, and was a case-control design avoided? AI models are diagnostic tests, and these same standards apply to them.
What Does Real-World Lab Data Actually Look Like?
Anyone who has worked a night shift knows the answer: messy. A consecutive dataset captures the variation that curated datasets quietly remove, including:
- Pre-analytical problems: hemolysis, lipemia, icterus, clotted samples, delayed processing, and underfilled tubes.
- Preparation artifacts: thick or thin smears, uneven staining, precipitate, smudge cells, folded or torn tissue sections, air bubbles, and pen marks on slides.
- Equipment variation: different analyzers, slide scanners, microscopes, reagent lots, and staining batches.
- Biological variation: pediatric and geriatric samples, patients on chemotherapy, post-transfusion samples, and mixed or overlapping conditions.
- Realistic prevalence: mostly normal or mildly abnormal findings, with rare critical results scattered throughout.
Large computational pathology studies have started embracing this reality. A landmark 2019 study in Nature Medicine by Campanella and colleagues trained deep learning models on 44,732 whole slide images from 15,187 patients without any data curation, deliberately keeping the technical variability of a real pathology archive. The resulting system could, in principle, let pathologists exclude 65–75% of slides from review while retaining 100% sensitivity in the cancer types tested. Real-world data made the model harder to train, but far more relevant to clinical practice.
How Does Prevalence Change an AI Model’s Real Performance?
This is where many published results become misleading, and lab professionals are well placed to spot it because you already think in terms of predictive values.
Imagine an AI tool that flags abnormal cells with 95% sensitivity and 95% specificity. It was tested on a dataset that was 50% positive, so its positive predictive value (PPV) looked excellent: about 95%. Now deploy it in a routine lab where only 2% of samples are truly positive. Out of 1,000 samples, it correctly flags 19 of the 20 positives but also falsely flags about 49 of the 980 negatives. The PPV drops to roughly 28%. Fewer than one in three flags is real.
Nothing about the model changed. Only the population did. A consecutive test set shows you the PPV you will actually live with, including the review workload the false positives will create for your staff.
Is Enriched or Curated Data Ever Appropriate?
Yes, and it’s important to be practical about this. If a condition appears in one of every 5,000 samples, a purely consecutive dataset might contain only a handful of examples, which is not enough for a model to learn from. Enriching the training data with additional rare cases is a legitimate and often necessary step.
The key distinction is between training and evaluation:
- Training data can be thoughtfully enriched, as long as it still contains plenty of consecutively collected routine samples, so the model learns what normal variation and artifacts look like.
- Validation and test data should be consecutive or random whenever possible, ideally from a different time period, and preferably from a different site. This is the dataset that tells you how the model will actually perform.
- Any enrichment should be documented clearly so that readers and regulators can judge how it affects the reported results.
In my experience, the best datasets use a consecutive backbone with targeted, well-documented enrichment, not an enriched collection dressed up as real-world data.
How Do You Build a Consecutive Dataset in a Clinical Lab or Pathology Department?
Done properly, consecutive sampling is more about discipline than technology. A typical workflow looks like this:
- Define inclusion criteria up front. Specify specimen type, test or stain, patient population, and time window before anyone looks at a single case.
- Pull cases from the laboratory information system, not from memory. An LIS query for every eligible accession number in the window prevents unconscious cherry-picking.
- Log every exclusion with a reason. Some exclusions are justified, such as missing reference results or unrecoverable slides. All of them should be visible.
- Cover enough time. A longer window captures reagent lot changes, staff rotations, equipment maintenance, and seasonal disease patterns.
- Split data by patient and by time. Keep all samples from the same patient in the same dataset split, and test on a later time period to mimic real deployment.
- De-identify before data leaves the lab. Remove patient identifiers from images, slide labels, and metadata in line with HIPAA, GDPR, or local regulations.
What Should Lab Professionals Ask Before Trusting an AI Tool?
If your lab is evaluating an AI-assisted hematology, microbiology, urinalysis, or digital pathology product, these questions quickly reveal how much the published accuracy is worth:
- Was the validation dataset consecutive, random, or curated?
- What was the prevalence of positives in the test set, and how does it compare with our lab’s case mix?
- Were samples with artifacts or poor quality excluded, and how many?
- Did the data come from multiple sites, analyzers, and scanners?
- Has the vendor reported performance on a separate external dataset?
- Can we run a local verification on our own consecutive samples before going live?
The last question matters most. Just as you would verify a new analyzer against your own patient population before reporting results, an AI tool deserves the same treatment.
Frequently Asked Questions
What is the difference between consecutive sampling and random sampling?
Consecutive sampling includes every eligible case in order of arrival during a defined period. Random sampling selects cases by chance from the full eligible population. Both produce representative datasets. Consecutive sampling is usually easier to carry out in a working lab because it follows the normal flow of specimens.
What is spectrum bias in diagnostic testing?
Spectrum bias occurs when a test is evaluated on a group of patients that doesn’t reflect the range of disease severity and conditions seen in practice. Testing only clear-cut positives and healthy controls makes a test, or an AI model, look more accurate than it will be in routine use.
Why do AI models perform worse after deployment in the lab?
The most common reason is a mismatch between training data and real-world samples. Differences in prevalence, sample quality, staining, equipment, and patient populations can all reduce performance. Consecutive, multi-site data helps reduce the gap.
Can AI models be trained with enriched datasets?
Yes. Enrichment helps models learn rare conditions. However, the model should still be validated on consecutive or random data that reflects real prevalence, and the enrichment strategy should be clearly reported.
Which reporting guidelines cover sampling in diagnostic AI studies?
STARD 2015 covers diagnostic accuracy studies in general, and QUADAS-2 is used to assess risk of bias in systematic reviews. AI-specific extensions and checklists, such as STARD-AI and CLAIM for medical imaging AI, build on these same principles.
How large should a consecutive dataset be for lab AI validation?
It depends on the prevalence of the target condition and how precisely you need to estimate sensitivity and specificity. Rare conditions require much larger consecutive series to capture enough positive cases. A biostatistician can help calculate the sample size before data collection begins.