Flatlines and silent gaps are the two failure modes that simple min/max range checks miss most often, and they are responsible for the majority of bad decisions traced back to sensor streams in environmental and process monitoring, per a 2020 systematic review of sensor data quality [S2].
The standard remedy is a layered rule set, not a single check: range tests catch hard faults, persistence/stuck-value tests catch flatlines, max-gap-duration rules catch communications dropouts, and statistical or model-based detectors catch drift and fouling that stay inside the valid range. Each rule fires a flag rather than overwriting the raw reading, and the raw channel is preserved untouched so that every corrected value traces back to a defensible source [S3].
What the rules actually target: gaps vs flatlines vs drift
A gap is a missing timestamp: the expected sample at time t is absent from the stream, usually because the data logger lost power, the radio link dropped, or the upstream buffer filled. A flatline is the opposite problem: timestamps arrive on schedule but the value stops changing, which is what happens when a probe fouls, a cable breaks to a fixed bias, or a transmitter hangs in its last valid output. Drift is a slow walk away from truth that still looks plausible, so range checks pass [S3].
These three modes require different statistical signatures. A gap is detected by a timestamp-interval test, typically a max-gap-duration rule such as "no more than 2.5 expected intervals between samples before flagging a dropout." A flatline is detected by a persistence test, often expressed as "value identical at LSB resolution for N consecutive samples," with N sized to the process dynamics. Drift is detected by a deviation test against a learned per-channel normal pattern, which is the only one of the three that handles the case of a fouled sensor staying inside its valid range for weeks [S3][S2].
The four-rule minimum rule set most spec sheets use
A working data quality rule set for sensor streams normally includes four rule families, applied in order from cheapest to most expensive to compute: range checks (physical plausibility, e.g. pH 0-14 or duct static -2000 to +2000 Pa), rate-of-change checks (|Δx/Δt| within a process-physical bound), persistence/stuck-value checks (no change over N samples), and gap-interval checks (Δt between consecutive timestamps within expected sampling tolerance) [S2][S4].
On top of those four, statistical detectors add value: rolling-window mean and standard deviation, seasonal deviation from a learned baseline, and inter-channel consistency tests (e.g. upstream level vs downstream flow should correlate within a lag). ESIP's environmental-sensor QA framework documents a similar structure, with near-real-time checks layered on top of preventative field measures, and the data published with explicit quality levels rather than just a pass/fail label [S1].
Selection criteria: which rules go on which channel
Rule selection should be driven by the sensor's physical behaviour, the process dynamics, and the cost of a missed fault. Persistence windows are typically sized at 5-10 times the dominant process time constant: a HVAC duct static pressure channel that swings on a 10-second damper cycle should not be flagged for identical readings over 30 seconds, but the same threshold on a slow tank level channel is a clear flatline signal [S3].
Gap rules are sized against the sampling cadence and the acceptable data-availability SLA. A 1-minute logger feeding a daily compliance report can tolerate 15-minute gaps without downstream impact, while a 100 ms power-quality channel feeding a protective relay cannot tolerate more than one missing interval. Rate-of-change rules must use physical limits of the measurand, not arbitrary thresholds: a 0-100 mbar pressure transmitter on a slow tank cannot physically change 50 mbar in 1 s, so that is a hard ceiling, while the same transmitter on a compressor pulse line can [S2][S4].
Comparison of the main detection approaches
The three dominant approaches, hard rule checks, statistical anomaly detection, and ML-based per-channel learning, trade off transparency, false-positive rate, and the ability to catch in-range faults. The table below lines them up against the criteria that matter when you are specifying a QA/QC stack. [S3]
On transparency, hard rules win because every flag is a boolean the auditor can replay. On in-range fault detection, ML-based per-channel learning is the only one of the three that reliably catches drift and fouling that stay inside the valid range, which is the failure mode that threshold checks systematically miss [S3]. On false-positive rate, statistical and ML detectors need a 2-4 week warm-up window of clean data before they are trustworthy, while hard rules are usable from day one if the range and rate-of-change bounds are grounded in process physics [S2][S5].
Who this is for, and where it does not fit
This rule pattern is for teams that own the QA/QC layer of a sensor pipeline, including process and instrumentation engineers, environmental monitoring leads, and data historians admins, and the right time to implement it is before a regulatory submission, not after the first audit finding. The pattern is well-suited to environmental field stations, water and wastewater SCADA, building automation, and any air quality monitor network feeding compliance reporting, because the failure modes are dominated by fouling, dropouts, and battery loss [S1][S3].
It is not a fit for high-rate vibration or power waveform analysis, where the gap/flatline vocabulary still applies but the rule thresholds and detectors are typically built into a power quality analyzer firmware rather than a downstream software layer, and the sample rates (kHz and above) make a per-channel ML warm-up window impractical. It is also not a substitute for instrument-level calibration: no QA/QC rule set recovers a sensor that has drifted out of calibration, and that failure mode still requires scheduled bench checks against a reference standard [S2][S5].
Implementation: raw channel preserved, flags travel with the data
The non-negotiable architectural rule is that raw sensor readings are never overwritten. Corrections are written to a separate edited channel, the original raw timestamp and value remain queryable, and quality flags travel with the data into every derived calculation, so an I&I estimate or a model calibration inherits the QA status of the channels it was built from [S3]. This pattern matches the way ESIP recommends publishing environmental sensor data: with an explicit quality level attached to each value rather than a single binary "good/bad" label on the whole stream, where Level 1 is the raw signal and Level 2 is gap-filled or interpreted data requiring careful downstream attention [S1].
Operationally, flags should be implemented as calculated channels rather than as a separate application, so the analyst can graph a flag alongside the raw value, alarm on it, and feed it into other calculations; this is also how ML-based detectors sit cleanly in the same pipeline rather than in a disconnected tool [S3]. The full edit history (what changed, who changed it, when) is part of the same rule, because the first time someone challenges a number, the answer needs to be a traceable chain rather than a reconstructed claim [S3]. Data contracts extend this further by pinning the agreed quality dimensions to specific data products, so the rule set becomes a binding spec between producer and consumer rather than a downstream best-effort [S4].
Limits and failure modes of the rule set itself
The rule set has known blind spots. Range checks will not catch a flatline inside the valid range, which is exactly why a fouled sensor can sit in tolerance for weeks before anyone notices; persistence checks will false-positive on a legitimately steady process, which is why the persistence window has to be sized against the process time constant rather than a generic default. Statistical detectors need a warm-up window and they degrade under non-stationary operation, so a rule that was tuned for summer flow will misfire in spring runoff until it is re-fit [S3][S2].
Inter-channel consistency tests are powerful but easy to over-specify: the more cross-checks you add, the more false positives you generate when one legitimate event (a pump trip, a slug of cold water) breaks the assumed correlation. The rule of thumb in practice is to start with range, persistence, gap, and rate-of-change, add inter-channel checks only for the 2-3 relationships that are physically enforced by the process, and add ML-based anomaly detection last, after the deterministic rule set is stable [S3][S5].
Signals to track next
Two trackable signals worth watching: the move toward metadata-driven rule application across pipelines, so a single rule definition propagates to every data product rather than being re-typed per channel, and the integration of agentic AI workflows that propose rule updates from incident postmortems rather than waiting for a human to revise the spec. Both are flagged as active directions in 2026 data-quality platform guidance [S6].
See also our earlier report, Data Historian vs Time-Series Database for IIoT: 2026 Decision Matrix.