Most metrologists prefer to have at least 10 pixel or subpixel "ticks" across the tolerance range, which is the working rule of thumb in machine vision gauging [S1].
That single number, 10 pixels per tolerance band, sets the chain of decisions: it forces a target object-space resolution (mm/pixel), which in turn sets the field of view, which in turn sets the sensor pixel count, the lens magnification, and the working distance [S5][S6]. If the 10× rule is broken, a one-pixel shift caused by focus drift, noise, or edge-threshold variation can consume the entire allowed tolerance on a critical dimension [S5].
Why 10 pixels per tolerance, and where the number comes from
The 10:1 rule is the same principle behind the metrological "rule of ten": the measuring instrument should be at least ten times more repeatable and accurate than the process specification it is asked to verify [S1]. A machine vision camera reports its output in discrete pixel (or subpixel) increments, so those increments play the role of the scale ticks on a caliper [S1]. A common practitioner restatement is that pixel size should be roughly one-tenth of the required measurement tolerance [S2]. A representative worked example: for a 30 µm process tolerance, the target resolution is 3 µm, i.e. one-tenth of the spec [S3].
The "10 ticks across the tolerance" figure is a measurement rule, not a detection rule, and the two are commonly confused. For defect detection, production inspection typically needs only 3 to 5 pixels across the smallest defect, because the question is "is it there?" rather than "what is its dimension?" [S2][S7]. The threshold rises to 10 or more pixels across the feature once the question becomes "how big is it?" or "is it within ±X?" [S7]. Using a detection-grade pixel budget for a gauging task is the most common root cause of false rejects on otherwise good parts.
How the 10-pixel rule cascades into hardware choices
The cascade starts with the tolerance and works backward. With a 50.00 mm nominal width and a ±0.10 mm tolerance, the total tolerance band is 0.20 mm; dividing by 10 gives a 0.02 mm/pixel object-space resolution target [S5]. A camera imaging a 30 mm part with 480 pixels across that span gives roughly 0.063 mm/pixel, which is enough for inspection but inadequate for tight gauging on the same part [S4]. A second worked example from a 2D measurement inspection reference: 0.024 mm/pixel × 5 pixels = 0.12 mm is the maximum distinguishable dimension step at that pixel scale [S9].
Two practical shortcuts capture the same logic. First, the spatial-resolution formula: spatial resolution (mm/pixel) equals field of view (mm) divided by the number of sensor pixels across that axis [S6][S8]. A 2000-pixel-wide sensor over a 100 mm field of view yields 0.05 mm/pixel, and to resolve a 0.02 mm feature the system needs roughly 0.01 mm/pixel or finer, i.e. 10,000+ pixels across the same field of view [S6]. Second, the defect-vs-measurement split: 3 to 5 pixels per defect for detection, 10+ pixels per tolerance for measurement [S2][S7]. Both formulations are equivalent once the tolerance and field of view are fixed.
Subpixel interpolation: pushing the effective count past 10
Raw pixel pitch is rarely the final resolution figure, because edge response spreads across several contiguous pixels and a subpixel-accurate edge can be inferred from the gray-scale profile [S1]. Standard edge-localisation schemes (second-derivative zero-crossing, centroid, fit-to-curve on the gray ramp) report subpixel increments smaller than the pixel pitch itself, and different algorithms yield different subpixel increments on the same image [S1]. In practice this means a 10-pixel-per-tolerance budget can be relaxed to something like 4 to 6 native pixels per tolerance when a well-characterised subpixel algorithm is applied, which is why a 5 MP or 10 MP industrial camera is often adequate where a naive mm/pixel calculation would suggest 25 MP [S5].
That relaxation is conditional, not free. Subpixel gain collapses if the edge is fuzzy (poor focus, wrong lighting), if the gray-scale profile is asymmetric (specular highlights, shadows), or if the calibration standard is not traceable [S1]. The conservative engineering position, and the one most metrologists default to in regulated industries, is to budget 10 native pixels across the tolerance before counting subpixel gain at all [S1][S2].
Inspection vs. gauging vs. high-precision metrology: where the rule bends
The 10-pixel rule is the middle of three tiers that practitioners use to size a machine vision system. For presence checks and barcode reads, pixel count is driven by feature recognisability, not by tolerance, and the system often runs with 1 to 2 pixels across the smallest meaningful feature. For dimensional gauging, the 10-pixel rule applies and is the industry default. For high-precision metrology in a lab or calibration cell, the budget commonly grows to 20 to 50 pixels across the tolerance, because the system is also the reference against which other instruments are compared [S1].
The 10-pixel rule also interacts with field-of-view geometry in a way that is easy to miss. A 2D vision measuring machine that images a 100 mm field of view with a 2000-pixel sensor hits 0.05 mm/pixel, which supports a 0.5 mm tolerance comfortably but cannot resolve a 0.05 mm tolerance without changing optics or sensor [S6]. A side-by-side comparison of the three pixel budgets clarifies this:
Detection (3 to 5 px/defect): best for "is it there?" tasks, lower sensor cost, tolerant of wider field of view, weaker on absolute dimension reporting [S2][S7].
Gauging (10 px/tolerance): the workhorse rule for inline dimensional measurement, balances sensor cost and tolerance margin, requires controlled lighting and edge algorithm choice [S1][S5].
High-precision metrology (20 to 50 px/tolerance, plus subpixel): reserved for lab-grade vision imaging and calibration cells, demands telecentric or corrected optics, traceable standards, and tighter environmental control [S1][S5].
What the rule does not cover, and where it fails
The 10-pixel rule is silent on three failure modes that routinely eat that margin on production lines. First, perspective error: if the part can sit at varying standoff distances within the field of view, the effective mm/pixel changes with depth, and a 10-pixel budget at the nominal standoff can become a 5-pixel budget 10 mm closer to the lens [S5]. Second, lens distortion: barrel or pincushion distortion shifts edge coordinates by several tenths of a pixel across the field, and a low-distortion lens is non-negotiable for sub-tolerance gauging on wide fields of view [S5]. Third, illumination: edge response, and therefore subpixel repeatability, depends on the lighting geometry; backlight gives the cleanest edge profile, whereas diffuse front light softens the gray ramp and costs subpixel precision [S1].
The 10-pixel rule also does not address what counts as a "tick" in a system that mixes pixel resolution, subpixel interpolation, and calibration. Most published guides collapse those into one number for simplicity, which is fine for a budget estimate and wrong for a spec sheet [S1][S5]. For a traceable measurement, the engineering chain is: tolerance → required object-space resolution → sensor pixel count and field of view → lens choice and distortion budget → subpixel algorithm → calibration standard, with each step independently justified [S5][S6]. A vision controller and the machine vision ID trace data that ride on top of that image are only as accurate as the pixel budget underneath.
Worked numbers and quick sizing checklist
Three reference calculations cover most of what a sizing engineer needs. Example 1, tight tolerance: ±0.05 mm on a 20 mm feature gives a 0.10 mm tolerance band; at 10 pixels across the band, target 0.01 mm/pixel, and over a 20 mm field of view a 2000-pixel sensor is exactly on the line, with no room for subpixel gain [S5][S6]. Example 2, moderate tolerance: ±0.10 mm on a 50 mm feature gives 0.20 mm tolerance, 0.02 mm/pixel, 2500 pixels across a 50 mm field of view, which a 5 MP sensor (≈2590 × 1944) handles with margin [S2][S5]. Example 3, detection only: a 1 mm defect on a 100 mm field of view needs 3 to 5 pixels on the defect, i.e. 0.2 to 0.33 mm/pixel, which a 640-pixel sensor satisfies cheaply [S2][S7].
A short sizing checklist for production engineers: (1) write the tolerance first, not the camera model; (2) divide the full tolerance band by 10 to get the target object-space resolution; (3) pick the field of view to fit the part plus a small margin; (4) solve sensor pixel count as FOV / target resolution; (5) budget extra pixels if the application is metrology-grade rather than gauging-grade; (6) verify the result with a calibration standard after the lens and lighting are fixed, because the on-paper figure and the measured figure routinely differ by 20 to 30 percent [S1][S5][S6][S10].
Standards, calibration, and the audit trail
No single ISO or ASME clause specifies a "10 pixels per tolerance" rule; the figure is an industry convention, not a regulated threshold. What the standards do require is traceability: the calibration object used to convert pixels to millimetres must be a referenced standard, and the calibration procedure must be documented with the same rigour as any other dimensional instrument [S1]. For inline gauging on a vision measuring machine, this typically means a certified calibration target, periodic re-verification, and a record of the subpixel algorithm and lighting geometry in use at the time of measurement [S1][S5].
The practical audit signal is whether the system's reported accuracy is at least 4 to 10 times better than the tolerance it is asked to police, matching the same rule-of-ten principle that drives the pixel budget itself [S1]. A system that reports a measurement uncertainty within the tolerance band is not a gauging system, it is a presence-check system, regardless of how many megapixels are on the sensor.
Trackable signals to watch in the next quarter: published subpixel-algorithm repeatability benchmarks on 5 MP and 10 MP industrial sensors with 1.6 to 3.76 µm pixel pitch, and any new application note that quantifies the gap between the on-paper 10-pixel-per-tolerance rule and the measured accuracy after lens distortion and illumination are included [S2][S5]. Related sizing problems on a production line, from strapping band tension verification to defect detection on battery cells, run into the same pixel-budget logic when a camera is added, and the same 10-pixel rule transfers [S7].
This topic is covered further in Acetylene vs Propane for Oxy-Fuel Cutting: Spec-First Selection.