REQUEST FOR QUOTE → Request a quote
SpecForge Editorial Team

Displacement Sensor Resolution, Accuracy and Repeatability

Table of Contents
  1. Separate the three datasheet quantities
  2. Compare the specifications against four decision criteria
  3. Read the rating basis and test conditions
  4. Match the sensor type to the control task
  5. Identify constraints that cause misleading performance
  6. Apply a practical datasheet acceptance test
Displacement Sensor Resolution, Accuracy and Repeatability

A displacement sensor can resolve 0.001 mm yet retain an error of several hundredths of a millimetre, because display resolution does not include nonlinearity, hysteresis, temperature effects or installation error [S4].

For a displacement sensor, resolution, accuracy and repeatability answer three different questions: how small a move can be seen, how close the reading is to truth, and how tightly repeated readings cluster.

Separate the three datasheet quantities

Resolution is the smallest change in displacement that produces a distinguishable output change. It depends on the sensor, signal conditioner and analog-to-digital converter, not merely the number of decimal places shown on a display [S1][S4]. A display that increments by 0.001 mm does not establish 0.001 mm measurement resolution when other error contributions are larger [S4].

Accuracy describes agreement between the indicated displacement and the true or expected displacement. For force-transducer terminology, FUTEK describes nonlinearity as the maximum deviation of the calibration curve from a straight line between no-load and rated-load outputs, expressed as a percentage of rated output and measured on increasing load [S1].

Repeatability is the spread of results when the same position is approached repeatedly under specified conditions. Balluff defines it as the range of actual positions taken while the system is repeatedly commanded to the same location under identical conditions [S2]. Sensor accuracy is influenced by non-linearity, hysteresis, and repeatability [S1].

Compare the specifications against four decision criteria

On minimum detectable movement, resolution wins: it defines the smallest position change that can be represented. On absolute error, accuracy is the controlling parameter. On scatter around a fixed point, repeatability is the controlling parameter. On total installed performance, none can be evaluated alone because mounting, alignment, thermal conditions and the acquisition chain can dominate the result [S4].

Resolution is appropriate when the task requires fine motion increments or servo-loop feedback. Accuracy is appropriate when absolute position relative to a reference matters and the sensor is properly calibrated. Repeatability is appropriate when a process repeatedly returns to the same position, such as a jig, actuator, or inspection station. A sensor with excellent absolute accuracy can still be unsuitable if its readings scatter, while a highly repeatable sensor can still report the wrong position [S3][S5].

Calibration can compensate systematic errors such as scale-pitch variation or offset, but repeatability must be sufficiently consistent for a correction to remain useful. Copley Controls notes that accurate but non-repeatable results are harder to correct effectively, whereas repeatable errors can be compensated through calibration [S3].

Read the rating basis and test conditions

resolution vs repeatability vs accuracy on a displacement sensor datasheet - Read the rating basis and test conditions
resolution vs repeatability vs accuracy on a displacement sensor datasheet - Read the rating basis and test conditions

First establish whether a specification applies to the sensor, the electronics, the complete probe and transmitter assembly, or the installed measurement chain [S4]. A stated accuracy figure is not automatically a complete uncertainty budget. Measurement uncertainty may include calibration, resolution, thermal behaviour, method and installation contributions, so it cannot be inferred from resolution alone [S4].

Check whether a percentage is based on full scale or on reading. For example, 0.1% of a 100 mm full-scale range equals 0.1 mm, but that conversion does not convert a linearity specification into total measurement uncertainty [S4]. Also determine whether the number is a maximum error, RMS error, linearity band, hysteresis band or repeatability statistic; those terms are not interchangeable [S1][S4].

Unless those conditions match the application, treat the accuracy value as a conditional figure rather than a universal system guarantee.

Match the sensor type to the control task

A laser displacement sensor with fine optical resolution still requires a stated accuracy band, beam target and mounting arrangement. A draw-wire sensor may provide repeatable linear travel, but mechanical alignment, cable force and installation movement can affect its result. The relevant choice is therefore the complete error performance under the intended motion, not the finest number printed in the resolution row. [S4]

Encoder-based specifications illustrate the same trap. A 16-bit encoder provides 65,536 counts per revolution, while a 360-count-per-revolution encoder represents 1° per count and a 90-count-per-revolution encoder represents 4° per count [S5]. A 16-bit encoder has 65,536 counts per revolution, while the cited examples compare encoders with 360-CPR (1°) and 90-CPR (4°) resolution [S5].

For magnetic encoders, direction-of-approach effects can create magnetic hysteresis. Copley Controls explains that calibration can correct errors caused by eccentricity or poor accuracy [S3]. Specifiers should ask whether repeatability is bidirectional or unidirectional and whether the test uses the same approach direction for every cycle.

Identify constraints that cause misleading performance

resolution vs repeatability vs accuracy on a displacement sensor datasheet - Identify constraints that cause misleading performance
resolution vs repeatability vs accuracy on a displacement sensor datasheet - Identify constraints that cause misleading performance

Nonlinearity, hysteresis and non-repeatability contribute to a sensor's accuracy, while electronic noise and mechanical tolerances contribute to repeatability [S1]. Temperature variation can shift the result, and a support that moves independently of the measured phenomenon introduces an installation error that the sensor cannot remove [S4].

For optical encoders, scale-pitch variation is often a major accuracy contributor when the scale is mounted concentrically. Gain, offset and phase variations in sin/cos signals create sub-divisional error, and harmonic distortion can add further error [S3]. Calibration can address scale-pitch errors, but high-frequency sub-divisional error is more difficult to correct [S3].

Uncertainty is the more complete procurement concept when decisions depend on a guaranteed tolerance. It characterizes the range of values that can reasonably be attributed to the measured quantity using available information, but it requires stated coverage or confidence information and specified conditions [S4]. Repeatability measured over months at changing temperatures or after changing the installation is not automatically a repeatability test under the short-time, fixed-condition definition [S4].

Apply a practical datasheet acceptance test

Require the supplier to state resolution in engineering units, accuracy as a signed or total-error limit, and repeatability with its statistical form and test conditions. For example, do not accept three labels alone: request “0.001 mm resolution,” “±0.02 mm total error band after installation,” and “0.01 mm repeatability, 10 consecutive bidirectional approaches” when those values represent the intended requirement. [S4]

Verify that the proposed sensor's error band remains smaller than the process tolerance at the selected span, not merely at zero or full scale. If the required displacement is 100 mm and the acceptable error is 0.1 mm, 0.1% of full scale is numerically 0.1 mm, but the selected device still needs an explicit total-error and repeatability budget [S4].

For a closed-loop positioning application, require both resolution and repeatability, then use accuracy or calibration to remove systematic bias. For an absolute dimensional measurement, prioritize total accuracy and uncertainty. For repeated return-to-position work, specify approach direction, dwell period, temperature and mechanical loading because those conditions determine whether repeatability data can be reproduced [S2][S3].

The next trackable procurement check is to attach the manufacturer's datasheet revision, calibration method, temperature range and measurement conditions to every quoted number, then verify the installed chain with at least 10 repeat approaches at the decision points.

Frequently asked questions

Does a displacement sensor with 0.001 mm resolution guarantee 0.001 mm accuracy?

No. A display increment of 0.001 mm does not establish 0.001 mm measurement resolution, and the sensor may still retain an error of several hundredths of a millimetre. Accuracy also includes nonlinearity, hysteresis, temperature effects and installation error.

Which displacement sensor specification should control selection for absolute position error?

Accuracy is the controlling specification when the required position must remain close to a true or reference position. Check whether the stated figure is a maximum error, RMS error, linearity band or hysteresis band, and whether it applies to full-scale range or the actual reading.

How should repeatability be specified for a system repeatedly returning to the same position?

Specify repeatability as the spread of actual positions obtained while repeatedly commanding the same location under identical conditions. Confirm whether it is stated as a standard deviation, an LSB value or a percentage of full-scale range, and whether bidirectional or unidirectional approach was used.

What temperature and setup conditions apply to a datasheet accuracy rating?

Treat that value as conditional rather than a universal system guarantee unless the application meets those conditions.

10 sources
  1. Sensor Resolution vs Accuracy vs Repeatability
  2. How to understand the difference between resolution, accuracy, and ...
  3. How Resolution, Accuracy and Repeatability Relate to System ... (Jan 7, 2026)
  4. Understanding sensor resolution, accuracy and uncertainty - advarna (Oct 19, 2026)
  5. Accuracy, Resolution and Repeatability
  6. Understanding precision: accuracy, repeatability, precision
  7. Encoder resolution, accuracy and repeatability: What's the difference? (Jul 28, 2019)
  8. Sensor Accuracy: Deviation Between True and Measured Value
  9. What are Resolution, Accuracy & Repeatability (Nov 2, 2023)
  10. Accuracy, Repeatability, and Resolution in Machine Vision (Mar 5, 2025)

Need to source matching manufacturers or get a quote?

SpecForge connects industrial buyers with verified manufacturers. Submit your requirement and we will route it to matched suppliers.

Submit RFQ now →
Ask SpecForge AI