By Jon Karafin, Edited by Steve Wright

March 25, 2024

Reading Time:
10 Minutes



In Part 2, the focus (no pun intended) was on establishing solid measurement criteria for evaluating the resolving power of an optical system, namely the mighty MTF (Modulation Transfer Function). We defined spatial frequencies and explored how the optical system’s resolving power degrades as the spatial frequency goes up and how the MTF% quantifies that degradation by calculating the resulting reduction in contrast percentage. We noted that the degradation was caused by the limits of diffraction, so up to this point the MTF% was due only to diffraction limitations. In this episode, we will see how the optical aberrations in physical optics (lenses) contribute to a much worse MTF% for the eye as well as camera lenses.

MTF vs. f/n

It turns out, due to our physics friend diffraction, there is a mathematical relationship between the MTF% and the f/n (f-stop)[1] of an optical system such that the maximum diffraction-limited resolving limits can be determined from the f/n value. This may not be totally surprising since the f/n relates to the size of the aperture, apertures diffract light, and the larger the aperture the greater the resolving power. But the real story is how useful it is to be able to characterize an optical system’s maximum theoretical resolving limit based on its f/n as illustrated in Figure 1 below.

Figure 1 - Diffraction-only-limited resolution of common optical apertures[2]


For each f/n in the Aperture column, Figure 1 reveals the maximum resolving power in lp/mm for several key MTF% values. Choosing, say f/2.8 as an example (outlined in green), the MTF 0% column represents the frequency limit (The Dawes Criterion) for a theoretically perfect optical system and shows 649 lp/mm as the maximum resolution. The MTF 9% column is the MTF value at the diffraction limit (the Rayleigh Criterion) discussed extensively within Part 2 and is more like the practical resolvable upper limit of optical systems, like cameras, showing 532 lp/mm while the MTF 50% and MTF 80% columns are other important industry benchmarks.

This would lead one to believe that the human eye, with an f/n approximately f/2.8, would be at its maximum resolving power when fully opened at ~6mm, and at the Rayleigh criteria of MTF 9% its maximum resolving power would be 532 lp/mm (Figure 1). This might be true, were it not for the unaccounted plethora of wavefront errors within the optical system.

When characterizing the real-world performance of an imaging system, it is very helpful to represent resolving power by the resolved spot size — that is, the smallest spot that can be resolved by an optical system. If you conceptually think of one line from a line pair as a single column of pixels, and a spot as a single pixel (ignoring a circle vs. a square pixel for a moment), then the relationship between lp/mm and spot size is that a line is a one-dimensional measurement whereas a spot is a two-dimensional measurement. Fundamentally, when the eye is seeing with greater resolution the lp/mm goes up (more line pairs per mm are resolvable) while the spot size goes down (a smaller spot is resolvable).

Figure 2 - Adapted illustration of diffraction-only limited spot size of the human eye[3]


Figure 2 shows the resolvable spot size for pupil sizes from 1 to 7 mm. The resolvable spot size gets smaller as the pupil gets larger for the perfect eye, meaning diffraction-only limited. But there are no perfect eyes and diffraction is not the only limit to resolving power. There are other perturbations to factor in before we get a truly representative model of the resolving power of an optical system.


I know what you are thinking. Yet another transfer function with another acronym. Ugly, but essential for understanding the optical aberrations of a real-world lens system. Up to now, we have considered the pupil as just an aperture and all of our errors were due to diffraction. But now it’s time to add a lens with all of the wavefront errors it introduces. Wavefront errors mean, of course, distortions to the incoming light of which there is a veritable zoo of different distortion devils. They include common distortions such as defocus and astigmatism, as well as obscure ones like pinching and coma. They pile on top of each other, but can be aggregated into a single wavefront distortion error characterized by the Aberration Transfer Function (ATF). While the MTF measures the optical system’s ability to transfer contrast, the ATF focuses on how the transfer process is further limited by distortions.

The causes of distortions range from manufacturing defects to design compromises. These must be controlled to industry standard tolerances in order to allow the lens designers to predict the performance of the optical system. A sample of these tolerances are listed in the table below.

Figure 3 - Typical optical system fabrication wavefront error tolerances for telescope and similar optics[4]


In the left-most column of Figure 3, LENS FORM refers to the shape of the lens while GLASS refers to its chemical makeup. The PARAMETER column lists factors to be controlled during fabrication, with the tolerances listed for low, medium and high quality which are labeled as COMMERCIAL, PRECISION, and HIGH PRECISION. Note that when the Figure Errors in the PARAMETRES column get to HIGH PRECISION the tolerance is λ/20, meaning 1/20th the wavelength of light. That’s a tight spec!

Fortunately, the total impact of all of these disparate sources of optical aberrations can be consolidated to a single total wavefront error variable ω (omega) used in the following Aberration Transfer Function (ATF) equation:

Where ω = the RMS wavefront error which quantifies the deviation of the resulting wavefront compared to the ideal wavefront in units of multiple wavelengths, and v = normalized spatial frequency between 0 (no frequency) and 1 (max frequency from Dawe’s limit). 

Figure 4 - Illustration of the Aberration Transfer Function (blue) = the product of ATF (red) * MTF%[5]

Figure 4 shows how the ATF affects the MTF by multiplying them together. As the ATF ω curve ranges from 0.37 to 0.149 (blue) it is multiplied by the MTF ω = 0 curve to produce the characteristic MTF curves labeled 0.37 to 0.149 (red) revealing the nature of the effect of the Aberration Transfer Function (ATF) wavefront errors over the MTF. As the ATF ω increases, it decreases the MTF contrast percentage resulting in poorer overall performance of the optical system.

Now that we have seen how the MTF curve is modified by a range of wavefront errors assessed by the ATF, you now are armed with a more accurate description of the potential resolution of an optical system that includes lens aberrations in addition to its diffraction limit. Going forward, when we speak of the MTF, it will be understood that it includes the ATF and no longer just represents the diffraction-limited value.

THE HUMAN VISUAL SYSTEM:  Good from far, but far from good

Noticeably worse than precision optics, the human visual system also suffers from both diffraction-limited losses plus ATF optical aberrations. What is important, however, is the inflection point or “sweet spot’ between them. Figure 5 graphs the Airy disc diameter (black line) vs. the RMS wavefront error for all aberrations (solid red line) graphed over the pupil diameter ranging 0 mm to 8 mm on the X axis. The various dotted red lines represent other RMS errors with selected aberrations removed. Keep in mind that as the Airy disc diameter goes up, the resolving power goes down.

Figure 5 - Illustration of the inflection point between diffraction-only limits and average common sources of wavefront errors[6]


As the pupil diameter gets larger than the inflection point, the Airy disc line goes down while the RMS error line goes up. The inflection point represents the very best compromise between the two where the greatest visual acuity occurs and is located right around 2.5 mm - 3 mm. If the pupil gets larger than that, the RMS errors go up, losing acuity. If the pupil gets smaller, the Airy disc diameter goes up, losing acuity. Like any optical system, the eye has its best performance when it is stopped down to limit the RMS wavefront error.

We can see the inflection point in action in Figure 6 below, which is a visual representation of the consequences of the graph in Figure 5, namely things get worse as you move away from the inflection point. Figure 6 illustrates resolved spot sizes for pupils ranging from 1 mm - 7 mm with the inflection point marked at 2.5 mm. The top row illustrates spot sizes for a diffraction-limited model (the perfect eye) which shows the resolved spot size getting larger as the pupil gets smaller, indicating reduced visual acuity below the 2.5 mm inflection point. The bottom row illustrates spot sizes for an aberration limited model (the typical eye), which shows the resolved spot size getting larger and more distorted as the pupil gets larger than 2.5 mm indicating reduced visual acuity above the inflection point.


Figure 6 - Comparison between the theoretical model of a perfect eye vs. physical eye actually resolves across minimum and maximum pupils (adapted)[7]

The takeaway from Figure 6 is, of course, that our visual acuity is best at a pupil diameter of 2.5 mm. The good news is that our pupil diameter hovers around 2.5 mm - 3 mm for most lighting conditions ranging from indoor to outdoor lighting. To get the pupil diameter down to 2 mm you need blazing direct sunlight. To get it larger than 4 mm will require a moonlit night. Not surprisingly, our vision has evolved for optimal acuity for the most common lighting conditions ranging from cave to savannah.


We began by documenting how the f/n is an indicator of the diffraction-limited resolving power of an optical system and indeed, determines the diffraction limits of that optical system because the pupil diameter dictates resolved spot size (Figure 1). We then learned there is no perfect optical system by introducing the Aberration Transfer Function (ATF) which introduced a zoo of optical aberrations that create wavefront distortions inherent in lenses.

The effect of the long list of available types of wavefront distortions on resolution was reduced to a single handy ATF equation that could then be used to factor the ATF into the MTF (Figure 4) to see how the MTF is degraded. Armed with a degraded MTF, we were ready to take on the human visual system and use the knowledge of our diffraction-only limits with our new-found wavefront errors to determine the inflection point for the sharpest human vision with a pupil diameter of 2.5 - 3 mm. (Figure 5). Figure 6 pulled it all together by illustrating how our diffraction-limited resolved spot size gets larger when our pupils get smaller, but our aberration-limited spot size gets uglier as our pupils get larger.

This culminated in the observation that our pupil size hovers in the 2.5 mm - 3 mm diameter for most lighting conditions which is the “sweet spot” of our visual acuity and is precisely the spot that should be used as the reference for calculating the resolution requirements of a holographic display. Tragically, we are not done maligning the eye. In Part 4 we will consider additional conditional and environmental factors, such as dynamic range, that further degrade our visual acuity and move the goalposts for the optimum holographic display resolution several more yards downfield.

[1] If you are a bit unclear about exactly what an f/n is just go back and review Part 1 of this series.


[3] Raymond A. Applegate, OD, PhD, “Aberration and Visual Performance: Part I: How aberrations affect vision,” University of Houston, TX, USA




[7] Raymond A. Applegate, OD, PhD, “Aberration and Visual Performance: Part I: How aberrations affect vision,” University of Houston, TX, USA