Published by: Research & Development Department, Technologie Optic.ca Inc., January 2026

Introduction

Optical fiber telecommunication relies on modulation – the process of encoding information onto light waves – to transmit digital data efficiently. In simple terms, modulation means changing some property of the light (such as its intensity, phase, or frequency) in a controlled way to represent binary 1s and 0s. Over the years, fiber-optic systems have evolved from basic on-off light signaling to very complex modulation formats in order to carry ever higher data rates across longer distances.

This paper provides an overview of the key modulation formats used in optical transceivers in the telecom sector, explaining how each works, along with its advantages, limitations, and typical data capacity. We will cover intensity modulation formats like Non-Return-to-Zero (NRZ) on-off keying and multi-level Pulse Amplitude Modulation (PAM4), phase-based modulation schemes such as Binary and Quadrature Phase-Shift Keying (BPSK, QPSK), and advanced combinations like Quadrature Amplitude Modulation (QAM). We also touch on polarization multiplexing as used in modern coherent optical systems.

Intensity Modulation – On-Off Keying (NRZ)

The simplest optical modulation format is On-Off Keying (OOK), which encodes data as the presence or absence of light. In digital form this is often called Non-Return-to-Zero (NRZ) signaling. As shown in Figure 1, NRZ uses two intensity levels: a high optical power for a "1" bit and a low (usually nearly zero) power for a "0" bit. Each bit is transmitted by either leaving the laser on or turning it off for the duration of a bit period. Only one bit is conveyed per symbol (bit period) in NRZ, making it a binary (2-level) modulation format.

The simplicity of NRZ-OOK has made it the workhorse of fiber optics for decades. Early optical networks from 1 Gb/s through 10 Gb/s per channel all used NRZ modulation, and it is still in use for many short-distance links and lower-speed interfaces. Key advantages of NRZ include its simplicity and robustness: it requires relatively simple transmitter and receiver designs (just turning a laser on/off and direct detection with a photodiode), and it has a large "eye opening" (signal amplitude difference between 0 and 1) which provides a good noise margin. NRZ signals also occupy a relatively narrow optical spectrum for a given bit rate, which historically allowed many wavelength channels to be dense packed in WDM systems (though still limited to about 10 Gb/s per wavelength on 50 GHz grids in early DWDM networks).

As data rates increase, NRZ-OOK reaches practical limits. Because NRZ carries one bit per symbol, achieving higher bit rates requires a proportional increase in symbol (baud) rate, which tightens transmitter/receiver bandwidth requirements and increases sensitivity to fiber impairments. For example, 40 Gb/s NRZ requires 40 Gbaud operation, making the signal highly vulnerable to dispersion and implementation penalties. As a result, NRZ was difficult to scale to long-haul 40G transmission and was progressively replaced by more advanced modulation formats.

Today, NRZ-OOK remains widely deployed at 10 Gb/s and, in some platforms, up to ~25 Gb/s per lane, mainly in short-reach and metro links. In the O-band (~1310 nm), where chromatic dispersion is near zero, NRZ can typically support 10–25 km without dispersion compensation, which aligns with common Ethernet optics. When additional link margin is needed, semiconductor optical amplifiers (SOAs) can provide short-reach power boosting. In practice, SOAs are often used with moderate gain (typically ~10–20 dB, depending on device and operating point), and they can help extend reach by providing several dB of extra budget; however, the achievable distance remains application-dependent because SOAs add noise and can introduce nonlinear distortion. For many NRZ O-band links, a realistic extended reach with SOA assistance is on the order of a few tens of kilometers, commonly ~20–40 km, assuming system penalties remain controlled.

In the C-band (~1550 nm), NRZ is still common in 10G DWDM transport. Without dispersion compensation, reach is often limited to ~20–40 km; with dispersion-compensating modules (DCMs) and optical amplification, ~80–100 km is achievable in DWDM links that include MUX/DMUX insertion losses. At ~25 Gb/s, dispersion tolerance becomes insufficient for long spans even with compensation, so deployment is typically limited to shorter distances. Overall, the maximum practical data rate for NRZ-OOK is ~25 Gb/s per lane, with the most robust reach performance occurring in short O-band links and legacy 10G C-band DWDM systems.

Multilevel Amplitude Modulation – PAM4

As per-channel data rates increased to 100 Gb/s and beyond, traditional NRZ-OOK became impractical, and PAM4 emerged as the dominant intensity-modulation format for high-speed short-reach transceivers. PAM4 uses four optical intensity levels per symbol (commonly labeled 0–3), so each symbol carries 2 bits mapped to 00, 01, 10, 11; a representative waveform is shown in Figure 2 (a). By packing two bits into one symbol, PAM4 delivers the same bit rate at half the baud rate of NRZ, reducing bandwidth demands on optics and electronics (e.g., ~50 Gbaud to carry ~100 Gb/s per wavelength). The corresponding PAM4 eye diagram in Figure 2 (b) shows three smaller eye openings, highlighting higher noise sensitivity than NRZ.

In practical terms, PAM4 enables 50 Gbaud signaling to carry 100 Gb/s per wavelength, whereas NRZ would require 100 Gbaud for the same throughput. This baud-rate reduction has been critical for scaling Ethernet and interconnect technologies. Today, PAM4 supports 100G, 200G, 400G, and even 800G optical modules. Typical implementations include 1×100G PAM4, 2×100G PAM4, or 4×100G PAM4 lanes, depending on the form factor and application. The effective maximum per-lane speed with PAM4 is currently ~100 Gb/s, corresponding to ~50–56 Gbaud signaling.

In the O-band (~1310 nm), chromatic dispersion is intrinsically low (≈0 to ≤1 ps/nm·km); therefore, dispersion-compensating modules (DCMs) are not required, even for long fiber spans. However, 100 km transmission with PAM4 in the O-band is not feasible, not because of dispersion, but due to insufficient power budget and OSNR limitations. Semiconductor optical amplifiers (SOAs) can provide ~10–20 dB gain, as mentioned earlier, enabling reach extension to ~10–20 km, but amplified spontaneous emission noise and nonlinear distortion prevent further scaling. Since DCMs provide no benefit in the O-band and high-gain amplification is unavailable, long-haul (≥100 km) PAM4 O-band links are impractical, and higher-sensitivity modulation formats are required.

In the C-band (~1550 nm), PAM4 experiences significant chromatic dispersion (~17 ps/nm·km), so without dispersion compensation the practical reach is short, typically ~2–5 km before dispersion degrades the signal. Unamplified PAM4 modules may extend to ~5–10 km based on link loss and receiver sensitivity alone. With EDFA amplification and dispersion compensation (e.g., DCM or electronic dispersion compensation), PAM4 DWDM links can reach ~80 km under favorable conditions, and some products claim up to ~80–120 km with careful optical power budgeting and compensation. Longer spans generally require advanced equalization and strict OSNR control to mitigate dispersion and noise effects.

The principal trade-off of PAM4 is its higher noise sensitivity. Relative to NRZ, PAM4 exhibits three smaller eye openings, requiring a higher SNR to achieve the same bit-error rate; therefore, Forward Error Correction (FEC) is effectively mandatory in PAM4-based links. Although PAM4 improves throughput within limited bandwidth, its reach is generally shorter than NRZ and significantly shorter than coherent transmission. In practice, O-band PAM4 is optimized for short-reach operation (typically ≤10–20 km), while C-band PAM4, including DWDM implementations, can be engineered to reach tens of kilometers and, in optimized systems with amplification and dispersion management, up to ~120 km. Beyond these ranges, coherent modulation is typically preferred due to superior sensitivity and dispersion tolerance.

Phase-Shift Keying (PSK) – Using Light's Phase for Data

Beyond intensity modulation, optical information can be encoded in the phase of the optical carrier using Phase-Shift Keying (PSK). Unlike OOK or PAM4, PSK maintains constant optical power and encodes data through controlled phase changes of the electromagnetic field. Detecting phase requires coherent receivers capable of measuring the phase relative to a local oscillator, increasing system complexity but enabling major gains in sensitivity and reach.

In Binary Phase-Shift Keying (BPSK), the optical phase toggles between two values separated by π radians (0° and 180°), representing binary 0 and 1 (see Figure 3). BPSK carries 1 bit per symbol, similar to NRZ, but with constant intensity, which improves tolerance to fiber nonlinearities. Coherent BPSK exhibits approximately 3 dB better sensitivity than NRZ or PAM4 for a given bit-error rate, making it one of the most power-efficient modulation formats. Compared to PAM4, BPSK requires significantly lower OSNR, does not require FEC by default, and supports much longer transmission distances.

The trade-off is spectral efficiency. While PAM4 supports 100G–400G+ per wavelength using multilevel amplitude modulation, BPSK is typically limited to ~100 Gb/s per wavelength in modern systems, often implemented as dual-polarization BPSK (DP-BPSK). This format sacrifices capacity in exchange for extreme reach and robustness.

In the C-band (~1550 nm), coherent BPSK is widely used in telecom DWDM long-haul systems and can achieve >1000 km reach in commercial networks. EDFA amplification with typical span lengths of 80–100 km is standard, while hybrid EDFA + Raman amplification is used to further improve OSNR and extend reach beyond ~2000 km. DCMs are not required, as coherent receivers provide full digital chromatic dispersion compensation. For 100G transmission, BPSK and PAM4 serve different regimes. 100G BPSK is preferred for ultra-long-haul links, where reach, OSNR margin, and nonlinear tolerance dominate over spectral efficiency. In contrast, 100G PAM4 in the C-band is favored for short-reach and metro applications (typically ≤40–120 km), where higher spectral efficiency is needed but dispersion and OSNR limitations require careful use of EDFA, DCM, and mandatory FEC. Thus, BPSK is chosen for maximum reach, while PAM4 is selected to maximize capacity over limited distances.

In the O-band (~1310 nm), BPSK is less commonly deployed for long-haul systems due to the lack of mature long-haul amplification infrastructure. While dispersion is near zero and DCMs are unnecessary, reach is primarily limited by available amplification and OSNR. As a result, ultra-long-haul BPSK systems are almost exclusively implemented in the C-band.

Differential BPSK (DPSK) avoids absolute phase detection by encoding information in phase transitions between symbols, enabling simpler receivers. DPSK provided ~3 dB sensitivity improvement over OOK and was widely used in 40 Gb/s systems before full coherent technology matured. In summary, BPSK outperforms PAM4 in reach, sensitivity, and nonlinear tolerance, but supports lower per-channel data rates. PAM4 is optimal for short-reach, high-capacity links, whereas BPSK remains a preferred choice for ultra-long-haul transmission where reach and robustness dominate over spectral efficiency.

Quadrature Phase-Shift Keying (QPSK)

QPSK (Quadrature Phase-Shift Keying) increases optical capacity by encoding data in four phase states of the carrier, enabling 2 bits per symbol. For reliable detection, the phases are chosen to be maximally separated: with 360° total phase and four symbols, the spacing is 90°, yielding the standard constellation at 45°, 135°, 225°, and 315° (Figure 4). The bit-to-phase assignment is arbitrary as long as transmitter and receiver agree, and Gray coding is commonly used to reduce errors between neighboring states. Because QPSK ideally keeps optical intensity constant, it is more tolerant to nonlinearities than intensity modulation.

By carrying two bits per symbol, QPSK doubles the spectral efficiency relative to BPSK at the same baud rate. For example, a 28–32 Gbaud polarization-multiplexed QPSK (DP-QPSK) signal supports ~100 Gb/s per wavelength, which became the foundation of commercial 100G coherent DWDM systems starting around 2010. This baud-rate reduction significantly improves dispersion tolerance and allows efficient operation within standard 50 GHz DWDM channel spacing.

In the C-band (~1550 nm), QPSK is extensively used for long-haul and metro DWDM networks. Chromatic dispersion (~17 ps/nm·km) is fully handled by digital signal processing (DSP) in coherent receivers; therefore, dispersion-compensating modules (DCMs) are not required. Optical amplification is essential, with EDFAs forming the backbone of commercial systems (typical spans of 80–100 km), while hybrid EDFA + Raman amplification is used for ultra-long-haul links. With amplification and coherent detection, DP-QPSK routinely achieves reach of 1,000–3,000 km, supporting data rates of 100G and 200G per wavelength.

In the O-band (~1310 nm), QPSK is less common for long-haul deployment. Although chromatic dispersion is near zero, long-distance transmission is limited by the lack of high-performance optical amplifiers. SOAs can be used for short reach with typical gains of ~10–20 dB, enabling tens of kilometers, but they add noise and nonlinear distortion. As a result, QPSK long-haul systems are almost exclusively deployed in the C-band, not the O-band.

Compared to PAM4, QPSK offers much higher reach and OSNR tolerance. While PAM4 enables higher per-lane rates (100G–400G+) using intensity modulation, it is noise-limited and dispersion-sensitive, restricting C-band PAM4 to ~40–120 km even with EDFA and DCM. QPSK, by contrast, trades spectral efficiency for orders-of-magnitude longer reach, making it the preferred choice for long-haul telecom transport, whereas PAM4 is favored for short-reach and metro applications.

Quadrature Amplitude Modulation (QAM): Increasing Bits per Symbol

While PSK modulates only the phase of the optical carrier, Quadrature Amplitude Modulation (QAM) exploits both phase and amplitude, forming a two-dimensional constellation in the in-phase (I) and quadrature (Q) plane. By increasing the number of constellation points, QAM enables more bits per symbol and thus higher spectral efficiency. In fact, PSK can be viewed as a special case of QAM with constant amplitude. Higher-order QAM formats (e.g., 16-QAM, 64-QAM) are the cornerstone of modern coherent optical communication systems.

16-QAM

16-QAM Performance and System Requirements (Telecom DWDM) — see Figure 5. Dual-polarization 16-QAM (DP-16QAM) encodes 4 bits/symbol per polarization, yielding 8 bits/symbol total. At ~30–32 Gbaud, DP-16QAM delivers ~200–240 Gb/s net throughput, while operation near ~60–64 Gbaud enables ~400 Gb/s per wavelength after overhead. With higher baud rates and advanced DSP/FEC, ~800 Gb/s per wavelength is achievable on short, high-OSNR routes. Relative to DP-QPSK, DP-16QAM provides a 2× spectral-efficiency gain at the same baud rate, but requires a higher OSNR (typically ~6–7 dB more for comparable BER), which limits transmission reach.

In commercial telecom networks, DP-16QAM is primarily deployed in the C-band (~1550 nm) for metro/regional DWDM because coherent receivers perform digital chromatic dispersion compensation, making DCMs unnecessary. Reach is therefore OSNR-limited rather than dispersion-limited; typical operational reach is ~100–500 km, depending on span loss, amplifier noise figure, ROADM filtering penalties, and FEC margin. Optical amplification is mandatory: EDFAs remain the default due to cost, maturity, and compatibility with 80–100 km span engineering. Raman (often hybrid EDFA+Raman) is preferred when higher OSNR is required—e.g., longer regional routes, higher baud rates, or denser ROADM chains—because distributed gain improves effective OSNR and can extend reach.

DWDM filtering must accommodate baud rate and roll-off: ~50 GHz channel grids are commonly paired with ~30–32 Gbaud DP-16QAM, while ~75–100 GHz spacing is typically required at ~60–64 Gbaud to reduce MUX/DMUX and ROADM passband penalties. Deployment in the O-band (~1310 nm) is uncommon for long-haul DP-16QAM; despite low dispersion, limited high-performance amplification and OSNR constraints make C-band the practical domain.

Higher-Order QAM (64-QAM and beyond)

Higher-order formats such as 64-QAM encode 6 bits per symbol (12 bits with PDM), enabling ~600 Gb/s per wavelength at ~60 Gbaud and even higher rates with advanced DSP and probabilistic shaping. However, these formats demand very high OSNR and are therefore limited to short distances, typically <100–200 km, such as data-center interconnect (DCI) or very clean metro links. Orders beyond 64-QAM are mostly confined to short-reach coherent links or experimental systems.

Comparison with QPSK and PAM4

Compared to QPSK, QAM formats (e.g., 16-QAM) offer higher capacity per wavelength but shorter reach due to higher OSNR sensitivity. QPSK remains preferred for long-haul (>1000 km) links, while 16-QAM is favored for metro and regional networks where capacity outweighs distance.

Compared to PAM4, QAM (with coherent detection) provides far superior dispersion tolerance and reach. PAM4, being intensity-modulated, is noise-sensitive and typically limited to ≤10–20 km in the O-band and tens of kilometers (up to ~120 km) in the C-band with amplification and dispersion management. In contrast, coherent QAM does not require DCMs, as chromatic dispersion is fully compensated digitally, enabling much longer distances.

Conclusion

This paper has shown that no single modulation format is best for every optical link—the right choice depends on bit rate, required reach, fiber band (O/C), and cost/complexity. NRZ-OOK remains the simplest and lowest-cost option for 10G and many 25G links. In practice, 10G NRZ can span ~10 km in O-band (typical LR-class optics) and, with high-budget C-band DWDM optics, can reach ~80–100 km over SMF without amplifiers or DCM in some designs; adding EDFA and (if needed) dispersion compensation mainly helps margin and system loss (e.g., MUX/DMUX). 25G NRZ in C-band is much more dispersion-limited, so long-reach DWDM is generally impractical.

PAM4 enables 100G–400G+ but is more OSNR-sensitive. It is typically short-reach in O-band, and in C-band it often needs FEC and dispersion management (electronic and/or optical) for longer spans. For longer distances and higher per-wavelength capacity, coherent phase-based formats are required. QPSK offers an optimal balance between reach and capacity, supporting 100–200G over thousands of kilometers in the C-band using EDFA/Raman amplification without DCMs. QAM formats (e.g., 16-QAM, 64-QAM) further increase capacity to 400G–800G+, but with reduced reach (~100–500 km for 16-QAM; <200 km for 64-QAM), making them ideal for metro and regional DWDM.

In practice, modern networks rely on adaptive coherent transceivers, dynamically selecting NRZ, PAM4, QPSK, or QAM to balance capacity, reach, and cost under real network conditions.