IPsec Protocol: 4 Building Blocks Behind Every Tunnel

Share:
Instrumentation
IPsec Protocol: 4 Building Blocks Behind Every Tunnel

Modbus and DNP3 traffic between a remote RTU site and the control room was never designed with a single bit of encryption in mind.

The IPsec Protocol is what closes that gap, wrapping an entire unprotected conversation inside a tunnel that an outside observer cannot read or tamper with.

IPsec Protocol ESP and AH IKE Site to Site VPN

IPsec Protocol combines authentication, encryption, and automated key exchange into one framework, most commonly used to build a site to site tunnel between a remote plant and a central SCADA control room.

Hello everyone, today we are going to walk through the actual building blocks of IPsec, from its two core protocols through its key exchange process, and why plants lean on it to protect Modbus and DNP3 traffic that carries no security of its own.

We will also cover transport versus tunnel mode, common encryption choices, NAT traversal, and how IPsec compares to newer options like WireGuard.
IPsec Protocol
Advertisement

4 Building Blocks Inside IPsec Protocol

1
ESP
Encapsulating Security Payload, the workhorse protocol handling encryption, authentication, and integrity together.
2
AH
Authentication Header, providing integrity and origin authentication only, with no encryption of the payload at all.
3
IKE
Internet Key Exchange, the negotiation process that sets up the encrypted channel before real data ever moves.
4
Security Association
A one way agreement on algorithms and keys, identified by a number that tells each side which rules apply.

ESP is by far the most widely deployed of these four, since it alone covers confidentiality, integrity, and authentication in a single pass, which is why AH sees comparatively little real world use today.

AH Versus ESP in Detail

FeatureAHESP
IP protocol number5150
ConfidentialityNoYes
Integrity and authenticationYesYes
NAT compatibilityPoorGood, with NAT traversal

AH struggles behind a NAT device because its integrity check covers parts of the outer IP header, and any address rewriting along the path breaks that check completely.

ESP avoids that problem since its protected fields sit entirely inside the packet, leaving room for NAT traversal to wrap the whole thing in an ordinary UDP datagram when needed. This is one reason ESP dominates real IPsec Protocol deployments today.

Transport Mode Versus Tunnel Mode

Transport Mode

Only the payload is protected, leaving the original IP header visible, suited to direct host to host communication with lower overhead.

Tunnel Mode

The entire original packet is encrypted and wrapped inside a new outer packet, hiding the internal network entirely from outside view.

Tunnel mode is the one that actually matters for a plant, since an IPsec Protocol site to site VPN between a remote RTU location and the central SCADA control room needs to hide whole subnets, not just a single host conversation.

Advertisement

How Internet Key Exchange Sets Up the Tunnel

Phase 1 of IKE builds a secure, authenticated control channel between the two gateways, confirming each side's identity and agreeing on the algorithms that will protect everything that follows.

Phase 2 then uses that already secured channel to negotiate the actual security associations that will carry real traffic, including which subnets are allowed through and whether a fresh key exchange happens for extra forward secrecy.

UDP 500
Carries IKE negotiation before NAT is detected
UDP 4500
Carries IKE and encapsulated ESP once NAT is found
IP Protocol 50
ESP traffic when no NAT sits in the path
IP Protocol 51
AH traffic, rarely seen in modern deployments
Did You Know
The newer IKEv2 standard finishes its entire Phase 1 negotiation in just four messages, compared to six for the older IKEv1 Main Mode. IKEv2 also builds in NAT traversal support natively and adds a cookie mechanism that resists denial of service attempts, which is why it is now the recommended choice for any new IPsec deployment.

Security Associations and the SPI Number

Every security association only ever protects traffic in one direction, so a normal two way conversation always needs a matching pair of them, one for each direction.

Incoming ESP packet header includes:
SPI = 0x3F2A9C10

Receiving gateway looks up SPI 0x3F2A9C10
in its security association table to find
the correct decryption key and algorithm.

The Security Parameter Index carried in that header is what lets a receiving gateway instantly pick the right set of rules out of potentially many active associations, much like a port number sorts traffic into the right application.

Choosing Modern Encryption and Integrity Algorithms

1
Use AES 128 or AES 256 for encryption, avoiding older 3DES on any new configuration.
2
Use SHA 256 or stronger for integrity checking, avoiding legacy SHA 1 and MD5 wherever possible.
3
Choose a modern Diffie Hellman group, such as an elliptic curve group, over an older short modular group.
4
Enable perfect forward secrecy so that a single compromised key cannot expose previously captured traffic.

A tunnel configured with weak legacy defaults can technically work for years without complaint, which is exactly why it is worth revisiting an old IPsec Protocol configuration rather than assuming it was set correctly the day it was built.

Why NAT Traversal Exists at All

ESP carries no port number of its own, which means an ordinary NAT device has nothing to rewrite and quietly breaks the connection for more than one client behind the same address.

NAT traversal solves this by wrapping ESP inside an ordinary UDP datagram on port 4500 whenever a NAT device is detected during the key exchange, restoring the port information that NAT needs to function correctly.

Tip
If a tunnel connects fine but data never actually passes, suspect a NAT device somewhere along the path that has not triggered NAT traversal correctly. Checking whether traffic is arriving on UDP port 4500 instead of raw ESP protocol 50 is often the fastest way to confirm the real cause.

Why OT Networks Lean on IPsec for Modbus and DNP3

Modbus TCP and DNP3, despite being two of the most widely deployed protocols in industrial control, were never built with any authentication, integrity checking, or encryption in their original specification.

That gap matters most the moment traffic leaves a trusted local network, such as a remote substation or a wellhead site connecting back to a central control room over a WAN link or the public internet.

A site to site IPsec tunnel closes that gap without touching a single line of RTU or PLC firmware, since the encryption happens entirely at the network layer between two gateways rather than inside the field devices themselves.

This fits neatly into the layered thinking behind the Purdue model, where a firewall and an IPsec tunnel typically sit at the boundary between the control network and anything reaching toward the enterprise or the outside world.

IPsec Protocol Against Newer Alternatives

OpenVPN runs in user space over TLS and is highly configurable, including the ability to run over TCP for traversing restrictive firewalls, but it generally trails IPsec and newer options on raw throughput.

WireGuard takes the opposite approach, using a tiny fixed codebase with modern cryptography baked in rather than negotiated, and it commonly benchmarks with lower latency and higher throughput than a comparable IPsec tunnel.

IPsec still holds its ground for site to site work because of how broadly it is supported across routers, firewalls, and legacy equipment, and because many compliance frameworks common in regulated industries explicitly reference it by name.

Advertisement

What Actually Gets Encrypted Inside a Packet

An unprotected IP packet leaving a remote RTU site normally exposes three things to anyone watching the wire, the source and destination addresses in the IP header, the port numbers in the TCP or UDP header, and the actual data itself.

In tunnel mode, ESP wraps that entire original packet, header and all, inside a brand new outer packet carrying the two VPN gateway addresses instead of the real internal ones.

An observer sitting between the two sites sees only two gateway addresses talking to each other, with no visibility into which internal host sent the traffic, which port it used, or what the payload actually contains.

That outer packet also carries an ESP trailer with padding and an authentication value, so the receiving gateway can confirm nothing was altered in transit before it ever decrypts and forwards the inner packet onward.

A Practical Checklist Before Turning On a Tunnel

1
Confirm both gateways agree on the same encryption, integrity, and Diffie Hellman group before the first negotiation attempt.
2
Define traffic selectors precisely, listing only the subnets that genuinely need to cross the tunnel rather than an overly broad range.
3
Check whether either gateway sits behind a NAT device, and confirm NAT traversal activates correctly during the key exchange.
4
Set a reasonable rekey interval so keys rotate periodically without triggering unnecessary renegotiation traffic.
5
Test failover behavior deliberately, since a tunnel that silently drops during a WAN outage can go unnoticed for hours in a quiet control room.

A tunnel that passes its first connectivity test is not the same thing as a tunnel that has actually been verified end to end, so treat that checklist as a minimum rather than a formality to rush through.

Every one of these checks exists because a misconfigured IPsec Protocol tunnel tends to fail quietly rather than loudly, often passing basic ping tests while still leaving real application traffic blocked or unprotected.

Common Problems That Break an IPsec Protocol Tunnel

Mismatched Phase 1 or Phase 2 proposals are the single most common cause of a tunnel that never comes up at all, since both gateways must agree on every algorithm exactly, not just approximately.

A pre shared key typed with a stray space or a mismatched case on one side will also fail authentication silently, producing log messages that are easy to misread as a completely different problem.

Clock drift between gateways can interfere with certificate based authentication, since a certificate that looks perfectly valid on one device can appear expired or not yet valid on a device with an incorrect system clock.

Overlapping traffic selectors, where two different tunnels both claim the same subnet range, will also cause unpredictable routing behavior that is often mistaken for a security association problem instead of a configuration conflict.

Watch: IPsec Explained

IPsec Protocol Questions Engineers Ask

Why is ESP used far more often than AH?
ESP covers encryption, integrity, and authentication together, while AH only offers integrity and skips confidentiality entirely.
Which mode should a site to site VPN use?
Tunnel mode, since it hides the entire internal packet and allows whole subnets to communicate securely across an untrusted link.
What does the SPI number actually do?
It tells the receiving gateway which security association and decryption key apply to that specific incoming packet.
Why does IPsec need NAT traversal at all?
ESP carries no port number, so a NAT device cannot rewrite it correctly without wrapping it inside UDP port 4500 first.
Is IKEv2 always better than IKEv1?
For new deployments yes, since it negotiates faster, includes native NAT traversal, and resists denial of service attempts better.
Does IPsec protect Modbus traffic by itself?
Yes, since the tunnel encrypts everything at the network layer, protecting Modbus or DNP3 traffic without changing the field devices at all.

Related Articles on This Site

External References

Advertisement

What We Learn Today

  • IPsec Protocol relies mainly on ESP for encryption and integrity together, with AH reserved for rare integrity only cases.
  • Tunnel mode is what protects a full site to site link, hiding entire subnets rather than a single host conversation.
  • A site to site tunnel is the standard way to protect Modbus and DNP3 traffic without touching field device firmware.
I hope you like above blog. There is no cost associated in sharing the article in your social media. Thanks for reading!! Happy Learning!!

Leave a Reply

Your email address will not be published. Required fields are marked *