By Ryan Tucker, Solutions Architect

Weigh up three popular ways to protect your data in transit—host level encryption (TLS/SSH), MACsec, and IPsec—to fortify your network security.

Encryption in transit refers to protecting data while it moves from one place to another, in contrast to “encryption at rest” which is used to protect data where it is stored. In this blog we’ll look at a few common options for encryption in transit, and the pros and cons of each from a network perspective.

Although there are many options available for encrypting data as it moves over a network, by far the most common are MACsec, IPsec, and what we’ll group together as host level encryption (most commonly Transport Layer Security, or TLS). Each of these approaches has its own advantages and disadvantages, and each operates at a different level of the network stack.

Using the OSI Model, MACsec operates at Layer 2, IPsec operates at Layer 3, and host level covers anything that operates at Layers 4-7. The reason for this is that Layer 4-7 encryption (including TLS and SSH) is typically handled within the clients and servers rather than at the network level.

Megaport works with customers using various combinations of all of these protocols, and many others.

Host level encryption

While host level encryption can be any kind of encryption implemented at Layers 4 through 7, in practice it is most commonly TLS (Transport Layer Security) and sometimes SSH (Secure Shell). Some applications may also use their own internal encryption schemes, for example the Signal Protocol used by several Instant Messaging services, but these are typically delivered inside of TLS and are transparent to the underlying network.

TLS, which may sometimes be referred to as SSL (Secure Sockets Layer), is by far the most common approach. This is the type of encryption that protects websites using HTTPS or QUIC, emails using SMTPS, file transfers using FTPS, Zero Trust solutions, some remote user VPNs, and various other services.

On top of encryption, TLS also provides identification, authentication, and data integrity and is now so common that many applications, including web browsers, will show warnings or fail to connect if a secure connection is not available.

One of the advantages of encryption being built into the clients and servers rather than handled by the network is that the data remains encrypted for as much of the path as possible. This is the goal of end-to-end encryption, where data is encrypted by the sender and decrypted only by the recipient. Each session with each client maintains its own keys and the data stays protected all the way from the client to the server.

Another advantage of this design is that the work of encrypting the data is distributed among all clients, making it very efficient and scalable as well as enabling other modern security features like certificate based identity and authentication.

Fortunately, almost all applications now use encrypted protocols and/or application level encryption. If this is the case in your environment, then you may not need to perform any separate network level encryption as the data will already be protected.

This application level encryption can simplify network design while maintaining the highest performance, scalability, and security. The catch is that this can be difficult or sometimes impossible to retrofit to existing legacy systems and applications, and needs to be managed at the application level rather than within the network.

IPsec

In some cases, you may be required to encrypt traffic between endpoints in an IP network. IPsec is the most common solution for this.

IPsec operates at Layer 3 and is often deployed to create a tunnel between two network devices over an untrusted network such as the internet. This is the technology used by most VPNs, although some do use TLS or other modern protocols like WireGuard. IPsec is a flexible and widely used solution that can create a secure path across multiple IP networks without requiring any changes to the applications using that path.

The main advantage of IPsec is its ubiquity and flexibility. Most routers and firewalls, and almost all service providers, support it as a way of establishing a secure path, and it can work across almost any IP network. The downside is that it can be complex to configure and troubleshoot, and there are significant performance overheads.

IPsec can present some implementation challenges for both the tunnel itself and the broader network architecture, and it should be noted that there are some configurations which may work but are no longer considered secure.

To successfully establish an IPsec connection both ends have to negotiate the details of the session, protocol parameters, authentication and encryption algorithms, subnets, etc. Then once the tunnel is established it has to be integrated into the routing design of the network, which can be complex if the same subnets are available inside and outside of the tunnel. The main drawback of IPsec, though, is the performance overhead. There are two separate ways in which IPsec can reduce the performance of network traffic:

1. The first factor is the performance of the encryption process itself. As the encryption is done by firewalls or routers as traffic passes through, the speed at which the device or service can encrypt packets is often a limiting factor. The total throughput depends on many variables including the algorithms used, device capacity and architecture, hardware acceleration, and packet size. This is often a constraint with cloud provider VPN services which are commonly restricted to around 1Gbps per tunnel.
2. The second way that IPsec can impact performance is by reducing maximum packet size due to the additional headers required. The headers themselves consume bandwidth, but also increase fragmentation which can reduce maximum transmission speeds further. The specifics of this depend heavily on the configuration of the tunnel and the traffic being sent over it, but an overhead of around 10% can be expected in typical deployments.

MACsec

Operating at Layer 2, MACsec is another approach to implementing encryption within a network. The MACsec header is inserted directly under the source and destination MAC address in the Ethernet Frame, and encrypts everything below that point including VLAN tags and IP headers.

This process is most commonly performed in dedicated hardware and therefore can be extremely fast, which is the primary advantage of MACsec for point-to-point connections. MACsec is able to maintain wire-speed performance while providing strong security. This makes it suitable for environments that require high throughput and low latency, such as data centers and high-performance computing networks.

However, its reliance on hardware support means that implementing MACsec can require new equipment and infrastructure investment. Although MACsec is a standard, not all devices implement it in the same way or support the same extensions, so interoperability has to be validated.

As the encryption is managed at the Ethernet layer, MACsec can only be used between devices that are directly connected via a Layer 2 path. It can’t be used across other networks or routed through multiple hops (without external encapsulation) and is typically used to handle high-speed encryption between two switches that are directly plugged into each other, with some devices supporting MACsec at up to 400Gbps.

This single-hop encryption is conceptually the opposite of end-to-end encryption, as the traffic is decrypted as it reaches each new network and must be re-encrypted by the next device. This prevents the source and destination hosts from validating the integrity of the path, and can be complex to manage as the number of connections and hops increases.

Conclusion

There are many options for encrypting traffic as it moves across a network – host level encryption, MACsec, and IPsec are just the most common. These three approaches operate at different levels of the network stack and each have their own pros and cons.

MACsec is very fast but limited in scope and requires supported hardware. IPsec is ubiquitous and flexible but has significant performance overheads. TLS is scalable and provides end-to-end protection of the connection, but can be difficult to add to existing systems.

In most cases the best solution is to ensure all applications natively use secure protocols. This provides the broadest protection, allows for other modern security features, is scalable, and reduces network complexity. If this isn’t an option, IPsec can be used in most situations to create a secure tunnel over which to send data. If you require very high throughput between two directly connected devices, MACsec will provide the highest performance.