Optimizing SMB Performance Over IPSec Tunnels with FortiGate and Cloudflare ZTNA

The Server Message Block (SMB) protocol is widely used in enterprise environments for file sharing. However, SMB is known for being extremely sensitive to network latency and fragmentation, making it a challenging protocol to optimize over wide-area networks (WANs), especially when IPSec tunnels, cloud security platforms like Cloudflare ZTNA, and next-generation firewalls such as FortiGate are involved. This document outlines best practices and field-proven techniques to optimize SMB performance in such environments.

Cloudflare-Fortigate

1. Challenges of Running SMB Over IPSec and ZTNA

SMB, particularly in versions v2/v3, relies on numerous small transactions and round-trip interactions between client and server. This "chatty" nature means that latency, packet fragmentation, and inspection overhead have a disproportionate impact on performance.

When using IPSec tunnels (e.g., FortiGate to Cloudflare Magic WAN), the tunnel overhead reduces the available MTU, increasing the likelihood of packet fragmentation. Similarly, ZTNA solutions may introduce additional latency, especially if they proxy or inspect SMB traffic.

2. MTU and MSS Tuning

A foundational step to mitigate fragmentation is adjusting the MTU and MSS values:

• Determine actual path MTU: Use tools like ping -f -l to test the maximum unfragmented packet size.

• Set tunnel MTU: On FortiGate IPSec interfaces, set the MTU to a conservative value like 1280 bytes to prevent fragmentation inside the tunnel:

  
  

config vpn ipsec phase1-interface 

edit <tunnel_name> 

set mtu 1280 

next 

end

• Clamp MSS: In FortiGate policies, explicitly set the TCP Maximum Segment Size (MSS) to MTU - 40 (e.g., 1240 for a 1280-byte MTU):

  
  

config firewall policy edit <policy_id>

set tcp-mss-sender 1240

set tcp-mss-receiver 1240 

next 

end

This ensures endpoints send TCP packets that will not exceed the tunnel's capacity, preventing fragmentation.

3. Ensure PMTUD is Functional

Path MTU Discovery (PMTUD) allows systems to automatically adjust packet sizes based on ICMP "Fragmentation Needed" messages. To support PMTUD:

• Enable ICMP Unreachable on FortiGate policies:

set icmp-unreachable enable

• Allow ICMP in firewall policies to and from IPSec interfaces.

• Verify that PMTUD is functioning using tools like ping -f -l and inspect ICMP behavior with diag debug flow.

  
  

4. Addressing Latency Sensitivity

Latency has a direct impact on SMB due to its sequential transaction pattern. Even with fragmentation eliminated, high round-trip times (RTTs) can cause significant delays:

• A 5MB PowerPoint file might open in 45 seconds over a 20ms RTT but take 4 minutes over a 90ms RTT.

• SMB's dependency on sequential request/response cycles means that each RTT compounds the delay.

  
  

Recommendations:

• Use SMB alternatives (if possible): WebDAV, SFTP, or RDP drive mapping.

• For persistent SMB use, prefer MPLS, local caching (BranchCache, Offline Files), or RDP-based access.

• Enable SMB Over QUIC if using Windows 11/Server 2022 to benefit from UDP and multiplexing.

  
  

5. Disable SMB Signing and Windows TCP Delays (if policy permits)

• SMB Signing can double the round-trip load per file operation. Disable if security policies allow:

  Set-SmbServerConfiguration -EnableSecuritySignature $false

• TCP Delay ACK & Nagle Algorithm: Use registry settings or PowerShell to disable these, improving responsiveness:

  New-ItemProperty -Path "HKLM:\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\{NIC-ID}" -Name "TcpAckFrequency" -Value 1 -PropertyType DWORD

  
  

6. Avoid L7 Inspection and Traffic Shaping for SMB

SMB traffic should not be subjected to Layer 7 deep inspection or security scanning, which can introduce unnecessary delays:

• Ensure FortiGate policies handling SMB traffic do not use SSL inspection, antivirus, or web filtering.

• Prioritize SMB traffic in traffic shaping policies if applicable.

  
  

Conclusion

Optimizing SMB over FortiGate-managed IPSec tunnels and Cloudflare ZTNA requires a multi-layered approach. By aligning MTU/MSS configurations, enabling PMTUD, tuning TCP behavior, and avoiding inspection overhead, organizations can significantly reduce latency and improve the end-user experience. While SMB is inherently sensitive to WAN conditions, careful tuning can mitigate the majority of performance bottlenecks.

Thanks For Reading.