8 Jul, 2025
Overlay Networking in Azure Local: VXLAN and Encapsulation Walkthrough
VMware , , ,

Table of Contents

  1. Introduction to Overlay Networking
  2. Azure Local SDN Architecture Recap
  3. VXLAN Basics: Format, Function, and Flow
  4. East-West Packet Walkthrough: Same Host
  5. East-West Packet Walkthrough: Cross Host
  6. Ingress & Egress: VM ↔ Physical Network
  7. Wireshark VXLAN Sample Analysis
  8. Troubleshooting Overlay Networking
  9. Conclusion & Key Takeaways

1. Introduction to Overlay Networking

Overlay networking enables virtual workloads to communicate over an abstracted logical network, regardless of physical IP schema or host location. How about we perform a day in the life of a packet? In Azure Local SDN, VXLAN (Virtual Extensible LAN) is used to encapsulate tenant packets, enabling:

  • Isolation across tenants and subnets
  • Flexible IP address reuse
  • Seamless east-west and north-south communication

2. Azure Local SDN Architecture Recap

Core Components

Component Description
Network Controller Central SDN brain; distributes intent-based policy across hosts
Host Agent On-host SDN agent that receives and enforces NC policy
vSwitch Hyper-V Virtual Switch; core data path connecting vNICs to VFP, HNS, NIC
VFP Kernel-mode filtering engine attached to vSwitch
HNS Host Networking Service; configures host-level virtual networks and endpoints
SLB Mux/Agent Software Load Balancer for SNAT, DNAT, and load distribution

Flow Hierarchy on Host (Simplified)

VM vNIC → vSwitch → VFP → HNS → NIC
  • The vSwitch is the bridge between VMs and the physical or overlay network
  • The VFP is injected into the vSwitch as a filtering extension
  • The HNS translates logical network intents into host-applied configurations

3. VXLAN Basics: Format, Function, and Flow

VXLAN Packet Format

VXLAN encapsulates Layer 2 frames inside UDP datagrams:

[Outer Ethernet] → [Outer IP] → [UDP] → [VXLAN Header] → [Inner Ethernet] → [Payload]
  • UDP Port: 4789 (IANA standard for VXLAN)
  • VXLAN Header (8 bytes):
    • Flags: 0x08 (I flag set)
    • VNI (VXLAN Network Identifier): 24-bit logical network ID

Overlay vs Underlay

Plane Format Purpose
Overlay Inner Ethernet/IP (VM packet) Application-layer communication
Underlay VXLAN-encapsulated UDP/IP over physical Cross-host transport over TOR

vSwitch Role

  • Position: Every VM NIC connects to the vSwitch
  • Responsibilities:
    • Port-level ACLs
    • QoS enforcement
    • Encapsulation via VFP extensions
  • Integration: VFP is injected as an extension to the vSwitch pipeline

4. East-West Packet Walkthrough: Same Host

Scenario

Two VMs (VM1 & VM2) are on the same host and virtual subnet.

Flow Sequence

  1. VM1 sends packet → enters vSwitch
  2. VFP applies flow policy (ACLs, QoS, etc.)
  3. No VXLAN encapsulation occurs
  4. Packet exits via vSwitch to VM2’s virtual NIC

ASCII Diagram

VM1
 ↓
vSwitch
 ↓
VFP (ACL/NAT/Bandwidth policies)
 ↓
Local Routing (same vSwitch instance)
 ↓
VM2
  • Encapsulation: ❌ None
  • vSwitch Role: Internal bridging and enforcement
  • VFP Role: Policy decision and enforcement per SDN Controller

5. East-West Packet Walkthrough: Cross Host

Scenario

VM1 on HostA sends a packet to VM2 on HostB.

Flow Sequence

  1. VM1 sends packet → hits vSwitch
  2. VFP applies ACLs, routes the flow to HostB
  3. VXLAN encapsulation added:
    • Outer IP = HostA → HostB
    • UDP Port = 4789
    • VNI = tenant subnet ID
  4. Packet sent via NIC → physical underlay → HostB
  5. HostB decapsulates packet via VFP
  6. Packet sent to VM2 via vSwitch

ASCII Diagram

VM1
 ↓
vSwitch
 ↓
VFP → Encapsulate (VXLAN)
 ↓
NIC → Network → HostB NIC
 ↓
VFP → Decapsulate
 ↓
vSwitch
 ↓
VM2
  • Encapsulation: Required
  • vSwitch Role: Entry/exit point, port policies
  • VFP Role: Encapsulation, routing, policy enforcement

6. Ingress & Egress: VM ↔ Physical Network

6.1 VM → Internet (Egress)

  1. VM packet enters vSwitch
  2. Processed by VFP, NAT performed via SLB Agent
  3. VXLAN-encapsulated packet sent to SDN Gateway
  4. Decapsulated and routed to TOR switch
VM
 ↓
vSwitch → VFP (SNAT)
 ↓
NIC → VXLAN → Gateway
 ↓
Decap → Physical Fabric → Internet

6.2 Internet → VM (Ingress)

  1. Public packet routed to SDN Gateway
  2. Encapsulated with VXLAN and VNI
  3. Sent to correct host
  4. Host decapsulates and routes to VM
Internet
 ↓
SDN Gateway → VXLAN Encapsulation
 ↓
Host NIC → VFP → vSwitch
 ↓
VM
  • vSwitch Role: Final/initial data path for VM
  • VFP Role: NAT, packet filtering, flow control
  • Encapsulation: VXLAN performed by Gateway, interpreted by host

7. Wireshark VXLAN Sample Analysis

Filter

udp.port == 4789

Synthetic Sample Breakdown

Frame 122: 142 bytes on wire
Ethernet II: Src MAC: 00:15:5d:01:02:03 → Dst MAC: 00:15:5d:04:05:06
IP: 192.168.100.10 → 192.168.100.12
UDP: Src Port: 52344 → Dst Port: 4789
VXLAN:
  Flags: 0x08 (Valid VNI)
  VNI: 0x0002fc
Inner Ethernet: 00:15:5d:aa:bb:cc → 00:15:5d:dd:ee:ff
IP: 10.0.0.4 → 10.0.0.5
TCP: HTTP

Notes:

  • Src/Dst IP: Host transport IPs (underlay)
  • Inner Payload: Unchanged L2/L3 packet between VMs
  • VNI: Represents tenant subnet

8. Troubleshooting Overlay Networking

Issue Root Cause Recommended Action
Packet drop (cross-host) MTU too large for VXLAN overhead Adjust MTU to 1450 or enable jumbo
No response from peer VNI mismatch or misconfigured endpoint Validate HNS and NC sync
Ingress never reaches VM NAT rule not applied or wrong public IP Check SLB/NAT rules
Same-host VMs not routing vSwitch ACLs misconfigured Use Get-VfpFlowEntry

Example Command:

Get-VfpFlowEntry -VMName "VM1" | Format-Table FlowName, FiveTuple, Action

9. Conclusion & Key Takeaways

  • VXLAN encapsulation is the foundation for Azure Local SDN overlay networking.
  • vSwitch is the operational hub — bridging vNICs and enabling consistent policy enforcement.
  • VFP acts as the programmable SDN extension inside the vSwitch, handling encapsulation, ACLs, NAT, and telemetry.
  • Cross-host and external flows leverage encapsulation; same-host flows are direct via the vSwitch.

Final Thought:
A deep understanding of how packets traverse Azure Local SDN, from VM to VFP to vSwitch to NIC, empowers architects and admins to build more resilient, scalable, and observable hybrid cloud environments.

*The thoughts and opinions in this article are mine and hold no reflect on my employer*

The post Overlay Networking in Azure Local: VXLAN and Encapsulation Walkthrough first appeared on Digital Thought Disruption.

The post Overlay Networking in Azure Local: VXLAN and Encapsulation Walkthrough appeared first on Digital Thought Disruption.