Networking¶
How VMs are wired together on a node and across the cluster, from an operator's point of view. For the step-by-step CLI workflow see the Networks guide; for where this sits in the wider system see the Architecture overview.
A VM NIC attaches to a network. There are two network types today:
bridge- a plain Linux bridge on a single node. VM tap devices are wired into it, and traffic stays local to that node (plus whatever the bridge is itself uplinked to).overlay- a cross-node layer-2 network. VMs on different hypervisors share one broadcast domain, carried by a VXLAN tunnel that rides an encrypted WireGuard underlay mesh.
Deferred types
VLAN and isolated networks are deferred extensions of the network-type enum,
not separate mechanisms. nat is an egress mode on bridge networks, not a
network type.
Bridge networks¶
For a bridge network the agent programs a Linux bridge on the host and attaches each VM's tap device to it. This is the simplest case: everything is local netlink, no tunnelling. Egress to the outside world is handled by the NAT mode below.
Overlay networks¶
An overlay network gives VMs on different nodes a single L2 segment. Each overlay network has:
- a VNI (VXLAN network identifier) that isolates it from other overlays, and
- a subnet that VM addresses are drawn from.
VM frames are encapsulated in VXLAN and tunnelled between nodes. Crucially, the control plane is authoritative for forwarding:
- The control plane computes the forwarding database (FDB) - the map of "which remote node hosts which MAC" - and distributes it to each agent on the heartbeat. VXLAN address learning is turned off; agents never learn MACs from the wire, they only apply the FDB the control plane hands them.
- A WireGuard mesh carries the overlay between nodes. The control plane recomputes the full peer set on every heartbeat and the agent simply applies it - there is no agent-to-agent peering negotiation.
This keeps the data plane deterministic: the source of truth for who-talks-to-whom lives in one place, and an agent restart re-converges from the next heartbeat.
MTUs¶
The overlay stacks two layers of encapsulation, so inner MTUs are sized down from the physical underlay to avoid fragmentation:
| Layer | MTU | Why |
|---|---|---|
| Physical underlay | 1500 | standard Ethernet |
WireGuard (otwg0) |
1440 | 1500 minus 60 bytes of WireGuard encapsulation (20 IP + 8 UDP + 32 WG) |
| Overlay inner (VM-facing) | 1390 | 1440 minus 50 bytes of VXLAN encapsulation |
If the cluster's underlay MTU is tuned away from 1500, the WireGuard and overlay MTUs are derived from it the same way.
Egress¶
Bridge networks can request managed egress with the nat mode, which installs
an nftables masquerade rule for the network's subnet. VMs then reach external
networks through the host's address.
Host device names¶
So you can recognise Otherix-managed interfaces on a node, the agent uses deterministic names:
| Device | Name | Notes |
|---|---|---|
| Bridge (overlay) | otb<vni> |
e.g. otb1000 for VNI 1000 |
| VXLAN VTEP | otvx<vni> |
the tunnel endpoint, e.g. otvx1000 |
| WireGuard interface | otwg0 |
one interface carries all overlay peers |
| VM tap | ot<12hex> |
ot + the first 12 hex digits of the NIC id |
All Otherix-managed taps share the ot prefix, which is how the agent
enumerates the ones it owns.