More Xen Tricks

In my previous post about Xen, I talked about how easy Xen is to configure and set up, particularly on Ubuntu and Debian. I'm still grateful that Xen remains easy; however, I've lately had a few Xen-related challenges that needed attention. In particular, I've needed to create some surprisingly messy solutions when using vif-route to route multiple IP numbers on the same network through the dom0 to a domU.

I tend to use vif-route rather than vif-bridge, as I like the control it gives me in the dom0. The dom0 becomes a very traditional packet-forwarding firewall that can decide whether or not to forward packets to each domU host. However, I recently found some deep weirdness in IP routing when I use this approach while needing multiple Ethernet interfaces on the domU. Here's an example:

Multiple IP numbers for Apache

Suppose the domU host, called webserv, hosts a number of websites, each with a different IP number, so that I have Apache doing something like¹:

Listen 192.168.0.200:80
Listen 192.168.0.201:80
Listen 192.168.0.202:80
...
NameVirtualHost 192.168.0.200:80
<VirtualHost 192.168.0.200:80>
...
NameVirtualHost 192.168.0.201:80
<VirtualHost 192.168.0.201:80>
...
NameVirtualHost 192.168.0.202:80
<VirtualHost 192.168.0.202:80>
...

The Xen Configuration for the Interfaces

Since I'm serving all three of those sites from webserv, I need all those IP numbers to be real, live IP numbers on the local machine as far as the webserv is concerned. So, in dom0:/etc/xen/webserv.cfg I list something like:

vif  = [ 'mac=de:ad:be:ef:00:00, ip=192.168.0.200',
         'mac=de:ad:be:ef:00:01, ip=192.168.0.201',
         'mac=de:ad:be:ef:00:02, ip=192.168.0.202' ]

… And then make webserv:/etc/iftab look like:

eth0 mac de:ad:be:ef:00:00 arp 1
eth1 mac de:ad:be:ef:00:01 arp 1
eth2 mac de:ad:be:ef:00:02 arp 1

… And make webserv:/etc/network/interfaces (this is probably Ubuntu/Debian-specific, BTW) look like:

auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
 address 192.168.0.200
 netmask 255.255.255.0
auto eth1
iface eth1 inet static
 address 192.168.0.201
 netmask 255.255.255.0
auto eth2
iface eth2 inet static
 address 192.168.0.202
 netmask 255.255.255.0

Packet Forwarding from the Dom0

But, this doesn't get me the whole way there. My next step is to make sure that the dom0 is routing the packets properly to webserv. Since my dom0 is heavily locked down, all packets are dropped by default, so I have to let through explicitly anything I'd like webserv to be able to process. So, I add some code to my firewall script on the dom0 that looks like:²

webIpAddresses="192.168.0.200 192.168.0.201 192.168.0.202"
UNPRIVPORTS="1024:65535"

for dport in 80 443;
do
  for sport in $UNPRIVPORTS 80 443 8080;
  do
    for ip in $webIpAddresses;
    do
      /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
        --syn -m state --state NEW \
        --sport $sport --dport $dport -j ACCEPT

      /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
        --sport $sport --dport $dport \
        -m state --state ESTABLISHED,RELATED -j ACCEPT

      /sbin/iptables -A FORWARD -o eth0 -s $ip \
        -p tcp --dport $sport --sport $dport \
        -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
    done  
  done
done

Phew! So at this point, I thought I was done. The packets should find their way forwarded through the dom0 to the Apache instance running on the domU, webserv. While that much was true, I now have the additional problem that packets got lost in a bit of a black hole on webserv. When I discovered the black hole, I quickly realized why. It was somewhat atypical, from webserv's point of view, to have three “real” and different Ethernet devices with three different IP numbers, which all talk to the exact same network. There was more intelligent routing needed.³

Routing in the domU

While most non-sysadmins still use the route command to set up local IP routes on a GNU/Linux host, iproute2 (available via the ip command) has been a standard part of GNU/Linux distributions and supported by Linux for nearly ten years. To properly support the situation of multiple (from webserv's point of view, at least) physical interfaces on the same network, some special iproute2 code is needed. Specifically, I set up separate route tables for each device. I first encoded their names in /etc/iproute2/rt_tables (the number