More Xen Tricks

In my previous post about Xen, I talked about how easy Xen is to configure and set up, particularly on Ubuntu and Debian. I'm still grateful that Xen remains easy; however, I've lately had a few Xen-related challenges that needed attention. In particular, I've needed to create some surprisingly messy solutions when using vif-route to route multiple IP numbers on the same network through the dom0 to a domU.

I tend to use vif-route rather than vif-bridge, as I like the control it gives me in the dom0. The dom0 becomes a very traditional packet-forwarding firewall that can decide whether or not to forward packets to each domU host. However, I recently found some deep weirdness in IP routing when I use this approach while needing multiple Ethernet interfaces on the domU. Here's an example:

Multiple IP numbers for Apache

Suppose the domU host, called webserv, hosts a number of websites, each with a different IP number, so that I have Apache doing something like¹:

        Listen 192.168.0.200:80
        Listen 192.168.0.201:80
        Listen 192.168.0.202:80
        ...
        NameVirtualHost 192.168.0.200:80
        <VirtualHost 192.168.0.200:80>
        ...
        NameVirtualHost 192.168.0.201:80
        <VirtualHost 192.168.0.201:80>
        ...
        NameVirtualHost 192.168.0.202:80
        <VirtualHost 192.168.0.202:80>
        ...
        

The Xen Configuration for the Interfaces

Since I'm serving all three of those sites from webserv, I need all those IP numbers to be real, live IP numbers on the local machine as far as the webserv is concerned. So, in dom0:/etc/xen/webserv.cfg I list something like:

        vif  = [ 'mac=de:ad:be:ef:00:00, ip=192.168.0.200',
                 'mac=de:ad:be:ef:00:01, ip=192.168.0.201',
                 'mac=de:ad:be:ef:00:02, ip=192.168.0.202' ]
        

… And then make webserv:/etc/iftab look like:

        eth0 mac de:ad:be:ef:00:00 arp 1
        eth1 mac de:ad:be:ef:00:01 arp 1
        eth2 mac de:ad:be:ef:00:02 arp 1
        

… And make webserv:/etc/network/interfaces (this is probably Ubuntu/Debian-specific, BTW) look like:

        auto lo
        iface lo inet loopback
        auto eth0
        iface eth0 inet static
         address 192.168.0.200
         netmask 255.255.255.0
        auto eth1
        iface eth1 inet static
         address 192.168.0.201
         netmask 255.255.255.0
        auto eth2
        iface eth2 inet static
         address 192.168.0.202
         netmask 255.255.255.0
        

Packet Forwarding from the Dom0

But, this doesn't get me the whole way there. My next step is to make sure that the dom0 is routing the packets properly to webserv. Since my dom0 is heavily locked down, all packets are dropped by default, so I have to let through explicitly anything I'd like webserv to be able to process. So, I add some code to my firewall script on the dom0 that looks like:²

        webIpAddresses="192.168.0.200 192.168.0.201 192.168.0.202"
        UNPRIVPORTS="1024:65535"
        
        for dport in 80 443;
        do
          for sport in $UNPRIVPORTS 80 443 8080;
          do
            for ip in $webIpAddresses;
            do
              /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
                --syn -m state --state NEW \
                --sport $sport --dport $dport -j ACCEPT
        
              /sbin/iptables -A FORWARD -i eth0 -p tcp -d $ip \
                --sport $sport --dport $dport \
                -m state --state ESTABLISHED,RELATED -j ACCEPT
        
              /sbin/iptables -A FORWARD -o eth0 -s $ip \
                -p tcp --dport $sport --sport $dport \
                -m state --state NEW,ESTABLISHED,RELATED -j ACCEPT
            done  
          done
        done
        

Phew! So at this point, I thought I was done. The packets should find their way forwarded through the dom0 to the Apache instance running on the domU, webserv. While that much was true, I now have the additional problem that packets got lost in a bit of a black hole on webserv. When I discovered the black hole, I quickly rea