PXE boot not working in virtual environment

The warning may be due to a related but separate problem: we get multiple wsgi processes. The system that tracks and displays persistent errors does not share its state between those processes.

So to diagnose the pxeboot problem we'll still need to know what other error you're hitting.

Why not try it?
This is trivially reproducible on installation and actual attempt to do something.
http://bazaar.launchpad.net/~smoser/maas/maas-pkg-test/view/head:/maas-ephemeral-test-quantal.txt

ok. it appears that the pxe boot issue might be user configuration error.
I'm not sure if it was my own misconfiguration or maas (or maas packaging) that left 'next-server' with a bad IP.

I'll dig more on that tomorrow.

OK. more info.
bug 1047061 made changes to 'next-server' setting to make it not be '127.0.0.1' (which was clearly broken), but now I get the IP address of the maas server, which for my use case was not sufficient.

Also, we recently started being required to pass '--ip' to 'maas config_master_dhcp'. I added a change to packaging to specify that, and had assumed that that would render into 'next-server' in the config. It appears not, and I'm not sure what that value is actually used for.

The end result is that right now for the networking described above, the pxe booted instances cannot reach 'next-server' although they can reach the dhcp server and the interface configured.

| subnet 192.168.77.0 netmask 255.255.255.0 {
| next-server 10.55.60.130;
| filename "pxelinux.0";
| option subnet-mask 255.255.255.0;
| option broadcast-address 192.168.77.255;
| option domain-name-servers 10.55.60.130;
| option routers 192.168.77.1;
| range dynamic-bootp 192.168.77.5 192.168.77.200;
|}

10.55.60.130 (the address of 'eth0') is better than 127.0.0.1, but is still
wrong.

Why is this address wrong? The standard package install puts a TFTP server on the same host as everything else and its listening address should match the appserver's listening address (they both need to be contacted by nodes).

I'm not exactly sure why it doesn't work, but in the test scenario that I'm using, it does not work.
I believe what is happening is that my networking isn't getting the tftp packets from the network that dhcpd is running on are not getting to the maas tftp server.

You can reproduce this easily enough given the link above (maas-ephemeral-test-quantal.txt).

Either way, we need to be able to specify the IP address of the 'next-server', and it is not necessarily the same as the maas primary server.

For more information, here is what the kvm console shows during failure:
 | DHCP (net0 52:54:00:12:34:01)...... ok
 | net0: 192.168.77.5/255.255.255.0 gw 192.168.77.1
 | Next server: 192.168.1.131
 | Filename: pxelinux.0
 | tftp://192.168.1.131/pxelinux.0... ok
 | !PXE entry point found (we hope) at 9A64:0456 via plan A
 | UNDI code segment at 9A64 len 08A8
 | UNDI data segment at 9AEF len 2D08
 | Getting cached packet 01 02 03
 | My IP address seems to be C0A84D05 192.168.77.5
 | ip=192.168.77.5:192.168.1.131:192.168.77.1:255.255.255.0
 | BOOTIF=01-52-54-00-12-34-01
 | SYSUUID=00000000-0000-0000-0000-000000000000
 | TFTP prefix:
 | Trying to load: pxelinux.cfg/00000000-0000-0000-0000-000000000000

So it seems like it does get the pxelinux.0 file, but something goes wrong after that. For comparison, after I change '192.168.1.131' (the IP of eth0) to '192.168.77.1', I see:

 | DHCP (net0 52:54:00:12:34:01)...... ok
 | net0: 192.168.77.5/255.255.255.0 gw 192.168.77.1
 | Next server: 192.168.77.1
 | Filename: pxelinux.0
 | tftp://192.168.77.1/pxelinux.0... ok
 | !PXE entry point found (we hope) at 9A64:0456 via plan A
 | UNDI code segment at 9A64 len 08A8
 | UNDI data segment at 9AEF len 2D08
 | Getting cached packet 01 02 03
 | My IP address seems to be C0A84D05 192.168.77.5
 | ip=192.168.77.5:192.168.77.1:192.168.77.1:255.255.255.0
 | BOOTIF=01-52-54-00-12-34-01
 | SYSUUID=00000000-0000-0000-0000-000000000000
 | TFTP prefix:
 | Trying to load: pxelinux.cfg/01-52-54-00-12-34-01 ok
 | Booting (amd64) under MAAS direction...
 | nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:maas-precise-12.04-amd64-ephemeral-20120424 iscsi_target_ip=192.168.1.131 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist ro root=LABEL=cloudimg-rootfs overlayroot=tmpfs cloud-config-url=http://192.168.1.131/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed log_host=192.168.1.131 log_port=514 console=tty1 console=ttyS0
 | Booting (i386) under MAAS direction...
 | nomodeset iscsi_target_name=iqn.2004-05.com.ubuntu:maas:maas-precise-12.04-i386-ephemeral-20120424 iscsi_target_ip=192.168.1.131 iscsi_target_port=3260 iscsi_initiator=maas-enlist ip=::::maas-enlist ro root=LABEL=cloudimg-rootfs overlayroot=tmpfs cloud-config-url=http://192.168.1.131/MAAS/metadata/latest/enlist-preseed/?op=get_enlist_preseed log_host=192.168.1.131 log_port=514 console=tty1 console=ttyS0
 | Loading amd64/generic/precise/commissioning/linux.......
 | Loading amd64/generic/precise/commissioning/initrd.gz...

Hi Scott

On Tuesday 18 September 2012 13:41:39 you wrote:
> I'm not exactly sure why it doesn't work, but in the test scenario that I'm
> using, it does not work. I believe what is happening is that my networking
> isn't getting the tftp packets from the network that dhcpd is running on
> are not getting to the maas tftp server.

I think we need to find out why it doesn't work before jumping to conclusions,
because ...

>
> You can reproduce this easily enough given the link above (maas-
> ephemeral-test-quantal.txt).
>
> Either way, we need to be able to specify the IP address of the 'next-
> server', and it is not necessarily the same as the maas primary server.

... the TFTP server will be running on the same IP as the API does, because
this is how the package installs it. If the node cannot contact TFTP then it
will also not be able to contact the API for metadata.

So "it is not necessarily the same as the maas primary server" is not true,
they *have* to be the same.

but it *can* get to it. Pay attention to the logs above and you'll see that the tftp works in both cases:
 | Filename: pxelinux.0
 | tftp://192.168.1.131/pxelinux.0... ok

Scott, you're going to have to help me more here because I cannot fathom what your problem is. I don't really understand how you've set this up so it would greatly help to have some sort of networking overview (in English) of your machine that hosts the MAAS stuff.

Specifically:
> I get the IP address of the maas server, which for my use case was not sufficient.
Why is it not sufficient?

> 10.55.60.130 (the address of 'eth0') is better than 127.0.0.1, but is still wrong.
Why is it wrong?

Are you running your own DHCP server? If so why is it on a different IP to the TFTP server when the default setup for MAAS is that they are both on the same address as installed by the package?

Did you set the DEFAULT_MAAS_URL to be the *internal* address that the nodes can access? If it's not, that will be causing problems a bit like the ones you see.

Thanks for helping me work out where the root problem is.

On Thu, 20 Sep 2012, Julian Edwards wrote:

> Scott, you're going to have to help me more here because I cannot fathom
> what your problem is. I don't really understand how you've set this up
> so it would greatly help to have some sort of networking overview (in
> English) of your machine that hosts the MAAS stuff.

I pointed to a script that will reproduce this environment for you.
Nothing I say in english is going to be so useful.

That said:
 eth0: normal network interface (only one on the system ip=$ETH0_ADDR)
 maasbr0: a linux bridge (ip=192.168.77.1) configured via libvirt, NAT is
   enabled. libvirt is running a caching only DNS server on the interface,
   but no dhcp.
   maas-dhcp is configured to run on that network interface, but insists
   on using $ETH0_ADDR for the 'next-server'.

kvm Instances have one nic attached to the 'maasbr0' bridge via tap, and
pxe boot from the maas-dhcp.

> Specifically:
> > I get the IP address of the maas server, which for my use case was not sufficient.
> Why is it not sufficient?

Because it doesn't work. pxe boot fails. It gets its IP, gets the
pxelinux.0 file, and then tries to get a file named 0000-0000-0000 (or
something) and times out.

> > 10.55.60.130 (the address of 'eth0') is better than 127.0.0.1, but is still wrong.
> Why is it wrong?

I dont know.

Additionally interesting is the fact that I'm telling maas-dhcp that its
IP should be 192.168.77.1 (via debconf) but i'm not sure what it does with
'maas-addr' that is given to it.

> Are you running your own DHCP server? If so why is it on a different IP
> to the TFTP server when the default setup for MAAS is that they are both
> on the same address as installed by the package?
>
> Did you set the DEFAULT_MAAS_URL to be the *internal* address that the
> nodes can access? If it's not, that will be causing problems a bit like
> the ones you see.

But clearly that is an insufficent solution. I one would need maas to
listen on 2 networks on a system that bridges 2 networks. If it only
listens on one, then I have to choose between being able to see the web UI
and interact with the maas api from outside the network or having the
nodes function with metadata.

> Thanks for helping me work out where the root problem is.

On Thursday 20 September 2012 00:45:48 you wrote:
> But clearly that is an insufficent solution. I one would need maas to
> listen on 2 networks on a system that bridges 2 networks. If it only
> listens on one, then I have to choose between being able to see the web UI
> and interact with the maas api from outside the network or having the
> nodes function with metadata.

We are not supporting multiple network interfaces in the precise branch
(although you can configure it manually), but quantal will soon deal with it.

I'll look into your setup in more detail later, it's not obvious what's going
on from a casual glance (at least to me it's not).

On 20 September 2012 01:45, Scott Moser <email address hidden> wrote:
...
> Because it doesn't work. pxe boot fails. It gets its IP, gets the
> pxelinux.0 file, and then tries to get a file named 0000-0000-0000 (or
> something) and times out.

This sounds like it could be a bug in PXELINUX, perhaps owing to the
virtualised environment. Unless you're seeing this in a non-virt
environment, why don't we put this on the back burner, for now?
Virtualised environments are not a target audience for 12.10.

On Thu, 20 Sep 2012, Gavin Panella wrote:

> On 20 September 2012 01:45, Scott Moser <email address hidden> wrote:
> ...
> > Because it doesn't work. pxe boot fails. It gets its IP, gets the
> > pxelinux.0 file, and then tries to get a file named 0000-0000-0000 (or
> > something) and times out.
>
> This sounds like it could be a bug in PXELINUX, perhaps owing to the
> virtualised environment. Unless you're seeing this in a non-virt
> environment, why don't we put this on the back burner, for now?
> Virtualised environments are not a target audience for 12.10.

It could be a bug in pxelinux, i agree. But I would doubt it has anything
to do with virtualization.

It feels more likely to me that it would be present in any network setup
like this (see pastebin at http://paste.ubuntu.com/1216671/ monospace font
for ascii art):

                    ___________________
                    | MAAS System |
 |NETWORK A| <=> | eth0=10.55.60.130 |
 10.0.1.0/24 | 192.168.77.1=eth1 | <=> [NETWORK B]
                    -------------------- 192.168.1.0/24
                                           Node1, Node2, Node3...

  - Maas Server listens on eth0 10.55.60.130 address.
    It's tftp server runs on this address.
  - Maas dhcp is told to use eth1 as its dhcp interface.
  - Maas Server configures dhcp 'next-server' to be 10.55.60.130.
  - In actual implementation with libvirt, 'eth1' is 'maasbr0' above.
  - Network B has NAT only access to network A.
  - Network A does not have direct access to Network B
  - I'm using xkvm wrapper script to more easily do tap devices
    to the 'maasbr0' bridge.
  - The pxe boot working and non-working output are shown at
    https://bugs.launchpad.net/maas/+bug/1051626/comments/7

Thats basically what I'm modelling. This seems like a common "real life"
scenario. No?

On 20 September 2012 14:02, Scott Moser <email address hidden> wrote:
> It feels more likely to me that it would be present in any network setup
> like this (see pastebin at http://paste.ubuntu.com/1216671/ monospace font
> for ascii art):
>
>                     ___________________
>                     |      MAAS System  |
>  |NETWORK A|   <=>  | eth0=10.55.60.130 |
>  10.0.1.0/24        | 192.168.77.1=eth1 | <=> [NETWORK B]
>                     --------------------     192.168.1.0/24
>                                            Node1, Node2, Node3...
>
>   - Maas Server listens on eth0 10.55.60.130 address.
>     It's tftp server runs on this address.

Fwiw, the TFTP server runs on all interfaces. There's no way to select
only a single interface right now.

>   - Maas dhcp is told to use eth1 as its dhcp interface.
>   - Maas Server configures dhcp 'next-server' to be 10.55.60.130.
>   - In actual implementation with libvirt, 'eth1' is 'maasbr0' above.
>   - Network B has NAT only access to network A.
>   - Network A does not have direct access to Network B

I've not been part of any discussion of MAAS running with anything
other than a fully routable network, i.e. the region controllers,
cluster controllers, and nodes can all route to one another. I don't
think this is a supported network layout, for 12.10 at least.

>   - I'm using xkvm wrapper script to more easily do tap devices
>     to the 'maasbr0' bridge.

I don't know what this means :)

>   - The pxe boot working and non-working output are shown at
>     https://bugs.launchpad.net/maas/+bug/1051626/comments/7
>
> Thats basically what I'm modelling.  This seems like a common "real life"
> scenario. No?

It sounds like something we could support for 13.04, if there's demand
for it. I guess it's a nice layout for a seed install.

Well, in an attempt to debug, I downloaded pxelinux.0 from hardy, lucid, natty, oneiric, precise, and tried each of them.

hardy syslinux_3.53-1ubuntu2.1_i386.deb
lucid syslinux_3.63+dfsg-2ubuntu3_i386.deb
natty syslinux-common_4.02+dfsg-7ubuntu1_all.deb
oneiric syslinux-common_4.04+dfsg-1ubuntu1_all.deb
precise syslinux-common_4.05+dfsg-2_all.deb
quantal syslinux-common_4.05+dfsg-6_all.deb

The same issue reproduces on each.

It does seem to be a networking issue at this point, given some tcpdump i've done on the maasbr0 interface in both working and non-working cases, but i haven't had time to more deeply debug that.

I see no fundamental reason why this setup won't work; in fact I have
something similar myself (but not virtual). I strongly suspect a kvm or
pxelinux bug.

On Thursday 20 September 2012 13:02:46 Scott Moser wrote:
> - Maas Server configures dhcp 'next-server' to be 10.55.60.130.

It only does this because you set DEFAULT_MAAS_URL to the wrong thing. You
need to set it to the *INTERNAL* address (192.168.77.1) as I previously
mentioned.

This means the nodes will all access MAAS via the internal IP.

External access will work via the "A" network regardless of what
DEFAULT_MAAS_URL is. DEFAULT_MAAS_URL only controls what the nodes talk to.

Reviewing this again. Does setting DEFAULT_MAAS_URL to the internal IP resolve the problem?

Hey guys, first post here and hope this is helpful.

I've been struggling with this issue (or related) for three days now. I suspected some kind of iptables issue because I kept getting a PXE timeout and for some reason the TFTP server for MAAS was only reachable via the localhost. Was my traffic dropped? Couldn't figure it out for the life of me. It was bizzarre.

The MAAS server is running in a Windows domain, (thus not serving DHCP) but setting the option 66/67 in the DHCP server worked just fine. The server seemed to be functioning and I could commission nodes with no issue (VirtualBox setup for all the nodes) but they would never move to the 'Ready' state. After checking all the PXE settings I could find for the TFTP server and pulling my hair out because the nodes wouldn't commission. I checked the PSERV (service maas-pserv status) it was running. I restarted it...still nothing.

In looking into possibly iptables issues, the freenode#Netfilter guys said something that triggered a thought, the TFTP server should be listening on the server IP, not just the localhost. A quick check of netstat showed...it was only running on localhost. Let's try a full-restart of the service. Boom. Listening on the correct IP, nodes commission and move to ready, music plays as the clouds part.

tl;dr - There seems to be an issue with the maas-pserv. It requires a service full-restart to begin functioning, allow nodes to commission and move to ready.

@Tony, what version of MAAS are you using?

Is the MAAS server in a different node from the nodes maas is trying to commission, separated with an physical network? remotely, can you connect to maas like:

:~s$ tftp <any-ip-of-maas>
tftp> get pxelinux.0
tftp>

If you see a similar output, then when you close the tftp console, you should see pxelinux.0 ther. If this is the case, it means that MAAS tftp server is reachable from the network, and the issue is either VBox or the Network.

Cheers.

On 13 June 2013 22:02, Tony Shireman <email address hidden> wrote:
> tl;dr - There seems to be an issue with the maas-pserv. It requires a
> service full-restart to begin functioning, allow nodes to commission and
> move to ready.

I think I know what this is. From maasserver.plugin:

  # Create a UDP server individually for each discovered network
  # interface, so that we can detect the interface via which we have
  # received a datagram.

The address on which the TFTP was contacted is passed to the region
controller and is used when composing kernel arguments. However,
listening on all interfaces, specified by 0.0.0.0 iirc, gives 0.0.0.0
for the incoming address, which is not useful. The solution
implemented was to start a TFTP service (all running in the same
process) for each network interface.

This is done once, at start-up, so if pserv is started early, before
all interfaces on the machine are configured, it will miss them and
not start a service on that interface's address.

Assuming this is the cause, one solution here might be to use upstart
to automatically restart pserv when a network interface comes up or
down. Indeed, you may be able to formulate an upstart script to
respond to net-device-up and net-device-down, restarting maas-pserv
each time, to workaround this bug for now.

Another solution might be to periodically check network interfaces
within pserv, and start or stop TFTP services as necessary. The
upstart solution would respond more rapidly to change, but this
solution would avoid downtime.

Perhaps a combination would work well: the new (or modified
maas-pserv) upstart job could send SIGHUP to maas-pserv, which could
then rescan network interfaces.

I'll file a bug.

Gavin.

[Expired for MAAS because there has been no activity for 60 days.]