cloud images in lxc get ipv6 address

Bug #1732002 reported by Scott Moser
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
nplan (Ubuntu)
Fix Released
Undecided
Unassigned
Xenial
Fix Released
Undecided
Unassigned
Artful
Fix Released
Undecided
Unassigned
systemd (Ubuntu)
Invalid
Undecided
Unassigned

Bug Description

[Impact]
All users of netplan.

[Test case]

== LXD containers ==
1) Start an LXD container (artful or bionic)
2) Verify that an IPv6 address is present
3) Verify that the system is brought up in a reasonable time (does not wait 2 minutes to be reachable).

== servers / cloud instances ==
1) Start a bionic system, with no IPv6 connectivity (no router to advertise a prefix)
2) Verify that the system boots quickly (does not wait 2 minutes to be reachable).

[Regression potential]
Since this changes default IPv6 behavior, care should be taken to validate that systems are maintaining IPv6 connectivity when it is available, and similarly that systems where no IPv6 connectivity is available on the network are behaving correctly: there should not be long boot delays, and no extra IPv6 addresses aside from link-local addresses generated by the kernel.

--

I noticed that lxd (lxc list) reports that an lxc container has an ipv6 address in artful or bionic. It does not list this in xenial or zesty. I suspect this change occurred in the switch over to netplan/networkd.

This may at first seem harmless or even desired, but note that the user configuration did not request ipv6 config, so its presence is a bug.

$ for rel in xenial zesty artful bionic; do
   lxc launch ubuntu-daily:$rel $rel-demo; done
Creating xenial-demo
Starting xenial-demo
..
Creating bionic-demo
Starting bionic-demo

$ sleep 10
$ lxc list
$ lxc list
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+
| artful-demo | RUNNING | 10.75.205.208 (eth0) | fd42:eee5:7c43:3d62:3a42:611c:3f6f:1184 (eth0) | PERSISTENT | 0 |
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+
| bionic-demo | RUNNING | 10.75.205.187 (eth0) | fd42:eee5:7c43:3d62:6f4:155b:39cc:fc3d (eth0) | PERSISTENT | 0 |
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+
| xenial-demo | RUNNING | 10.75.205.143 (eth0) | | PERSISTENT | 0 |
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+
| zesty-demo | RUNNING | 10.75.205.123 (eth0) | | PERSISTENT | 0 |
+-------------+---------+----------------------+------------------------------------------------+------------+-----------+

## Here is the config that was provided by lxd
$ lxc exec bionic-demo cat /var/lib/cloud/seed/nocloud-net/network-config
version: 1
config:
    - type: physical
      name: eth0
      subnets:
          - type: dhcp
            control: auto

## Here is the config that cloud-init rendered.
$ lxc exec bionic-demo -- grep -v '^#' /etc/netplan/50-cloud-init.yaml
network:
    version: 2
    ethernets:
        eth0:
            dhcp4: true

$ lxc exec bionic-demo cat /run/systemd/network/10-netplan-eth0.network
[Match]
Name=eth0

[Network]
DHCP=ipv4

[DHCP]
UseMTU=true
RouteMetric=100

$ lxc exec bionic-demo -- systemctl status --no-pager --full systemd-networkd
● systemd-networkd.service - Network Service
   Loaded: loaded (/lib/systemd/system/systemd-networkd.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2017-11-13 18:37:34 UTC; 8min ago
     Docs: man:systemd-networkd.service(8)
 Main PID: 118 (systemd-network)
   Status: "Processing requests..."
    Tasks: 1 (limit: 4915)
   Memory: 2.0M
      CPU: 19ms
   CGroup: /system.slice/systemd-networkd.service
           └─118 /lib/systemd/systemd-networkd

Nov 13 18:37:34 bionic-demo systemd[1]: Starting Network Service...
Nov 13 18:37:34 bionic-demo systemd-networkd[118]: eth0: Gained IPv6LL
Nov 13 18:37:34 bionic-demo systemd-networkd[118]: Enumeration completed
Nov 13 18:37:34 bionic-demo systemd[1]: Started Network Service.
Nov 13 18:37:37 bionic-demo systemd-networkd[118]: eth0: DHCPv6 address fd42:eee5:7c43:3d62:6f4:155b:39cc:fc3d/128 timeout preferred 3600 valid 3600
Nov 13 18:37:37 bionic-demo systemd-networkd[118]: eth0: DHCPv4 address 10.75.205.187/24 via 10.75.205.1
Nov 13 18:37:37 bionic-demo systemd-networkd[118]: Not connected to system bus, ignoring transient hostname.
Nov 13 18:37:39 bionic-demo systemd-networkd[118]: eth0: Configured
Nov 13 18:38:09 bionic-demo systemd-networkd[118]: Could not set hostname: Method call timed out

ProblemType: Bug
DistroRelease: Ubuntu 18.04
Package: nplan 0.30
ProcVersionSignature: Ubuntu 4.13.0-16.19-generic 4.13.4
Uname: Linux 4.13.0-16-generic x86_64
ApportVersion: 2.20.7-0ubuntu4
Architecture: amd64
Date: Mon Nov 13 18:27:53 2017
ProcEnviron:
 TERM=xterm-256color
 PATH=(custom, no user)
 LANG=C.UTF-8
SourcePackage: nplan
UpgradeStatus: No upgrade log present (probably fresh install)

Related branches

Revision history for this message
Scott Moser (smoser) wrote :
Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I can't remember if the host kernel also has changed defaults, thus host kernel might be a spanner in the works too. All of the above was tested on the same host kernel, right?

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

Yeah, i guess it is, given steps to reproduce are all in one go.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

I wonder if cloud images need to ship an extra netplan snippet to disable ipv6 addresses by default, of if netplan default need changing, or kernel default needs changing.....

Revision history for this message
Ryan Harper (raharper) wrote :

I've got a branch which would have cloud-init disable accept-ra unless cloud-init has an explicit ipv6 configuration (static or dhcp6). This only affects the solicitation effort, ipv6 link-local is of course, unaffected as is explicit ipv6 configuration. I believe LXD is in a position to know if it has an ipv6 enabled bridge; it could control the template to use a dhcp6 config instead of dhcp (which is ipv4) and not the unconfigured IPV6 RA, which on older releases like Xenial and Trusty won't see without configuration anyhow.

https://code.launchpad.net/~raharper/cloud-init/+git/cloud-init/+merge/339437

Revision history for this message
Ryan Harper (raharper) wrote :

After some discussion, it appears that networkd is kinda of making a boolean (on|off) for IPV6 RA, when it's really a tri-state (on|off|kernel).

Upstream networkd indicates there is a tri-state; like so:

Enable or disable IPv6 Router Advertisement (RA) reception support for the interface. Takes a boolean parameter. If true, RAs are accepted; if false, RAs are ignored, independently of the local forwarding state. When not set, the kernel default is used, and RAs are accepted only when local forwarding is disabled for that interface. When RAs are accepted, they may trigger the start of the DHCPv6 client if the relevant flags are set in the RA data, or if no routers are found on the link.

While in netplan, we've only a boolean, which could be fine, except netplan defaults to AcceptRA=True which means we have no way of allowing the kernel configuration to work.

Netplan needs to know if the yaml includes an accept-ra key, and if so, uses the value set (off or on); but if the yaml does not specify an accept-ra key, it should *NOT* render a default value.

This allows hosts to defer the the kernel settings. This key was introduced as away to resolve a bug where "unconfigured" interfaces got an IPV6 address due to kernel setting and an IPV6 router present.

https://bugs.launchpad.net/maas/+bug/1655440

Revision history for this message
Ryan Harper (raharper) wrote :

It would be nice for both nplan and networkd to accept a tri-state:

accept-ra: [off|on|kernel]

I think that would make it clear rather than an unset value to indicate that it's controlled by the kernel.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

What's the point of having 'kernel'? You have a boolean here specifically because you'll likely want to override the default, whatever it might be. I'm fine with *not writing* accept-ra if that helps, but for most cases, this is just the kind of option that people should avoid touching.

FWIW, as far as I know this is expected and proper behavior, you *do* want to make the best effort attempt to bring up IPv6 whenever possible. In that sense, having RAs accepted and handled by default is the correct behavior. Please do not disable this unless there's a very good, well discussed and understood reason.

Also:

 mtrudel@demeter  ~  lxc launch ubuntu:16.04
Creating the container
Container name is: perfect-shrimp
Starting perfect-shrimp
 mtrudel@demeter  ~  lxc ls
+----------------+---------+---------------------+-----------------------------------------------+------------+-----------+
| NAME | STATE | IPV4 | IPV6 | TYPE | SNAPSHOTS |
+----------------+---------+---------------------+-----------------------------------------------+------------+-----------+
| perfect-shrimp | RUNNING | 10.3.110.252 (eth0) | fd42:9d0d:d7be:83fe:216:3eff:feda:54f7 (eth0) | PERSISTENT | 0 |
+----------------+---------+---------------------+-------------------

Revision history for this message
Ryan Harper (raharper) wrote :

The kernel value is equivalent to not writing it out, yes; I thought it'd be better to be explicit in the config.

W.r.t disabling by default; no plans to do so at this point but as it is now, all Artful and Bionic images will block for 10 seconds, unless an RA comes in sooner. So ideally we stop writing the Accept-RA unless we have a value provided in the yaml.

Revision history for this message
Stéphane Graber (stgraber) wrote :

I'd like to +1 what cyphermox said, the expected behavior on Ubuntu is that if you do receive a RA, you let the kernel configure it.

That's how Ubuntu has been ever since IPv6 support was enabled and I personally have about 200 systems that very much rely on this (no specific IPv6 configuration, just RA based config).

If you turn off IPv6 on your host, this may or may not affect your containers, depending on how you've turned it off (globally at kernel level, through the default policy or through the all policy).

Revision history for this message
Ryan Harper (raharper) wrote : Re: [Bug 1732002] Re: cloud images in lxc get ipv6 address

Guys, I'm no longer suggesting that we disable RA here. After our
discussion I updated this bug
to indicate that we really want some way to *leave* RA alone.

At this point that means netplan needs to *NOT* emit AcceptRA values
into the .network configuration
files by default (as it does now), but only if the input yaml included
an accept-ra value set.

On Mon, Apr 9, 2018 at 5:22 PM, Stéphane Graber <email address hidden> wrote:
> I'd like to +1 what cyphermox said, the expected behavior on Ubuntu is
> that if you do receive a RA, you let the kernel configure it.
>
> That's how Ubuntu has been ever since IPv6 support was enabled and I
> personally have about 200 systems that very much rely on this (no
> specific IPv6 configuration, just RA based config).
>
> If you turn off IPv6 on your host, this may or may not affect your
> containers, depending on how you've turned it off (globally at kernel
> level, through the default policy or through the all policy).
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1732002
>
> Title:
> cloud images in lxc get ipv6 address
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/nplan/+bug/1732002/+subscriptions

Revision history for this message
Ryan Harper (raharper) wrote :

The MP has been moved to a github PR:

https://github.com/CanonicalLtd/netplan/pull/19

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

This got landed in netplan.io 0.35 in bionic. The changes still should be part of a SRU.

Changed in nplan (Ubuntu):
status: New → Fix Released
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Scott, or anyone else affected,

Accepted nplan into artful-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nplan/0.32~17.10.4 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-artful to verification-done-artful. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-artful. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nplan (Ubuntu Artful):
status: New → Fix Committed
tags: added: verification-needed verification-needed-artful
Revision history for this message
Brian Murray (brian-murray) wrote :

Hello Scott, or anyone else affected,

Accepted nplan into xenial-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/nplan/0.32~16.04.5 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed.Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested and change the tag from verification-needed-xenial to verification-done-xenial. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-xenial. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance!

Changed in nplan (Ubuntu Xenial):
status: New → Fix Committed
tags: added: verification-needed-xenial
Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Verification-done on xenial: nplan 0.32~16.04.5

nplan-configured LXD container appears to behave as expected: on a setup with IPv6 enabled, I get an IPv6 address on the container without a startup delay. In a host with IPv6 disabled on the lxd bridge, the container also starts up appropriately, with no IPv6 address, as expected.

Revision history for this message
Mathieu Trudel-Lapierre (cyphermox) wrote :

Verification-done on artful: nplan 0.32~17.10.4

As above for the xenial tests: I get an IPv6 address when I should, and no IPv6 address when I shouldn't, based on the setup of the LXD bridge. Behavior is as expected.

tags: added: verification-done-artful verification-done-xenial
removed: verification-needed verification-needed-artful verification-needed-xenial
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nplan - 0.32~17.10.4

---------------
nplan (0.32~17.10.4) artful; urgency=medium

  * bond/bridge: Support suffixes for time-based values so things like
    "mii-monitor-interval" can support milliseconds. (LP: #1745597)
  * debian/postinst: Write breadcrumbs on disk in /etc/network/interfaces to
    denote the migration to using netplan. (LP: #1756742)
  * DHCPv4: add a "dhcp-identifier: mac" field that can be set to fix interop
    with Windows Server-based DHCP servers which don't support RFC 4361.
    (LP: #1738998)
  * IPv6: accept-ra should default to being unset, so that the kernel default
    can be used. (LP: #1732002)
  * doc/netplan.md: Clarify the behavior for time-based values for bonds
    and bridges. (LP: #1756587)
  * critical: provide a way to set "CriticalConnection=true" on a networkd
    connection, especially for remote-fs scenarios. (LP: #1769682)
  * networkd: don't wipe out /run/netplan on generate: we do want to keep any
    YAML configurations in that directory, we just need to remove generated
    wpasupplicant configs. (LP: #1764869)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 08 May 2018 11:04:30 -0400

Changed in nplan (Ubuntu Artful):
status: Fix Committed → Fix Released
Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for nplan has completed successfully and the package has now been released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package nplan - 0.32~16.04.5

---------------
nplan (0.32~16.04.5) xenial; urgency=medium

  * bond/bridge: Support suffixes for time-based values so things like
    "mii-monitor-interval" can support milliseconds. (LP: #1745597)
  * Do not attempt to rebind driver 'qeth'. (LP: #1756322)
  * Allow setting ClientIdentifier=mac for networkd-renderered devices
    (LP: #1738998)
  * IPv6: accept-ra should default to being unset, so that the kernel default
    can be used. (LP: #1732002)
  * doc/netplan.md: Clarify the behavior for time-based values for bonds
    and bridges. (LP: #1756587)
  * critical: provide a way to set "CriticalConnection=true" on a networkd
    connection, especially for remote-fs scenarios. (LP: #1769682)
  * networkd: don't wipe out /run/netplan on generate: we do want to keep any
    YAML configurations in that directory, we just need to remove generated
    wpasupplicant configs. (LP: #1764869)

 -- Mathieu Trudel-Lapierre <email address hidden> Tue, 08 May 2018 10:36:24 -0400

Changed in nplan (Ubuntu Xenial):
status: Fix Committed → Fix Released
Revision history for this message
Dan Streetman (ddstreet) wrote :

please reopen if this is still an issue

Changed in systemd (Ubuntu):
status: New → Invalid
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.