NGINX fails to start/install/upgrade if IPv6 is completely disabled.

Debian bug discussion:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942817

Debian proposal patch:
https://bugs.debian.org/cgi-bin/bugreport.cgi?att=2;bug=942817;filename=0001-Allow-nginx-to-start-on-ipv6-disabled-system.patch;msg=15

So IMHO there is 2 ways to fix this:

(1) - We remove the ipv6 listen from the vhost (just like Centos/Upstream does atm)

# nginx.vh.default.conf -> "nginx-1.16.0-1.el8.ngx.src.rpm"
server {
    listen 80;
    server_name localhost;

So as upstream nginx:
https://github.com/nginx/nginx/blob/master/conf/nginx.conf#L36

This won't introduce behaviour change for those who upgrade nginx, but it will introduce a behaviour change for those used to have ipv6 listen in the default vhost at fresh package installation.

But IMHO, the default vhost is a starting point/example, that the admin will anyway have to modify manually or via some orchestration tool (ansible, chef, puppet, ...)

So this will only imply that the admin will need to update their recipe to add the ipv6 listen in the vhost. I don't see any big blocker here.

(2) We implement a ipv6 detection as describe in the debian bug and provide a noipv6 vhost and/or a ipv6 vhost depending if ipv6 is [en|dis]abled on the system. But for me, that seems hacky.

I have a preference for scenario #1.

I have proposed both scenarios to debian, let's see what they say. In the absence of response from their part, I'll have SRU scenario #1 into Ubuntu.

I'll give debian maintainer team a week or so before.

- Eric

I am in favor of solution #1, and will drive this for Focal. However, I have no call in whether this is accepted by the SRU team for SRU.

More to come though, I have a priority on the nginx package in Focal and determining which branch we track heading up to release first.

@teward, sound good I'll let you drive the focal upload since you were already in the middle of it and include solution #1 in it.

For the SRU, I'll take care of the discussion internally, and will keep the bug updated on the outcome.

Eric

If I understand the situation correctly, my subjective opinion is that I don't think a "fix" for this is suitable for an SRU, but I welcome discussion if you disagree.

It's not uncommon to come across a situation where the user configures the system in some non-default way in one configuration file, and something else breaks without the user altering some other configuration file.

A couple of unrelated examples of this pattern:

1) A user configures a service to bind to a specific address, but then finds that the service fails to start because this introduces a dependency on having an interface with that address configured first. In this example the service hasn't been written to handle the interface coming up after service start, which would be the ideal solution to the problem. In the meantime, the user needs to also configure the init system to order the service start after the interface configuration.

2) A user configures a service to use a non-standard filesystem location, but then finds that AppArmor blocks access to that location. The user needs to also adjust the AppArmor profile to permit access to that location.

So here we have (I think):

3) A user disables IPv6. The user also needs to adjust the nginx configuration to not attempt to bind to IPv6 addresses.

In all of these cases, I agree that it would be nice if the appropriate thing could automatically detect the other configuration change and adjust itself accordingly by default. Doing this would certainly be a valuable usability improvement. However, I don't think these are bugs as such; this is expected behaviour albeit that the expected behaviour could be improved upon. While we might be able to make individual enhancements, in the general case it's clearly impossible to infer every possible necessary adjustment automatically. In the meantime, a user who adjusts away from default configuration in one place is expected to adjust other dependent configuration to match.

My conclusion is that this is a valid Wishlist-level bug in that in this specific case it would be nice if the default nginx configuration would continue to work without needing adjustment to match the global IPv6 disablement. However, for the reasons above I don't think an SRU is warranted, since the current behaviour is expected behaviour and doesn't prevent the user from achieving what they want. The currently correct way of handling this is to adjust nginx configuration, which is possible without needing any changes to the existing packaging.

I appreciate the efforts to make this experience better for users. That should take effect in future releases (but not change behaviour in existing stable releases). I therefore propose "Won't Fix" for the Ubuntu stable release bug tasks.

This bug was fixed in the package nginx - 1.17.5-0ubuntu1

---------------
nginx (1.17.5-0ubuntu1) focal; urgency=medium

  * New upstream release (1.17.5) - full changelog available from
    http://nginx.org/en/CHANGES
  * Remaining Ubuntu-specific changes:
    - debian/patches/ubuntu-branding.patch: add Ubuntu branding (refreshed)
    - d/{control,rules,nginx-core.*}: add new binary package for main,
      nginx-core, which contains only source-tarball-included modules
      and no third-party modules.
    - debian/tests/control: add nginx-core test.
    - debian/apport/source_nginx.py: Add apport hooks for additional bug
      information gathering.
    - debian/nginx-common.install: Add install rule for apport hooks.
    - d/nginx-{core,light,full,extras}.postinst: Add checks for whether
      port 80 is in use or not to determine whether or not to attempt
      starting of the NGINX service during install/upgrade
    - d/control: Add dependencies to nginx-{core,light,full,extras} on
      `iproute2` as the postinst scripts now use `ss` to determine if
      Port 80 is open or not.
    - d/rules: Enable --with-compat build option for all nginx package
      flavors
    - d/{control,rules,copyright,modules/http-geoip2*}: Add GeoIP2 third party
      module to nginx-full and nginx-extras (and use proper DEP5 syntax for
      d/copyright).
  * New Ubuntu-specific changes:
    - d/conf/sites-available/default: Update default nginx site configuration
      file to remove the IPv6 listening line so that servers running without
      IPv6 enabled at all on the system will start nginx properly.
      (LP: #1743592)

 -- Thomas Ward <email address hidden> Fri, 01 Nov 2019 11:55:10 -0400

Opting for #1 is a stupid idea.

Yes, personally I hate it because many times I have been wondering why it's not listening on IPv6 only to find out that the configuration line "listen 80 default_server;" doesn't in fact mean to just listen on port 80, but to listen only on port 80 for IPv4. In my opinion this configuration line doesn't make it clear, and if it stays like this, then "#listen [::]:80 default_server;" needs to be included in this way at the very least (note the hash symbol)!

What is another problem is hosting providers that are dual-stacked, and where maybe AAAA-records already exist. In those cases the client would first try to connect to the IPv6 address which has no nginx listening on port 80. Happy eyeballs should take care of this and fallback to IPv4, but that doesn't always seem to work properly.

On another note this is another step towards cannibalizing the deployment of IPv6 by making it opt-in instead of opt-out. Either implement option #2 or return to the way it was before and leave it up to the admin to figure out why his config doesn't work on the system he fiddled with.

T Jeske:

That needs its own bug tracking to try and change the functionality. It also needs broader discussions as well outside of here.s

Thomas

So from now on my IPv6-only installs will not work? That's just great, let's go back to 1992, when IPv6 did not exist yet.

I should create a bug for this..

If you disable IPv6, expect your system not to work. This is the 21st century. My systems don't even have 127.0.0.1 anymore, only ::1 and guess what. Modern software just deals with it.

I consider this a huge regression.

IPv6 should not be completely disabled on any system. This is not in keeping with Best Current Practices. See RFC 6540 (BCP 177). If an admin has manually disabled IPv6 on their system, it is their responsibility to also disable any services listening on IPv6 by default. This "fix" is antithetical to a modern Internet.

This change would break IPv6 support for nginx on all of my ISO spins. It's also doubly aggravating that edits to nginx.conf drive apt up the wall asking for 3 way merges; so I simply stopped editing nginx.conf and moved to using nginx/conf.d for *everything* because it kept breaking unattended-upgrades.

This will be reverted in an upload tomorrow.

Hm. After thinking about this a little, perhaps there is a way to have cake and consume it justly.
Since `ss` is now being used to check ports, I suggest a postinst check with `ss -6` to see if the icmp6 entry exists or not. If it does not exist, write a marker file out on disk that ipv6-appears-disabled. Check the marker during each postinst and select between the two "default" conditions automatically. Duplicate the `sites-available/default` with an additional copy `sites-available/default-ipv6`, with the inverse listen commented out, and the appropriate sites-enabled symlink. If either symlink is removed, the postinst doesn't recreate it.

This way `default` remains true to upstream, while the debian/ubuntu ipv6 listen addition is obvious to any operators. A quirk like this could easily be pointed out in the focal release notes as an ubuntuism.

As far as I am aware, during deployments of nginx by site operators it is common to remove the `sites-enabled/default` symlink to disable it without apt freaking out about 3 way merges.

Currently, I don't even utilize the sites-available or sites-enabled at all with my ISO spins.
The last line in my `/etc/nginx/conf.d/autoconfig.conf` is:

`include /opt/project/*/resources/latest/config/http/*/*.conf;`

It would be no trouble at all for me to add the scripted removal of the `sites-enabled/default-ipv6` symlink alongside my existing scripted removal of `sites-enabled/default`.

Means a slightly mungier postinst; but appropriate comments detailing the idea there would go a long way.

Thomas Ward:

Which part are you referring to exactly?

I agree automating the detection of everything seems a bit over the top. But for something so fundamental as the different IP-stacks, I think it is well worth considering it, though I am also not really happy as I still believe whoever disables IPv6 system-wide should consider the consequences.

Addition: Actually I think the best solution would be to test for IPv6 at runtime and produce just a warning instead of failing. Same for systems where IPv4 is disabled but the config still contains the IPv4 directive.

Any objections?

Using a single "listen [::]:80 default_server ipv6only=off;" seems to do the right thing in all situations: IPv4-only, IPv6-only or dual-stack.

The drawback of using this single socket is that IPv4 clients have their IPs represented as IPv4-Mapped IPv6 in access_log:

==> /var/log/nginx/access.log <==
::ffff:10.139.167.1 - - [20/Feb/2020:17:15:25 +0000] "GET / HTTP/1.1" 200 612 "-" "Wget/1.19.4 (linux-gnu)"

That said, if you go to extra lengths to disable IPv6 on your box, why not also tune the default vhost? Your config is already not aligned with the default of IPv6 enabled :)

Upon additional discussion among the Server Team, we have come to this conclusion:

The changes introduced as a result of this bug are no longer valid and are going to be reverted in the next upload of NGINX to the repositories.

For 20.04, we do not want to introduce additional complexity at this time to the postinst logic to 'alter' the default file depending on if IPv4 or IPv6 are actually available. Further, we do not want to break 'default log parsing' by using the solution that was provided by sdeziel.

Because we are *reverting* this change, I am setting the bug's status to "Opinion" - at a later time we may introduce something which actually fixes this problem in the future.

To clarify what just happened:

nginx should serve on IPv6 by default, just as IPv6 is available on Ubuntu by default. The previous upload broke this, so this revert restores it. Thank you to all who participated in the discussion.

The default nginx configuration will fail if IPv6 is disabled, as before. This is unfortunate, and this experience could be improved, so this bug remains open.

If you wish to disable IPv6, you must currently also adjust service configuration to match. While this could be improved and we are open to discussion on how to achieve that, the current situation is by design, so I suggested that the bug should be Wishlist to Thomas.

Nothing prevents a user from disabling IPv6. Just that if you do, you must also adjust some services to not listen on IPv6. Another way of looking at this is that adjusting those services is part of "disabling IPv6".

For those coming here because their installation or upgrade of the 'nginx' package fails due to their system using ipv6.disabled=1, you need to edit the file "/etc/nginx/sites-enabled/default" and remove or comment out this line:

        listen [::]:80 default_server;

Alternately, you can remove that file completely (it is actually a symlink to "/etc/nginx/sites-available/default"). However, you will of course need to create your own nginx configuration, for it to do anything useful.

Then you can restart the nginx service with:
$ sudo systemctl restart nginx
or just reboot.

You will also need to fix your broken installation of nginx:

$ sudo apt install --fix-broken

Note that apt may complain with a warning like:

W: APT had planned for dpkg to do more than it reported back (3 vs 7).
   Affected packages: nginx-core:amd64

I believe that warning can be ignored.

On IRC, Simon suggested this:

16:43 <sdeziel> teward: to add to possible solutions, using "listen [::]:80 default_server ipv6only=off;" alone seems to do the right thing no matter what is present, v4 only, v6 only or both

There's one known downside, which is that IPv4 connections are logged as IPv6 like this:

18:48 <teward> yes, but that makes the logs all log IPv6 addresses ::ffff:10.1.2.3 as an example

However as far as I understand that doesn't lose anything.

Opinions on this as a way of resolving this bug? Summary: we change the default listen directive to "listen [::]:80 default_server ipv6only=off;"

Feature freeze is on Thursday, so ideally we should make a decision on this very soon if we want any change to release in Focal.

Will it log as ::ffff:10.1.2.3 if IPv6 is disabled on the system?

I could imagine this notation could confuse some tools for automatic analyses of logfiles. I fear that this would break something in the background without people even realizing it until it is too late. Imo failing hard should be the preferred option.

On Tue, Feb 25, 2020 at 01:41:42PM -0000, T Jeske wrote:
> Will it log as ::ffff:10.1.2.3 if IPv6 is disabled on the system?

I believe so, yes.

> I could imagine this notation could confuse some tools for automatic
> analyses of logfiles. I fear that this would break something in the
> background without people even realizing it until it is too late. Imo
> failing hard should be the preferred option.

I agree this isn't ideal. But ultimately, log file syntax does change
over time, tooling needs to be updated, and the time to make changes is
in new Ubuntu releases (rather than existing ones). So in principle I
think it's acceptable to make this change even though some log analysis
may fail until those analysers are updated.

However, it does seem quite late in the day to make this change for
Focal. It gives us less time to find tooling in the archive that may
need updating, and less time for the out-of-archive community to adjust,
too. Thomas has expressed the same opinion to me on IRC. So on that
basis, it looks like we'll be keeping the status quo for Focal, and
maybe changing it in Focal+1.

Dan Streetman (ddstreet) wrote on 2020-02-21:
> For those coming here because their installation or upgrade of the 'nginx' package fails
> due to their system using ipv6.disabled=1, you need to edit the file...

FWIW, that doesn't work, and here I have all servers, including default servers, configured to listen only on IPv4 addresses. When "ipv6.disabled=1" is passed to the kernel, NGINX fails to start...period, even with no IPv6 servers configured. I also tried adding "ipv6=off" to the "resolver" directive, but that has no effect; IPv6 must be enabled on my server for NGINX to start.

It is one thing to ipv6/conf/all/disabled=1, that is still supported. But ipv6.disable=1 should not be supported in any way. It is in direct conflict with modern software and proper deployments which should default to IPv6, in accordance with IETF BCP177.

I found this thread when trying to simplify automated nginx installation on Ubuntu 20.04. In my case, IPv6 is disabled with ipv6.disable=1, which breaks a fresh nginx installation from proceeding with "nginx: [emerg] socket() [::]:80 failed (97: Address family not supported by protocol)". Specifically, apt fails when trying to install nginx, which break the automation workflow.

The workaround that works for me and can be easily automated:

1. Manually create sites-available/default without a "listen [::]:80 default_server;" directive _before_ installing nginx so that the problematic file is not created and the workflow is not broken.
2. Install nginx with apt. Because sites-available/default does not contain the problematic directive, the installation finishes successfully.
3. Delete sites-available/default (and corresponding sites-enabled/default).
4. Proceed with the rest of the configuration.

This workaround is definitely clumsy and I would appreciate if the fresh nginx install did not fail when IPv6 is disabled. Nginx failing to start with the default config when IPv6 is disabled is perfectly fine, however this bug breaks the installation process itself, which is not fine.

Regarding ipv6.disable=1 should not be supported, let me politely disagree with this assessment. RFC6540/IETF BCP177 says that new IP stack implementations must support IPv6, should support 4/6 dual-stack and must not require IPv4 stack for IPv6 operation. It does not require dropping application support for IPv4-only systems at all. IPv4 is not nearly EOL yet.

There are legitimate use cases when IPv6 must be disabled, for example in IPv4-only environments that do not use IPv6, such as in many if not most corporate networks behind the NAT. Disabling unused systems and protocols is a long-standing best practice, both from infosec and operational perspectives and is still required by the infosec policies in many organisations.

Is it feasible to re-open the bug and fix it to the level so that the nginx installation completes successfully on an IPv4-only system and does not fail when nginx is simply unable to start with the default site? The nginx itself unable to start with the default configuration is perfectly fine and expected.

@Nikita

If you're using automation to install and configure nginx, then you should already be using policy-rc.d to disable the service start when you install it with apt. Then nginx shouldn't fail to start because it doesn't attempt to start. You can then configure nginx and start it manually.

For example, with ansible, you can use policy_rc_d=101 when calling the apt module.

In my opinion automation tooling that doesn't do this by default is buggy. That's unfortunately all of them. But they can generally be arranged to do it properly in configuration.

Does this work for you?

This is a pretty old issue but in Debian they made a decision to not accomodate this back in 2015 for when IPv6 is disabled on systems on the basis of "not only is it non-standard but it's nonsensical":

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=779825#97

The workaround for this is when dpkg or apt fail is to edit /etc/nginx/sites-available/default, remove the IPv6 listen line, and then run `apt install -f` - this can even be automated too. Automation tooling that is not able to do this is the fault of the tooling.

I'm going to close this as 'won't fix' because there's really no actions needed by the teams here. And this has sat without any activity for over a year.