[SRU] TLS is not enabled for memcached>=1.5.13

Bug #1887943 reported by Moisés Guimarães de Medeiros
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
memcached (Ubuntu)
Fix Released
Wishlist
Unassigned
Focal
Fix Released
Wishlist
Unassigned

Bug Description

[Impact]

TLS enablement allows Memcached to both encrypt cached data on the wire as well as to provide authentication of clients and servers according to the specified TLS configuration.

TLS is a feature enabled via configuration or command-line arguments, therefore existing deployments of Memcached will not be affected and will continue to work as expected. Such deployments would then have the choice to opt-in TLS usage by providing the extra TLS configuration.

TLS support is required to safely run Memcached on cloud environments where the user does not have total control over the network.

According to [1], support for TLS was added in version 1.5.13 while Focal ships 1.5.22. The feature is just not enabled during compile time.

[Test Case]

$ apt install memcached
$ memcached -Z -v
Error loading the certificate chain: (null)

That is enough to check if TLS capabilities are enabled in Memcached.

[Regression Potential]

Enabling TLS as an SRU will introduce a new protocol in certain environments. This may be problematic for a small number of users, but the benefit of having TLS enabled greatly outweighs that.

From an update point of view, this only enables the capability to run Memcached with TLS, and as this is an opt-in feature, services that do not choose to opt-in should stay the same.

[Fix]
This simply needs --enable-tls passed to the configure script to enable TLS. The change has been reviewed and accepted by Debian and sync'd to Ubuntu groovy. The upstream commit is https://github.com/docker-library/memcached/blob/4538128227a0e422e59df735d67b03ee23d39637/debian/Dockerfile#L56

[Discussion]

[Original Report]
At OpenStack we use ubuntu (currently 20.04) at our CI jobs.

There is a current demand for TLS enablement in order to be able to cache sensitive information such as access tokens.

Related branches

Bryce Harrington (bryce)
Changed in memcached (Ubuntu):
importance: Undecided → Wishlist
Bryce Harrington (bryce)
Changed in memcached (Ubuntu Focal):
importance: Undecided → Wishlist
tags: added: server-triage-discuss
Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi @Bryce,

Is there anything I can do to help to push this forward like patches or attending to open meetings?

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Bryce Harrington (bryce)
tags: removed: server-triage-discuss
Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Moisés,

Yes, patches would help for sure. That'd help us know what's needed for getting this into groovy technically. There are also questions about the implications for supportability, and what Debian's plans might be, but patches would also assist in getting answers to those as well.

Backporting this to focal via SRU is a separate question, which can be tackled after it's fixed in groovy, and is likely to be more in the realm of the security team. It is enabling a feature rather than just being a bug fix, so there may be some additional considerations to make once the groovy fix is known.

Changed in memcached (Ubuntu):
status: New → Incomplete
Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi Bryce,

Can you point me to the right repo/spec file? This should be as simple as adding --enable-tls like in here:

https://github.com/docker-library/memcached/blob/4538128227a0e422e59df735d67b03ee23d39637/debian/Dockerfile#L56

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :
Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi Bryce,

I can confirm that my patch for TLS enablement already landed in grovy.

Can you walk me through the SRU?

Bryce Harrington (bryce)
description: updated
Changed in memcached (Ubuntu Focal):
status: New → Triaged
Revision history for this message
Bryce Harrington (bryce) wrote :

Thanks for landing it in Debian, and great to hear it sync'd into groovy without issue. I'm marking the groovy task as Fix Released, and set the status on the focal task to Triaged.

I'll be glad to walk you through the SRU process. The first step is to fill in the "SRU paperwork". I've pasted in the template into the bug description, which you should be able to edit (see the yellow pencil icon on the right of the Bug Description). You can see some examples of other TLS enablement SRUs that got accepted here, for an idea of what needs to be said:

https://bugs.launchpad.net/ubuntu/+source/apache2/+bug/1845263
https://bugs.launchpad.net/ubuntu/+source/openssl/+bug/1797386

The apache2 is probably the right level of detail to follow. The openssl one is far more verbose than SRU's usually are, but useful to see the types of concerns TLS enablement SRUs have considered in the past.

For the Impact section, especially for wishlist SRUs you should indicate other users besides OpenStack that will be benefited by this change. The SRU reviewers will be considering the breadth of the impact.

For the Regression Potential, the section from the apache2 bug linked above is a good starting point. The main thing to think about is if there was a regression, how would testers/users notice it, and distinguish it as a TLS-caused regression vs. some other random bug?

The Test Case section is IMHO the most crucial for SRUs and should try to give a 'paint by numbers' way to both reproduce the issue and to verify the fix works.

The [Fix] section is optional but I've gone ahead and added it.

The next step is to prepare a PPA with the patch applied on top of the focal version of memcached. Essentially, apt-get source memcached, then apply https://salsa.debian.org/lamby/pkg-memcached/-/commit/39c91c8d5eb9fc48fda31723923659c518d6577f. Add a changelog entry, with the version set to "1.5.22-2ubuntu0.1", and run 'update-maintainer' which will set debian/control properly. Create a PPA and upload your package there, verify it builds ok, and link to it from this bug report. If you prefer us to handle any of that, just ask.

At that point, myself or one of the server team members will take over. We'll review and sponsor the upload into focal-proposed, and submit it for SRU review. For the SRU, a security team review is probably also needed.

For reference, the official SRU policy is here: https://wiki.ubuntu.com/StableReleaseUpdates, mainly just section 3.

Good luck, and if you need help just ask.

Changed in memcached (Ubuntu):
status: Incomplete → Fix Released
Bryce Harrington (bryce)
summary: - TLS is not enabled for memcached>=1.5.13
+ [SRU] TLS is not enabled for memcached>=1.5.13
description: updated
Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi Bryce, I got stuck at the PPA part. I tried following both your steps and the steps found here: https://packaging.ubuntu.com/html/fixing-a-bug.html

But I got this out of your steps:

root@ab85fe1cd213:/tmp/memcached-1.5.22# update-maintainer
The Maintainer email is set to an ubuntu.com address. Doing nothing.

and trying the other steps I got:

root@ab85fe1cd213:/tmp/memcached-1.5.22# debuild -S -d -us -uc
...
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed

Revision history for this message
Bryce Harrington (bryce) wrote :

The update-maintainer is fine, that is just saying it was already updated so no extra change needed. You can ignore it.

As to the info: fuzz message, if that's just a warning and debuild is successfully producing a *source.changes, that's fine, you can go ahead and put it into the PPA via:

  dput ppa:<ppa-name> *source.changes

If it isn't producing a *source.changes file, not sure what's happening... copy in a larger snippet of the build log and we'll go from there.

Btw, I'll be out of town next week on PTO. I've asked my co-workers to keep an eye on this bug and help out, but if you need someone to chat with directly, head to IRC on freenode, channel #ubuntu-devel.

Revision history for this message
Dimitri John Ledkov (xnox) wrote :

@bryce no fuzz is allowed, it's a fatal error which means that source package is not produced.

@moguimar you need to refresh the patches to ensure they are without fuzz.

$ quilt refresh

should do it. Not it will regenerate the patch with matching context for the current source.

Then building source package should work.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

full debuild -S output

---

root@4895276aec39:/tmp/enable-tls/memcached-1.5.22# debuild -S
 dpkg-buildpackage -us -uc -ui -S
dpkg-buildpackage: info: source package memcached
dpkg-buildpackage: info: source version 1.5.22-2ubuntu0.2
dpkg-buildpackage: info: source distribution focal
dpkg-buildpackage: info: source changed by Moises Guimaraes de Medeiros <email address hidden>
 dpkg-source --before-build .
 debian/rules clean
dh clean --with autoreconf
   debian/rules override_dh_auto_clean
make[1]: Entering directory '/tmp/enable-tls/memcached-1.5.22'
dh_auto_clean
rm --force /tmp/enable-tls/memcached-1.5.22/debian/memcached.init /tmp/enable-tls/memcached-1.5.22/debian/memcached.service
make[1]: Leaving directory '/tmp/enable-tls/memcached-1.5.22'
   dh_clean
 dpkg-source -b .
dpkg-source: info: using source format '3.0 (quilt)'
dpkg-source: info: building memcached using existing ./memcached_1.5.22.orig.tar.gz
dpkg-source: info: using patch list from debian/patches/series
patching file debian/control
Reversed (or previously applied) patch detected! Skipping patch.
2 out of 2 hunks ignored
patching file debian/control.orig
patching file debian/rules
Reversed (or previously applied) patch detected! Skipping patch.
1 out of 1 hunk ignored
dpkg-source: info: the patch has fuzz which is not allowed, or is malformed
dpkg-source: info: if patch 'enable-tls-capabilities-by-default.patch' is correctly applied by quilt, use 'quilt refresh' to update it
dpkg-source: error: LC_ALL=C patch -t -F 0 -N -p1 -u -V never -E -b -B .pc/enable-tls-capabilities-by-default.patch/ --reject-file=- < memcached-1.5.22.orig.UFGHGq/debian/patches/enable-tls-capabilities-by-default.patch subprocess returned exit status 1
dpkg-buildpackage: error: dpkg-source -b . subprocess returned exit status 2
debuild: fatal error at line 1182:
dpkg-buildpackage -us -uc -ui -S failed

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

I'm not sure if I messed up some stuff here. I uploaded the PPA, but as I had some signing issues I removed it from my account and tried to upload it again. I keep getting emails that the version is already accepted, but I can't see it in my ppa package listing.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

Good to hear you made good progress Moisés, feel free to ask if you find any blocker on your side.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi @Lucas, @Dimitri, and @Bryce,

So now that I have the PPA, what is the next step to get it closer to land in Focal? I remember something about someone having to sponsor my build.

Revision history for this message
Bryce Harrington (bryce) wrote :
Revision history for this message
Seth Arnold (seth-arnold) wrote :

What's this bit for?

+ifeq (,$(filter $(DEB_HOST_ARCH),armhf))

Thanks

Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Moisés,

I've submitted a merge proposal for this change. It will be reviewed by a ubuntu-server team member and if approved they or I will sponsor the upload to focal-proposed. Technically I could review it myself, but figure it's better to get a second set of eyes. I did attempt to build it myself in an LXC container but it failed here:

ok 16 - strtoull
/bin/bash: line 7: 24639 Alarm clock ./testapp
Failed to startup/connect to memcached server. at ./t/binary.t line 10.
./t/binary.t ............
Dubious, test returned 25 (wstat 6400, 0x1900)
...

However given that it passed in the PPA I'll assume something's just quirky about my lxc container.

One question I had was that I noticed this is added to Build-Depends:

+ libio-socket-ssl-perl <!nocheck>,

What is the rationale for the !nocheck? I assume this is from the Debian change, but this sounds like something that might need to be mentioned in the changelog.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi Bryce,

Thanks for the support. Yes, that !nocheck was just copied from the debian change. I think Christian's suggestion to put in the changelog is good. The testing part also didn't work in my first build, IIRC the tests have a bit of flakiness there.

Seth, that is for not enabling it for armhf platforms, it wasn't in my plans, but the debian folks put it there, I'm not aware what problems they ran into.

Revision history for this message
Seth Arnold (seth-arnold) wrote : Re: [Bug 1887943] Re: [SRU] TLS is not enabled for memcached>=1.5.13

On Tue, Sep 01, 2020 at 08:21:05AM -0000, Moisés Guimarães de Medeiros wrote:
> Seth, that is for not enabling it for armhf platforms, it wasn't in my
> plans, but the debian folks put it there, I'm not aware what problems
> they ran into.

Thanks, when it's not part of our delta that makes it a bit easier to
accept, but I still find it very strange.

Thanks

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hey guys, sorry for the delay here, I've been sick since last week. Today I'll be busy with some doctor appointments. Tomorrow I should be able to continue with this backport.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Sure, get well first and ping back once you are really ready.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :
Revision history for this message
Bryce Harrington (bryce) wrote :

Hi Moisés, actually the MP has been reviewed, accepted and uploaded.
I've subscribed the SRU team to review and accept the package.

Next step after that will be for you to validate the package in -proposed, so stay tuned for an automated message requesting that validation.

Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Moisés, or anyone else affected,

Accepted memcached into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/memcached/1.5.22-2ubuntu0.2 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in memcached (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (memcached/1.5.22-2ubuntu0.2)

All autopkgtests for the newly accepted memcached (1.5.22-2ubuntu0.2) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

php-doctrine-cache/1.10.0-3 (amd64)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#memcached

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Bryce Harrington (bryce) wrote :
Revision history for this message
Bryce Harrington (bryce) wrote :

The test case is failing only on amd64, with this test case error:

1) Doctrine\Tests\Common\Cache\ChainCacheTest::testLifetime
Data should be expired
Failed asserting that true is false.

/tmp/autopkgtest.RGHUkG/build.W9o/src/tests/Doctrine/Tests/Common/Cache/CacheTest.php:286

I'm going to try re-running the test in case it's just something intermittent. From the package's test history looks like it's had intermittent test failures in the past.

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

There was some back and forth in the Debian package with tests too. I was able to connect the RHEL/Fedora package maintainer with the Debian package maintainer and they were able to get the tests to a stable state.

You can see the whole thread here:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=968603

Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

I just verified that the proposed build has TLS support enabled.

Bryce, we need to wait on the other builds status before
changing the tag to verification-done-focal, right?

Revision history for this message
Bryce Harrington (bryce) wrote :

As far as I know, the builds, test runs, and verification can be done in any order, they just all must be properly working before the package will complete migration.

tags: added: verification-done verification-done-focal
removed: verification-needed verification-needed-focal
Revision history for this message
Moisés Guimarães de Medeiros (moguimar) wrote :

Hi Bryce,

Is there anything I could do on my end to help fixing the tests?

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Hey! So I see that the package FTBFS on arm64, even though the previous version in focal-updates built fine on that architecture. Therefore I can't really release this SRU before this build failure is resolved.

Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Switching to Incomplete until the build failure gets addressed.

Changed in memcached (Ubuntu Focal):
status: Fix Committed → Incomplete
Revision history for this message
Bryce Harrington (bryce) wrote :

The package is building fine on arm64 now, but there is a new FTBFS on riscv64. I don't think the build results change is due to any change in the package itself. The error in the build log mentions dns troubles, so suspect the failures could be intermittent network trouble. I've resubmitted the riscv64 build.

https://launchpad.net/ubuntu/+source/memcached/1.5.22-2ubuntu0.2

Changed in memcached (Ubuntu Focal):
status: Incomplete → New
Revision history for this message
Bryce Harrington (bryce) wrote :

The riscv64 rebuild failed as before. However, 1.5.22-2ubuntu0.1's riscv64 build shows the same error, so this is not a regression introduced by this update.

It looks like this update is ready to be considered for migration.

Revision history for this message
Chris Halse Rogers (raof) wrote : Update Released

The verification of the Stable Release Update for memcached has completed successfully and the package is now being released to -updates. Subsequently, the Ubuntu Stable Release Updates Team is being unsubscribed and will not receive messages about this bug report. In the event that you encounter a regression using the package from -updates please report a new bug using ubuntu-bug and tag the bug report regression-update so we can easily find any regressions.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package memcached - 1.5.22-2ubuntu0.2

---------------
memcached (1.5.22-2ubuntu0.2) focal; urgency=medium

  * Enable TLS capabilities present in Memcached since version 1.5.13.
    TLS support is required in order to harden Memcached deployments in
    cloud environments or similar scenarios where the network is not to
    be entirely trusted. (LP: #1887943)
    - d/rules: set --enable-tls
    - d/control: add libio-socket-ssl-perl to run build time ssl tests
    - d/control: add libssl dependency

 -- Moisés Guimarães de Medeiros <email address hidden> Mon, 24 Aug 2020 15:14:26 +0200

Changed in memcached (Ubuntu Focal):
status: New → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.