sssd_be is leaking memory

I have this bug in Ubuntu 16.04.

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Presumably this exists only in Xenial and Yakkety, since Zesty has 1.15.0-3ubuntu4? Marking Fix Released for Zesty accordingly, and creating tasks for Xenial and Yakkety.

To have Xenial updated, please first read https://wiki.ubuntu.com/StableReleaseUpdates. We'll either need a backport of the fix or we'll need to ensure that all changes in updating to 1.13.5 are acceptable to automatically update users under the policy.

If you could check and document this by following as much of https://wiki.ubuntu.com/StableReleaseUpdates#Procedure as you can, this would be most helpful.

Status changed to 'Confirmed' because the bug affects multiple users.

I am seeing this is in 1.13.4-1ubuntu1.9. Can we get a version bump in Xenial?

There is no upstream release for 1.13.5 yet, and a major version dump to 1.14 or 1.15 is not recommended for an LTS release.

I also checked the 1-13 branch in upstream and couldn't see any obvious immediate fix for this issue. Are you using group and user enumeration?

No, both group and user enumeration is explicitly turned off in our config. 1.13.4-1ubuntu1.9 seems to start spawning a bunch of sssd_be processes which slows auth to a crawl

I have backported it myself and multiple issues (including this one) have been resolved as a result. I am still confused by the reasoning behind being unwilling to backport versions of this package in a "Long Term Support" release of Ubuntu. This bug is not fixed until a newer version is backported (as mentioned in the original Fedora ticket).

> I am still confused by the reasoning behind being unwilling to backport versions of this package in a "Long Term Support" release of Ubuntu.

This is not correct. We are quite willing to accept fixes for this bug in all our stable releases, including LTS releases. The policy and rationale are documented at https://wiki.ubuntu.com/StableReleaseUpdates. Which part of this confuses you?

I feel that sssd_be spiraling out of control (spawning hundreds to thousands of processes) and consuming all of the available RAM would be considered a "High-impact" bug, when that same behavior is not observed in 14.04.

16.04 has a xenial-backport component, is it not possible to put a newer version there instead of placing it in xenial-updates or xenial-security? What dictates what is put where? If upstream never releases 1.13.5 (though they seem to have tagged this 1.13 release: https://github.com/SSSD/sssd/releases/tag/sssd-1_13_91), will 16.04 never get an sssd upgrade?

On Thu, Feb 22, 2018 at 08:33:19PM -0000, sean wrote:
> I feel that sssd_be spiraling out of control (spawning hundreds to
> thousands of processes) and consuming all of the available RAM would be
> considered a "High-impact" bug, when that same behavior is not observed
> in 14.04.

Indeed, which is why I explained what process to follow to get the bug
fixed in xenial-updates for 16.04 in comment 2.

> 16.04 has a xenial-backport component, is it not possible to put a newer
> version there instead of placing it in xenial-updates or xenial-
> security?

Sure, if somebody follows the process, prepares a suitable upload, and
so forth. Somebody will also need to continue maintaining the backports
package for security, assuming you care about that, since Canonical's
security cover doesn't cover the backports pocket. But in any case this
is a bug, so fixing it in xenial-updates would be preferable.

> What dictates what is put where? If upstream never releases
> 1.13.5 (though they seem to have tagged this 1.13 release:
> https://github.com/SSSD/sssd/releases/tag/sssd-1_13_91), will 16.04
> never get an sssd upgrade?

Did you read https://wiki.ubuntu.com/StableReleaseUpdates? We absolutely
can update sssd in xenial-updates in 16.04 to fix this bug. Somebody
just needs to figure out the appropriate fix and follow the process to
get the fix landed.

Right now, I note that only two people have reported themselves as
affected and the bug has been open a year. We don't have specific steps
to reproduce the problem, and no specific fix has been identified. So it
seems to me rather difficult to tackle for someone who does not know how
to reproduce the problem. But if someone affected can find the required
development effort to fix this, Ubuntu will welcome that contribution.

Sean, could you please share your sssd.conf file?

The upstream bug you linked to mentions two specific features in use:
- enumeration turned on (which you disabled)
- AD backend

The other option is a libtevent bug they mention, and one debian user has patched locally, but only an update to 1.13.5(git) doesn't hit the bug.

There are some mentions of leaks in the git log since 1.13.4 that we could try patching. If you see the problem quickly and all the time, and are willing to test packages from a PPA, I could try to prepare some for you to test.

If you want to give it a try, this PPA (https://launchpad.net/~ahasenack/+archive/ubuntu/sssd-memleak-1676328) has a xenial build with this patch:

commit e22cbe9326073e6d42fe2118661fa6daaed8638c
Author: Sumit Bose <email address hidden>
Date: Tue Jul 12 13:16:43 2016 +0200

    AD: avoid memory leak in netlogon_get_domain_info() and make it public

    Reviewed-by: Jakub Hrozek <email address hidden>
    (cherry picked from commit 74bef2150c76c8814bf4c1654ecd3660604eb4e6)

I too have this problem. If it's a matter people reporting it as a problem, count me in. It's taken out production services multiple times. We've had to create monitors and alerts for it, it is a huge problem.

Can you give the packages from comment #13 a try please?

I have given the package in comment #13 a try. it takes a couple weeks for the problem to get bad enough to trip the monitor we have set up for it, but today is that day:

```
$ dpkg -l | grep sssd
ii sssd 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- metapackage
ii sssd-ad 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- Active Directory back end
ii sssd-ad-common 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- PAC responder
ii sssd-common 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- common files
ii sssd-ipa 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- IPA back end
ii sssd-krb5 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- Kerberos back end
ii sssd-krb5-common 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- Kerberos helpers
ii sssd-ldap 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- LDAP back end
ii sssd-proxy 1.13.4-1ubuntu1.11~ppa1 amd64 System Security Services Daemon -- proxy back end
```

we see today:

```
$ ps axf | grep sssd_be
15972 pts/0 S+ 0:00 \_ grep --color=auto sssd_be
  445 ? S 0:08 \_ /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
65342 ? S 0:03 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
29323 ? S 0:01 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
 4004 ? S 0:00 \_ /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
58378 ? S 0:00 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
58502 ? S 0:00 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
29326 ? S 0:00 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
 3000 ? S 0:00 \_ /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
29354 ? S 0:00 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
 3703 ? S 0:00 \_ /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
29481 ? S 0:01 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
 3304 ? S 0:00 \_ /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starbucks.net --uid 0 --gid 0 --debug-to-files
48643 ? S 0:00 /usr/lib/x86_64-linux-gnu/sssd/sssd_be --domain starb...

Thanks for trying it out.

It's "wontfix" for yakkety. The xenial task is still open.

I ll comb again through the changes in 1.16 to see if I can find some commits that might have fixed this, but it's quite possible the fix is a change that relies on other bigger parts of the new 1.16 code.

hi, any update?

Is this git release https://github.com/SSSD/sssd/releases/tag/sssd-1_13_91 can be upgradable from `1.13.4-1ubuntu1.15` on ubuntu Xenial 16.04 LTS . I am facing memory leaks with this version on multiple servers. Thanks

Xenial has reached end of standard support, so I'm marking this bug as Won't Fix.