Memory usage issues with large Apache httpd configuration files
| Affects | Status | Importance | Assigned to | Milestone | |
|---|---|---|---|---|---|
| Augeas |
Fix Released
|
Unknown
|
|||
| augeas (Ubuntu) |
Fix Released
|
High
|
Unassigned | ||
| Bionic |
Fix Released
|
High
|
Miriam España Acebal | ||
| Cosmic |
Won't Fix
|
High
|
Unassigned | ||
Bug Description
[SRU]
[Impact]
The use of memory grows as a site conf file enlarges when using augeas.
[Test Plan]
- Copy the attached conf file here to /etc/apache2/
- Mime module has to be enabled as well (check it with a2query -m | grep mime). If not, enabling it with a2enmod mime.
- Reload apache2 configuration ( systemctl reload apache2 ). If something wrong here, maybe a non-graphical web browser like lynx is need to be installed.
- Open htop in a different terminal
- Run augtool -I httpd.aug
- In augtool prompt use: match /files/
- Check in htop that augtool is consuming:
- Before fixing the issue: near 2238MB
- After fixing the issue (updated packages needed are augeas-lenses, augeas-tools and libaugeas0): near 179 MB
PPA with built package with fix for testing: ppa:mirespace/
[Where problems could occur]
This patch is cherry picked from upstream, so the bug is corrected in following augeas package versions and no other conflicts have arisen with this change.
Also, the change itself acts as a cache of evaluated expressions (substituting the abstract syntax tree in memory for the value it produces), so the way expressions are evaluated is the same before and after the change and no other part of augeas has to be affected by this.
[Other Info]
Fixed upstream in version 1.11.0, commit id : bbf31f719db5491
Commit description: Replace pure function invocations in path expressions with their result.
In path expressions, we generally need to evaluate functions against
every node that we consider for the result set. For example, in the path
expression /files/
function was evaluated against every entry in /etc/hosts. Evaluating that
function requires the construction and compilation of a new regexp.
Because of how memory is managed during evaluation of path expressions, the memory used by all these copies of the same regexp is only freed after we are done evaluating the path expression. This causes unacceptable memory usage in large files (see hercules-team#569).
To avoid these issues, we now distinguish between pure and impure functions in the path expression interpreter. When we encounter a pure function, we change the AST for the path expression so that the function invocation is replaced with the result of invoking the function. With the example above, that means we only construct and compile the regexp '127\\.' once, regardless of how many nodes it gets checked against. That leads to a dramatic reduction in the memory required to evaluate path expressions with such constructs against large files.
Fixes hercules-team#569
Bug on upstream: https:/
[Original Report]
This bug in libaugeas was fixed in version 1.11.0, but is not in the 18.04.2 repo.
See: https:/
I found it using certbot and started a bug report there.
See: https:/
Which led me here to see if the newer version could be back-ported.
Related branches
- Utkarsh Gupta (community): Approve
- Canonical Server MOTU reviewers: Pending requested
-
Diff: 206 lines (+162/-1)5 files modifieddebian/changelog (+13/-0)
debian/control (+2/-1)
debian/patches/0005-Replace-pure-funcs-calls-in-path-expr-w-result.patch (+143/-0)
debian/patches/series (+1/-0)
debian/rules (+3/-0)
| Robie Basak (racb) wrote | #1 |
| Changed in augeas (Ubuntu): | |
| status: | New → Triaged |
| importance: | Undecided → High |
| status: | Triaged → Fix Released |
| Changed in augeas (Ubuntu Bionic): | |
| status: | New → Triaged |
| Changed in augeas (Ubuntu Cosmic): | |
| status: | New → Triaged |
| Changed in augeas (Ubuntu Bionic): | |
| importance: | Undecided → High |
| Changed in augeas (Ubuntu Cosmic): | |
| importance: | Undecided → High |
| Changed in augeas: | |
| status: | Unknown → Fix Released |
| tags: | added: bitesize |
| Bryce Harrington (bryce) wrote | #2 |
| Changed in augeas (Ubuntu Cosmic): | |
| status: | Triaged → Won't Fix |
| Changed in augeas (Ubuntu Bionic): | |
| assignee: | nobody → Miriam España Acebal (mirespace) |
| Changed in augeas (Ubuntu Bionic): | |
| status: | Triaged → In Progress |
| description: | updated |
| Miriam España Acebal (mirespace) wrote | #3 |
| description: | updated |
| description: | updated |
| Robie Basak (racb) wrote Please test proposed package | #4 |
| Changed in augeas (Ubuntu Bionic): | |
| status: | In Progress → Fix Committed |
| tags: | added: verification-needed verification-needed-bionic |
| Miriam España Acebal (mirespace) wrote | #5 |
| tags: |
added: verification-done verification-done-bionic removed: verification-needed verification-needed-bionic |
| Launchpad Janitor (janitor) wrote | #6 |
| Changed in augeas (Ubuntu Bionic): | |
| status: | Fix Committed → Fix Released |
| Brian Murray (brian-murray) wrote Update Released | #7 |
Thank you for taking the time to report this bug and helping to make Ubuntu better.
We can cherry-pick a fix to Ubuntu 18.04. Does the upstream commit apply to the 18.04 Ubuntu packaging cleanly? It would help if someone could prepare the upload please.