X1Carbon comes to a crawl during high CPU usage tasks

This change was made by a bot.

Confirming, downgrading to 4.4 fixes the issue.

Would it be possible for you to test the latest upstream kernel? Refer to https://wiki.ubuntu.com/KernelMainlineBuilds . Please test the latest v4.8 kernel[0].

If this bug is fixed in the mainline kernel, please add the following tag 'kernel-fixed-upstream'.

If the mainline kernel does not fix this bug, please add the tag: 'kernel-bug-exists-upstream'.

Once testing of the upstream kernel is complete, please mark this bug as "Confirmed".

Thanks in advance.

[0] http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8

So I tested linux 4.8 final and the regression still exists. On another note I also tested linux 4.7 and the issue does not exist there.

Can you give the v4.8-rc1 kernel a try? It can be downloaded from:
http://kernel.ubuntu.com/~kernel-ppa/mainline/v4.8

We can perform a kernel bisect, once we know the last good kernel and first bad one.

I tried linux 4.8-rc1 and it does have the said regression.

I started a kernel bisect between v4.7 final and v4.8-rc1. The kernel bisect will require testing of about 7-10 test kernels.

I built the first test kernel, up to the following commit:
1c88e19b0f6a8471ee50d5062721ba30b8fd4ba9

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Omer reports the first test kernel is bad on IRC.

I built the next test kernel, up to the following commit:
f7816ad0f878dacd5f0120476f9b836ccf8699ea

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Tested, the latest kernel build that you provided is bad as well.

I built the next test kernel, up to the following commit:
55392c4c06204c8149dc333309cf474691f1cc3c

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Same results. This build is infected as well.

I built the next test kernel, up to the following commit:
52770c37db2c0ee5585dae2de3d19c8453f1e8dc

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Ok, that is a good kernel. My test is `stress -c 4` and it works fine i.e. system performance does not degrade.

I built the next test kernel, up to the following commit:
5048c2af078d5976895d521262a8802ea791f3b0

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

I filed bug 1626436 which is similar; I wanted to try http://kernel.ubuntu.com/~jsalisbury/lp1627108 but that is empty?

To add the missing comment, I tried that last kernel last night and that is a good one as well. I notified Joseph on IRC.

I built the next test kernel, up to the following commit:
0f657262d5f99ad86b9a63fb5dcd29036c2ed916

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Hi! That is a bad kernel, i.e. it contains the regression.

I built the next test kernel, up to the following commit:
c86ad14d305d2429c3da19462440bac50c183def

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

That's a good kernel build. Seems there are multiple builds in there, so I tested 201610061038 which is newer.

I built the next test kernel, up to the following commit:
3ebfd81f7fb3e81a754e37283b7f38c62244641a

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/3ebfd81f7fb3e81a754e37283b7f38c62244641a

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

I built the next test kernel, up to the following commit:
766fd5f6cdaf1d558afba19850493b2603c9625d

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/766fd5f6cdaf1d558afba19850493b2603c9625d

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

That is a bad kernel.

I built the next test kernel, up to the following commit:
55e16d30bd99510900caec913c90f53bc2b35cba

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/766fd5f6cdaf1d558afba19850493b2603c9625d

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

Thanks, that's a bad kernel.

I pointed you to the wrong kernel in comment #26. The URL should have been:

http://kernel.ubuntu.com/~jsalisbury/lp1627108/55e16d30bd99510900caec913c90f53bc2b35cba/

Can you test that kernel?

If that kernel is also bad, then the next one is already ready and can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1627108/3d89e5478bf550a50c99e93adf659369798263b0/

However, if it is good, I'll have to build the next kernel with a good result.

Ok, the one is comment 28 is bad, while the one in #29 is Good.

I built the next test kernel, up to the following commit:
ea86cb4b7621e1298a37197005bf0abcc86348d4

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/ea86cb4b7621e1298a37197005bf0abcc86348d4

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

FTR, I tested #29 which works for Omer, but not for me, so bug 1626436 is not a duplicate.

I built the next test kernel, up to the following commit:
3d30544f02120b884bba2a9466c87dba980e3be5

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/3d30544f02120b884bba2a9466c87dba980e3be5

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

As reported on IRC, that's a bad kernel.

I built the next test kernel, up to the following commit:
7dc603c9028ea5d4354e0e317e8481df99b06d7e

The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/7dc603c9028ea5d4354e0e317e8481df99b06d7e

Can you test that kernel and report back if it has the bug or not? I will build the next test kernel based on your test results.

Thanks in advance

The bisect reported this as the first bad commit:

commit 3d30544f02120b884bba2a9466c87dba980e3be5
Author: Peter Zijlstra <email address hidden>
Date: Tue Jun 21 14:27:50 2016 +0200

    sched/fair: Apply more PELT fixes

I built a Yakkety test kernel with a revert of commit 3d30544f02120b884bba2a9466c87dba980e3be5. The test kernel can be downloaded from:

http://kernel.ubuntu.com/~jsalisbury/lp1627108/

Can you test that kernel and report back if it has the bug or not?

Note with this kernel, you need to install both the linux-image and linux-image-extra .deb packages.

As notified on on IRC, that is a good kernel.

The issue is easy to reproduce on an X1 Carbon just run `stress -c 4` and within a few seconds mouse starts to lag and everything becomes slow.

The patch author requested that we provide a /proc/sched_debug dump while the issue is happening. Can you collect that data?

Upstream also requested testing of the following repo:

git://git.kernel.org/pub/scm/linux/kernel/git/tip/tip.git

I built a test kernel with this tree. It can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/upstream/

Can you test this kernel and see if it exhibits the bug or not?

I tested the kernel tip from your debs and the issue is very much alive. Here is the requested dump: http://paste.ubuntu.com/23312351/

Update and request from upstream:

"I have looked at the dump and there is something very odd for
system.slice task group where the display manager is running.
system.slice->tg_load_avg is around 381697 but tg_load_avg is
normally equal to Sum of system.slice[cpu]->tg_load_avg_contrib
whereas Sum of system.slice[cpu]->tg_load_avg_contrib = 1013 in our
case. We can have some differences because the dump of
/proc/shed_debug is not atomic and some changes can happen but nothing
like this difference.

The main effect of this quite high value is that the weight/prio of
the sched_entity that represents system.slice in root cfs_rq is very
low (lower than task with the smallest nice prio) so the system.slice
task group will not get the CPU quite often compared to the user.slice
task group: less than 1% for the system.slice where lightDM and xorg
are running compared 99% for the user.slice where the stress tasks are
running. This is confirmed by the se->avg.util_avg value of the task
groups which reflect how much time each task group is effectively
running on a CPU:
system.slice[CPU3].se->avg.util_avg = 8 whereas
user.slice[CPU3].se->avg.util_avg = 991

This difference of weight/priority explains why the system becomes
unresponsive. For now, I can't explain is why
system.slice->tg_load_avg = 381697 whereas is should be around 1013
and how the patch can generate this situation.

Is it possible to have a dump of /proc/sched_debug before starting
stress command ? to check if the problem is there from the beginning
but not seen because not overloaded. Or if it the problem comes when
user starts to load the system"

I have attached two dumps. One with the buggy kernel and the other with the kernel that does not show the lag.

From Upstream:

"> Here is the dump before stress is started:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1627108/+attachment/4760437/+files/dump_nonbuggy

This one is ok.
The dump indicates Sched Debug Version: v0.11, 4.8.0-11-generic
#12~lp1627108Commit3d30544Reverted
so this is without the culprit commit

>
> Here it is after:
> https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1627108/+attachment/4760436/+files/dump_buggy
>

This one has the exact same odds values for system.slice->tg_load_avg
than the 1st dump that you sent yesterday
The dump indicates Sched Debug Version: v0.11, 4.8.0-22-generic #24-Ubuntu
So this dump has been done with a different kernel than for the dump above.
As I can't find any stress task in the dump, i tend to believe that
the dump has been done before starting the stress tasks and not after
starting them. Can you confirm ?

If i'm right, it mean that the problem was already there before
starting stress tasks."

That's correct, that dump_buggy is without stress running.

I built a test kernel with the proposed patch from upstream. The test kernel can be downloaded from:
http://kernel.ubuntu.com/~jsalisbury/lp1627108/upstream/

Can you test this kernel and see if it resolves this bug?