memory allocate issue of windows vm

Bug #1916313 reported by HYSong
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
qemu (Ubuntu)
Expired
Undecided
Unassigned

Bug Description

I found the windows vm that I boot with 2C16G40G will allocate all memory in compute node, and it could only be set memory allocation manually with `virsh qemu-monitor-command <vm-id> --hmp "balloon 8192"` to change actual memory usage in compute node.
In the meantime, the reduced memory in compute node will be increase in the inner windows system.
In contrast, The memory of Linux vm can be allocated automatically by balloon, and allocate actual memory by check top command in compute node.

I think the windows vm should allocate memory automatically like Linux vm.
And if I set memory manually, the reduced memory usage in compute node should not transform into the usage memory of inner windows system and reduce the available memory.
Is it a qemu bug or my incorrect configurations?

Here is my env info:
----------------------
root@cmp001:/# virsh version
Compiled against library: libvirt 6.0.0
Using library: libvirt 6.0.0
Using API: QEMU 6.0.0
Running hypervisor: QEMU 4.2.1

root@cmp001:~# virsh qemu-monitor-command instance-000112f4 --hmp "info balloon"
balloon: actual=16384

root@cmp001:~# top -p 8977
top - 10:12:37 up 14 days, 16:05, 1 user, load average: 5.12, 5.29, 5.09
Tasks: 1 total, 0 running, 1 sleeping, 0 stopped, 0 zombie
%Cpu(s): 9.6 us, 5.5 sy, 0.0 ni, 84.5 id, 0.0 wa, 0.0 hi, 0.4 si, 0.0 st
KiB Mem : 13188148+total, 23065860 free, 88282144 used, 20533472 buff/cache
KiB Swap: 8388604 total, 7387032 free, 1001572 used. 38870928 avail Mem

  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
 8977 libvirt+ 20 0 18.316g 0.015t 30668 S 9.7 12.5 1938:22 qemu-system-x86

root@cmp001:~# virsh qemu-monitor-command instance-000112f4 --hmp "balloon 8192"

root@cmp001:~# virsh qemu-monitor-command instance-000112f4 --hmp "info balloon"
balloon: actual=8192

the in-used memory of inner windows system will be increase and available memory decreased in windows resource monitor.

Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thank you for your bug report. I've subscribe Christian to it, who is our qemu expert. He will take a look as time permits.

Revision history for this message
HYSong (songhongyuan) wrote :

@Sergio Durigan Junior (sergiodj) Thanks for all your assistance, I'm looking forward to your recovery.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi, this somewhat sounds like beign affected by translation issues. So if I misunderstood you, please let me know and try try to clarify the misunderstanding.

> I found the windows vm that I boot with 2C16G40G

Which number is that?
Is this especially about very huge instances or was that meaning you see the effects no matter if you use 2G or 16G or 40G?

> will allocate all memory in compute node, and it could only be set memory allocation manually with `virsh qemu-monitor-command <vm-id> --hmp "balloon 8192"` to change actual memory usage in compute node.

Be careful about what you see.
The balloon is likely doing exactly what it was told to. That will from the guest look like it would have allocated 8GB memory (preventing that other bits in the guest can grab it). But from the Host POV this memory isn't really allocated and can be used for something else.
The WIndows OS (just like Linux) I'd expect to allocate from the remaining memory on demand, but again be aware that both will massively use caching - so over time all non-balloon-consumed memory can often be expected to be fully allocated by the guest.

> In the meantime, the reduced memory in compute node will be increase in the inner windows
> system.. In contrast, The memory of Linux vm can be allocated automatically by balloon, and
> allocate actual memory by check top command in compute node.

Again the balloon is meant to appear as if allocating/consuming memory from the guests POV.
So that does not seem to be wrong unless I misunderstood you.

> I think the windows vm should allocate memory automatically like Linux vm.

It should already ...

> And if I set memory manually, the reduced memory usage in compute node should not transform
> into the usage memory of inner windows system and reduce the available memory.

If you set the balloon size you set the size of memory that is fake-consumed from the guests POV.
That means:
- if you increase the balloon size then less will be left usable from the guests POV. And since thereby it prevents the guest from really allocating that memory from the hosts POV the host can use it elsewhere
- if you reduce the balloon size then at first no further allocation happens. All that happens is the balloon deflating which appears to the guest as if more memory now would be free. If the guest consumes this or not is depending on configuration and workload.

> Is it a qemu bug or my incorrect configurations?

Right now to me it feels more like a misconfig - or actually a conceptual misunderstanding or wrong exectations - than a bug.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Thank you for taking the time to report this bug and helping to make Ubuntu better.

Since it seems likely to me that this is a local configuration problem, rather than a bug in Ubuntu, I'm marking this bug as Incomplete.

If indeed this is a local configuration problem, you can find pointers to get help for this sort of problem here: http://www.ubuntu.com/support/community

Or if you believe that this is really a bug, then you may find it helpful to read "How to report bugs effectively" http://www.chiark.greenend.org.uk/~sgtatham/bugs.html. We'd be grateful if you would then provide a more complete description of the problem, explain why you believe this is a bug in Ubuntu rather than a problem specific to your system, and then change the bug status back to New.

Changed in qemu (Ubuntu):
status: New → Incomplete
Revision history for this message
HYSong (songhongyuan) wrote :

Hi Christian, thanks for your explanations.

First of all, 2C16G40G is my vm flavor specification(2 vcpus, 16G ram, 40G disk), and this issue occurred regardless of what ram configured.

In short, this issue is that the compute node will allocated 16G ram by `top -p <vm-process>` when I boot a 16G ram windows vm instead of the actual ram.

On the contrary, the linux VM I boot will allocate actual ram by top command.

And I have to execute `virsh qemu-monitor-command <instance-id> --hmp "balloon <size>"` manually to decrease the ram in order to achieve allocate more ram.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote : Re: [Bug 1916313] Re: memory allocate issue of windows vm

> First of all, 2C16G40G is my vm flavor specification(2 vcpus, 16G ram,
> 40G disk), and this issue occurred regardless of what ram configured.

Thanks for clarifying

> In short, this issue is that the compute node will allocated 16G ram by
> `top -p <vm-process>` when I boot a 16G ram windows vm instead of the
> actual ram.

Ok, that is odd but then a windows behavior and nothing that qemu/kvm "do".
There are only a few things the host can do to influence this, mostly the so
called HV-enlightenments which you can control.
https://github.com/qemu/qemu/blob/master/docs/hyperv.txt
But none of those I'd know to be about memory allocation.

And if not this, then this is about windows configuration. But I can't
add a windows bug-task here :-)

I'd be more concerned if you'd say up to Ubuntu/qemu version X this
was fine, but now with version Y the guest behaves differently.
But as far as I understood this always was the case for your setup right?

> On the contrary, the linux VM I boot will allocate actual ram by top
> command.

Still you'll see it converge onto all-ram over it's life-cycle due to caches.

Also be careful that for checking memory consumption top is probably
the least reliable tool.
There is always "more detail", but I'd at least use the following:
For a guest I had "h-qemu-modules"
  $ pidstat -p $(pgrep -f 'name guest=h-qemu-modules') -T ALL -tr 5
And in general only care about RSS not VSS in any of those tools.
Actually even RSS can't be just added up, but that is a different topic.

> And I have to execute `virsh qemu-monitor-command <instance-id> --hmp
> "balloon <size>"` manually to decrease the ram in order to achieve
> allocate more ram.

Well, it would not have any other chance. By increasing the balloon
you steal away
that much from the guest and free it up from the Hosts POV. No
surprise on this one.

Revision history for this message
zhaoleilc (zhaoleilc) wrote :

I encountered the same question too, the issue is severe!

Revision history for this message
HYSong (songhongyuan) wrote :

Hi Christian, I have tested this problem recently and found that the windows VM memory usage in the new compute node will turn to actual value if I execute `VM live migrate` between two compute node which is created by openstack or virshinstall.

I guess it is a problem with qemu memory allocate.
Could you please boot a windows VM and migrate it to test is issue?

In addition, If I boot a windows VM with 16G memory and migrate it, the actual memory usage is decrease to 1.3G in new compute node, it sounds like a good news, but the actual memory usage will up to 8G in compute node if I execute `virsh qemu-monitor-command instance-000112f4 --hmp "balloon 8192"` instead of the actual usage, I'm really confused about it.

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :
Download full text (41.0 KiB)

Summary on top to be more readable
TBH - while I could confirm and no finally see myself what you reported
I didn't seen anything of that to be likely in qemu/kvm but instead this is
the windows memory management. Qemu/KVM just responds with the allocations
for qemu to the pages touched (or freed) by the Guest OS.
And using an OS I somewhat understand and can follow the memory management
actions it totally behaves as expected.

I'd think you would need to talk to a Windows Expert (or their support)
and ask about that behavior. If anything KVM related then maybe the
virtio-balloon driver might be affected, but AFAICS it seems it would not "do"
this but only might "trigger" the windows behavior that we see in effect here.

An alternative might be to report it to qemu mailing [1] list as I'm sure
many others have stumbled, wondered and accepted this as windows behavior.
If that assumption is true they might have the right pointers to explain it for
you.

[1]: https://lists.nongnu.org/mailman/listinfo/qemu-discuss

What can we see in the detailed log below
#1.1 shows that host and guest agree with ~900M of the 12G being used after boot
#1.2 increasing the balloon does neither change VSS (max process allocation) nor
       RSS (actual allocation)
#1.2 The guest accessible memory is reduced to 8G as intended
#1.3 we see RSS rising to match the real usage (4g plus some overhead).
#1.3 we also see that ~3.2G are still totally unused by the guest
#1.4.1 we see that the host does not realize it could free some mem again
       (still 5.8G RSS)
#1.4.2 we see the mem is back free again from the guest POV
#1.5 as expected inflating the balloon in the guest makes it let go of memory
       and real usgae on the host (RSS) shrinks again
#1.6 We see that the host took a chance to reduce effective allocation a bit
       (expected) Think of things like de-duplictaion or pruning for page where
       flags allow it to do so.
       But from the Guest POV nothing changed at all (expected)
       Migrating back it even had a bit more RSS again.

So far all cases behaved as expected, nothing but the max mem and the baloon
size is really static. Qemu does fault-in pages on demand and can only let them
go if the guest let's them go (or via deup and such). Now is this different
with Windows ?

#2.1 right off boot it is more memory hungry than a linux and it seems to
     always consume ~8G. The installer allocated the same and this is without
     any workload and seems independent to the sizing.
#2.2.1 We see RSS drop as expected as response to the balloon increase
#2.2.2 We see that the guest "available" memory was reduced (as expected)
       The balloon seems to be accounted to the "in use" section of the windows
       mem stats which is ok, from the guests POV
#2.5 Increasing the balloon is pushing down the effective memory allocation on
     the host as expected
#2.6 as soon as the memory in the balloon is let go the guest consumes it again
     and scales up to 8G
     I checked the numbers and this does not seem anything that qemu/kvm does
     it is windows that touches this memory as soon as it is made available
     by the balloon deflatin...

Changed in qemu (Ubuntu):
status: Incomplete → Confirmed
Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

I didn't have any luck finding a root cause or having another case that would shed more light on this. I want to reflect this properly that it isn't really actionable for me atm and set it to incomplete.

Changed in qemu (Ubuntu):
status: Confirmed → Incomplete
Revision history for this message
Launchpad Janitor (janitor) wrote :

[Expired for qemu (Ubuntu) because there has been no activity for 60 days.]

Changed in qemu (Ubuntu):
status: Incomplete → Expired
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.