Bug #1980401 “Qemu memory leak” : Bugs : qemu package : Ubuntu

Revision history for this message

Lena Voytek (lvoytek) wrote on 2022-07-01:

#1

Hello,
Thank you for submitting this bug report. I attempted to reproduce the issue on my 22.04 system with 32gb of RAM, allocating 12 to a qemu vm instance, but didn't run into a memory leak. Although it may just be difficult to encounter unless the right conditions are met.

As the RAM usage is increasing are you able to see any specific process taking up a large amount of it either in the system monitor processes tab or in htop?

Also would you be able to share the XML config of your virtual system? It may help determine the cause of the leak.

Thanks!

Changed in qemu (Ubuntu):
status:	New → Incomplete

Revision history for this message

Lucas Zanella (lucaszanella) wrote on 2022-07-01:

#2

Download full text (8.0 KiB)

No processes are taking this ammount of RAM. firefox is taking 332mb, proton vpn 800mb, and qemu shows with 149mb. I have no other programs in my host ubuntu, only virt-manager, I do everything inside ubuntu VMs for safety.

I left my computer running without virt-manager and it consumed around 7gb of RAM. Once I start my VM, ram starts filling until 32gb are reached after 1 hour or so, and then firefox starts closing tabs and I get freezes. It's getting difficult to work with this problem.

I tried without the virgl OpenGL renderer as well and same problem.

Here's my XML

<domain type="kvm">
  <name>myvm1</name>
  <uuid>7b22391a-ee8c-4259-8883-daa88a4559ac</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">12288000</memory>
  <currentMemory unit="KiB">12288000</currentMemory>
  <memoryBacking>
    <source type="memfd"/>
    <access mode="shared"/>
  </memoryBacking>
  <vcpu placement="static">6</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-6.2">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/orwell.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller...

No processes are taking this ammount of RAM. firefox is taking 332mb, proton vpn 800mb, and qemu shows with 149mb. I have no other programs in my host ubuntu, only virt-manager, I do everything inside ubuntu VMs for safety.

I left my computer running without virt-manager and it consumed around 7gb of RAM. Once I start my VM, ram starts filling until 32gb are reached after 1 hour or so, and then firefox starts closing tabs and I get freezes. It's getting difficult to work with this problem.

I tried without the virgl OpenGL renderer as well and same problem.

Here's my XML

<domain type="kvm">
  <name>myvm1</name>
  <uuid>7b22391a-ee8c-4259-8883-daa88a4559ac</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">12288000</memory>
  <currentMemory unit="KiB">12288000</currentMemory>
  <memoryBacking>
    <source type="memfd"/>
    <access mode="shared"/>
  </memoryBacking>
  <vcpu placement="static">6</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-6.2">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/orwell.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <interface type="network">
      <mac address="52:54:00:1f:e1:80"/>
      <source network="default"/>
      <model type="virtio"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </interface>
    <serial type="pty">
      <target type="isa-serial" port="0">
        <model name="isa-serial"/>
      </target>
    </serial>
    <console type="pty">
      <target type="serial" port="0"/>
    </console>
    <channel type="unix">
      <target type="virtio" name="org.qemu.guest_agent.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="1"/>
    </channel>
    <channel type="spicevmc">
      <target type="virtio" name="com.redhat.spice.0"/>
      <address type="virtio-serial" controller="0" bus="0" port="2"/>
    </channel>
    <input type="tablet" bus="usb">
      <address type="usb" bus="0" port="1"/>
    </input>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice">
      <listen type="none"/>
      <image compression="off"/>
      <gl enable="yes" rendernode="/dev/dri/by-path/pci-0000:00:02.0-render"/>
    </graphics>
    <sound model="ich9">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1b" function="0x0"/>
    </sound>
    <audio id="1" type="spice"/>
    <video>
      <model type="virtio" heads="1" primary="yes">
        <acceleration accel3d="yes"/>
      </model>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </memballoon>
    <rng model="virtio">
      <backend model="random">/dev/urandom</backend>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </rng>
  </devices>
</domain>

Revision history for this message

Joril (jorilx) wrote on 2022-07-26:

#3

I may have a similar problem... Can you post the output of
slabtop -o -sc
?

Revision history for this message

Lucas Zanella (lucaszanella) wrote on 2022-07-26:

#4

Download full text (17.1 KiB)

Active / Total Objects (% used) : 7182629 / 7337467 (97,9%)
Active / Total Slabs (% used) : 146316 / 146316 (100,0%)
Active / Total Caches (% used) : 157 / 216 (72,7%)
Active / Total Size (% used) : 1159973,65K / 1193486,03K (97,2%)
Minimum / Average / Maximum Object : 0,01K / 0,16K / 16,00K

OBJS ACTIVE 5384128 5355671 99% 10632 10626 99% 126060 126036 99% 10748 10674 99% 57792 57568 99% 63072 62335 98% 150003 135089 90% 167832 167254 99% 73521 70943 96% 36525 33211 90% 71370 61357 85% 25284 23666 93% 57816 57780 99% 124176 108407 87% 2976 2817 94% 1384 1310 94% 70528 69835 99% 121984 113143 92% 3136 2995 95% 11936 9739 81% 57330 51105 89% 160384 149885 93% 6348 6070 95% 5733 4807 83% 4080 1774 43% 16832 14145 84% 936 59136 59136 100% 123930 120281 97% 15057 14798 98% 40704 29007 71% 7825 6235 79% 3042 2565 84% 4960 2801 56% 9696 8076 83% 8672 8398 96% USE OBJ SIZE SLABS OBJ/SLAB CACHE SIZE NAME
0,06K 84127 64 336508K vmap_area
16,00K 5316 2 170112K zio_buf_comb_16384
0,97K 3820 33 122240K dnode_t
8,00K 2687 4 85984K kmalloc-8k
1,12K 2064 28 66048K zfs_znode_cache
1,00K 1971 32 63072K kmalloc-1k
0,38K 7143 21 57144K arc_buf_hdr_t_full_crypt
0,19K 7992 21 31968K dentry
0,38K 3501 21 28008K dmu_buf_impl_t
0,62K 1461 25 23376K inode_cache
0,20K 1830 39 14640K vm_area_struct
0,57K 903 28 14448K radix_tree_node
0,24K 1752 33 14016K sa_cache
0,10K 3184 39 12736K abd_t
4,00K 372 8 11904K kmalloc-4k
7,88K 346 4 11072K task_struct
0,12K 2204 32 8816K kernfs_node_cache
0,06K 1906 64 7624K kmalloc-64
2,00K 196 16 6272K kmalloc-2k
0,50K 373 32 5968K kmalloc-512
0,09K 1365 42 5460K kmalloc-96
0,03K 1253 128 5012K kmalloc-32
0,70K 276 23 4416K proc_inode_cache
0,74K 273 21 4368K shmem_inode_cache
1,06K 136 30 4352K dmaengine-unmap-128
0,25K 526 32 4208K filp
614 65% 4,00K 117 8 3744K zfs_btree_leaf_cache
0,06K 924 64 3696K kmalloc-rcl-64
0,02K 729 170 2916K lsm_file_cache
0,19K 717 21 2868K kmalloc-192
0,06K 636 64 2544K anon_vma_chain
0,31K 313 25 2504K arc_buf_hdr_t_full
0,81K 78 39 2496K sock_inode_cache
0,50K 155 32 2480K zio_buf_comb_512
0,25K 303 32 2424K kmalloc-256
0,25K 271 ...

Active / Total Objects (% used)    : 7182629 / 7337467 (97,9%)
 Active / Total Slabs (% used)      : 146316 / 146316 (100,0%)
 Active / Total Caches (% used)     : 157 / 216 (72,7%)
 Active / Total Size (% used)       : 1159973,65K / 1193486,03K (97,2%)
 Minimum / Average / Maximum Object : 0,01K / 0,16K / 16,00K

OBJS ACTIVE  USE OBJ SIZE  SLABS OBJ/SLAB CACHE SIZE NAME                   
5384128 5355671  99%    0,06K  84127       64    336508K vmap_area              
 10632  10626  99%   16,00K   5316        2    170112K zio_buf_comb_16384     
126060 126036  99%    0,97K   3820       33    122240K dnode_t                
 10748  10674  99%    8,00K   2687        4     85984K kmalloc-8k             
 57792  57568  99%    1,12K   2064       28     66048K zfs_znode_cache        
 63072  62335  98%    1,00K   1971       32     63072K kmalloc-1k             
150003 135089  90%    0,38K   7143       21     57144K arc_buf_hdr_t_full_crypt
167832 167254  99%    0,19K   7992       21     31968K dentry                 
 73521  70943  96%    0,38K   3501       21     28008K dmu_buf_impl_t         
 36525  33211  90%    0,62K   1461       25     23376K inode_cache            
 71370  61357  85%    0,20K   1830       39     14640K vm_area_struct         
 25284  23666  93%    0,57K    903       28     14448K radix_tree_node        
 57816  57780  99%    0,24K   1752       33     14016K sa_cache               
124176 108407  87%    0,10K   3184       39     12736K abd_t                  
  2976   2817  94%    4,00K    372        8     11904K kmalloc-4k             
  1384   1310  94%    7,88K    346        4     11072K task_struct            
 70528  69835  99%    0,12K   2204       32      8816K kernfs_node_cache      
121984 113143  92%    0,06K   1906       64      7624K kmalloc-64             
  3136   2995  95%    2,00K    196       16      6272K kmalloc-2k             
 11936   9739  81%    0,50K    373       32      5968K kmalloc-512            
 57330  51105  89%    0,09K   1365       42      5460K kmalloc-96             
160384 149885  93%    0,03K   1253      128      5012K kmalloc-32             
  6348   6070  95%    0,70K    276       23      4416K proc_inode_cache       
  5733   4807  83%    0,74K    273       21      4368K shmem_inode_cache      
  4080   1774  43%    1,06K    136       30      4352K dmaengine-unmap-128    
 16832  14145  84%    0,25K    526       32      4208K filp                   
   936    614  65%    4,00K    117        8      3744K zfs_btree_leaf_cache   
 59136  59136 100%    0,06K    924       64      3696K kmalloc-rcl-64         
123930 120281  97%    0,02K    729      170      2916K lsm_file_cache         
 15057  14798  98%    0,19K    717       21      2868K kmalloc-192            
 40704  29007  71%    0,06K    636       64      2544K anon_vma_chain         
  7825   6235  79%    0,31K    313       25      2504K arc_buf_hdr_t_full     
  3042   2565  84%    0,81K     78       39      2496K sock_inode_cache       
  4960   2801  56%    0,50K    155       32      2480K zio_buf_comb_512       
  9696   8076  83%    0,25K    303       32      2424K kmalloc-256            
  8672   8398  96%    0,25K    271       32      2168K skbuff_head_cache      
  3105   3105 100%    0,69K    135       23      2160K squashfs_inode_cache   
 21620  15078  69%    0,09K    470       46      1880K anon_vma               
  9988   9554  95%    0,18K    454       22      1816K kvm_mmu_page_header    
   104    104 100%   12,00K     52        2      1664K zio_buf_comb_10240     
  4800   4747  98%    0,31K    192       25      1536K mnt_cache              
   660    652  98%    2,06K     44       15      1408K sighand_cache          
  1320   1248  94%    1,06K     44       30      1408K UNIX                   
 16524  13685  82%    0,08K    324       51      1296K arc_buf_t              
  1216   1009  82%    1,00K     38       32      1216K kmalloc-cg-1k          
  1064    956  89%    1,12K     38       28      1216K signal_cache           
  1875   1377  73%    0,62K     75       25      1200K task_group             
 16520  16520 100%    0,07K    295       56      1180K Acpi-Operand           
  9184   7798  84%    0,12K    287       32      1148K kmalloc-128            
   735    651  88%    1,50K     35       21      1120K zio_buf_comb_1536      
   340    300  88%    3,00K     34       10      1088K zio_buf_comb_3072      
   306    270  88%    3,50K     34        9      1088K zio_buf_comb_3584      
  1056    940  89%    1,00K     33       32      1056K zio_buf_comb_1024      
   512    512 100%    2,00K     32       16      1024K kmalloc-cg-2k          
   992    864  87%    1,00K     31       32       992K RAW                    
  1500   1207  80%    0,62K     60       25       960K i915_request           
   232    232 100%    4,00K     29        8       928K biovec-max             
   464    416  89%    2,00K     29       16       928K zio_buf_comb_2048      
   728    602  82%    1,19K     28       26       896K RAWv6                  
   728    728 100%    1,22K     28       26       896K zio_cache              
   336    288  85%    2,50K     28       12       896K zio_buf_comb_2560      
  1760   1760 100%    0,50K     55       32       880K kmalloc-cg-512         
    52     52 100%   12,00K     26        2       832K zio_buf_comb_12288     
    50     44  88%   16,00K     25        2       800K zio_buf_comb_14336     
  3864   3864 100%    0,19K    184       21       736K proc_dir_entry         
   176    176 100%    4,00K     22        8       704K names_cache            
  1408   1250  88%    0,50K     44       32       704K skbuff_fclone_cache    
   168    147  87%    4,00K     21        8       672K kmalloc-cg-4k          
   168    131  77%    4,00K     21        8       672K zio_buf_comb_4096      
    84     72  85%    8,00K     21        4       672K zio_buf_comb_6144      
 14110  14110 100%    0,05K    166       85       664K ftrace_event_field     
    80     80 100%    8,00K     20        4       640K zio_buf_comb_8192      
    76     60  78%    8,00K     19        4       608K zio_buf_comb_5120      
    76     64  84%    8,00K     19        4       608K zio_buf_comb_7168      
 35328  31520  89%    0,02K    138      256       552K kmalloc-16             
  2709   2709 100%    0,19K    129       21       516K cred_jar               
  4032   3556  88%    0,12K    126       32       504K pid                    
  5612   5612 100%    0,09K    122       46       488K trace_event_file       
   360    360 100%    1,31K     15       24       480K UDPv6                  
  1824   1824 100%    0,25K     57       32       456K pool_workqueue         
  2100   2100 100%    0,19K    100       21       400K kmalloc-cg-192         
   168    168 100%    2,19K     12       14       384K TCP                    
  7990   7990 100%    0,05K     94       85       376K zio_link_cache         
 11136   9778  87%    0,03K     87      128       348K fsnotify_mark_connector
  1075    747  69%    0,31K     43       25       344K nf_conntrack           
   882    757  85%    0,37K     42       21       336K zil_lwb_cache          
   270    270 100%    1,06K      9       30       288K mm_struct              
    63     63 100%    4,25K      9        7       288K net_namespace          
   504    504 100%    0,56K     18       28       288K kioctx                 
    32     32 100%    8,00K      8        4       256K kmalloc-cg-8k          
   216    216 100%    1,16K      8       27       256K perf_event             
   256    256 100%    1,00K      8       32       256K biovec-64              
   120    120 100%    2,08K      8       15       256K request_queue          
   272    272 100%    0,94K      8       34       256K mqueue_inode_cache     
   104    104 100%    2,38K      8       13       256K TCPv6                  
    16     16 100%   16,00K      8        2       256K lz4_cache              
   322    322 100%    0,69K     14       23       224K files_cache            
   140    140 100%    1,56K      7       20       224K bdev_cache             
    21     21 100%   10,62K      7        3       224K kvm_vcpu               
  2310   2310 100%    0,09K     55       42       220K kmalloc-rcl-96         
   234    234 100%    0,81K      6       39       192K fuse_inode             
    72     72 100%    2,61K      6       12       192K x86_emulator           
  1287   1287 100%    0,10K     33       39       132K buffer_head            
   208    208 100%    0,61K      8       26       128K user_namespace         
 15360  15360 100%    0,01K     30      512       120K kmalloc-8              
   259    259 100%    0,42K      7       37       112K uts_namespace          
  1728   1728 100%    0,06K     27       64       108K kmalloc-cg-64          
  1326   1326 100%    0,08K     26       51       104K sigqueue               
  6400   6400 100%    0,02K     25      256       100K kmalloc-cg-16          
   384    384 100%    0,25K     12       32        96K kmem_cache             
   117    117 100%    0,81K      3       39        96K dax_cache              
    81     81 100%    1,15K      3       27        96K ext4_inode_cache       
   672    672 100%    0,12K     21       32        84K kmalloc-rcl-128        
   320    320 100%    0,25K     10       32        80K kmalloc-cg-256         
   867    867 100%    0,08K     17       51        68K Acpi-State             
   184    184 100%    0,34K      8       23        64K taskstats              
   296    296 100%    0,21K      8       37        64K file_lock_cache        
    32     32 100%    2,00K      2       16        64K biovec-128             
    96     96 100%    0,65K      4       24        64K hugetlbfs_inode_cache  
   264    264 100%    0,24K      8       33        64K tw_sock_TCP            
    78     78 100%    0,81K      2       39        64K QIPCRTR                
   294    294 100%    0,19K     14       21        56K skbuff_ext_cache       
    63     63 100%    0,76K      3       21        48K fat_inode_cache        
  1408   1408 100%    0,03K     11      128        44K kmalloc-cg-32          
   730    730 100%    0,05K     10       73        40K file_lock_ctx          
   576    576 100%    0,06K      9       64        36K kmem_cache_node        
  4096   4096 100%    0,01K      8      512        32K kmalloc-cg-8           
   336    336 100%    0,09K      8       42        32K kmalloc-cg-96          
   256    256 100%    0,12K      8       32        32K kmalloc-cg-128         
   312    312 100%    0,10K      8       39        32K Acpi-ParseExt          
   448    448 100%    0,07K      8       56        32K nsproxy                
   272    272 100%    0,12K      8       34        32K seq_file               
   816    816 100%    0,04K      8      102        32K pde_opener             
    15     15 100%    2,06K      1       15        32K dmaengine-unmap-256    
  2048   2048 100%    0,02K      8      256        32K ep_head                
   584    584 100%    0,05K      8       73        32K ip_fib_alias           
   680    680 100%    0,05K      8       85        32K ip_fib_trie            
    11     11 100%    2,75K      1       11        32K iommu_domain           
   208    208 100%    0,15K      8       26        32K fuse_request           
   256    256 100%    0,12K      8       32        32K i915_dependency        
   168    168 100%    0,19K      8       21        32K kcf_context_cache      
   208    208 100%    0,15K      8       26        32K zil_zcw_cache          
   368    368 100%    0,09K      8       46        32K zfs_znode_hold_cache   
    34     34 100%    0,94K      1       34        32K bio-936                
   240    240 100%    0,13K      8       30        32K kvm_async_pf           
   144    144 100%    0,11K      4       36        16K khugepaged_mm_slot     
    63     63 100%    0,19K      3       21        12K kmalloc-rcl-192        
   306    306 100%    0,04K      3      102        12K bio_crypt_ctx          
    31     31 100%    0,26K      1       31         8K numa_policy            
    32     32 100%    0,25K      1       32         8K dquot                  
   204    204 100%    0,04K      2      102         8K ext4_extent_status     
    30     30 100%    0,13K      1       30         4K pid_namespace          
    73     73 100%    0,05K      1       73         4K mbcache                
   256    256 100%    0,02K      1      256         4K jbd2_revoke_table_s    
    34     34 100%    0,12K      1       34         4K jbd2_journal_head      
   170    170 100%    0,02K      1      170         4K mod_hash_entries       
    21     21 100%    0,19K      1       21         4K ext4_groupinfo_4k      
     0      0   0%    0,01K      0      512         0K kmalloc-rcl-8          
     0      0   0%    0,02K      0      256         0K kmalloc-rcl-16         
     0      0   0%    0,03K      0      128         0K kmalloc-rcl-32         
     0      0   0%    0,25K      0       32         0K kmalloc-rcl-256        
     0      0   0%    0,50K      0       32         0K kmalloc-rcl-512        
     0      0   0%    1,00K      0       32         0K kmalloc-rcl-1k         
     0      0   0%    2,00K      0       16         0K kmalloc-rcl-2k         
     0      0   0%    4,00K      0        8         0K kmalloc-rcl-4k         
     0      0   0%    8,00K      0        4         0K kmalloc-rcl-8k         
     0      0   0%    0,09K      0       42         0K dma-kmalloc-96         
     0      0   0%    0,19K      0       21         0K dma-kmalloc-192        
     0      0   0%    0,01K      0      512         0K dma-kmalloc-8          
     0      0   0%    0,02K      0      256         0K dma-kmalloc-16         
     0      0   0%    0,03K      0      128         0K dma-kmalloc-32         
     0      0   0%    0,06K      0       64         0K dma-kmalloc-64         
     0      0   0%    0,12K      0       32         0K dma-kmalloc-128        
     0      0   0%    0,25K      0       32         0K dma-kmalloc-256        
     0      0   0%    0,50K      0       32         0K dma-kmalloc-512        
     0      0   0%    1,00K      0       32         0K dma-kmalloc-1k         
     0      0   0%    2,00K      0       16         0K dma-kmalloc-2k         
     0      0   0%    4,00K      0        8         0K dma-kmalloc-4k         
     0      0   0%    8,00K      0        4         0K dma-kmalloc-8k         
     0      0   0%    0,12K      0       34         0K iint_cache             
     0      0   0%    0,78K      0       20         0K x86_lbr                
     0      0   0%    0,30K      0       26         0K request_sock_TCP       
     0      0   0%    1,00K      0       32         0K PING                   
     0      0   0%    0,75K      0       21         0K xfrm_state             
     0      0   0%    0,31K      0       25         0K xfrm_dst_cache         
     0      0   0%    0,37K      0       21         0K request_sock_subflow   
     0      0   0%    1,88K      0       17         0K MPTCP                  
     0      0   0%    0,20K      0       20         0K ip4-frags              
     0      0   0%    0,03K      0      128         0K dnotify_struct         
     0      0   0%    0,19K      0       21         0K userfaultfd_ctx_cache  
     0      0   0%    0,03K      0      128         0K ext4_pending_reservation
     0      0   0%    0,06K      0       64         0K ext4_io_end            
     0      0   0%    0,14K      0       28         0K ext4_allocation_context
     0      0   0%    0,08K      0       51         0K ext4_fc_dentry_update  
     0      0   0%    0,04K      0      102         0K fat_cache              
     0      0   0%    0,81K      0       39         0K ecryptfs_auth_tok_list_item
     0      0   0%    0,02K      0      256         0K ecryptfs_file_cache    
     0      0   0%    1,00K      0       32         0K ecryptfs_inode_cache   
     0      0   0%    2,82K      0       11         0K dm_uevent              
     0      0   0%    3,16K      0       10         0K kcopyd_job             
     0      0   0%    0,30K      0       26         0K request_sock_TCPv6     
     0      0   0%    0,24K      0       33         0K tw_sock_TCPv6          
     0      0   0%    1,19K      0       26         0K PINGv6                 
     0      0   0%    0,18K      0       22         0K ip6-frags              
     0      0   0%    2,00K      0       16         0K MPTCPv6                
     0      0   0%    0,13K      0       30         0K fscrypt_info           
     0      0   0%    0,25K      0       32         0K fsverity_info          
     0      0   0%    0,06K      0       64         0K execute_cb             
     0      0   0%    0,19K      0       21         0K kcf_sreq_cache         
     0      0   0%    0,50K      0       32         0K kcf_areq_cache         
     0      0   0%    0,07K      0       56         0K metaslab_alloc_trace_cache
     0      0   0%    0,44K      0       36         0K ddt_entry_cache        
     0      0   0%    0,09K      0       42         0K arc_buf_hdr_t_l2only   
     0      0   0%    0,13K      0       30         0K sio_cache_0            
     0      0   0%    0,15K      0       26         0K sio_cache_1            
     0      0   0%    0,16K      0       24         0K sio_cache_2

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2022-07-26:

#5

Well - there is "only" ~1.2GB in the slab.
That isn't all you are looking for.

Let us check in general if we look at the kernel or userspace.
Maybe you could run and report here the output of

$ cat /proc/meminfo
$ sudo smem -tk

That might help further to include/exclude some things in out considerations.

Revision history for this message

Lucas Zanella (lucaszanella) wrote on 2022-07-26:

#6

Download full text (13.0 KiB)

cat /proc/meminfo
MemTotal: 32582696 kB
MemFree: 5706384 kB
MemAvailable: 5720280 kB
Buffers: 4564 kB
Cached: 2417768 kB
SwapCached: 120580 kB
Active: 1312224 kB
Inactive: 8002812 kB
Active(anon): 1095264 kB
Inactive(anon): 7849040 kB
Active(file): 216960 kB
Inactive(file): 153772 kB
Unevictable: 1155116 kB
Mlocked: 112 kB
SwapTotal: 2097148 kB
SwapFree: 1428220 kB
Dirty: 0 kB
Writeback: 0 kB
AnonPages: 7950296 kB
Mapped: 521400 kB
Shmem: 2053184 kB
KReclaimable: 86320 kB
Slab: 969548 kB
SReclaimable: 86320 kB
SUnreclaim: 883228 kB
KernelStack: 18688 kB
PageTables: 48116 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 18388496 kB
Committed_AS: 17846100 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 834476 kB
VmallocChunk: 0 kB
Percpu: 9664 kB
HardwareCorrupted: 0 kB
AnonHugePages: 4524032 kB
ShmemHugePages: 0 kB
ShmemPmdMapped: 0 kB
FileHugePages: 0 kB
FilePmdMapped: 0 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
HugePages_Surp: 0
Hugepagesize: 2048 kB
Hugetlb: 0 kB
DirectMap4k: 6036656 kB
DirectMap2M: 26173440 kB
DirectMap1G: 2097152 kB

sudo smem -tk
PID User Command Swap USS PSS RSS
4682 lz sh -c /usr/bin/ibus-daemon 104.0K 88.0K 95.0K 1.1M
4274 root fusermount3 -o rw,nosuid,no 4.0K 112.0K 121.0K 1.3M
2486 root /usr/sbin/acpid 0 252.0K 261.0K 1.4M
2535 avahi avahi-daemon: chroot helper 0 128.0K 273.0K 948.0K
2492 root /usr/sbin/cron -f -P 0 292.0K 302.0K 1.7M
3105 root /usr/sbin/dnsmasq --conf-fi 0 112.0K 330.0K 1.2M
2988 rtkit /usr/libexec/rtkit-daemon 0 396.0K 415.0K 2.1M
4385 lz /usr/libexec/goa-identity-s 1.7M 420.0K 465.0K 5.6M
4363 lz /usr/libexec/gdm-wayland-se 128.0K 444.0K 471.0K 4.1M
4797 lz /usr/libexec/ibus-memconf 276.0K 460.0K 494.0K 4.6M
4440 lz /usr/libexec/gnome-session- 0 484.0K 508.0K 3.7M
5147 kernoops /usr/sbin/kerneloops 4.0K 472.0K 516.0K 1.9M
5139 kernoops /usr/sbin/kerneloops --test 0 480.0K 523.0K 1.9M
4481 lz /usr/bin/dbus-daemon --conf 240.0K 452.0K 545.0K 2.7M
2508 root /usr/sbin/irqbalance --fore 0 540.0K 560.0K 2.4M
4814 lz /usr/libexec/ibus-portal 364.0K 532.0K 565.0K 4.6M
17121 lz /usr/bin/gpg-agent --superv 0 516.0K 571.0K 2.1M
3104 libvirt-dnsmasq /usr/sbin/dnsmasq --conf-fi 0 356.0K 584.0K 2.0M
4468 lz /usr/libexec/at-spi-bus-lau 348.0K 548.0K 584.0K 4.9M
4261 lz /usr/libexec/xdg-permission 0 632.0K 660.0K 4.3M
3756028 root sudo smem -tk 0 11...

cat /proc/meminfo
MemTotal:       32582696 kB
MemFree:         5706384 kB
MemAvailable:    5720280 kB
Buffers:            4564 kB
Cached:          2417768 kB
SwapCached:       120580 kB
Active:          1312224 kB
Inactive:        8002812 kB
Active(anon):    1095264 kB
Inactive(anon):  7849040 kB
Active(file):     216960 kB
Inactive(file):   153772 kB
Unevictable:     1155116 kB
Mlocked:             112 kB
SwapTotal:       2097148 kB
SwapFree:        1428220 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:       7950296 kB
Mapped:           521400 kB
Shmem:           2053184 kB
KReclaimable:      86320 kB
Slab:             969548 kB
SReclaimable:      86320 kB
SUnreclaim:       883228 kB
KernelStack:       18688 kB
PageTables:        48116 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    18388496 kB
Committed_AS:   17846100 kB
VmallocTotal:   34359738367 kB
VmallocUsed:      834476 kB
VmallocChunk:          0 kB
Percpu:             9664 kB
HardwareCorrupted:     0 kB
AnonHugePages:   4524032 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
FileHugePages:         0 kB
FilePmdMapped:         0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
Hugetlb:               0 kB
DirectMap4k:     6036656 kB
DirectMap2M:    26173440 kB
DirectMap1G:     2097152 kB

sudo smem -tk
  PID User     Command                         Swap      USS      PSS      RSS 
 4682 lz       sh -c /usr/bin/ibus-daemon    104.0K    88.0K    95.0K     1.1M 
 4274 root     fusermount3 -o rw,nosuid,no     4.0K   112.0K   121.0K     1.3M 
 2486 root     /usr/sbin/acpid                    0   252.0K   261.0K     1.4M 
 2535 avahi    avahi-daemon: chroot helper        0   128.0K   273.0K   948.0K 
 2492 root     /usr/sbin/cron -f -P               0   292.0K   302.0K     1.7M 
 3105 root     /usr/sbin/dnsmasq --conf-fi        0   112.0K   330.0K     1.2M 
 2988 rtkit    /usr/libexec/rtkit-daemon          0   396.0K   415.0K     2.1M 
 4385 lz       /usr/libexec/goa-identity-s     1.7M   420.0K   465.0K     5.6M 
 4363 lz       /usr/libexec/gdm-wayland-se   128.0K   444.0K   471.0K     4.1M 
 4797 lz       /usr/libexec/ibus-memconf     276.0K   460.0K   494.0K     4.6M 
 4440 lz       /usr/libexec/gnome-session-        0   484.0K   508.0K     3.7M 
 5147 kernoops /usr/sbin/kerneloops            4.0K   472.0K   516.0K     1.9M 
 5139 kernoops /usr/sbin/kerneloops --test        0   480.0K   523.0K     1.9M 
 4481 lz       /usr/bin/dbus-daemon --conf   240.0K   452.0K   545.0K     2.7M 
 2508 root     /usr/sbin/irqbalance --fore        0   540.0K   560.0K     2.4M 
 4814 lz       /usr/libexec/ibus-portal      364.0K   532.0K   565.0K     4.6M 
17121 lz       /usr/bin/gpg-agent --superv        0   516.0K   571.0K     2.1M 
 3104 libvirt-dnsmasq /usr/sbin/dnsmasq --conf-fi        0   356.0K   584.0K     2.0M 
 4468 lz       /usr/libexec/at-spi-bus-lau   348.0K   548.0K   584.0K     4.9M 
 4261 lz       /usr/libexec/xdg-permission        0   632.0K   660.0K     4.3M 
3756028 root     sudo smem -tk                      0   116.0K   672.0K     2.2M 
 2516 root     /usr/libexec/switcheroo-con        0   656.0K   684.0K     4.2M 
 5106 lz       /usr/libexec/gvfsd-metadata        0   708.0K   743.0K     4.8M 
 4240 lz       /usr/libexec/gvfsd-fuse /ru   256.0K   680.0K   745.0K     4.9M 
 2489 avahi    avahi-daemon: running [pc.l        0   620.0K   780.0K     2.5M 
 4683 lz       /usr/libexec/gsd-a11y-setti        0   816.0K   851.0K     4.8M 
 4357 lz       /usr/libexec/gvfs-goa-volum     4.0K   828.0K   857.0K     4.7M 
 4361 lz       /usr/libexec/goa-daemon         6.5M   752.0K   920.0K     9.1M 
1895953 root     /usr/libexec/ubuntu-advanta        0   860.0K   946.0K     5.8M 
 4699 lz       /usr/libexec/gsd-rfkill            0   920.0K   951.0K     4.8M 
 4605 lz       /usr/libexec/dconf-service         0   920.0K   952.0K     4.7M 
 4661 lz       /usr/libexec/at-spi2-regist        0   920.0K   955.0K     5.2M 
 2512 root     /usr/libexec/power-profiles     4.0K   936.0K   971.0K     5.0M 
 4227 lz       /usr/libexec/gvfsd            188.0K   900.0K   981.0K     5.4M 
 4256 lz       /usr/libexec/xdg-document-p    28.0K   964.0K     1.0M     5.4M 
 4409 lz       /usr/libexec/gvfs-mtp-volum     4.0K  1016.0K     1.0M     5.5M 
 4869 lz       /usr/libexec/ibus-engine-si   192.0K   976.0K     1.0M     5.3M 
 4420 lz       /usr/libexec/gvfs-gphoto2-v        0     1.0M     1.1M     5.2M 
 2408 systemd-oom /lib/systemd/systemd-oomd          0   892.0K     1.1M     4.3M 
16852 lz       /usr/lib/bluetooth/obexd           0     1.0M     1.1M     4.8M 
 4690 lz       /usr/libexec/gsd-housekeepi        0     1.1M     1.1M     5.5M 
 4730 lz       /usr/libexec/gsd-smartcard         0     1.1M     1.2M     5.4M 
 2728 root     /usr/libexec/boltd                 0     1.1M     1.2M     5.3M 
 2419 systemd-timesync /lib/systemd/systemd-timesy        0   988.0K     1.2M     4.7M 
 2485 root     /usr/libexec/accounts-daemo        0     1.2M     1.2M     5.1M 
 2529 root     /usr/sbin/zed -F                   0     1.2M     1.2M     3.2M 
 4700 lz       /usr/libexec/gsd-screensave        0     1.3M     1.3M     5.2M 
 4732 lz       /usr/libexec/gsd-sound          4.0K     1.3M     1.3M     6.0M 
 4393 lz       /usr/libexec/gvfs-afc-volum     4.0K     1.2M     1.3M     5.7M 
1418389 lz       /usr/libexec/gvfsd-dnssd --        0     1.2M     1.4M     7.0M 
3756027 root     sudo smem -tk                      0   576.0K     1.4M     4.4M 
1418340 lz       /usr/libexec/gvfsd-network         0     1.3M     1.5M     6.9M 
 2803 root     /usr/sbin/gdm3                     0     1.5M     1.5M     5.9M 
3706031 root     /usr/lib/NetworkManager/nm-        0     1.2M     1.5M     7.5M 
 4777 lz       /usr/libexec/gsd-disk-utili    12.0K     1.4M     1.6M     5.6M 
 5057 lz       /usr/libexec/gvfsd-trash --        0     1.5M     1.6M     6.3M 
 4698 lz       /usr/libexec/gsd-print-noti   104.0K     1.5M     1.6M     7.2M 
 4207 lz       /usr/bin/pipewire-media-ses     4.0K     1.5M     1.6M     3.7M 
 4590 lz       /usr/libexec/gnome-shell-ca     1.8M     1.4M     1.7M     8.7M 
 2490 root     /usr/lib/bluetooth/bluetoot        0     1.7M     1.7M     4.1M 
 4206 lz       /usr/bin/pipewire               4.0K     1.6M     1.7M     3.9M 
 2524 root     /usr/sbin/thermald --system     4.0K     1.6M     1.8M     7.4M 
 2507 root     /usr/libexec/iio-sensor-pro     4.0K     1.7M     1.8M     5.4M 
 2520 root     /lib/systemd/systemd-logind        0     1.5M     1.8M     5.5M 
 4366 lz       /usr/libexec/gnome-session-    96.0K     1.8M     1.9M     8.1M 
 2521 root     /lib/systemd/systemd-machin        0     1.7M     1.9M     5.1M 
2043690 lz       /usr/libexec/gvfsd-mtp --sp        0     1.8M     2.0M     7.8M 
 4882 lz       /usr/libexec/gsd-printer       56.0K     2.0M     2.1M     7.8M 
 4276 lz       /usr/bin/gnome-keyring-daem     8.0K     2.0M     2.1M     5.7M 
 4726 lz       /usr/libexec/gsd-sharing        8.0K     2.0M     2.2M     7.6M 
 4352 lz       /usr/libexec/gvfs-udisks2-v     4.0K     2.0M     2.2M     7.1M 
16149 lz       /snap/firefox/1551/usr/lib/     7.0M     1.2M     2.2M    10.2M 
3337246 root     /usr/sbin/cups-browsed             0     1.9M     2.2M     8.9M 
 4687 lz       /usr/libexec/gsd-datetime       4.0K     2.2M     2.2M     8.7M 
 3021 root     /usr/libexec/upowerd               0     2.2M     2.3M     7.0M 
 4219 lz       /usr/bin/dbus-daemon --sess   236.0K     2.2M     2.3M     4.6M 
 4453 lz       /usr/libexec/gnome-session-   936.0K     2.3M     2.4M     8.9M 
 6445 lz       /snap/firefox/1551/usr/lib/     8.1M     1.4M     2.4M    10.7M 
3664143 lz       bash                               0     2.4M     2.5M     4.0M 
 5402 root     /usr/sbin/virtlogd                 0     1.9M     2.6M    10.0M 
 2513 syslog   /usr/sbin/rsyslogd -n -iNON     8.0K     2.8M     2.9M     4.4M 
3337243 root     /usr/sbin/cupsd -l                 0     2.5M     2.9M    10.2M 
 4010 root     gdm-session-worker [pam/gdm     8.0K     2.8M     3.0M     8.0M 
 2494 messagebus @dbus-daemon --system --add   404.0K     2.9M     3.0M     5.2M 
 1368 root     /lib/systemd/systemd-udevd         0     3.1M     3.1M     4.7M 
 4976 lz       /usr/libexec/xdg-desktop-po        0     3.2M     3.3M     9.2M 
3706036 nm-openvpn /usr/sbin/openvpn --remote         0     2.7M     3.3M     7.7M 
 4685 lz       /usr/bin/ibus-daemon --pane     1.4M     3.5M     3.6M     7.8M 
 2526 root     /sbin/wpa_supplicant -u -s         0     3.2M     3.6M     7.0M 
 2727 root     /usr/sbin/ModemManager          8.0K     3.6M     3.7M     8.1M 
 4174 lz       /lib/systemd/systemd --user        0     3.0M     3.8M     8.2M 
 3634 root     /usr/libexec/packagekitd           0     3.6M     4.0M    12.0M 
 3912 colord   /usr/libexec/colord           112.0K     3.7M     4.0M     8.9M 
 2525 root     /usr/libexec/udisks2/udisks    28.0K     3.8M     4.0M     9.2M 
 4176 lz       (sd-pam)                      144.0K     3.4M     4.1M     5.8M 
13404 lz       /snap/firefox/1551/usr/lib/     8.6M     2.9M     4.2M    13.5M 
 4948 lz       /usr/bin/gjs /usr/share/gno     2.0M     3.4M     4.3M    12.8M 
 4610 lz       /usr/libexec/evolution-sour   260.0K     4.3M     4.7M    12.9M 
 2511 root     /usr/libexec/polkitd --no-d     4.0K     4.7M     4.8M     9.2M 
 4863 lz       /usr/libexec/evolution-addr   124.0K     4.0M     5.1M    14.4M 
 4692 lz       /usr/libexec/gsd-keyboard     708.0K     5.1M     5.3M    13.6M 
 4734 lz       /usr/libexec/gsd-wacom        540.0K     5.2M     5.4M    13.9M 
 4697 lz       /usr/libexec/gsd-power        672.0K     5.5M     5.9M    16.2M 
    1 root     /sbin/init splash                  0     4.5M     5.9M    10.3M 
 6562 lz       /usr/bin/snap userd             6.5M     5.9M     5.9M     6.7M 
 4686 lz       /usr/libexec/gsd-color        732.0K     5.6M     5.9M    16.0M 
 4651 lz       /usr/libexec/evolution-cale   372.0K     4.9M     6.0M    15.1M 
 4694 lz       /usr/libexec/gsd-media-keys   860.0K     5.7M     6.3M    17.2M 
 2416 systemd-resolve /lib/systemd/systemd-resolv   364.0K     7.0M     7.3M    11.7M 
 2497 root     /usr/sbin/NetworkManager --   136.0K     7.1M     7.3M    13.6M 
 5697 lz       /usr/libexec/ibus-x11         484.0K     7.1M     7.5M    17.0M 
 2510 root     /usr/bin/python3 /usr/bin/n     1.8M     7.5M     8.0M    14.8M 
 2760 root     /usr/bin/python3 /usr/share     1.4M     7.2M     8.0M    16.1M 
 5025 lz       /usr/libexec/xdg-desktop-po     1.6M     7.7M     8.0M    17.1M 
10773 lz       update-notifier                 1.5M     7.4M     8.0M    19.1M 
 4317 lz       /usr/libexec/tracker-miner-   824.0K     8.1M     8.6M    16.8M 
 4209 lz       /snap/snapd-desktop-integra     1.7M     8.3M    10.3M    18.5M 
 4798 lz       /usr/libexec/ibus-extension     2.2M    10.4M    10.7M    19.6M 
3662614 lz       /usr/bin/python3 /usr/lib/p        0    10.9M    11.6M    19.7M 
 5620 lz       /usr/libexec/gsd-xsettings      5.8M    11.8M    12.3M    23.7M 
 4208 lz       /usr/bin/pulseaudio --daemo     1.1M    10.6M    12.8M    23.1M 
 1317 root     /lib/systemd/systemd-journa        0    10.6M    14.3M    20.9M 
 4781 lz       /usr/libexec/evolution-data     3.1M    13.8M    14.9M    29.5M 
1359938 lz       /usr/bin/seahorse --gapplic        0    14.4M    15.3M    29.1M 
 2757 root     /usr/sbin/libvirtd              3.1M    15.8M    16.9M    26.3M 
 4658 lz       /usr/bin/gjs /usr/share/gno     3.8M    19.1M    19.9M    28.3M 
3756029 root     /usr/bin/python3 /usr/bin/s        0    19.5M    20.0M    24.1M 
1895803 root     python3 /usr/lib/software-p        0    20.5M    21.9M    34.1M 
 2515 root     /usr/lib/snapd/snapd            2.8M    22.6M    22.6M    24.0M 
 6496 lz       /snap/firefox/1551/usr/lib/     5.6M    19.5M    29.2M    74.5M 
3664117 lz       /usr/libexec/gnome-terminal   300.0K    18.1M    33.9M    66.5M 
3661515 lz       gjs /usr/share/gnome-shell/        0    19.9M    40.7M    77.8M 
 5598 lz       /usr/bin/Xwayland :0 -rootl    18.2M    10.6M    42.9M    80.7M 
2149136 lz       gnome-control-center            1.7M    40.3M    55.1M    95.5M 
3740716 lz       /snap/firefox/1551/usr/lib/        0    40.8M    59.7M   121.6M 
 6739 lz       /snap/firefox/1551/usr/lib/     8.0M    58.7M    67.5M   109.8M 
3742281 lz       /snap/firefox/1551/usr/lib/        0    63.1M    88.0M   156.0M 
 4982 lz       /usr/libexec/xdg-desktop-po    19.0M    75.2M   109.0M   158.2M 
3699404 lz       /snap/firefox/1551/usr/lib/     1.8M   179.0M   191.3M   242.0M 
 4479 lz       /usr/bin/gnome-shell           60.4M   197.2M   265.0M   355.8M 
 5156 lz       /usr/bin/python3 /usr/bin/v    65.5M   297.4M   329.2M   388.1M 
 5428 lz       /snap/firefox/1551/usr/lib/    56.6M   381.9M   405.1M   472.5M 
 5720 lz       /usr/bin/python3 /usr/bin/p   116.0M     1.5G     1.5G     1.5G 
3732310 libvirt-qemu /usr/bin/qemu-system-x86_64        0     5.9G     5.9G     6.0G 
-------------------------------------------------------------------------------
  144 14                                     436.8M     9.2G     9.6G    10.9G

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2022-07-26:

#7

So of your
32G there are:

Re-usable:
- 5.7G free
- 2.3G cache

Allocated
- 1.3G Active
- 8.0G Inactive
- 2G shmem
- 1G slab
- 4.5G anon-huge

A lot of those things might double account and overlap, so I do not got to more detail.
But that doesn't look to bad to me.
But to be fair, your swap really has some pages in it, so at some point there was pressure pushing things out of memory onto swap for sure.

The qemu process only has ~6GB which is totally fine, I see no leak there.
Or does this grow out of the allocated ~12G at some point and keeps growing continuously?

Revision history for this message

Lucas Zanella (lucaszanella) wrote on 2022-07-26:

#8

I left the computer 2 days without opening qemu, and it stays at 7gb of usage maximum. When I open Qemu, suddenly it grows to 31gb and something, even if the VM has only 6gb of RAM, and then firefox starts closing my tabs and etc.

There is a mismatch between gnome-system-monitor RAM, which is ~31gb and htop which shows 'real' usage. Anyway, my RAM is indeed full even if htop says it's 16gb only, cause firefox will close all tabs, and I get some freezes when it's possibly swapping some memory. So I don't know if those results are true. I only know that before 22.04 I never had so much RAM usage.

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2022-07-26:

#9

Interesting, I'm re-reading your XMl with that behavior in mind ...

Revision history for this message

Christian Ehrhardt  (paelzer) wrote on 2022-07-26:

#10

The two things that might be related would be:
a) backing definition
  <memoryBacking>
    <source type="memfd"/>
    <access mode="shared"/>
  </memoryBacking>

b) 3d enablement
    <graphics type="spice">
      <listen type="none"/>
      <image compression="off"/>
      <gl enable="yes" rendernode="/dev/dri/by-path/pci-0000:00:02.0-render"/>
    </graphics>
    <audio id="1" type="spice"/>
    <video>
      <model type="virtio" heads="1" primary="yes">
        <acceleration accel3d="yes"/>
      </model>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>

To be clear, none "should" grow it to 32GB, but they are the most related or oncommon things in case they now use other allocations or pre-allocate some buffers or that. You mentioned that you removed virtglrender already and it keeps doing that.

Could you please
1. remove both above sections
2. start your guest again and check if it still behaves in the bad way
2.1 if it does keep the guest without those sections as it ust makes it easier to debug)
2.2 if it stops behaving that way try which setting exactly it was by adding them back one by one
3. Once you have found the minimal set which exposes the issue please start your guest while running the smem call of above in a loop - maybe every 2 seconds. We'd want to see if it is qmeu or anything else that initially grows that much.

Revision history for this message

Lucas Zanella (lucaszanella) wrote on 2022-07-30:

#11

Download full text (7.3 KiB)

Same problem. Here's my new XML:

<domain type="kvm">
  <name>supernova</name>
  <uuid>7b22391a-ee8c-4259-8883-daa88a4559ac</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">20480000</memory>
  <currentMemory unit="KiB">20480000</currentMemory>
  <vcpu placement="static">8</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-6.2">hvm</type>
    <boot dev="hd"/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <vmport state="off"/>
  </features>
  <cpu mode="host-passthrough" check="none" migratable="on"/>
  <clock offset="utc">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="qcow2" discard="unmap"/>
      <source file="/var/lib/libvirt/images/orwell.qcow2"/>
      <target dev="vda" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="cdrom">
      <driver name="qemu" type="raw"/>
      <target dev="sda" bus="sata"/>
      <readonly/>
      <address type="drive" controller="0" bus="0" target="0" unit="0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="...

Ubuntu
qemu package

Qemu memory leak

Bug Description

Other bug subscribers

Remote bug watches

Ubuntuqemu package

Qemu memory leak

Bug Description

Other bug subscribers

Remote bug watches

Ubuntu
qemu package