targetcli fails iscsi setup after upgrade from 18.04 to 20.04

Flagging this as server next to be sure upgrade paths are cleared, specially now that targetcli-fb was MIR'ed.

In Bionic

$ dpkg -l | egrep -E "(rtslib-fb|targetcli-fb)" | awk '{print $3}'
2.1.57+debian-4
2.1.57+debian-4
2.1.43-1

targetcli-fb in Bionic:

https://pastebin.ubuntu.com/p/fJg8zRQ3bG/

/> saveconfig
Last 10 configs saved in /etc/rtslib-fb-target/backup.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

after a reboot:

https://pastebin.ubuntu.com/p/dGn5hy7Vnx/

/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/rtslib-fb-target/backup.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

All working good in Bionic.. from Bionic to Focal we had:

targetcli-fb from 2.1.43-1 to 1:2.1.51-0ubuntu1
python-rtslib-fb from 2.1.57+debian-4 to 2.1.71-0ubuntu1

Let's test...

Right after the upgrade I had the same running LIO setup:

https://pastebin.ubuntu.com/p/VXb6JS98fy/

/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/rtslib-fb-target/backup/.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

After a reboot...

My LIO configuration was empty... I could only get back after doing:

/> restoreconfig
Configuration restored from /etc/rtslib-fb-target/saveconfig.json

Problem here is that targetcli-fb has, by default the following option:

/> set global auto_save_on_exit=false

and if the user does not restore previous config and exit targetcli, it will save the empty configuration.

Only way to restore the previous configuration (the working one) would be:

/> restoreconfig /etc/rtslib-fb-target/backup/saveconfig-20200908-19:34:15-json
Configuration restored from /etc/rtslib-fb-target/backup/saveconfig-20200908-19:34:15-json

and then I have my configuration back.

So this bug is about having "targetcli-fb" to load by default its default settings. Something like a service running "sudo targetcli restoreconfig", for example. Let me check Groovy's default behavior on this...

In Groovy:

Same targetcli-fb configuration with 2 luns, acls and iscsi IQN configured...

/> saveconfig
Configuration saved to /etc/rtslib-fb-target/saveconfig.json
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/rtslib-fb-target/backup/.
Configuration saved to /etc/rtslib-fb-target/saveconfig.json

Since Groovy has the "target.service", we will have to enable it:

[rafaeldtinoco@targetclifbnew ~]$ sudo systemctl enable --now target.service
Created symlink /etc/systemd/system/multi-user.target.wants/target.service → /lib/systemd/system/target.service.

<reboot>

And sudo targetcli gives me all the configuration ready.

Focal does not have a systemd unit called "targetd.service" which is responsible to restore previous LIO configuration. Groovy has it. In Focal, targetcli-fb was not yet in MAIN archive, it has been put in MAIN after Focal (for a reason, like explained in LP: #1854362).

In order to mitigate this, one can create a systemd unit file in Focal with the following:

[rafaeldtinoco@targetclifb ~]$ systemctl edit --force --full targetd.service

----
[Unit]
Description=Restore LIO kernel target configuration
Requires=sys-kernel-config.mount
After=sys-kernel-config.mount network.target local-fs.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/usr/bin/targetctl restore
ExecStop=/usr/bin/targetctl clear
SyslogIdentifier=target

[Install]
WantedBy=multi-user.target
----

[rafaeldtinoco@targetclifb ~]$ systemctl enable --now targetd.service
Created symlink /etc/systemd/system/multi-user.target.wants/targetd.service → /etc/systemd/system/targetd.service.

This will make Focal to behave just as in Groovy.. after a <reboot>, the LIO configuration is restored like the last configuration set in targetcli-fb:

[rafaeldtinoco@targetclifb ~]$ sudo targetcli
targetcli shell version 2.1.51
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.

/> ls /
o- / ...............................................
  o- backstores ....................................
  | o- block .......................................
  | o- fileio ......................................
  | o- pscsi .......................................
  | o- ramdisk .....................................
  | o- lun01 .....................................
  | | o- alua ....................................
  | | o- default_tg_pt_gp ......................
  | o- lun02 .....................................
  | o- alua ....................................
  | o- default_tg_pt_gp ......................
  o- iscsi .........................................
  | o- iqn.2003-01.org.linux-iscsi.targetclifb.x8664
  | o- tpg1 ......................................
  | o- acls ....................................
  | | o- iqn.1993-08.org.debian:01:c6fc64976698
  | | o- mapped_lun1 .........................
  | | o- mapped_lun2 .........................
  | o- luns ....................................
  | | o- lun1 ..................................
  | | o- lun2 ..................................
  | o- portals .................................
  | o- 0.0.0.0:3260 ..........................
  o- loopback ......................................
  o- vhost .........................................
  o- xen-pvscsi ....................................
/>

I added a tag to discuss this with server team in order to check if adding a new systemd unit to Focal is something okay to be done as SRU (Stable Release Update).

Wearing my SRU hat, my answer is "I don't know, depends on the specifics". In a discussion with the server team we weren't clear on the specifics. How is it that this regresses on upgrade from Bionic? Did a system unit that previously existed accidentally vanish? Or something else more complicated?

However also we noted that this is in universe in Focal.

I have tested the scenario Rafael described above and it works as expected in Impish but in Focal, as he described, the configuration is lost. The workaround presented in comment #5 is needed.

From my understanding, after the upgrade from Bionic to Focal there is no regression apparently but after the first reboot the configuration seems to be gone because the variable 'auto_save_on_exit' is set to false by default and there is no systemd unit file to call restore when starting the service. In Bionic there is no systemd service at all, the systemd support was added after Bionic.

So the fix targeting Focal would be to set 'auto_save_on_exit' to true and add the systemd unit file to call restore when starting the service. Do you think this is SRU-able?

Robie, from the SRU team, asked for more deep understanding on why this was working before (without systemd) and in Focal (with systemd) stopped working. This is still in our backlog.