rsync works bad with encfs now

Hi,
I'm not 100% sure but to me that is a setup and configuration issue with the new version being a bit more insisting that it can read/access paths it is supposed to back up.

The new behavior can be disabled with --ignore-errors, but I'm tempted to consider this dangerous as you could have any other I/O error and it would still delete things.

Did you try to use --exclude=PATTERN and if that matches your use case --delete-excluded to make the new version behave as you were used from the older one?

I'm marking this incomplete waiting for feedback on the suggested options or a discussion why this really should be an error to be fixed in the source (instead of configuration).

Hi Christian and thank you for answer.
As you said --ignore-errors could work, but il would be very very
dangerous.

I had tried --exclude=PATTERN (more precisely
--filter "- /home/claude/Documents_chiffres/") and it seemed to work
better because there's not more the "IO error encountered -- skipping
file deletion" error.
rsync did not save the virtual directory corresponding to the encrypted
directory but that's no matter because the user's script does it.

It could be good BUT because of the --delete-excluded option,
rsync deletes the backup of the existing virtual directory
corresponding to the encrypted directory.
I've found a way modifying my backup's script : first it moves the
backup of the virtual directory to a safe place before rsync acts and
after it moves it back to the previous place.
It works but it could be better I think.
Best regards.
Claude MICOUIN

Le lundi 03 janvier 2022 à 14:05 +0000, Christian Ehrhardt  a écrit :
> Hi,
> I'm not 100% sure but to me that is a setup and configuration issue
> with the new version being a bit more insisting that it can
> read/access paths it is supposed to back up.
>
> The new behavior can be disabled with --ignore-errors, but I'm
> tempted
> to consider this dangerous as you could have any other I/O error and
> it
> would still delete things.
>
> Did you try to use --exclude=PATTERN and if that matches your use
> case
> --delete-excluded to make the new version behave as you were used
> from
> the older one?
>
> ** Changed in: rsync (Ubuntu)
> Status: New => Incomplete
>

It seems to be like this is a bug in encfs rather than in rsync? I appreciate newer rsync might have changed behaviour, but ultimately encfs needs to provide a regular (if virtual) filesystem. Where it doesn't, rsync can't be expected to work.

Is there any specific reason to think that encfs is working correctly in your failure case?

No. It's only this changing of behavior who makes me think of a bug.
(of rsync or of encfs?) Why before rsync and encfs worked well together
and not now?
I noticed that encfs is still working correctly when encrypting. I also
noticed that rsync is now refusing the file deletion because of the IO
error security while before he wasn't and he was going on saving, only
printing a simple warning.
But I understand perfectly that it could be a voluntary security added
in rsync and not a bug. It is better to be too careful than not enough.
Recently I’ve found a page who talks about an old bug of rsync entitled
"I/O errors other than IOERR_GENERAL should not suppress deletion"
https://bugzilla.samba.org/show_bug.cgi?id=7809
Maybe the same type of problem??
Thank you very much for reading me.

Hi Micouin,

The bug you linked does include a patch which is available in focal (20.04). This means that the error you are seing is indeed caused by a IOERR_GENERAL error. As for

> Why before rsync and encfs worked well together and not now?

would you mind describing which versions of both rsync and encfs you were using when the previous behavior was observed? Was that still in focal as well?

Hi Athos,
so you mean that this patch is in focal and the error I'm talking about
is a IOERR_GENERAL error?
I thought no, but I may be mistaken (surely) :-)

today's versions of rsync and encfs are
rsync 3.1.3-8ubuntu0.1
encfs 1.9.5-1build2
and Ubuntu's version was (and is) focal Ubuntu 20.04.3 LTS

I've looked for the date of the first problem : it was about the 2021-
11-07.
I've searched in my /var/log/apt/history.log* and the unique file which
contains rsync is dated 2021-11-07 (the oldest log in my machine is
dated 2021-03-02).
However there is no encfs in the logs of apt. That's why I thought that
the origin of the change/bug was in the update of rsync.
I've found an old log of synaptic 2020-03-28 who says:
rsync (3.1.2-2.1ubuntu1) to 3.1.2-2.1ubuntu1.1

I've looked in ubuntu server site and found that the previous version
of rsync is dated 2021-11-06
rsync_3.1.2-2.1ubuntu1.2_amd64.deb 2021-11-06 01:09 327K
and the date of the actual encfs is 2020-03-23.

Thank you for reading me.
Claude MICOUIN

Hi Claude,

Are you able to roll rsycn back to version 3.1.3-8 to see if this gets fixed? In 2021-11-16, we got an update to fix LP #1896251. FWIW, this is the upstream patch that was applied:

https://github.com/WayneD/rsync/commit/af6118d98b3482cbcfc223bf2a0777bc19eccb02

Hi Lucas,
I wanted to do this to make the test, but because of all the
dependencies, I'm scared to make a big mistake if I do it using apt ou
dpkg.
I'd like to install the previous and the actuel versions side by side,
but I have to learn to do it before. Maybe with a tar archive?
Thanks.
Claude

Hi Claude,

Were you able to downgrade to see if the prior version works properly, as Lucas suggested? If you haven't yet done so, I'd recommend using apt rather than co-installation from a tarball in this case both because it'd be a better apples-to-apples check, and because the behavior will be more predictable (co-installation can mess up config files and such). For example, downgrade using a command somewhat like this:

    $ apt-cache policy rsync
    rsync:
      Installed: 3.1.3-8ubuntu0.1
      Candidate: 3.1.3-8ubuntu0.1
      Version table:
     *** 3.1.3-8ubuntu0.1 500
        500 http://us.archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     3.1.3-8 500
        500 http://us.archive.ubuntu.com/ubuntu focal/main amd64 Packages

    $ sudo apt-get install rsync=3.1.3-8
    [sudo] password for bryce:
    Reading package lists... Done
    Building dependency tree
    Reading state information... Done
    The following packages will be DOWNGRADED:
      rsync
    0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 26 not upgraded.
    Need to get 322 kB of archives.
    After this operation, 12.3 kB of additional disk space will be used.
    Do you want to continue? [Y/n]

Let us know when you've completed the check.

Hello Claude,

It should be straightforward to downgrade the package on Focal, because the previous version is still in the -release pocket. You can simply:

# apt install rsync=3.1.3-8

and it should do the trick.

While at it, would it be possible for you to provide a step-by-step reproducer for this bug? It would be easier for us if we had this.

Thanks in advance.

Hi Bryce and Sergio
I did not downgrade rsync yet, so I will do it using
apt-get install rsync=3.1.3-8

I will do the test with this previous version, but how can I provide a
step-by-step reproducer for this bug ?
With --verbose option or something else ?

Thanks.
Claude

Le lundi 24 janvier 2022 à 19:16 +0000, Bryce Harrington a écrit :
> Hi Claude,
>
> Were you able to downgrade to see if the prior version works
> properly,
> as Lucas suggested? If you haven't yet done so, I'd recommend using
> apt
> rather than co-installation from a tarball in this case both because
> it'd be a better apples-to-apples check, and because the behavior
> will
> be more predictable (co-installation can mess up config files and
> such).
> For example, downgrade using a command somewhat like this:
>
> $ apt-cache policy rsync
> rsync:
> Installed: 3.1.3-8ubuntu0.1
> Candidate: 3.1.3-8ubuntu0.1
> Version table:
> *** 3.1.3-8ubuntu0.1 500
> 500 http://us.archive.ubuntu.com/ubuntu focal-updates/main
> amd64 Packages
> 100 /var/lib/dpkg/status
> 3.1.3-8 500
> 500 http://us.archive.ubuntu.com/ubuntu focal/main amd64
> Packages
>
> $ sudo apt-get install rsync=3.1.3-8
> [sudo] password for bryce:
> Reading package lists... Done
> Building dependency tree
> Reading state information... Done
> The following packages will be DOWNGRADED:
> rsync
> 0 upgraded, 0 newly installed, 1 downgraded, 0 to remove and 26
> not upgraded.
> Need to get 322 kB of archives.
> After this operation, 12.3 kB of additional disk space will be
> used.
> Do you want to continue? [Y/n]
>
> Let us know when you've completed the check.
>

Hi Claude,

You could provide step-by-step reproducer by mentioning the exact steps that one could follow to reproduce the bug in a clean environment.

If you could provide steps/commands that I can copy-paste in a Docker or an LXD container, then that'd really help us and we can take it from there. \o/

Hi Utkarsh Gupta,
I understand better what is a step-by-step reproducer :-)
I have tried to downgrade rsync with apt-get install rsync=3.1.3-8 and
there is no change. But the version who seemed to work on my PC was
older rsync_3.1.2-2.1
I've tried to install it but apt-get refused.
Is there a way to do this installation?
Thanks.

Hi Claude,

to reproduce the steps you could do it in a newly installed 20.04 up to date install.

Once installed you will start fresh with a newly encrypted directory then you write on your note sheet (text editor for instance):
"Creating directory "MyEncryptedDir" (for instance),
then continue with the install of encfs, then encrypt "MyEncryptedDir",

make it one line per step,

then show from there the rsync command line (from > to and options if any, where "from" and "to" are the source and the destination).

Copy paste in your note sheet all the command lines used in the terminal, until you get the errors.

This is what the devs can use to reproduce your steps, as if they were right besides you while you are doing it.

Hello,
perdon for the delay, but the Covid is not a sympatical illness. :-(
I have just finished the test of encfs with the older rsync's version (3.1.2-2.1) on a virtual machine. The problem (IO error encountered -- skipping file deletion) is still there, so it looks like there is no bug with rsync. I don't know why there is this new problem on my machine, but it is not because of the new version of rsync. So, I think we can close the report.
Thank you very much for your help and attention and sorry for wasting time.
Claude MICOUIN

I let the details of the test in the following message.

* Installation of Ubuntu kubuntu-18.04-desktop-amd64 in VirtualBox
* Creation of user "toto" passwd is "clown". The name of virtual machine is toto-VirtualBox
* Installation of rsync 3.1.2-2.1ubuntu1.4 and encfs 1.9.2-2build2
* Creation of two directories (by toto) "Backup" and "Directory_to_save" inside of /home/toto and creation of two directories "Crypt" and ".Crypt" inside of the directory "Directory_to_save"
$ mkdir -p /home/toto/Backup /home/toto/Directory_to_save/Crypt /home/toto/Directory_to_save/.Crypt
* Filling the directory "Directory_to_save" with diverse files and directories
$ cd /home/toto/Directory_to_save/ ; mkdir Countries Downloads Felines Fishes Images Models Mountains Musics Public Videos
$ cd /home/toto/Directory_to_save/Countries/ ; touch China.txt France.txt Germany.txt Japan.txt
$ cd /home/toto/Directory_to_save/Felines ; touch Cats.txt Lions.txt Tigers.txt
$ cd /home/toto/Directory_to_save/Fishes ; touch Sharks.txt Trouts.txt
$ cd /home/toto/Directory_to_save/Mountains ; touch Alps.txt Atlas.txt Himalayas.txt
Or with one ligne only
mkdir -p /home/toto/Backup /home/toto/Directory_to_save/Crypt /home/toto/Directory_to_save/.Crypt ; cd /home/toto/Directory_to_save/ ; mkdir Countries Downloads Felines Fishes Images Models Mountains Musics Public Videos ; cd /home/toto/Directory_to_save/Countries/ ; touch China.txt France.txt Germany.txt Japan.txt ; cd /home/toto/Directory_to_save/Felines ; touch Cats.txt Lions.txt Tigers.txt ; cd /home/toto/Directory_to_save/Fishes ; touch Sharks.txt Trouts.txt ; cd /home/toto/Directory_to_save/Mountains ; touch Alps.txt Atlas.txt Himalayas.txt
* Test of the command rsync by root (or sudo)
1) With rsyc 3.1.3-8
# rsync -a --progress --delete-during /home/toto/Directory_to_save/ /home/toto/Backup/
sending incremental file list
.Crypt/
Countries/
Countries/China.txt
              0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=11/25)
Countries/France.txt
              0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=10/25)
Countries/Germany.txt
              0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=9/25)
Countries/Japan.txt
              0 100% 0.00kB/s 0:00:00 (xfr#4, to-chk=8/25)
etc, etc, etc.
OK. There is no "IO error encountered -- skipping file deletion" message.
2) With rsync 3.1.2-2.1
# rsync -a --progress --delete-during /home/toto/Directory_to_save/ /home/toto/Backup/
sending incremental file list
.Crypt/
Countries/
Countries/China.txt
              0 100% 0.00kB/s 0:00:00 (xfr#1, to-chk=11/25)
Countries/France.txt
              0 100% 0.00kB/s 0:00:00 (xfr#2, to-chk=10/25)
Countries/Germany.txt
              0 100% 0.00kB/s 0:00:00 (xfr#3, to-chk=9/25)
Countries/Japan.txt
              0 100% 0.00kB/s 0:00:00 (xfr#4, to-chk=8/25)

etc, etc, etc.
OK. There is no "IO error encountered -- skipping file deletion" message.
* Mount of the directory ".Crypt" on "Crypt" (empty) by toto
encfs /home/toto/Directory_to_save/.Crypt/ /home/toto/Directory_to_save/Crypt/ #(give a passwd)
* Filling the directory "Crypt" with diverse files and directories
$ cd /home/toto/Directory_to_save/Crypt/ ; mkdi...

Hello,

> [...] I don't know why there is this new problem on my
> machine, but it is not because of the new version of
> rsync. So, I think we can close the report.

Indeed, this doesn't look like a bug in the package itself so marking this as "Invalid". But that said, from your logs..

> rsync: readlink_stat("/home/toto/Directory_to_save/Crypt")
> failed: Permission denied (13)

This looks like you don't have the correct (write) permission to where you're trying to do this. I suggest trying to look there and fix the permission and re-try again?