dovecot silently fails to deliver email when user's mailbox is malformed

Thanks Robert for taking the time to report this bug. There isn't really enough information here for a developer to confirm this issue is a bug, or to begin working on it. Could you try to dig a bit more and try to understand if anything useful gets logged? Best of all would be to have some steps to reproduce the problem from scratch.

I'm marking this report as Incomplete for the moment, please change its status after commenting to New and we'll look at it again. Thanks!

      I further discovered the mailbox in question was malformed, had a space
before the 'From' in the first message. Fixing that, and then deleting the
index file and allowing it to rebuild got customer back in service. I further
found the mailbox got malformed by a users ineptidude, they tried to manually
as in using an editor, move a mail from another box. So that was the cause,
but it would be good for it to handle a corrupted mailbox more gracefully,
perhaps scan to the first good From and go from there.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 4 May 2020, Paride Legovini wrote:

> Date: Mon, 04 May 2020 16:08:54 -0000
> From: Paride Legovini <email address hidden>
> Reply-To: Bug 1876537 <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1876537] Re: dovecot eating mail indexes /var/dovecot-index/user
>
> Thanks Robert for taking the time to report this bug. There isn't really
> enough information here for a developer to confirm this issue is a bug,
> or to begin working on it. Could you try to dig a bit more and try to
> understand if anything useful gets logged? Best of all would be to have
> some steps to reproduce the problem from scratch.
>
> I'm marking this report as Incomplete for the moment, please change its
> status after commenting to New and we'll look at it again. Thanks!
>
> ** Changed in: dovecot (Ubuntu)
> Status: New => Incomplete
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1876537
>
> Title:
> dovecot eating mail indexes /var/dovecot-index/user
>
> Status in dovecot package in Ubuntu:
> Incomplete
>
> Bug description:
> I will get reports of users being able to connect, authenticate, but then not have any mail
> available when there is new mail in their INBOX. Something is happening to the dovecot-index file
> for that user because deleting it restores their ability to get mail.
>
> I have never had this happen before in six years of using dovecot, it is only since upgrade
> to 20.04 that this started happening. I made no local conf changes.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: dovecot-core 1:2.3.7.2-1ubuntu3 [modified: lib/systemd/system/dovecot.service]
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27
> Architecture: amd64
> CasperMD5CheckResult: skip
> Date: Sat May 2 17:23:01 2020
> ProcEnviron:
> TERM=xterm-color
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: dovecot
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dovecot/+bug/1876537/+subscriptions
>

      I kind of figured this would go ignored, unfortunately Canonical seems
to be going the way of Redhat these days.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Tue, 5 May 2020, Lucas Kanashiro wrote:

> Date: Tue, 05 May 2020 21:29:59 -0000
> From: Lucas Kanashiro <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1876537] Re: dovecot eating mail indexes /var/dovecot-index/user
>
> ** Changed in: dovecot (Ubuntu)
> Status: Incomplete => Invalid
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1876537
>
> Title:
> dovecot eating mail indexes /var/dovecot-index/user
>
> Status in dovecot package in Ubuntu:
> Invalid
>
> Bug description:
> I will get reports of users being able to connect, authenticate, but then not have any mail
> available when there is new mail in their INBOX. Something is happening to the dovecot-index file
> for that user because deleting it restores their ability to get mail.
>
> I have never had this happen before in six years of using dovecot, it is only since upgrade
> to 20.04 that this started happening. I made no local conf changes.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: dovecot-core 1:2.3.7.2-1ubuntu3 [modified: lib/systemd/system/dovecot.service]
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27
> Architecture: amd64
> CasperMD5CheckResult: skip
> Date: Sat May 2 17:23:01 2020
> ProcEnviron:
> TERM=xterm-color
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: dovecot
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dovecot/+bug/1876537/+subscriptions
>

Hello Robert,

This is a community supported bug system and sometimes bugs don't get expected visibility for many reasons. If you're looking for support services, with different support levels and expected servicing times, you can always look for Ubuntu Advantage options:

https://ubuntu.com/legal/ubuntu-advantage-service-description

      I will just compile it myself, no big deal. Thank you.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Fri, 8 May 2020, Rafael David Tinoco wrote:

> Date: Fri, 08 May 2020 01:58:21 -0000
> From: Rafael David Tinoco <email address hidden>
> Reply-To: Bug 1876537 <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1876537] Re: dovecot eating mail indexes /var/dovecot-index/user
>
> Hello Robert,
>
> This is a community supported bug system and sometimes bugs don't get
> expected visibility for many reasons. If you're looking for support
> services, with different support levels and expected servicing times,
> you can always look for Ubuntu Advantage options:
>
> https://ubuntu.com/legal/ubuntu-advantage-service-description
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1876537
>
> Title:
> dovecot eating mail indexes /var/dovecot-index/user
>
> Status in dovecot package in Ubuntu:
> Invalid
>
> Bug description:
> I will get reports of users being able to connect, authenticate, but then not have any mail
> available when there is new mail in their INBOX. Something is happening to the dovecot-index file
> for that user because deleting it restores their ability to get mail.
>
> I have never had this happen before in six years of using dovecot, it is only since upgrade
> to 20.04 that this started happening. I made no local conf changes.
>
> ProblemType: Bug
> DistroRelease: Ubuntu 20.04
> Package: dovecot-core 1:2.3.7.2-1ubuntu3 [modified: lib/systemd/system/dovecot.service]
> Uname: Linux 5.6.0 x86_64
> ApportVersion: 2.20.11-0ubuntu27
> Architecture: amd64
> CasperMD5CheckResult: skip
> Date: Sat May 2 17:23:01 2020
> ProcEnviron:
> TERM=xterm-color
> PATH=(custom, no user)
> XDG_RUNTIME_DIR=<set>
> LANG=en_US.UTF-8
> SHELL=/bin/bash
> SourcePackage: dovecot
> UpgradeStatus: No upgrade log present (probably fresh install)
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/ubuntu/+source/dovecot/+bug/1876537/+subscriptions
>

I'm wondering about the temper here, you got three responses in four days.
Where is that a lack of attention or "being ignored".

All I see is:
- Paride was asking for more info than "Something is happening to the dovecot-index file" which seems fine to me.
- Lucas later marked it invalid after your debugging showed that it doesn't appear as a bug in the dovecot code or its packaging.
- Rafael tried to explain options if you want a higher support level. You might not want that, but there is nothing bad in him telling you I'd think.

Finally you said " I will just compile it myself" - does that imply there is indeed a fix that makes it handle corrupted inboxes more gracefully? If so could you point to a commit or upstream discussion.
I'm asking as we are most likely going to bump 2.3.7.2 to 2.3.10 this cycle and if your fix is upstream we should get it by that for Ubuntu 20.10 and later. But furthermore the questions will then become if there is an isolated fix that is SRU-backportable - so if you have identified one please let us know.

      Look, I haven't chanced the conf, but I've had these problems since
19.10:

May 11 11:11:32 mail dovecot: imap(mac)<1488494><mmGrRmOllcjMehAG>: Error:
Mailbox Trash: copy: i_stream_read(/var/spool/mail/mac) failed: Cached message
size larger than expected (5548 > 5547, box=INBOX, UID=27882)
May 11 11:12:05 mail dovecot: imap(mac)<1488582><XkqwSGOlzcjMehAG>: Error:
Corrupted record in index cache file
/var/dovecot-index/mac/.imap/INBOX/dovecot.index.cache: UID 20762: Broken
physical size in mailbox INBOX: read(/var/spool/mail/mac) failed: Cached
message size larger than expected (5174 > 5173, box=INBOX, UID=20762)
May 11 11:12:05 mail dovecot: imap(mac)<1488582><XkqwSGOlzcjMehAG>: Error:
Mailbox Trash: copy: i_stream_read(/var/spool/mail/mac) failed: Cached message
size larger than expected (5174 > 5173, box=INBOX, UID=20762)
May 11 11:27:11 mail dovecot: imap(ariex)<1491563><3ZvEfmOlPBeu2Bgu>: Error:
Mailbox Trash: lseek() failed with mbox: Illegal seek
May 11 11:42:18 mail dovecot: imap(ariex)<1491563><3ZvEfmOlPBeu2Bgu>: Error:
Mailbox Trash: lseek() failed with mbox: Illegal seek
May 11 11:57:23 mail dovecot: imap(ariex)<1491563><3ZvEfmOlPBeu2Bgu>: Error:
Mailbox Trash: lseek() failed with mbox: Illegal seek
May 11 12:05:42 mail dovecot: imap(wstewart)<1499301><8/6ICGSlwswYEDrL>: Error:
Next message unexpectedly corrupted in mbox file /var/spool/mail/wstewart at
492309796

      Surely something isn't right.

-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-_-
  Eskimo North Linux Friendly Internet Access, Shell Accounts, and Hosting.
    Knowledgeable human assistance, not telephone trees or script readers.
  See our web site: http://www.eskimo.com/ (206) 812-0051 or (800) 246-6874.

On Mon, 11 May 2020, Christian Ehrhardt  wrote:

> Date: Mon, 11 May 2020 11:38:01 -0000
> From: Christian Ehrhardt  <email address hidden>
> To: <email address hidden>
> Subject: [Bug 1876537] Re: dovecot eating mail indexes /var/dovecot-index/user
>
> I'm wondering about the temper here, you got three responses in four days.
> Where is that a lack of attention or "being ignored".
>
> All I see is:
> - Paride was asking for more info than "Something is happening to the dovecot-index file" which seems fine to me.
> - Lucas later marked it invalid after your debugging showed that it doesn't appear as a bug in the dovecot code or its packaging.
> - Rafael tried to explain options if you want a higher support level. You might not want that, but there is nothing bad in him telling you I'd think.
>
> Finally you said " I will just compile it myself" - does that imply there is indeed a fix that makes it handle corrupted inboxes more gracefully? If so could you point to a commit or upstream discussion.
> I'm asking as we are most likely going to bump 2.3.7.2 to 2.3.10 this cycle and if your fix is upstream we should get it by that for Ubuntu 20.10 and later. But furthermore the questions will then become if there is an isolated fix that is SRU-backportable - so if you have identified one please let us know.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.la...

Hi Robert,

Sorry if you felt ignored here but it was not the case, we are just trying to make sure your bug report is actionable.

I did not quite understand your problem considering you mentioned in Comment #3 it was an issue due to a manual intervention in your mailbox. Are you talking about a feature to handle a corrupted mailbox more gracefully? If this is the case it is a feature request and you should update the bug title/description and mark it as New and Wishlist.

Please let me know if I correctly understood your intention here.

     At this point this is what I know.  There was manual intervention
on behalf of a user that ended up with an extra space in the box.  This
caused dovecot to blow up with no real intelligent error.  It should
have given some error like mail header corrupt or some such.  Second, it
seems to be doing some corrupting of its own, or postfix, not sure which
but occasionally NUL characters are also making their way into the INBOX
and causing troubles.

On 5/12/20 12:16 PM, Lucas Kanashiro wrote:
> Hi Robert,
>
> Sorry if you felt ignored here but it was not the case, we are just
> trying to make sure your bug report is actionable.
>
> I did not quite understand your problem considering you mentioned in
> Comment #3 it was an issue due to a manual intervention in your mailbox.
> Are you talking about a feature to handle a corrupted mailbox more
> gracefully? If this is the case it is a feature request and you should
> update the bug title/description and mark it as New and Wishlist.
>
> Please let me know if I correctly understood your intention here.
>

There is a very similar (to yours) problem described here:

https://dovecot.org/pipermail/dovecot/2014-May/095998.html

one of the developers tell end-user to move away from mbox,
and there seems to exist a correct answer stating that mbox
is a supported format and there is no current documentation
saying that mbox format is broken.

Unfortunately that thread did not have a better ending.

Several discussions, dated from before 2014, stating that
from time to time the cache gets corrupted. There might be
more than 1 reason on why the cache gets corrupted - doing a
parallel to any other kind of journaling scheme - so...

without having an exact reproducer it would be very difficult
to check why the cache gets corrupted :\.

Moving this to confirmed as it seems to still exist.

I've updated the title to match what I gather is the underlying problem here.

An example mailbox and/or more explicit steps to set up dovecot to reproduce the issue are still needed before this can be worked.

The fix for this would likely be applicable upstream as it is not simply a packaging problem, so filing a bug report with dovecot upstream about this would make a lot of sense. They may be able to fix it faster and better than we could.

[Expired for dovecot (Ubuntu) because there has been no activity for 60 days.]