ClamAV error: CL_EFORMAT: Bad format or broken data

Users have reported that scanning MS compressed files are returning "CL_EFORMAT: Bad format or broken data ERROR". The patch for bug #4626 introduced this behavior. We are currently investigating the correct solution to these reports.

The return value noted here is a scanning error code. This behavior has been seen on files that do not decompress correctly. If using clamav-milter, messages with attachments flagged as CL_EFORMAT trigger the OnFail Action specified in the clamav-milter.conf configuration file. The default Action for OnFail is Defer, so these messages will be rescanned at intervals and never delivered.

There is a workaround to break the loop of re-scanning these broken files. To reject messages with these attachments, change the OnFail Action to Reject. If you prefer the old behavior of the previous release, change the OnFail Action to Accept.

These solutions should be used as a temporary workaround only and in an environment where the connection to clamd is stable. Changing this configuration line will also affect any messages that fail scanning because of transient issues such as dropped connections or out of memory errors. If OnFail is set to Accept, malicious mail may be passed by the scanner. If OnFail is set to Reject, non-malicious mail may be rejected by the scanner. This workaround should be used with caution.

The root cause has been traced to an errorcode-handling change in the patch for bug #4669 and is not LZX-compression specific. Other file formats which have reported to trigger CL_EFORMAT errors include Excel & Word documents as well as PDF files.

Hi, guys,

Is the patch at http://git.clamav.net/gitweb?p=clamav-devel.git;a=commit;h=cdbe27798612df93351a9d0da2a2e377ce4118aa a fix for this?

Regards,

David.

I am getting ready to try it on FreeBSD. If anyone wants the ports patch let me know.
This applies cleanly to 0.97.5.

The patch did not compile for me. This patch did:

diff --git a/libclamav/scanners.c b/libclamav/scanners.c
index e84d735..1055b2c 100644
--- a/libclamav/scanners.c
+++ b/libclamav/scanners.c
@@ -2369,7 +2369,19 @@ static int magic_scandesc(int desc, cli_ctx *ctx, cli_file_t type)
      ctx->fmap--;
      cli_bitset_free(ctx->hook_lsig_matches);
      ctx->hook_lsig_matches = old_hook_lsig_matches;
- ret_from_magicscan(ret);
+ /* Same switch as end of magic_scandesc function */
+ switch(ret) {
+ case CL_EFORMAT:
+ case CL_EMAXREC:
+ case CL_EMAXSIZE:
+ case CL_EMAXFILES:
+ cli_dbgmsg("Descriptor: %s\n", cl_strerror(ret));
+ case CL_CLEAN: /* here, only from cli_checkfp() */
+ cache_add(hash, hashed_size, ctx);
+ ret_from_magicscan(CL_CLEAN);
+ default:
+ ret_from_magicscan(ret);
+ }
  }
     }

on of (us?) is off by 30 lines.
The original patch applied fine for me against 0.97.5 source tarball.

are you saying 'it didn't compile' ? or are you saying the patch did not apply?

(I am compiling now, both AMD64 and I386/32)

ok, I see:

cli_dbgmsg("Descriptor[%d]: %s\n", fmap_fd(*ctx->fmap), cl_strerror(ret));

gives me:

  CCLD clamscan
../libclamav/.libs/libclamav.so: undefined reference to `fmap_fd'
gmake[2]: *** [clamscan] Error 1
gmake[2]: Leaving directory `/work/a/ports/security/clamav/work/clamav-0.97.5/clamscan'
gmake[1]: *** [all-recursive] Error 1

you changed it to:

cli_dbgmsg("Descriptor: %s\n", cl_strerror(ret));

(still off by 30 lines.. strange)

> (still off by 30 lines.. strange)

I may have had some other patches... can't remember. I build Debian packages and there may be patches from the Debian maintainer in the mix.

Regards,

David.

There is a one line difference between the versions of the patch for the master code branch and 0.97 branch, and that debug line is it.

Here is the commit for the 0.97 version (the diff David posted):
http://git.clamav.net/gitweb?p=clamav-devel.git;a=commit;h=6a879ad98460303b23a6fc119769a3b463a902f8

The patch should apply cleanly to 0.97.5. Regression tests are running as we speak so we can be confident in the results. I'll give more status details as I have them.

*** Bug 5346 has been marked as a duplicate of this bug. ***

I hit same error with clamav-0.97.5-1600.fc16 :

https://admin.fedoraproject.org/updates/FEDORA-2012-9577/clamav-0.97.5-1600.fc16

this is a show stopper

Is there any ETA for shipping this fix ?

We've been hitting this error with certain Thunderbird and Firefox .tar.bz2 archives:

http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/14.0b4-candidates/build1/linux-i686/en-GB/thunderbird-14.0b4.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/14.0b4-candidates/build1/linux-i686/lt/thunderbird-14.0b4.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/14.0b4-candidates/build1/linux-i686/hr/thunderbird-14.0b4.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/14.0b4-candidates/build1/linux-i686/uk/thunderbird-14.0b4.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/thunderbird/nightly/14.0b4-candidates/build1/linux-i686/hu/thunderbird-14.0b4.tar.bz2
http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/14.0b11-candidates/build1/linux-x86_64/bn-IN/firefox-14.0b11.tar.bz2

If you decompress the .bz2 and scan the resulting .tar file, the scan completes ok.

We're hitting this problem with clamscan 0.97.5.
Downgrading to 0.97.4 scans the .tar.bz2 files OK.

If you are seeing this error, then I suggest completing the patch. It tested on my debian 6.0 compiled 0.97.5 with the patch, the issue no longer occurs. I can't wait till the regression tests are finished.

(In reply to comment #14)
> If you are seeing this error, then I suggest completing the patch. It tested on
> my debian 6.0 compiled 0.97.5 with the patch, the issue no longer occurs. I
> can't wait till the regression tests are finished.

1 - where is the patch ?
2 - Still not have the answer, when we have a new release ?

The fix for bug is currently in regression testing.

I'm having the same false positive with clamav-0.97.5 + current db, but with an mp4 file. Worked with older db's.

(In reply to comment #16)
> The fix for bug is currently in regression testing.

and what is the date ( in a number ) that regression testing finish ?

2 - Still not have the answer, when we have a new release ?

we be one week , one month or one year ?

Yes, the slow reaction to this (IMO) critical bug is very disappointing. Something as serious as this should definitely prompt a new release.

For our customers, we build packages with the patch in Comment #5 or Comment #9 and it works fine.

Regards,

David.

(In reply to comment #15)
> (In reply to comment #14)
> > If you are seeing this error, then I suggest completing the patch. It tested on
> > my debian 6.0 compiled 0.97.5 with the patch, the issue no longer occurs. I
> > can't wait till the regression tests are finished.
>
> 1 - where is the patch ?
> 2 - Still not have the answer, when we have a new release ?

Would you please file a release critical bug against the clamav package in Debian for this issue? I'm one of the Debian clamav maintainers and that will help me get the fix into Wheezy (and Squeeze).

Please give some indication of the steps to reproduce the issue so I can better demonstrate the problem/solution and get it approved.

This problems happens on RPMs as well.
I applied the patch and when I try to compile is get:
../libclamav/.libs/libclamav.so: undefined reference to `fmap_fd'

Environment Redhat AS 5.8
GCC gcc44-4.4.6-3

(In reply to comment #20)
> (In reply to comment #15)
> > (In reply to comment #14)
> > > If you are seeing this error, then I suggest completing the patch. It tested on
> > > my debian 6.0 compiled 0.97.5 with the patch, the issue no longer occurs. I
> > > can't wait till the regression tests are finished.
> >
> > 1 - where is the patch ?
> > 2 - Still not have the answer, when we have a new release ?
>
> Would you please file a release critical bug against the clamav package in
> Debian for this issue? I'm one of the Debian clamav maintainers and that will
> help me get the fix into Wheezy (and Squeeze).
>
> Please give some indication of the steps to reproduce the issue so I can better
> demonstrate the problem/solution and get it approved.

I use and test it in Fedora 16, 0.97.5 have been push to stable updates , due a security concerns, I had to exclude clamav from updates ...

Still not see, where is the patch, neither when we expect a need release ...
At least let me know when was released by writing in this bug report.

FYI after a regular patching of Ubuntu 11.04, clam has started giving this error too (I hit it on a squid proxy scanning attachments).
The Ubuntu tracking of this bug is:
https://bugs.launchpad.net/ubuntu/+source/clamav/+bug/1015405
Any news on when 0.97.6 might appear? Since I'm using standard Ubuntu packages I'd prefer an official clam release that gets pull into Ubuntu, as opposed to compiling from source.

We need this fix too.
This is really urgent.

Please give us a rough date for 0.97.6.
We have to supply our customers with a rough guess when this error will be fixed.

I've uploaded an updated package with the patch for this issue to Debian Unstable. From there it will propagate to Debian Wheezy/Squeeze Updates and Ubuntu.

Since I know many of you on this CC list have been asking, the complete 0.97.6 update including this patch is coming very soon.

For Debian/Ubuntu users, there is an update to 0.97.5 with the patch that fixes this issue. For Debian it's in Unstable and should get to Wheezy over the weekend and it's in stable-proposed-updates and should get to stable-updates today. For Ubuntu the update is available as of today for all releases.

yup, got it already thx :)

On Thu, Aug 16, 2012 at 8:28 AM, Scott Kitterman <email address hidden>wrote:

> For Debian/Ubuntu users, there is an update to 0.97.5 with the patch
> that fixes this issue. For Debian it's in Unstable and should get to
> Wheezy over the weekend and it's in stable-proposed-updates and should
> get to stable-updates today. For Ubuntu the update is available as of
> today for all releases.
>
> --
> You received this bug notification because you are subscribed to the bug
> report.
> https://bugs.launchpad.net/bugs/1015405
>
> Title:
> ClamAV error: CL_EFORMAT: Bad format or broken data
>
> Status in The ClamAV Antivirus Scanner:
> In Progress
> Status in Hardy Heron Backports:
> Fix Released
> Status in Lucid Backports:
> Fix Released
> Status in “clamav” package in Ubuntu:
> Fix Released
> Status in “clamav” source package in Natty:
> Fix Released
> Status in “clamav” source package in Oneiric:
> Fix Released
> Status in “clamav” source package in Precise:
> Fix Released
> Status in “clamav” source package in Quantal:
> Fix Released
> Status in “clamav” package in Debian:
> Fix Released
>
> Bug description:
> [IMPACT]
>
> * Scanning errors on some files of some important types like PDF,
> DOC, XLS, and tar.bz2.
>
> * Can cause major disruption of mail servers and other applications
>
> * Regression from previous releases
>
> * Bug fix is upstream fix that will be included in the next clamav
> release.
>
> [TESTCASE]
>
> * sudo apt-get install clamav (if not already installed) and once the
> package is installed and signatures are downloaded you are ready to
> test.
>
> * With an appropriate test file (the thunderbird bz2 file that is
> attached to the bug is one such file) run $ clamscan $FILENAME and you
> should get an error as described in the original bug.
>
> * Install the updated packages from -proposed
>
> * Run the test again and it should test OK (no error and no virus
> found).
>
> [Regression Potential]
>
> * Nil. Patch taken from upstream and given where the patch is in the
> code it would be very difficult to regress.
>
> Original bug:
>
> got this in my syslog after update of clamav dunno if its a bug or not
>
> Jun 19 21:47:29 server dansguardian[]: ClamAV error: CL_EFORMAT: Bad
> format or broken data
> Jun 19 21:47:29 server dansguardian[]: scanFile/Memory returned error: -1
>
> update version
> 0.97.5
>
> ubuntu 11.04
>
> To manage notifications about this bug go to:
> https://bugs.launchpad.net/clamav/+bug/1015405/+subscriptions
>

For Debian/Ubuntu users, there is an update to 0.97.5 with the patch
that fixes this issue. For Debian it's in Unstable and should get to
Wheezy over the weekend and it's in stable-proposed-updates and should
get to stable-updates today. For Ubuntu the update is available as of
today for all releases.

I just built and installed clamav 0.97.5 on a Solaris 11 system that had 0.97.4 running successfully and within a few hours I encountered this issue.

When will 0.97.6 be available and does it include a fix? How can I get the fix prior to 0.97.6 release?

Thanks in advance.

Kirk

ClamAV 0.97.6 is available. Thanks for your patience.

(In reply to comment #30)
> ClamAV 0.97.6 is available. Thanks for your patience.

Thanks David! Do you happen to know when the fix will be available for CentOS?

(In reply to comment #31)
> (In reply to comment #30)
> > ClamAV 0.97.6 is available. Thanks for your patience.
>
> Thanks David! Do you happen to know when the fix will be available for CentOS?

With a quick online search, I see several package repositories have rpm files available for 0.97.6 already. You should be able to find the build that you need without much trouble.

*** Bug 5735 has been marked as a duplicate of this bug. ***