squid does not accept WCCP of Cisco router since 3.5.27-1ubuntu1.12

Bug #1952158 reported by amk
10
This bug affects 1 person
Affects Status Importance Assigned to Milestone
Squid
Unknown
Unknown
squid (Ubuntu)
Confirmed
Undecided
Unassigned
Bionic
Invalid
Undecided
Unassigned
Impish
Won't Fix
Undecided
Unassigned
Jammy
Confirmed
Undecided
Unassigned
squid3 (Ubuntu)
Invalid
Undecided
Unassigned
Bionic
Confirmed
Undecided
Unassigned
Impish
Invalid
Undecided
Unassigned
Jammy
Invalid
Undecided
Unassigned

Bug Description

WCCP peering between squid and CIsco IOS 15.8(3)M2 stopped as of

Start-Date: 2021-10-07 06:27:37
Commandline: /usr/bin/unattended-upgrade
Upgrade: squid-common:amd64 (3.5.27-1ubuntu1.11, 3.5.27-1ubuntu1.12)

1) The release of Ubuntu you are using: 18.04
2) The version of the package you are using: 3.5.27-1ubuntu1.12
3) What you expected to happen:

Unattended upgrade will not break working setup. Valid wccp packets from the router continue to get accepted and processed by squid.

4) What happened instead

The squid cache.log is logging a loop of ERROR messages:

ERROR: Ignoring WCCPv2 message: ntohl(wccp2_i_see_you.type) == WCCP2_I_SEE_YOU
ERROR: Ignoring WCCPv2 message: !security_info
ERROR: Ignoring WCCPv2 message: !security_info
ERROR: Ignoring WCCPv2 message: !security_info

Router logged Oct 7 04:28:45.918: %WCCP-1-SERVICELOST: Service web-cache lost on WCCP client x.x.x.x
Since then debug wccp logs periodically WCCP-EVNT:IPv4:S0: HIA from x.x.x.x with bad rcv_id 0 (expected yy)

wccp service detail shows: WCCP Client information: State: NOT Usable (initializing)

CVE References

Changed in squid (Ubuntu):
status: New → Invalid
Changed in squid (Ubuntu Bionic):
status: New → Invalid
Changed in squid3 (Ubuntu):
status: New → Invalid
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thank you for taking the time to file a bug report.

I noticed that the latest update of the squid3 package on Bionic was a security fix that touched exactly the WCCP code:

squid3 (3.5.27-1ubuntu1.12) bionic-security; urgency=medium

  * SECURITY UPDATE: information disclosure via OOB read in WCCP protocol
    - debian/patches/CVE-2021-28116.patch: validate packets better in
      src/wccp2.cc.
    - CVE-2021-28116

 -- Marc Deslauriers <email address hidden> Mon, 04 Oct 2021 08:32:25 -0400

I'm trying to understand here how to reproduce this bug. I don't have access to Cisco hardware, and I'm not an expert on WCCP (far from it).

Given the description of the changelog entry above, I would double check to see if your Cisco hardware is properly configured and running the latest version of its firmware/software.

Based on the logs you posted, the following is one of the assertions that is failing on squid:

  Must(ntohl(wccp2_i_see_you.type) == WCCP2_I_SEE_YOU);

This means that the packet received by squid don't have the expected type, apparently. This check wasn't here before the patch.

This is another assertion that is failing:

         case WCCP2_SECURITY_INFO:
             Must(!security_info); // <----- THIS ASSERTION HERE
             SetField(security_info, itemHeader, itemHeader, itemSize,
                      "security definition truncated");
             break;

This case statement has been rewritten, and the assertion is now in place there.

In fact, this whole function has been overhauled and is quite different than what it was before this latest squid3 version. I am not sure if what you're seeing is in fact a bug in squid, or is actually squid being more careful regarding what it accepts as WCCP packets.

Either way, I would need a way to reproduce this error locally in order to further investigate it. Could you please provide some help in this regard? It would also be great if could try squid from newer Ubuntu releases to see if you can reproduce this problem.

I am setting this as Incomplete for now.

Changed in squid3 (Ubuntu Bionic):
status: New → Incomplete
Revision history for this message
amk (9-launchpad-mikus-sk) wrote :

Thank you for looking into the issue.

Let me first test current versions of squid against my router. If that works I shall dig into the ubuntu code. Already tried to enable wccp debug in squid but it did not help much. Ended up running a standalone wccp client as a workaround.

Where is the patch coming from? Official patches for the issue I could find are for squid 4 and 5 only.

Revision history for this message
Marc Deslauriers (mdeslaur) wrote :

Hi,

The patch was backported from Squid 4 as no patch for Squid 3 was available. The code in wccp2.cpp is almost identical. The resulting code in wccp2.cpp is almost identical to the code in 4.13 in impish, so I suspect you'll hit the same regression with current versions of Squid.

The only two commits that are different are the two following commits, which I don't believe could be causing the regression you are seeing:

https://github.com/squid-cache/squid/commit/7f7b4fd3f9af404d5bc528f7a73320f3ed1cc7d4
https://github.com/squid-cache/squid/commit/43b6575c9823248357a1eca8a55db76fd6c848ca

It would help to be able to test your environment with current versions of Squid to determine if this is caused by the upstream fix or not. Thanks!

Revision history for this message
amk (9-launchpad-mikus-sk) wrote :

4.13-10ubuntu5 in 21.10 and 5.2-1ubuntu1 in jammy are failing as well, with debug log different when compared to version 3 involved here:

2021/12/05 19:58:41.705 kid1| 80,6| wccp2.cc(1580) wccp2HereIam: wccp2HereIam: Called
2021/12/05 19:58:41.705 kid1| 80,5| wccp2.cc(1599) wccp2HereIam: wccp2HereIam: sending to service id 0
2021/12/05 19:58:41.705 kid1| 80,3| wccp2.cc(1630) wccp2HereIam: Sending HereIam packet size 144
2021/12/05 19:58:41.707 kid1| 80,6| wccp2.cc(1202) wccp2HandleUdp: wccp2HandleUdp: Called.
2021/12/05 19:58:41.707 kid1| 80,3| wccp2.cc(1226) wccp2HandleUdp: Incoming WCCPv2 I_SEE_YOU length 128.
2021/12/05 19:58:41.707 kid1| ERROR: Ignoring WCCPv2 message: duplicate security definition
    exception location: wccp2.cc(1249) wccp2HandleUdp

This looks like a problem with squid itself, the packet does not have duplicate security definition. In the code at http://www.squid-cache.org/Doc/code/wccp2_8cc_source.html I miss some debug output in the loop processing the packet /* Go through the data structure */ so would need to rebuild the package or to involve debugger.

I was not able to find any documentation of squid listing supported/tested wccp servers but at this point this looks like an issue to be reported upstream. There is no reason to consider wccp packets from IOS 15.8(3)M2 invalid.

Revision history for this message
amk (9-launchpad-mikus-sk) wrote :
Changed in squid3 (Ubuntu Bionic):
status: Incomplete → Confirmed
Changed in squid3 (Ubuntu):
status: Invalid → Confirmed
Revision history for this message
Sergio Durigan Junior (sergiodj) wrote :

Thanks for the further investigation, amk. And thanks for following up with upstream. We will track the progress of their bug and act accordingly (likely backporting a patch to fix the issue).

Changed in squid (Ubuntu Impish):
status: New → Invalid
status: Invalid → Confirmed
Changed in squid (Ubuntu Jammy):
status: Invalid → Confirmed
Changed in squid3 (Ubuntu Impish):
status: New → Invalid
Changed in squid3 (Ubuntu Jammy):
status: Confirmed → Invalid
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

It seems that upstream did not merge any fix yet. Let's keep an eye on it.

Revision history for this message
Brian Murray (brian-murray) wrote :

Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release.

Changed in squid (Ubuntu Impish):
status: Confirmed → Won't Fix
Revision history for this message
Lucas Kanashiro (lucaskanashiro) wrote :

I still do not see anything merged upstream to address this issue.

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

There is some progress going on https://github.com/squid-cache/squid/pull/970. Let's continue to monitor the patch progress in there.

Revision history for this message
Athos Ribeiro (athos-ribeiro) wrote :

The referred PR is still a draft. No recent updates here.

To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.