CURLFile POST missing Content-Length header

Bug #1887826 reported by Stephen Rees
14
This bug affects 1 person
Affects Status Importance Assigned to Milestone
php
Unknown
Unknown
php7.4 (Ubuntu)
Fix Released
Medium
Bryce Harrington
Focal
Fix Released
Medium
Bryce Harrington

Bug Description

[Impact]

 * PHP7.4.3 has an issue with cURL where it is defaulting to chunked mode when transferring a file via POST. Setting the HTTP version does not impact it. For third-party servers that don't support chunked but expect a content-length, there is no way to make it work.

 * See https://bugs.php.net/bug.php?id=79013 for this exact issue.

[Test Case]

* Here is a sample similar to what I'm doing, with url/token/file removed:

-----------------

<?php
$url = '';
$access_token = '';
$file = '';
$ch = curl_init($url);
$options = [
   CURLOPT_CUSTOMREQUEST => 'POST,
   CURLOPT_RETURNTRANSFER => true,
   CURLOPT_HTTPHEADER => [
      'Authorization: Bearer ' . $access_token,
      'Content-Type: multipart/form-data',
   ],
   CURLOPT_SSL_VERIFYHOST => 2,
   CURLOPT_FOLLOWLOCATION => true,
   CURLOPT_POSTREDIR => 3,
   CURLINFO_HEADER_OUT => true,
   CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,//this has no effect
];
$file = curl_file_create(realpath($file), null, basename($file));
$data = ['file' => $file, 'name' => basename($file)];
$options[CURLOPT_POSTFIELDS] = $data;
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
$curl_info = curl_getinfo($ch);
$curl_error = curl_error($ch);
curl_close($ch);
?>

-----------------

This works in Ubuntu 18, but not Ubuntu 20.

[Regression Potential]

The fix itself for this bug is reasonably straightforward, but it builds on an earlier patch that involved a fair bit of refactoring code.

This includes code which detects and adapts to libcurl > 7.56.0, so behavioral regressions narrowable to the newer versions of libcurl could potentially indicate a regression caused by this SRU.

[Other Info]

Upgrading Ubuntu's supported PHP version to 7.4.4 *should* fix the issue, or backporting the patch in the link I included.

package: php7.4 (7.4.3-4ubuntu2.2)(possibly related, libapache2-mod-php7.4d, php7.4-curl)
release: Ubuntu 20.04 LTS

Expect: PHP-cURL to POST HTTP1.1 multipart/form-data with automatically generated Content-Length and empty Transfer-Encoding headers.

Actual: PHP-cURL POSTS HTTP1.1 multipart/form-data with Transfer-Encoding: chunked and empty Content-Length headers.

Related branches

CVE References

Revision history for this message
Christian Ehrhardt  (paelzer) wrote :

Hi Stephen - thanks for the report,
a merge of 7.4.5 is currently going on and will resolve this in groovy.
From there SRUs can be considered as needed.

Changed in php7.4 (Ubuntu):
status: New → Triaged
assignee: nobody → Bryce Harrington (bryce)
Bryce Harrington (bryce)
tags: added: server-next
Changed in php7.4 (Ubuntu):
importance: Undecided → Medium
Revision history for this message
Bryce Harrington (bryce) wrote :

Here is the patch from upstream's PR https://github.com/php/php-src/pull/5228

Revision history for this message
Bryce Harrington (bryce) wrote :

Stephen Rees, can you provide steps to reproduce the problem? We'll need to document that to get the SRU approved.

Changed in php7.4 (Ubuntu):
status: Triaged → Incomplete
status: Incomplete → Triaged
Changed in php7.4 (Ubuntu Focal):
status: New → Incomplete
tags: added: patch
Revision history for this message
Stephen Rees (srees304) wrote :

I'm not quite sure what to say about how to reproduce.
POST a file using PHP-cURL to a server that doesn't support chunked mode.

I'm using https://sandbox.zenodo.com, but this requires a free account and utilizing their API, which seems beyond the scope of what I should put in a comment. Here is a sample similar to what I'm doing, with url/token/file removed:
-----------------
<?php
$url = '';
$access_token = '';
$file = '';
$ch = curl_init($url);
$options = [
   CURLOPT_CUSTOMREQUEST => 'POST,
   CURLOPT_RETURNTRANSFER => true,
   CURLOPT_HTTPHEADER => [
      'Authorization: Bearer ' . $access_token,
      'Content-Type: multipart/form-data',
   ],
   CURLOPT_SSL_VERIFYHOST => 2,
   CURLOPT_FOLLOWLOCATION => true,
   CURLOPT_POSTREDIR => 3,
   CURLINFO_HEADER_OUT => true,
   CURLOPT_HTTP_VERSION => CURL_HTTP_VERSION_1_1,//this has no effect
];
$file = curl_file_create(realpath($file), null, basename($file));
$data = ['file' => $file, 'name' => basename($file)];
$options[CURLOPT_POSTFIELDS] = $data;
curl_setopt_array($ch, $options);
$response = curl_exec($ch);
$curl_info = curl_getinfo($ch);
$curl_error = curl_error($ch);
curl_close($ch);

This works in Ubuntu 18, but not Ubuntu 20. To make it work in 20 I have to additionally empty the Transfer-Enconding header and determine and set the Content-Length header. This should not be necessary:
$options[CURLOPT_HTTPHEADER][] = "Transfer-Encoding: ";
//not sure how to determine $size
$options[CURLOPT_HTTPHEADER][] = "Content-Length: $size"

description: updated
Changed in php7.4 (Ubuntu Focal):
status: Incomplete → Triaged
Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package php7.4 - 7.4.5-1ubuntu2

---------------
php7.4 (7.4.5-1ubuntu2) groovy; urgency=medium

  * No-change rebuild against libicu67

 -- Steve Langasek <email address hidden> Tue, 28 Jul 2020 16:14:33 +0000

Changed in php7.4 (Ubuntu):
status: Triaged → Fix Released
Bryce Harrington (bryce)
Changed in php7.4 (Ubuntu Focal):
importance: Undecided → Medium
assignee: nobody → Bryce Harrington (bryce)
Bryce Harrington (bryce)
description: updated
Revision history for this message
Brian Murray (brian-murray) wrote : Please test proposed package

Hello Stephen, or anyone else affected,

Accepted php7.4 into focal-proposed. The package will build now and be available at https://launchpad.net/ubuntu/+source/php7.4/7.4.3-4ubuntu2.3 in a few hours, and then in the -proposed repository.

Please help us by testing this new package. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation on how to enable and use -proposed. Your feedback will aid us getting this update out to other Ubuntu users.

If this package fixes the bug for you, please add a comment to this bug, mentioning the version of the package you tested, what testing has been performed on the package and change the tag from verification-needed-focal to verification-done-focal. If it does not fix the bug for you, please add a comment stating that, and change the tag to verification-failed-focal. In either case, without details of your testing we will not be able to proceed.

Further information regarding the verification process can be found at https://wiki.ubuntu.com/QATeam/PerformingSRUVerification . Thank you in advance for helping!

N.B. The updated package will be released to -updates after the bug(s) fixed by this package have been verified and the package has been in -proposed for a minimum of 7 days.

Changed in php7.4 (Ubuntu Focal):
status: Triaged → Fix Committed
tags: added: verification-needed verification-needed-focal
Revision history for this message
Ubuntu SRU Bot (ubuntu-sru-bot) wrote : Autopkgtest regression report (php7.4/7.4.3-4ubuntu2.3)

All autopkgtests for the newly accepted php7.4 (7.4.3-4ubuntu2.3) for focal have finished running.
The following regressions have been reported in tests triggered by the package:

php-sabre-vobject/unknown (armhf)
php-league-commonmark/unknown (armhf)
cacti/unknown (armhf)
jsonlint/unknown (armhf)
composer/unknown (armhf)
plinth/unknown (armhf)
php-doctrine-dbal/unknown (armhf)
php-parser/4.2.2-2 (ppc64el, amd64, s390x, arm64, armhf)
php-codecoverage/unknown (armhf)
php-imagick/unknown (armhf)
pkg-php-tools/unknown (armhf)

Please visit the excuses page listed below and investigate the failures, proceeding afterwards as per the StableReleaseUpdates policy regarding autopkgtest regressions [1].

https://people.canonical.com/~ubuntu-archive/proposed-migration/focal/update_excuses.html#php7.4

[1] https://wiki.ubuntu.com/StableReleaseUpdates#Autopkgtest_Regressions

Thank you!

Revision history for this message
Bryce Harrington (bryce) wrote :

Stephen, now that this has been accepted to -proposed, could you verify the fix is working properly? You'll need to enable proposed on a 20.04 based system, install the package, and verify the issue no longer reproduces.

Thanks ahead of time.

Revision history for this message
Stephen Rees (srees304) wrote :

Verified as fixed.

The instructions on the listed wikipage (https://wiki.ubuntu.com/Testing/EnableProposed) for testing a package from proposed were complete rubbish. Couldn't get those to work any way I could imagine. I finally got it installed using the SRU Verification wiki page for instructions -- just adding the one line to sources.list.

Steps to verify:
- removed PHP 7.4.9 and external PPA it was sourced from
- reinstalled PHP 7.4.3 from normal ubuntu packages
- reproduced problem
- removed PHP 7.4.3
- enabled proposed repo
- reinstalled PHP 7.4.3 from -proposed
- verified the issue is corrected!

Thanks!

Revision history for this message
Stephen Rees (srees304) wrote :

Greater specificity:
Also included php7.4-curl with the above testing, and the specific version of packages tested were
(7.4.3-4ubuntu2.3)

tags: added: verification-done-focal
removed: verification-needed-focal
Revision history for this message
Łukasz Zemczak (sil2100) wrote :

Thank you for the verification! Bryce, I see there's still a set of autopkgtest failures for this upload (php-parser for all arches) - can you take a look and assess if this is now broken in release and/or unrelated? Thanks!

Revision history for this message
Bryce Harrington (bryce) wrote :

Łukasz, yes, see LP: #1895878 and LP: #1878102 which are additional SRUs to get this thru. I'm not sure what causes those failures but they are reproducible independently of this so evidently are unrelated.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package php7.4 - 7.4.3-4ubuntu2.4

---------------
php7.4 (7.4.3-4ubuntu2.4) focal-security; urgency=medium

  * SECURITY UPDATE: Incorrect encryption data
    - debian/patches/CVE-2020-7069.patch: fix wrong ciphertext/tag
      in AES-CCM encryption for a 12 bytes IV in ext/openssl/openssl.c,
      ext/openssl/tests/cipher_tests.inc, ext/openssl/openssl_*_ccm.phpt.
    - CVE-2020-7069
  * SECURITY UPDATE: Possibly forge cookie
    - debian/patches/CVE-2020-7070.patch: do not decode cookie names anymore
      in main/php_variables.c, tests/basic/022.phpt, tests/basic/023.phpt,
      tests/basic/bug79699.phpt.
    - CVE-2020-7070

 -- <email address hidden> (Leonidas S. Barbosa) Tue, 06 Oct 2020 12:47:56 -0300

Changed in php7.4 (Ubuntu Focal):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.