dput's sftp method is very slow

Bug #1814791 reported by Roman Shipovskij
14
This bug affects 2 people
Affects Status Importance Assigned to Milestone
dput (Ubuntu)
Fix Released
Low
Unassigned

Bug Description

Distributor ID: Ubuntu
Description: Ubuntu 18.04.1 LTS
Release: 18.04
Codename: bionic
dput: 1.0.1ubuntu1

New dput's sftp method is very slow, ~1.5Mbit/s, after some investigation I found that increasing buffer size up to 65536 increase upload speed up to ~8Mbit/s, but this is still much slower than old method.

Revision history for this message
Roman Shipovskij (roman-shipovskij) wrote :
Revision history for this message
Launchpad Janitor (janitor) wrote :

Status changed to 'Confirmed' because the bug affects multiple users.

Changed in dput (Ubuntu):
status: New → Confirmed
tags: added: disco
Changed in dput (Ubuntu):
importance: Undecided → Low
Mathew Hodson (mhodson)
tags: added: patch
Revision history for this message
Julian Andres Klode (juliank) wrote :

We should probably also or only adjust the reads to be 64 KiB large. I'm not sure if the sftp code is always buffering - I'd hope if I pass it really large buffers like that, that it just uses those directly, rather than buffer again.

Changed in dput (Ubuntu):
status: Confirmed → In Progress
Revision history for this message
Julian Andres Klode (juliank) wrote :

Changed it to use 256 KiB large buffers when reading and turned of buffering in sftp, in order to get things to hopefully speed up; although progress reporting quality will suffer obviously.

Changed in dput (Ubuntu):
status: In Progress → Fix Released
status: Fix Released → Fix Committed
Revision history for this message
Julian Andres Klode (juliank) wrote :

Not sure why I picked 256 KiB vs 64 KiB, though. Let me know how it performs for you.

Revision history for this message
Roman Shipovskij (roman-shipovskij) wrote :

I just installed dput 1.0.3ubuntu1 on Bionic from https://launchpad.net/ubuntu/+source/dput/1.0.3ubuntu1 and now upload speed is ~11Mbit/s.

For example, upload speed of Midnight Commander or console sftp is 100Mbit/s which is the physical limit of my network.

Revision history for this message
Launchpad Janitor (janitor) wrote :

This bug was fixed in the package dput - 1.0.3ubuntu1

---------------
dput (1.0.3ubuntu1) eoan; urgency=medium

  [ Julian Andres Klode ]
  * Merge from Debian unstable. Remaining changes:
    - dput:
      + Recognize 0ubuntu1 as a debian version that requires
        orig.tar.gz to be included in the upload.
      + Correctly handle cases where a non-existant host or no host at all is
        supplied when -o option is passed. (Thanks to David Futcher for
        spotting)
      + Be more careful about setting a variable in a section that does not
        exist in host argument handling.
    - dput.1: Updated to document host argument feature and sftp support.
    - dput.cf:
      + Set 'default_main_host = ubuntu'
      + Set 'progress_indicator = 2'
      + Updated ppa stanza to make use of argument support.
    - dput.cf.5: Updated to note support for sftp transport and host args.
    - dput.cf: Switch ubuntu stanza to upload to "/ubuntu" rather than "/"
      (LP: #1340130).
    - dput.cf: Drop trailing "/ubuntu" from ppa stanza, to support the new
      form of the upload path needed for PPAs based on derived distributions
      (LP: #1340130).
    - Implement a new sftp method that connects via openssh and then
      uses paramiko's sftp support. Some code copied from bzr.
  * sftp: Use bigger chunks when copying files (LP: #1814791)
  * sftp: Handle exceptions during upload, try to remove file

  [ David Lechner ]
  * sftp: fix atomic upload regression for sftp method (LP: #1762572)

dput (1.0.3) unstable; urgency=medium

  * The “سعد راشد محمد الفقيه‎‎ (Sa'ad Rashed Mohammad al-Faqih)” release.

  [ Ben Finney ]
  * Specify current VCS for this code base.
  * Declare “Standards-Version: 4.3.0”. No additional changes needed.
  * Declare Debhelper compatibility level 12.
  * debian/compat:
    * Remove obsolescent configuration file.
  * Update publication years in copyright notices.

  [ Ondřej Nový ]
  * Remove specification of minimum Python versions.
    The required versions are now in all supported Debian releases.

 -- Julian Andres Klode <email address hidden> Mon, 22 Apr 2019 12:31:01 +0200

Changed in dput (Ubuntu):
status: Fix Committed → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.