Launchpad itself

bzr push reporting AssertionError on terminal

Bug #674305 reported by Michael Hudson-Doyle
68
This bug affects 13 people
Affects Status Importance Assigned to Milestone
Launchpad itself
Fix Released
High
Unassigned

Bug Description

mwh@grond:testtools$ bzr push
Using saved push location: bzr+ssh://bazaar.launchpad.net/~mwhudson/launchpad/testtools
Traceback (most recent call last):
  File "/srv/bazaar.launchpad.net/production/launchpad-rev-11738/eggs/Twisted-10.1.0-py2.6-linux-x86_64.egg/twisted/internet/defer.py", line 441, in _runCallbacks
...
  File "/usr/lib/python2.6/xmlrpclib.py", line 838, in close
    raise Fault(**self._stack[0])
xmlrpclib.Fault: <Fault -1: 'Unexpected Zope exception: AssertionError: '>
No new revisions to push.
HPSS calls: 12 (0 vfs) SmartSSHClientMedium(bzr+ssh://<email address hidden>/)

OOPS-1777XMLP119 is if not my oops, an example of the same problem.

Things to note:

1) the traceback is being printed to stderr by the bzr serve process on the server
2) the assertionerror is being raised by a call to transaction.doom() that's in an "except (RequestExpired, TimeoutError):" block
3) the sql log in the oops reports a gap of 15s between sql requests (so a timeout is legitimate, although something funny is going on)
4) that assertionerror is raised when the transaction is not "active" or already "doomed". The other possible statuses are "committing", "committed" or "commitfailed". I don't know which.

The XMLRPC internal server was suffering a high rate of OOPSes / timeouts and its these that are being shown on the codehosting front end and passed onto the user.

We believe it is a load/timeout issue on one specific internal server.

=======

We think this is fixed: we've reconfigured the backend xmlrpc service to be served from our primary appserver cluster, so there are now 13 times the resources available for it; we've further fine tuning and capacity management to do, but the basic situation should be remedied.

*please* comment here if you encounter zope assertion error, so that we can tell the bug isn't fixed.

See original description
Revision history for this message
Tim Penhey (thumper) wrote :

There is an RT to have all the app servers provide internal xmlrpc access. This should fix what we are mostly guessing right now is the thread starvation that is going on here.

tags: added: codehosting-ssh xmlrpc
Changed in launchpad-code:
status: New → Triaged
importance: Undecided → High
Revision history for this message
Bilal Akhtar (bilalakhtar) wrote :

This happened to me just now as well! http://paste.ubuntu.com/530504/

I noticed a small delay in scanning the branch, so could this error be the reason? I guess bzr pokes LP to scan the branch, which isn't done when this error comes. Nevertheless, the branch gets pushed by SSH.

(just trying to provide extra information for LP devs to work on a fix)

Revision history for this message
Bilal Akhtar (bilalakhtar) wrote :

Continuation of my previous comment:

On the pastebin link, we can see that when I tried to push again, and even though there were no new revs to push, the error came.

Revision history for this message
Guewen Baconnier @ Camptocamp (gbaconnier-c2c) wrote :
Download full text (4.8 KiB)

Hello,

I got the same error but with a commit.
The commit is well done but I got the following traceback :

gbaconnier@yushan:~/code/c2cXA/extra-addons$ bzr commit -m"[MRG] from LP"
Committing to: bzr+ssh://bazaar.launchpad.net/~c2c/openobject-addons/stable_5.0-extra-addons-c2c-official/
added c2c_scanned_packing
modified base_sale_multichannels/sale.py
modified base_sale_multichannels/sale_view.xml
modified c2c_block_customer_sale/c2c_block_customer_sale_view.xml
added c2c_scanned_packing/__init__.py
added c2c_scanned_packing/__terp__.py
added c2c_scanned_packing/server_patch.diff
added c2c_scanned_packing/stock.py
added c2c_scanned_packing/stock_view.xml
added c2c_scanned_packing/wizard
added c2c_scanned_packing/wizard/__init__.py
added c2c_scanned_packing/wizard/c2c_scan_product.py
added c2c_scanned_packing/wizard/c2c_scan_product_view.xml
added c2c_scanned_packing/wizard/last_packing_confirm.py
added c2c_scanned_packing/wizard/last_packing_confirm_view.xml
added c2c_scanned_packing/wizard/validate_partial_packing.py
added c2c_scanned_packing/wizard/validate_partial_packing_view.xml
modified c2c_webkit_report/webkit_report.py
modified fleet_maintenance/stock.py
modified kettle_connector/kettle.py
modified kettle_connector/terminator_install.py
added marketing/i18n/sr.po
added marketing/i18n/zh_CN.po
modified nan_product_pack/pack.py
added partner_spam/i18n/pt.po
added paydays/i18n/pt.po
added product_expiry/i18n/el.po
modified product_expiry/i18n/hr.po
modified product_gs1_128/i18n/fr_FR.po
added product_price_update/i18n/pt.po
added product_qt/i18n/pt.po
added product_related_pricelist_items/i18n/pt.po
modified product_tax_include/i18n/pt.po
added product_variant_multi/i18n/pt.po
added product_visible_discount/i18n/el.po
added product_visible_discount/i18n/pt.po
added purchase_discount/i18n/pt.po
added purchase_payment/i18n/pt.po
added purchase_tax_include/i18n/pt.po
added pxgo_bank_statement_analytic/i18n/pt.po
added pxgo_bank_statement_running_balance/i18n/pt.po
added pxgo_cash_statement/i18n/pt.po
added pxgo_cash_statement_analytic/i18n/pt.po
modified pxgo_openoffice_reports/openoffice_report.py
added pxgo_openoffice_reports/i18n/pt.po
added radiotv/i18n/pt.po
added sale_credit_limit/i18n/pt.po
added sale_margin/i18n/pt.po
added sale_payment/i18n/pt.po
added sale_retails/i18n/pt.po
added survey/i18n/nl.po
added survey/i18n/pt.po
modified training/training.py
modified training/training_sequence.xml
modified training/i18n/ca_ES.po
modified training/i18n/es_ES.po
modified training/i18n/training.pot
modified training/wizard/wizard_create_offer.py
added wiki_extension/i18n/pt.po
Committed revision 3925.
Traceback (most recent ...

Read more...

Revision history for this message
Andrew Bennetts (spiv) wrote :

~gbaconnier-c2c: yes, commit in a bound branch is essentially the same as push.

Someone on #twisted just reported seeing:

    bzr: ERROR: Server sent an unexpected error: ('error', "<Fault -1: 'Unexpected Zope exception: RequestExpired: request expired.'>")

Which may be related. Basically, it appears some part of Launchpad is timing out.

Revision history for this message
Jonathan Lange (jml) wrote :

Bumped to critical. We need to fix this pronto.

Changed in launchpad-code:
importance: High → Critical
Revision history for this message
Francis J. Lacoste (flacoste) wrote :

Although bad as it looks, this doesn't warrant the Critical priority.

Changed in launchpad-code:
importance: Critical → High
Revision history for this message
Jonathan Lange (jml) wrote : Re: [Bug 674305] Re: bzr push occasionally reports AssertionError on terminal

On Fri, Nov 12, 2010 at 6:59 PM, Francis J. Lacoste
<email address hidden> wrote:
> Although bad as it looks, this doesn't warrant the Critical priority.

I strongly disagree.

Revision history for this message
Robert Collins (lifeless) wrote :

On Sat, Nov 13, 2010 at 8:08 AM, Jonathan Lange <email address hidden> wrote:
> On Fri, Nov 12, 2010 at 6:59 PM, Francis J. Lacoste
> <email address hidden> wrote:
>> Although bad as it looks, this doesn't warrant the Critical priority.
>
> I strongly disagree.

We need a mechanism for dealing with problems in the weekend. Lets
discuss next week.

Robert Collins (lifeless)
summary: - bzr push occasionally reports AssertionError on terminal
+ bzr push reporting AssertionError on terminal
description: updated
description: updated
Revision history for this message
Robert Collins (lifeless) wrote :

We've disabled code imports for the weekend, because we think that will reduce load enough to permit this to settle and work. Fingers crossed.

Robert Collins (lifeless)
description: updated
Cody A.W. Somerville (cody-somerville)
tags: added: oem-services
Robert Collins (lifeless)
Changed in launchpad-code:
status: Triaged → In Progress
description: updated
Revision history for this message
Colin Watson (cjwatson) wrote :

Is there any cleanup that needs to be done in the aftermath of this? I encountered this assertion error while pushing to lp:~cjwatson/upstart/state-changed, and while 'bzr log lp:~cjwatson/upstart/state-changed' shows the correct final revision, https://code.launchpad.net/~cjwatson/upstart/state-changed does not.

The final revision should be:

revno: 1244
committer: Colin Watson <email address hidden>
branch nick: state-changed
timestamp: Sat 2010-11-13 01:47:34 +0000
message:
  new Failed signal - works better than trying to deduce failure from combinations of goals and states

Revision history for this message
Robert Collins (lifeless) wrote : Re: [Bug 674305] Re: bzr push reporting AssertionError on terminal

On Sun, Nov 14, 2010 at 2:48 AM, Colin Watson <email address hidden> wrote:
> Is there any cleanup that needs to be done in the aftermath of this?  I

Its very likely that we need to use a find script on the codehosting
filesystem to identify changed branches and trigger a branchChanged
event for them all, for the period when this started - with a fuzz
factor - say 1/2 day before the bug report was opened.

Revision history for this message
Tim Penhey (thumper) wrote :

Clean up has been completed. All branches touched during the issue tie have been poked and scanned if necessary.

Revision history for this message
Robert Collins (lifeless) wrote :

Thanks Tim.

Changed in launchpad-code:
status: In Progress → Fix Released
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Subscribing...

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.