udd importer should make tea while launchpad is down
Affects | Status | Importance | Assigned to | Milestone | |
---|---|---|---|---|---|
Launchpad itself |
Invalid
|
High
|
Unassigned | ||
Ubuntu Distributed Development |
Fix Released
|
High
|
Vincent Ladeuil |
Bug Description
Launchpad goes offline or readonly roughly every month for about an hour.
At the moment the package importer handles this quite poorly by failing every single package that it tries to import during that time. It can take some time for them to get retried; istr in some cases manual intervention is needed.
It might be better if, before starting an import, it checked if Launchpad is up (and writable) at all. If not, it should probably just not trying anything else for a few minutes.
I don't think Launchpad exports a specific machine-readable interface to say "I'm down" or "I'm readonly". We could look for "readonly" in the html of the front page or try some other requests. Possibly that code should go into lplib if it's reusable.
Related branches
- James Westby: Approve
-
Diff: 587 lines (+397/-17)7 files modifiedmass_import.py (+107/-13)
selftest.py (+1/-0)
udd/circuit_breaker.dot (+23/-0)
udd/circuit_breaker.py (+109/-0)
udd/icommon.py (+25/-4)
udd/tests/test_circuit_breaker.py (+129/-0)
udd/threads.py (+3/-0)
Changed in udd: | |
assignee: | Jonathan Riddell (jr) → Vincent Ladeuil (vila) |
status: | Confirmed → In Progress |
On Fri, 10 Jun 2011 00:09:47 -0000, Martin Pool <email address hidden> wrote:
> Public bug reported:
>
> Launchpad goes offline or readonly roughly every month for about an
> hour.
>
> At the moment the package importer handles this quite poorly by failing
> every single package that it tries to import during that time. It can
> take some time for them to get retried; istr in some cases manual
> intervention is needed.
>
> It might be better if, before starting an import, it checked if
> Launchpad is up (and writable) at all. If not, it should probably just
> not trying anything else for a few minutes.
There's a small issue here in that there are two processes involved.
The mass_import process is what should ideally sleep for a while. This
doesn't communicate with LP at all currently.
The import_package.py could check whether LP is up and exit with a "LP
is down" exit code, which would then trigger a sleep.
Another way to do it would just be to keep a tally of the number of
failures in the last N minutes and sleep if it's 100% failures for a
while.
Perhaps even both should be implemented?
Thanks,
James