glance caching fails with webob 1.2.0

Bug #1045179 reported by Itsuro Oda
24
This bug affects 4 people
Affects Status Importance Assigned to Milestone
Glance
Fix Released
Medium
Sascha Peilicke
Folsom
Fix Released
Medium
Sascha Peilicke
Grizzly
Fix Released
Medium
Sascha Peilicke

Bug Description

get devstack at 2012/09/03. set "flavor=keystone+caching" in glance-api.conf.
---
$ glance -d image-list
curl -i -X GET -H 'X-Auth-Token: 9523d7980eb045c283cc127922e7cef4' -H 'Content-Type: application/json' -H 'User-Agent: python-glanceclient' http://172.17.190.3:9292/v1/images/detail?limit=20

HTTP/1.1 500 Internal Server Error
date: Mon, 03 Sep 2012 05:25:56 GMT
content-length: 2234
content-type: text/plain
connection: close

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line 382, in handle_one_response
    result = self.application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 320, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1295, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1259, in call_application
    app_iter = application(self.environ, start_response)
  File "/opt/stack/keystone/keystone/middleware/auth_token.py", line 278, in __call__
    return self.app(env, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 320, in __call__
    response = req.get_response(self.application)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1295, in send
    application, catch_exc_info=False)
  File "/usr/local/lib/python2.7/dist-packages/webob/request.py", line 1259, in call_application
    app_iter = application(self.environ, start_response)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 130, in __call__
    resp = self.call_func(req, *args, **self.kwargs)
  File "/usr/local/lib/python2.7/dist-packages/webob/dec.py", line 195, in call_func
    return self.func(req, *args, **kwargs)
  File "/opt/stack/glance/glance/common/wsgi.py", line 321, in __call__
    return self.process_response(response)
  File "/opt/stack/glance/glance/api/middleware/cache.py", line 145, in process_response
    image_id = request.environ['api.cache.image_id']
AttributeError: 'NoneType' object has no attribute 'environ'

Request returned failure status.
None
HTTPInternalServerError (HTTP 500)
---

note that "flavor=keystone" work OK. result should be:
---
$ glance -d image-list
curl -i -X GET -H 'X-Auth-Token: c1fcc2fa11094db5a4e325254882edf2' -H 'Content-Type: application/json' -H 'User-Agent: python-glanceclient' http://172.17.190.3:9292/v1/images/detail?limit=20

HTTP/1.1 200 OK
date: Mon, 03 Sep 2012 05:21:54 GMT
content-length: 1513
content-type: application/json; charset=UTF-8

{"images": [{"status": "active", "name": "cirros-0.3.0-x86_64-uec-ramdisk", "deleted": false, "container_format": "ari", "created_at": "2012-09-03T05:20:19", "disk_format": "ari", "updated_at": "2012-09-03T05:20:19", "properties": {}, "min_disk": 0, "protected": false, "id": "e76ee014-2bfc-4b2a-9fcb-751958f560b6", "checksum": "2cec138d7dae2aa59038ef8c9aec2390", "owner": "ee946e6701e241b58e24c21f2260c6b5", "is_public": true, "deleted_at": null, "min_ram": 0, "size": 2254249}, {"status": "active", "name": "cirros-0.3.0-x86_64-uec", "deleted": false, "container_format": "ami", "created_at": "2012-09-03T05:20:19", "disk_format": "ami", "updated_at": "2012-09-03T05:20:20", "properties": {"kernel_id": "2651798a-cd03-46c1-8cba-c83cdff7a4a9", "ramdisk_id": "e76ee014-2bfc-4b2a-9fcb-751958f560b6"}, "min_disk": 0, "protected": false, "id": "910dfb67-5261-46fc-aa41-d2f9067516b2", "checksum": "2f81976cae15c16ef0010c51e3a6c163", "owner": "ee946e6701e241b58e24c21f2260c6b5", "is_public": true, "deleted_at": null, "min_ram": 0, "size": 25165824}, {"status": "active", "name": "cirros-0.3.0-x86_64-uec-kernel", "deleted": false, "container_format": "aki", "created_at": "2012-09-03T05:20:18", "disk_format": "aki", "updated_at": "2012-09-03T05:20:18", "properties": {}, "min_disk": 0, "protected": false, "id": "2651798a-cd03-46c1-8cba-c83cdff7a4a9", "checksum": "cfb203e7267a28e435dbcb05af5910a9", "owner": "ee946e6701e241b58e24c21f2260c6b5", "is_public": true, "deleted_at": null, "min_ram": 0, "size": 4731440}]}

+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| ID | Name | Disk Format | Container Format | Size | Status |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
| 2651798a-cd03-46c1-8cba-c83cdff7a4a9 | cirros-0.3.0-x86_64-uec-kernel | aki | aki | 4731440 | active |
| 910dfb67-5261-46fc-aa41-d2f9067516b2 | cirros-0.3.0-x86_64-uec | ami | ami | 25165824 | active |
| e76ee014-2bfc-4b2a-9fcb-751958f560b6 | cirros-0.3.0-x86_64-uec-ramdisk | ari | ari | 2254249 | active |
+--------------------------------------+---------------------------------+-------------+------------------+----------+--------+
---

Revision history for this message
Itsuro Oda (oda-g) wrote :

self investigation:
---
$ cd /opt/stack
$ find . -name pip-requires | xargs egrep -i webob
./keystone/tools/pip-requires:WebOb==1.0.8
./glance/tools/pip-requires:webob==1.0.8
./cinder/tools/pip-requires:WebOb==1.0.8
./nova/tools/pip-requires:WebOb==1.0.8
./quantum/tools/pip-requires:webob==1.2.0
$ dpkg -l | egrep -i webob
ii python-webob 1.1.1-1ubuntu0 Python module providing WSGI request and response objects
$ pip freeze | egrep -i webob
Warning: cannot find svn location for distribute==0.6.24dev-r0
WebOb==1.2
---
is webob==1.2 wrong for glance ?

Revision history for this message
Brian Waldon (bcwaldon) wrote :

Can you try installing v1.0.8 to see if that solves the problem?

Revision history for this message
Itsuro Oda (oda-g) wrote :

It seems webob 1.0.8 solves the problem.
I tried to remove Quantum (and related services) from 'ENABLED_SERVICES' in devstack.
I got:
---
$ pip freeze | egrep -i webob
Warning: cannot find svn location for distribute==0.6.24dev-r0
WebOb==1.0.8
---
and 'glance image-list' with 'flavor=keystone+caching' works no problem.

Brian Waldon (bcwaldon)
summary: - glance image-list gets Internal Server Error when using
- flavor=keystone+caching
+ glance caching fails with webob 1.2.0
Revision history for this message
Brian Waldon (bcwaldon) wrote :

So it turns out webob deprecated acces to response.request in v1.2a1. They reversed that deprecation in v1.2b1 but it is no longer being automatically populated with the original Request object. The caching code will have to be rewritten to work without depending on that original request object.

Way to go, webob!

Changed in glance:
milestone: none → folsom-rc1
status: New → Triaged
importance: Undecided → Medium
Revision history for this message
Brian Waldon (bcwaldon) wrote :

I don't think this is the time to update the dep to v1.2.0, but it would be great to work with it in an unofficial capacity.

Changed in glance:
assignee: nobody → Eddie Sheffield (eddie-sheffield)
status: Triaged → In Progress
Revision history for this message
Brian Waldon (bcwaldon) wrote :

Quantum review that lowers its webob dep to v1.0.8: https://review.openstack.org/#/c/12514/

Brian Waldon (bcwaldon)
Changed in glance:
milestone: folsom-rc1 → none
Changed in glance:
assignee: Eddie Sheffield (eddie-sheffield) → nobody
Changed in glance:
assignee: nobody → Sascha Peilicke (saschpe)
Revision history for this message
OpenStack Infra (hudson-openstack) wrote : Fix merged to glance (master)

Reviewed: https://review.openstack.org/17794
Committed: http://github.com/openstack/glance/commit/7324ffd4c54fef9e5363d1a8a3432ac51d2fbf29
Submitter: Jenkins
Branch: master

commit 7324ffd4c54fef9e5363d1a8a3432ac51d2fbf29
Author: Sascha Peilicke <email address hidden>
Date: Mon Dec 10 19:20:26 2012 +0100

    wsgi.Middleware forward-compatibility with webob 1.2b1 or later

    Response.request is None by default
    (http://docs.webob.org/en/latest/news.html#b1), but is used in the
    CacheFilter WSGI middleware.

    Fixes bug 1045179

    Change-Id: I28f5ca92fe517f4f56af934799db32650e079ba7

Changed in glance:
status: In Progress → Fix Committed
Thierry Carrez (ttx)
Changed in glance:
milestone: none → grizzly-2
status: Fix Committed → Fix Released
Revision history for this message
Thierry Carrez (ttx) wrote :
To post a comment you must log in.
This report contains Public information  
Everyone can see this information.

Duplicates of this bug

Other bug subscribers

Remote bug watches

Bug watches keep track of this bug in other bug trackers.