HUD autopilot tests failing

I am checking this with fginther it seems to be related to an issue w/ the autolanding test environment, not with the test themselves afaik,

Running locally, I see this errors:

======================================================================
FAIL: unity_webapps_qml.tests.test_hud.UnityWebappsHudTestCase.test_clearAction
----------------------------------------------------------------------
_StringException: Empty attachments:
  process-stdout

process-return-code: {{{-15}}}
process-stderr: {{{
Loading testability driver.
Ignoring argument: "/usr/bin/unity-webapps-qml-launcher"
Ignoring argument: "-testability"
Adding property: "url" , value: "file:///home/vrruiz/webapps/lp/unity-webapps-qml-fix-hud-autopilot/tests/integration/autopilot/html/test_webapps_hud.html"
Adding property: "webappName" , value: "AutopilotTest"

WARNING: This project is using the experimental QML API extensions for QtWebKit and is therefore tied to a specific QtWebKit release.
WARNING: The experimental API will change from version to version, or even be removed. You have been warned!

Could not open manifest file: "/usr/share/unity-webapps/userscripts/unity-webapps-googleplus/manifest.json"
Invalid webapps installation found: "/usr/share/unity-webapps/userscripts/unity-webapps-googleplus"
[object Object]
Trying to initialize the MessagingMenu binding with invalid context
Could not retrieve domain for WebApp: "AutopilotTest"
onLoadingChanged: loading changed: file:///home/vrruiz/webapps/lp/unity-webapps-qml-fix-hud-autopilot/tests/integration/autopilot/html/test_webapps_hud.html, status: 0
WebApps received message: {"target":"unity-webapps-call","name":"init","args":"[{\"name\":\"HUD TEST\",\"domain\":\"localhost\",\"onInit\":{\"callbackid\":\"ubuntu-webapps-api0.5815204377286136\"},\"iconUrl\":\"icon://myicon\"}]"}
Backend ready: base
onLoadingChanged: loading changed: file:///home/vrruiz/webapps/lp/unity-webapps-qml-fix-hud-autopilot/tests/integration/autopilot/html/test_webapps_hud.html, status: 2
WebApps received message: {"target":"unity-webapps-call","name":"addAction","args":"[\"This is an action\",{\"callbackid\":\"ubuntu-webapps-api0.7348437742330134\"}]"}
WebApps received message: {"target":"unity-webapps-call","name":"addAction","args":"[\"Another action\",{\"callbackid\":\"ubuntu-webapps-api0.6678710402920842\"}]"}
Now watching for emissions of the "resultUpdated(QString)" signal on object with id 3
Signal emissions 0 QObject(0x22b7090)
Reply sent.
Method parameter names: ("expr")
Method parameter types: ("QVariant")
Method signature: "evalInPageUnsafe(QVariant)" return type: QVariant
Method Invoked.
Result: actionadded
Signal emissions 1 QObject(0x22b7090)
Reply sent.
Signal emissions 1 QObject(0x22b7090)
Reply sent.
Signal emissions 1 QObject(0x22b7090)
Reply sent.
Method parameter names: ("expr")
Method parameter types: ("QVariant")
Method signature: "evalInPageUnsafe(QVariant)" return type: QVariant
Method Invoked.
Signal emissions 1 QObject(0x22b7090)
Reply sent.
Result: true
Signal emissions 2 QObject(0x22b7090)
Reply sent.
Signal emissions 2 QObject(0x22b7090)
Reply sent.
Signal emissions 2 QObject(0x22b7090)
Reply sent.
Method parameter names: ("expr")
Method parameter types: ("QVariant")
Method signature: "evalInPageUnsafe(QVariant)" return ty...

  There are a number of problems going on with unity-webapps-qml's Autopilot tests. They don't pass on the desktop and I posted the failures in a comment to this bug: https://bugs.launchpad.net/unity-webapps-qml/+bug/1208355 The component appears to return unexpected results. Also, there is known issue with the call to a "log" function without being declared or imported, which Alex fixed in a branch and is pending to land.

  But the problem with the Jenkins VM has been another one. The failures in my desktop and in the VM are different. The call to ensure_unity_is_running() fails on the VM:

ERROR: hud.tests.test_hud.HudTestCase.test_hud
----------------------------------------------------------------------
  File "/usr/lib/python2.7/dist-packages/unity/emulators/__init__.py", line 42, in ensure_unity_is_running
  File "/usr/lib/python2.7/dist-packages/autopilot/introspection/backends.py", line 101, in introspection_iface
    "Application under test exited before the test finished!"
RuntimeError: Application under test exited before the test finished!

  This happens when AutopilotTestCase is used as the class parent, instead of UnityTestCase.

## This fails in the Jenkins VM
#!/usr/bin/env python

from unity.tests import UnityTestCase
from unity.emulators import ensure_unity_is_running
from unity.emulators.unity import Unity

class ATestCase(UnityTestCase):

    def setUp(self):
        super(ATestCase, self).setUp()
        ensure_unity_is_running()

    def test_hud(self):
        self.unity.hud.ensure_visible()
        self.addCleanup(self.unity.hud.ensure_hidden)
        self.assertTrue(True)

## This is OK.
#!/usr/bin/env python

from autopilot.testcase import AutopilotTestCase
from unity.emulators import ensure_unity_is_running

from unity.emulators.unity import Unity

class HudTestCase(AutopilotTestCase):

    def setUp(self):
        super(HudTestCase, self).setUp()
        ensure_unity_is_running()
        self.unity = Unity.get_root_instance()

    def test_hud(self):
        self.unity.hud.ensure_visible()
        self.addCleanup(self.unity.hud.ensure_hidden)
        self.assertTrue(True)

  UnityTestCase takes some steps to connect via DBUS to Unity, already has a call to ensure_unity_is_running(), and provides self.unity, no need to replicate that code.

  Why it passes in my computer but not in the VM? It may be a different Unity version (7.1.0 vs 7.0.2).

VM:
unity-autopilot 7.1.0+13.10.20130805
unity: 7.1.0+13.10.20130805-0ubuntu1

Desktop:
unity-autopilot 7.1.0+13.10.20130805-0ubuntu1
unity: 7.0.2+13.10.20130705.1-0ubuntu3

  So for this tests to run smoothly, reparent UnityWebappsTestCaseBase to UnityTestCase and fix the other errors.

Followup: Why it passes in my computer but not in the VM? I upgraded Unity to 7.1.0 and is still ok. The AutopilotTestCase failure can be reproduced ssh'ing to your own desktop (of course, DISPLAY must be properly set). I checked whether setting the DBUS env variables would avoid the failure, but not.

Correction: it's related to DBUS env variables.

$ export DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-tsy8Cc7bV0
$ autopilot run hud
Tests running...
======================================================================
ERROR: hud.tests.test_hud.HudTestCase.test_hud
[...]
RuntimeError: Application under test exited before the test finished!

$ export DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-tsy8Cc7bV0
$ autopilot run hud
Loading tests
Tests running...
Ran 2 tests in 2.018s
OK

The problem described above with AutopilotTestCase vs UnityTestCase was solved. But we still have issues with the HUD. On my desktop and in the VM, the tests are supposed to add and remove actions from the HUD. However, the HUD won't do that. With the help of Ted, I debugged on Friday and seems that there are DBUS connection problems with the HUD when running the tests.
Seen in .cache/upstart/dbus.log:

Failed to activate service 'com.canonical.hud': timed out

To make things worse, Alex Abreu is not able to reproduce the bug in his computer.

This failures are also related to this bug: https://bugs.launchpad.net/unity-webapps-qml/+bug/1211352

As seen in the daily-release, this problem has been fixed recently - therefore unblocking the component.