MQOpts unpack in pymqi.py doesn't unflatten arrays

Hey,

as a quick hint, without trying your code yet, does the explanation from http://packages.python.org/pymqi/#caveats do the trick for you?

No. I have taken care of my problem outside of pymqi.py. I am just
pointing out that the unpack doesn't handle arrays like pack does and
how I found it. It is probably not a big deal as it doesn't look like
anything that is unpacked currently has arrays. New features in future
MQ versions may have array parameters.

Brent

Brent
On Thu, 2010-02-25 at 20:49 +0000, Dariusz Suchojad wrote:
> Hey,
>
> as a quick hint, without trying your code yet, does the explanation from
> http://packages.python.org/pymqi/#caveats do the trick for you?
>

I'm a bit confused, what sort of arrays are you thinking of? Can you post a sample code that fails with curent .unpack? Thanks.

Hm, I'm sorry, I have somehow missed "if type(v) is types.ListType:" line in the your code - still, can you send some sample failing code? Thanks again.

Nothing is failing yet. But it could in future versions of MQ if they
decide to add an array to the md parameters.

The current pack allows for the handling of arrays:

    def pack(self):
        """ pack()

        Pack the attributes into a 'C' structure to be passed to MQI
        calls. The pack order is as defined to the MQOpts
        ctor. Returns the structure as a string buffer"""

        # Build tuple for struct.pack() argument. Start with format
        # string.
        args = [self.__format]
        # Now add the current attribute values to the tuple
        for i in self.__list:
            v = getattr(self, i[0])
            # Flatten attribs that are arrays
            if type(v) is types.ListType:
                for x in v:
                    args.append(x)
            else:
                args.append(v)
        return apply(struct.pack, args)

The current unpack does not:

    def unpack(self, buff):
        """unpack(buff)

        Unpack a 'C' structure 'buff' into self."""

        # Unpack returns a tuple of the unpacked data, in the same
        # order (I hope!) as in the ctor's list arg.
        r = struct.unpack(self.__format, buff)
        x = 0
        for i in self.__list:
            setattr(self, i[0], r[x])
            x = x + 1

Currently the only things that are unpacked are of class md and pmo
        mDesc.unpack(rv[0])
        putOpts.unpack(rv[1])
 in various locations.

An example of a class that does call pack and does have arrays as
arguments is cd.

        if "6.0" in pymqe.__mqlevels__:
            opts += [['HdrCompList', [0L, -1L], '2' + MQLONG_TYPE],
                     ['MsgCompList', [0] + 15 * [-1L], '16' +
MQLONG_TYPE],
                     ['CLWLChannelRank', 0L, MQLONG_TYPE],
                     ['CLWLChannelPriority', 0L, MQLONG_TYPE],
                     ['CLWLChannelWeight', 50L, MQLONG_TYPE],
                     ['ChannelMonitoring', 0L, MQLONG_TYPE],
                     ['ChannelStatistics', 0L, MQLONG_TYPE]]

HdrCompList and MsgCompList are examples of options that are arrays.

If in the future, if MQ decides to add an option that is an array in md
or pmo, and the options are added to pymqi.py, it will break because the
current unpack does not unpack arrays.

Here is a way you can test if you would like:

Since cd has arrays for arguments, it can be used for testing.
In the connectWithOptions function after the line ocd = cd() add the
following:

print 'cd structure before packing and unpacking'
print ocd
print 'cd structure after packing and unpacking'
print ocd

Then run some test code that calls connectWithOptions or call
connectTCPClient that calls connectWithOptions.

In the output, look at HdrCompList and MsgCompList before and after.
Notice before they are arrays and after they are not. You can see in
the after that the array values are spread to the following arguments
which totally messes up the rest of the arguments.

Next, make the changes I suggested to the unpack to unflatten the arrays
and rerun your test and you will see the before and after printouts of
the cd structure are the same.

    def unpack(self, buff):
        """unpack(b...

OK, I get it now, thanks for reporting it. I'm reclassifying it as a wishlist short of a way to specify a 'Request for enhancement'.

With 'HdrCompList', I think you've actually touched on a bigger issue of passing pointers from Python's PyMQI side to low-level pymqe extension. It currently 'works' because a list of 0's gets evaluated into a list of C NULLs so both pymqe and MQ are happy about that. But, if you really wanted to do advanced tricks with MQ, such as using HdrCompList, you wouldn't currently have any way to do it.

I've been bitten by it some time ago when I wanted to add MQ distribution lists to PyMQI, a mildly exotic feature which needs exactly the thing I've just mentioned. I don't know what to do about it; on one hand, 99,999% of users won't ever need 99,999% of advanced MQ features, on the other hand, that's no real excuse for not letting them use whatever's needed for the task at hand. I think it simply needs to wait for PyMQI 2.0 which I hope will be written in Cython (but that's another story, what about all those poor AIX/HP-UX/zOS users stuck with their exotic compilers.. PyMQI is truly cross-platform now and I won't let it change).

Anyway, it's a wishlist for now.

Converted to a blueprint instead of a bug https://blueprints.launchpad.net/pymqi/+spec/mqopts-unpack-in-pymqi-py-doesnt-unflatten-arrays