Python 3 support

The most important dependency to port over to Python 3 before work can start on pyexiv2 itself is boost-python.

According to http://www.boost.org/doc/libs/1_47_0/libs/python/doc/news.html, the work is done in the trunk but it hasn’t been released yet.

Hi Olivier,

Actually, I have theses dependencies installed on Archlinux:

/local/boost-libs 1.47.0-1
/local/exiv2 0.3.0-3
/local/scons 2.0.1-1

Is it a way to force scons to use python3.2/site-packages for install instead of python2.7 ?

I don't see anything in files SConstruct & SConscript.

I've also tried : scons install /usr/lib/python3.2/site-packages without success.

For Ubuntu, with Oneiric the default version of python will be version 3, can we hope the pyexiv2 package wil be updated ?

Hi Vincent,

The first important thing to check is whether your version of boost-python is compiled against python 3.
On my Ubuntu Natty box, libboost-python1.42.0 installs the following files:

    /usr/lib/libboost_python-py27.so.1.42.0
    /usr/lib/libboost_python-py26.so.1.42.0

I guess a version compiled against py3k would exhibit it in the name of the .so file.
I just checked http://www.archlinux.org/packages/extra/i686/boost-libs/ and the package contains /usr/lib/libboost_python3.so.1.47.0, so it looks like it’s going to work!

Now, I’m pretty sure that scons won’t run with python3, but it shouldn’t be a problem in itself, we can modify src/SConscript to hardcode the compilation flags to point to python3 instead of the current version in use by scons.

In src/SConscript, there are references to get_python_inc and get_python_lib, you will need to replace those code by hardcoded values that you will obtain from executing them in a python3 shell.

Please let me know if this helps, or the problems you encounter, I’ll gladly help. Having pyexiv2 work with py3k would be awesome!

Note that as far as I can tell the default version of Python in Ubuntu Oneiric is still 2.7.

Ok, I've updated libboost_python to 1.47.0

After theses change in src/SConscript

    env.Append(CPPPATH=['usr/include/python3.2mu']
    install_dir = 'usr/lib/python3.2/site-packages'

reprocess scons

[vincent@myhost pyexiv2-0.3.0]$ scons
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
g++ -o build/exiv2wrapper.os -c -fPIC -I/usr/include/python3.2mu src/exiv2wrapper.cpp
g++ -o build/exiv2wrapper_python.os -c -fPIC -I/usr/include/python3.2mu src/exiv2wrapper_python.cpp
g++ -o build/libexiv2python.so -shared build/exiv2wrapper.os build/exiv2wrapper_python.os -lboost_python -lexiv2
scons: done building targets.
[vincent@myhost pyexiv2-0.3.0]$ su
Password:
[root@myhost pyexiv2-0.3.0]# scons install
scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
Install file: "build/libexiv2python.so" as "/usr/lib/python3.2/site-packages/libexiv2python.so"
Install file: "src/pyexiv2/__init__.py" as "/usr/lib/python3.2/site-packages/pyexiv2/__init__.py"
Install file: "src/pyexiv2/exif.py" as "/usr/lib/python3.2/site-packages/pyexiv2/exif.py"
Install file: "src/pyexiv2/iptc.py" as "/usr/lib/python3.2/site-packages/pyexiv2/iptc.py"
Install file: "src/pyexiv2/metadata.py" as "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py"
Install file: "src/pyexiv2/preview.py" as "/usr/lib/python3.2/site-packages/pyexiv2/preview.py"
Install file: "src/pyexiv2/utils.py" as "/usr/lib/python3.2/site-packages/pyexiv2/utils.py"
Install file: "src/pyexiv2/xmp.py" as "/usr/lib/python3.2/site-packages/pyexiv2/xmp.py"
scons: done building targets.
[root@myhost pyexiv2-0.3.0]#

Trying a little appli:

[vincent@myhost magnifier_3]$ python main.py
Traceback (most recent call last):
  File "main.py", line 13, in <module>
    from miniView import MiniView
  File "/home/vincent/magnifier_3/miniView.py", line 11, in <module>
    import pyexiv2
  File "/usr/lib/python3.2/site-packages/pyexiv2/__init__.py", line 60, in <module>
    import libexiv2python
ImportError: /usr/lib/libboost_python.so.1.47.0: undefined symbol: PyClass_Type

Where is called this last import ? in libexiv2python ?

It should be /usr/lib/libboost_python3.so.1.47.0 wich exist.

Right, you also need to modify the following line in src/SConscript from:

    libs = [ARGUMENTS.get('BOOSTLIB', 'boost_python'), 'exiv2']

to:

    libs = [ARGUMENTS.get('BOOSTLIB', 'boost_python3'), 'exiv2']

and recompile (maybe run `scons -c` first to start afresh).

It seems we are on a good way.

I've also convert the .py files with 2to3, see attached log.

Trying with following code:

import pyexiv2

print ("pyexiv2 version: ", pyexiv2.version_info)

metadatas = pyexiv2.ImageMetadata('/home/vincent/magnifier_3/DSCN0502.JPG')
metadatas.read()
print ("Date:\t",metadatas['Exif.Image.DateTime'])
print ("ISO:\t",metadatas['Exif.Photo.IsoSpeedRatings'])
print ("Fnumber:\t",metadatas['Exif.Photo.FNumber'])

 I've this error:

[vincent@myhost magnifier_3]$ python metatest.py
pyexiv2 version: (0, 3, 0)
Traceback (most recent call last):
  File "metatest.py", line 13, in <module>
    metadatas.read()
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 107, in read
    self.__image = self._instantiate_image(self.filename)
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 79, in _instantiate_image
    return libexiv2python._Image(filename)
Boost.Python.ArgumentError: Python argument types in
    _Image.__init__(_Image, bytes)
did not match C++ signature:
    __init__(_object*, std::string, long)
    __init__(_object*, std::string)
[vincent@myhost magnifier_3]$

Indeed it’s starting to look good!

It looks like the 'filename' argument passed to libexiv2python._Image(…) is of type 'bytes' in python3, whereas I would expect it to be of type 'str'.

Does applying the following patch improve things?

=== modified file 'src/pyexiv2/metadata.py'
--- src/pyexiv2/metadata.py 2011-08-17 16:35:49 +0000
+++ src/pyexiv2/metadata.py 2011-09-16 11:08:32 +0000
@@ -76,7 +76,7 @@
         stat = os.stat(filename)
         self._atime = stat.st_atime
         self._mtime = stat.st_mtime
- return libexiv2python._Image(filename)
+ return libexiv2python._Image(str(filename))

     @classmethod
     def from_buffer(cls, buffer):

And now for the good news: Ubuntu Oneiric ships libboost-python 1.46.1, which is compiled against python 3.2:

$ dpkg -c libboost-python1.46.1_1.46.1-5ubuntu2_amd64.deb
[…]
-rw-r--r-- root/root 311136 2011-06-03 21:30 ./usr/lib/libboost_python-py26.so.1.46.1
-rw-r--r-- root/root 311136 2011-06-03 21:30 ./usr/lib/libboost_python-py27.so.1.46.1
-rw-r--r-- root/root 307040 2011-06-03 21:30 ./usr/lib/libboost_python-py32.so.1.46.1
[…]

This means I’ll soon be able to work seriously on porting whatever needs to be ported to make pyexiv2 work with py3k.

In the meantime, Vincent’s help is invaluable. Thanks Vincent!

> It looks like the 'filename' argument passed to libexiv2python._Image(…) is
> of type 'bytes' in python3, whereas I would expect it to be of type 'str'.

This may very well be https://svn.boost.org/trac/boost/ticket/4609.

Hi,

I've strange effects, the following code is working nice with python3

# -*- coding: utf-8 -*-

import os
import sys

paths = ['/home/vincent/image.jpg', '/home/vincent/àéèîö.jpg']

for path in paths:
    print('Path: {0}, Type: {1}'.format(path, type(path)))
    if not os.path.exists(path) or not os.path.isfile(path):
        print ('File not found: {0}'.format(path))
    path = path.encode(sys.getfilesystemencoding())
    print('Type: {0}'.format(type(path)))
    stat = os.stat(path)
    print(stat.st_atime)
    print(stat.st_mtime)

So, I've seen this morning the problem in 'src/pyexiv2/metadata.py' with sys.getfilesystemencoding() and I've commented this line.

Now if I read the metadatas of the image '/home/vincent/àéèîö.jpg' I get this error:

<class 'str'>
Traceback (most recent call last):
  File "/home/vincent/magnifier_3/miniView.py", line 69, in get_image_file
    self.open_image(img)
  File "/home/vincent/magnifier_3/miniView.py", line 78, in open_image
    self.read_datas(path)
  File "/home/vincent/magnifier_3/miniView.py", line 98, in read_datas
    metas.read()
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 111, in read
    self.__image = self._instantiate_image(self.filename)
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 75, in _instantiate_image
    if not os.path.exists(filename) or not os.path.isfile(filename):
  File "/usr/lib/python3.2/genericpath.py", line 18, in exists
    os.stat(path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-29: ordinal not in range(128)

The first line '<class 'str'>' is a print I've inserted in 'def _instantiate_image()'
Here the error appears with 'os.path.exist(filename)' but is the same with 'os.path.isfile(filename)' or 'os.path.exist(str(filename))' and 'os.path.isfile(str(filename))'

I just tested your patch:

IOError: /home/vincent/\xe9\xe0\xe8\xf4.JPG: Failed to open the data source: (errno = 2)

In my example codes theses functions don't return any error, class 'str' == unicode.

Really strange

It seems it is a problem with Python.

Example

# -*- coding: utf-8 -*-

import os
import sys
import platform

print('\nPython version: ', sys.version.split()[0])
print(platform.platform())

paths = ['/home/vincent/image.jpg', '/home/vincent/àéèîö.jpg']

for path in paths:
    print('\nPath: {0}, Type: {1}'.format(path, type(path)))
    if not os.path.exists(path) or not os.path.isfile(path):
        print('File not found: {0}'.format(path))
    else:
        print('File exists')

My pc ubuntu
--------------------------------------------------------------------------------------
vincent@tiemoko:~/Python/oqapy/devel$ python3 string_2.py

Python version: 3.1.2
Linux-2.6.32-33-generic-i686-with-Ubuntu-10.04-lucid

Path: /home/vincent/image.jpg, Type: <class 'str'>
File exists

Path: /home/vincent/àéèîö.jpg, Type: <class 'str'>
File exists
vincent@tiemoko:~/Python/oqapy/devel$
------------------------------------------------------------------------------------------------
- pc Archlinux:
-------------------------------------------------------------------------------------------------
<class 'str'>
Traceback (most recent call last):
  File "/home/vincent/magnifier_3/miniView.py", line 69, in get_image_file
    self.open_image(img)
  File "/home/vincent/magnifier_3/miniView.py", line 78, in open_image
    self.read_datas(path)
  File "/home/vincent/magnifier_3/miniView.py", line 98, in read_datas
    metas.read()
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 111, in read
    self.__image = self._instantiate_image(self.filename)
  File "/usr/lib/python3.2/site-packages/pyexiv2/metadata.py", line 75, in _instantiate_image
    if not os.path.exists(filename) or not os.path.isfile(filename):
  File "/usr/lib/python3.2/genericpath.py", line 18, in exists
    os.stat(path)
UnicodeEncodeError: 'ascii' codec can't encode characters in position 26-29: ordinal not in range(128)
--------------------------------------------------------------------------------------------

So, I'll post on python-list ...

Thanks for the investigation Vincent.

So there are issues with non-ascii characters in filenames when commenting out the code that handles them, which doesn’t surprise me much.

But does it work with a filename that doesn’t have non-ascii characters?

Also, it looks like the 'unicode' type doesn’t exist in python3 any longer: strings (of type 'str') are unicode by default. Non-unicode strings have the type 'bytes'. This means that the test in ImageMetadata.__init__(…) should become:

    if filename is not None and isinstance(filename, str):
        self.filename = filename.encode(sys.getfilesystemencoding())

And in ImageMetadata._instantiate_image(…), since we are bitten by https://svn.boost.org/trac/boost/ticket/4609, we probably want to do the following:

    return libexiv2python._Image(filename.decode('utf-8'))

Vincent, can you test if the above works for you?

Sorry, second log not complete

Archlinux
--------------------------------------------------------------------------------------------------------------------------------
[vincent@myhost ~]$ python string_2.py

Python version: 3.2.2
Linux-3.0-ARCH-x86_64-Pentium-R-_Dual-Core_CPU_T4500_@_2.30GHz-with-glibc2.2.5

Path: /home/vincent/image.jpg, Type: <class 'str'>
File exists
Traceback (most recent call last):
  File "string_2.py", line 13, in <module>
    print('\nPath: {0}, Type: {1}'.format(path, type(path)))
UnicodeEncodeError: 'ascii' codec can't encode characters in position 21-25: ordinal not in range(128)
[vincent@myhost ~]$
-----------------------------------------------------------------------------------------------------------------------------------

don't works

    filename = filename.decode('utf-8')
AttributeError: 'str' object has no attribute 'decode'

same on v3.1.2 and v3.2.2

It should be better to solve this question at the beggining of Metaadata.__init__()

self.filename = filename.magically_transform()

because all functions or methods reurns the same error

- sys.getfilesystemencoding()
- os.path.exists()
- os.path.isfile()
- os.stat()
- etc

... be zen

Now, I am certain that it is a bug python.

Reported here:

    http://bugs.python.org/issue12995

Wait and see.

Not a python bug but the config of archlinux himself.

Now, I reverse all changes in /usr/lib/python3.2/site-packages/pyexiv2/metadata.py
and my test is running

[vincent@myhost ~]$ python main.py

open file: /home/vincent/magnifier_3/Jeanne-48.jpg

Date: <Exif.Image.DateTime [Ascii] = 2009:08:24 13:17:18>
ISO: <Exif.Photo.ISOSpeedRatings [Short] = 66>
Fnumber: <Exif.Photo.ExposureTime [Rational] = 10/3337>
Orientation: <Exif.Image.Orientation [Short] = 1>

open file: /home/vincent/magnifier_3/àéèîö.JPG

Date: <Exif.Image.DateTime [Ascii] = 2009:11:28 14:34:58>
ISO: <Exif.Photo.ISOSpeedRatings [Short] = 800>
Fnumber: <Exif.Photo.ExposureTime [Rational] = 10/370>
Orientation: <Exif.Image.Orientation [Short] = 1>

I've also tried yours tests (test/TestRunner.py) but these files needs to more changes for python3

Hi,

Now, I can read and write the exif, iptc and xmp with pyexiv2 compiled for Python3
under Ubuntu-12.04

Preview (thumbnail) writting not tested yet.

Diff file attached