Translate function raises error when dealing with extended characters

[mfogels]

Calling the translate '_()' function to translate a string whose return-value contains non-ascii charactes raises a UnicodeDecodeError?.

Setting Myghty's 'encoding_errors' option to 'htmlentityreplace' does not appear to have any effect on the error; perhaps the myghty handlers don't get a chance to catch the error?

Unfortunately, the error traceback isn't very helpful:

Traceback (most recent call last):
  File "/home/mars/work/xxx/trunk/XXX/xxx/tests/functional/te
st_dealer.py", line 62, in test_french_login
    res.follow() # we should be redirected to the index page
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/fix
ture.py", line 506, in follow
    return self.test_app.get(location, **kw)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/fix
ture.py", line 195, in get
    return self.do_request(req, status=status)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/fix
ture.py", line 322, in do_request
    raise_on_wsgi_error=raise_on_wsgi_error,
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/wsg
ilib.py", line 347, in raw_interactive
    app_iter = application(basic_environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/reg
istry.py", line 309, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/rec
ursive.py", line 81, in __call__
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/err
ordocument.py", line 181, in __call__
    app_iter = self.application(environ, change_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/cas
cade.py", line 92, in __call__
    return self.apps[-1](environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/eva
lexception/middleware.py", line 175, in __call__
    return self.respond(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/err
or.py", line 210, in respond
    return self.application(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/htt
pexceptions.py", line 623, in __call__
    self.send_http_response, catch=HTTPException)
  File "/usr/local/lib/python2.4/site-packages/Paste-0.9.8.1-py2.4.egg/paste/wsg
ilib.py", line 232, in catch_errors_app
    app_iter = application(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/PasteDeploy-0.9.6-py2.4.egg/paste
/deploy/config.py", line 160, in __call__
    app_iter = self.application(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/wsg
iapp.py", line 315, in __call__
    return self.app(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Beaker-0.6.1-py2.4.egg/beaker/cac
he.py", line 120, in __call__
    return self.app(environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Beaker-0.6.1-py2.4.egg/beaker/ses
sion.py", line 100, in __call__
    response = self.wrap_app(environ, session_start_response)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/wsg
iapp.py", line 91, in __call__
    response = self.dispatch(controller, environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/wsg
iapp.py", line 237, in dispatch
    return controller(environ, start_response)
  File "/home/mars/work/stihl/dpms/trunk/StihlDPMS/stihldpms/lib/base.py", line 
18, in __call__
    response = WSGIController.__call__(self, environ, start_response)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/con
trollers.py", line 134, in __call__
    response = self._dispatch_call()
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/con
trollers.py", line 79, in _inspect_call
    return func(*args)
  File "<string>", line 2, in index
  File "/home/mars/work/stihl/dpms/trunk/StihlDPMS/stihldpms/lib/base.py", line 
64, in protected
    return func(self, *args, **kwds)
  File "/home/mars/work/stihl/dpms/trunk/StihlDPMS/stihldpms/controllers/dealer.
py", line 17, in index
    return render_response('/viewbossinfo.myt')
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/tem
plating.py", line 288, in render_response
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/templating.py", line 273, in render
    namespace=kargs, **cache_args)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/templating.py", line 196, in render
    return engine_config['engine'].render(namespace, template=full_path, **options)
  File "/usr/local/lib/python2.4/site-packages/Pylons-0.9.2-py2.4.egg/pylons/templating.py", line 234, in render
    global_args=global_args, out_buffer=buf,
  File "/home/mars/lib/python/Myghty-1.1-py2.4.egg/myghty/interp.py", line 156, in execute
    return self.make_request(component = component, **params).execute()
  File "/home/mars/lib/python/Myghty-1.1-py2.4.egg/myghty/request.py", line 275, in execute
    raise error
Error: Error(UnicodeDecodeError): 'ascii' codec can't decode byte 0xc3 in position 1: ordinal not in range(128) at /home/mars/lib/python/Myghty-1.1-py2.4.egg/myghty/requestbuffer.py line 367

To recreate the error:

Use the attached .po file, which contains utf-8 french-language characters in the 'msgstr' field. Call the translate function from a unicode template that will return the unicode string. Watch the flames.

Speculation: The translate function, '_()', in pylons.util uses the GNUgettext object's gettext() method, which returns an ascii string. The function should probably be calling the object's ugettext() method instead, which returns a unicode string.

[mfogels]

UTF-8 .po file containing extended ascii characters

[pjenvey]

(In [1558]) _ now uses gettext (returns unicode strings) fixes #126

[bbangert]

(In [1561]) Added ungettext to helpers for use with pluralized i18n, and added the ungettext and translator objects to be globals for templates. Refs #126.