Ticket #11725 (confirmed Bug)
Non-ascii character in title of Plone Service Provider causes UnicodeEncodeError
Reported by: | frisi | Owned by: | plone-website@… |
---|---|---|---|
Priority: | minor | Milestone: | |
Component: | Website | Version: | |
Keywords: | Cc: |
Description
when viewing the sites bulit with plone and following the link to the provider i get an error for our company: http://plone.org/support/providers/webmeisterei
i tried several other companies - all worked well.
thanks for you help!
Change History
comment:2 follow-up: ↓ 4 Changed 5 years ago by frisi
hey ross
maybe it's just because the defaultencoding is set differently on your machine and on the server.
try to compare the output of
import sys sys.getdefaultencoding()
lines like this can be dangerous as soon as they will be casted to unicode later on.
returning safe_unicode in such cases tries to guess the correct encoding when encoding a string to a unicodestring.
i guess the problem raises from rendering the image in provider_view.pt#L45
the title and the navigation should be umlaut-aware and a tal:content for the homepagelink should be pretty save too...
comment:3 Changed 5 years ago by frisi
btw: I replaced the umlauts with ue so the page does not raise an error in the meantime.
comment:4 in reply to: ↑ 2 ; follow-up: ↓ 5 Changed 5 years ago by rossp
Replying to frisi:
maybe it's just because the defaultencoding is set differently on your machine and on the server.
try to compare the output of
import sys sys.getdefaultencoding()
Yup, locally it returns 'utf-8' and on plone.org it returns 'ascii', both of those under "bin/instance1 debug". But when I simply fire up the python used in bin/instance1 I get 'ascii' on *both*. What's that about? How come it's different under "bin/instance1 debug" and how can I change my local to match the server to reproduce this?
lines like this can be dangerous as soon as they will be casted to unicode later on.
returning safe_unicode in such cases tries to guess the correct encoding when encoding a string to a unicodestring.
That's only used in CSV export of multiple providers, not in the single provider view where you're seeing the error.
i guess the problem raises from rendering the image in provider_view.pt#L45
But that line is only rendering the logo image and, as I said earlier, I confirmed that changing the title fixes the issue. Are you seeing the title involved in that line somehow?
comment:5 in reply to: ↑ 4 ; follow-up: ↓ 6 Changed 5 years ago by frisi
hi ross
Replying to rossp:
Yup, locally it returns 'utf-8' and on plone.org it returns 'ascii', both of those under "bin/instance1 debug". But when I simply fire up the python used in bin/instance1 I get 'ascii' on *both*. What's that about? How come it's different under "bin/instance1 debug" and how can I change my local to match the server to reproduce this?
a bit off-topic: you can set the encoding for your python interpreters on several ways. years ago i used a site.py file in the site-packages directory of my python installation. site.py: import sys; sys.setdefaultencoding('utf-8')
i can imagine you can set this in your zope.conf as well. this would explain why bin/instance behaves differently than the python interpreter the instance is using.
lines like this can be dangerous as soon as they will be casted to unicode later on.
returning safe_unicode in such cases tries to guess the correct encoding when encoding a string to a unicodestring.
That's only used in CSV export of multiple providers, not in the single provider view where you're seeing the error.
yes - i've seen that it's csv related. since it was the first file i looked for possible causes i still mentioned the line as potentially harmfull. i'm quite sure the csv export won't work as soon we change the ue to ü in our title again.
i guess the problem raises from rendering the image in provider_view.pt#L45
But that line is only rendering the logo image and, as I said earlier, I confirmed that changing the title fixes the issue. Are you seeing the title involved in that line somehow?
sure: the alt attribute of the image will be the provider's title.
could you try to customize the template so it does not include the image, change back the umlauts and see if it works for you?
thanks a lot!
comment:6 in reply to: ↑ 5 ; follow-up: ↓ 7 Changed 5 years ago by rossp
Replying to frisi:
hi ross
Replying to rossp:
Yup, locally it returns 'utf-8' and on plone.org it returns 'ascii', both of those under "bin/instance1 debug". But when I simply fire up the python used in bin/instance1 I get 'ascii' on *both*. What's that about? How come it's different under "bin/instance1 debug" and how can I change my local to match the server to reproduce this?
a bit off-topic: you can set the encoding for your python interpreters on several ways. years ago i used a site.py file in the site-packages directory of my python installation. {{{site.py: import sys; sys.setdefaultencoding('utf-8')}}}
Nope, has no effect. Some pdb stepping shows me that the default encoding is being changed while loading the ZCML.
i can imagine you can set this in your zope.conf as well. this would explain why bin/instance behaves differently than the python interpreter the instance is using.
Specifics, please?
lines like this can be dangerous as soon as they will be casted to unicode later on.
returning safe_unicode in such cases tries to guess the correct encoding when encoding a string to a unicodestring.
That's only used in CSV export of multiple providers, not in the single provider view where you're seeing the error.
yes - i've seen that it's csv related. since it was the first file i looked for possible causes i still mentioned the line as potentially harmfull. i'm quite sure the csv export won't work as soon we change the ue to ü in our title again.
I don't know what the CSV code is about at all. IMO, if no one knows what the use case is for this custom CSV code, it should be ripped out. Either way, that's not what this ticket is about.
i guess the problem raises from rendering the image in [ http://dev.plone.org/collective/browser/Products.PloneServicesCenter/trunk/Products/PloneServicesCenter/skins/plone_services_center/provider_view.pt#L45 provider_view.pt#L45]
But that line is only rendering the logo image and, as I said earlier, I confirmed that changing the title fixes the issue. Are you seeing the title involved in that line somehow?
sure: the alt attribute of the image will be the provider's title.
could you try to customize the template so it does not include the image, change back the umlauts and see if it works for you?
I need to be able to reproduce the problem in tests, hence the need to figure out what is changing the default encoding during loading ZCML.
comment:7 in reply to: ↑ 6 ; follow-up: ↓ 8 Changed 5 years ago by frisi
- Priority changed from major to minor
Replying to rossp:
a bit off-topic: you can set the encoding for your python interpreters on several ways. years ago i used a site.py file in the site-packages directory of my python installation. {{{site.py: import sys; sys.setdefaultencoding('utf-8')}}}
Nope, has no effect. Some pdb stepping shows me that the default encoding is being changed while loading the ZCML.
you should be able to change the default encoding for your python installation using site.py as mentioned above. sure you're patching the correct python installation. http://blog.ianbicking.org/illusive-setdefaultencoding.html might be helpful
maybe the zpublisher encoding setting changes this later.
i can imagine you can set this in your zope.conf as well. this would explain why bin/instance behaves differently than the python interpreter the instance is using.
Specifics, please?
i thought about default-zpublisher-encoding ( http://pypi.python.org/pypi/plone.recipe.zope2instance/) when mentioning that. but it's not very likely to change sys.getdefaultencoding
i guess the problem raises from rendering the image in [ http://dev.plone.org/collective/browser/Products.PloneServicesCenter/trunk/Products/PloneServicesCenter/skins/plone_services_center/provider_view.pt#L45 provider_view.pt#L45]
But that line is only rendering the logo image and, as I said earlier, I confirmed that changing the title fixes the issue. Are you seeing the title involved in that line somehow?
sure: the alt attribute of the image will be the provider's title.
could you try to customize the template so it does not include the image, change back the umlauts and see if it works for you?
I need to be able to reproduce the problem in tests, hence the need to figure out what is changing the default encoding during loading ZCML.
i hope any of my hints will help you to be able to reproduce the problem on your machine. otherwhise i'm running out of ideas...
a quick and dirty approach would be to customize the view on plone.org, remove the img tag temporarily to check wheter its the root of all evil here.
for me, removing the umlauts in the title more or less fixes the issue: our company details can be accessed in the provider section.
comment:8 in reply to: ↑ 7 Changed 5 years ago by rossp
Replying to frisi:
Replying to rossp:
a bit off-topic: you can set the encoding for your python interpreters on several ways. years ago i used a site.py file in the site-packages directory of my python installation. {{{site.py: import sys; sys.setdefaultencoding('utf-8')}}}
Nope, has no effect. Some pdb stepping shows me that the default encoding is being changed while loading the ZCML.
you should be able to change the default encoding for your python installation using site.py as mentioned above.
I fired up the same exact python as being used by bin/instance, did "import site" and then looked at the path for the module to find the one to modify.
But this suggestion misses the point. That change is happening *after* python fires up, when I just fire python up, it's the same as on plone.org, see the original comment. As I said, I confirmed that it's happening while loading ZCML *by stepping through in pdb*.
sure you're patching the correct python installation.
could you try to customize the template so it does not include the image, change back the umlauts and see if it works for you?
I need to be able to reproduce the problem in tests, hence the need to figure out what is changing the default encoding during loading ZCML.
i hope any of my hints will help you to be able to reproduce the problem on your machine. otherwhise i'm running out of ideas...
a quick and dirty approach would be to customize the view on plone.org, remove the img tag temporarily to check wheter its the root of all evil here.
I need to be able to reproduce the problem in tests, hence the need to figure out what is changing the default encoding during loading ZCML.
If I'm going to fix it, this is how I will fix it. If I can't get help figuring out how to control the default encoding in the test fixture, then someone else will have to fix this.
comment:9 Changed 4 years ago by rossp
Others are being bitten by this bug, see #12309. Can someone please weigh in on what might be changing the encoding during ZCML loading?
comment:10 Changed 4 years ago by eleddy
- Status changed from new to closed
- Version set to 4.1
- Resolution set to fixed
- severity set to Untriaged
Can't reproduce this anymore. Please reopen if you still see it
comment:11 Changed 4 years ago by frisi
- Status changed from closed to reopened
- Resolution fixed deleted
this one is still valid: if you change the title of a company/provider so it contains non-ascii characters (in our case german umlauts in "Büro") you'll get a site error
comment:12 Changed 4 years ago by davisagli
- Status changed from reopened to closed
- Resolution set to duplicate
Closing as a duplicate in favor of #11613 which has a link that still exhibits the problem.
comment:13 Changed 3 years ago by kleist
- Status changed from closed to reopened
- Resolution duplicate deleted
comment:14 Changed 3 years ago by kleist
- Status changed from reopened to confirmed
- Version 4.1 deleted
- Summary changed from http://plone.org/support/providers/webmeisterei leads to an error to Non-ascii character in title of Plone Service Provider causes UnicodeEncodeError
Reopening, and closed #11613 which contains less technical information.
First I verified that if I replaced the umlauts in the provider's title with "u"s, the error goes away and returns when I restore the umlauts so this is definitely a unicode issue.
I spent some time trying to reproduce this in tests but couldn't. Finally I fired up "bin/instance1 debug" on plone.org and I found that it was treating things much differently than my local environment.
So I'm guessing the issue is unicode+environment related. Unicode is not my strong point, can someone else help?