DreamHost "Announce List" feature & text encoding

I maintain a mailing list using DreamHost's "Announce List" feature. Today I sent out a bulletin, only to discover that once received, accented characters (e with acute accent, for instance) and some non-alphanumeric characters (bullet, for instance) were not displayed properly (getting éa in place of an é, etc.). Having issued dozens of bulletins in the past, often with accented & non-alphanumeric characters, without problem, this was a surprising thing to see.

Contacted DreamHost support to ask if anything at their end had recently changed, explaining the experience, and asking what users can do to sidestep the issue. A DreamHost rep explained:

"We actually just recently changed our encoding to UTF-8. In order to get characters to display properly, please make sure to specify the appropriate charset that you're using when submitting an announcement to your members."

But from what I can tell, no field or area for specifying charset exists in the Announce List area of the control panel. I wrote back pointing this out but have so far not received a response, and the issue is somewhat time-sensitive.

Anyone know what steps need to be taken to get these characters to show up correctly at the recipients' end?

Thanks in advance.

In reply to:

---

But from what I can tell, no field or area for specifying charset exists in the Announce List area of the control panel. I wrote back pointing this out but have so far not received a response, and the issue is somewhat time-sensitive.

Anyone know what steps need to be taken to get these characters to show up correctly at the recipients' end?

---

Apparently not everybody in Support is on the same page. The option to specify a charset has been removed. In the past you could specify the charset in the Web Panel; you could also specify the charset when posting a message using the DreamHost API.

Since the DreamHost Web Panel itself is being served as UTF-8, you should only have to make sure that if you are pasting text, the text was created in an editor as UTF-8.

For example, I use Notepad++ on Windows XP and it has a "Format > Convert to UTF-8 without BOM" function. So I'm covered. And there are ways to convert text in many programming langauges as well.


Customer since 2000 openvein.org

Thanks for the response. Taking your advice, I conducted some tests today (using a closed Announce list that sends only to a handful of my own personal e-mail addresses), with inconsistent results.

Using WordPad (on a Windows XP Professional PC), I created the following text content, saved the file selecting the "Unicode Text Document" option, closed it, reopened it (to verify that everything looked good so far), and then cut & pasted the content into the "Your message" field of the Announce List area of the DH control panel.

Experiment: With Windows XP used Thunderbird to send 'Unicode (UTF-8)' message with characters copy and pasted from Character map. Message was sent with header Content-Type: text/plain; charset=UTF-8

Result: DreamHost-provided Squirrelmail serves web page as ISO-8859-1 and converted the characters to HTML entities: • bullet É capital E acute accent é lower-case e acute accent, and the correct glyphs appear when using Bitstream Vera Serif font. Chose 'Download this as a file" and Notepad confirmed 'ANSI as UTF-8' showing the correct glyphs. Thunderbird also shows message correctly both in sender 'Sent' folder and recipient 'Inbox'.

Experiment: Posted Announcement as plain text.
Result: DreamHost-provided Squirrelmail serves web page as ISO-8859-1 and DID NOT CONVERT the characters to HTML entities, multiple characters and multiple glyphs appeared. Chose 'Download this as a file" and Notepad confirmed 'ANSI as UTF-8' showing the correct glyphs. Thunderbird also shows message correctly both in sender 'Sent' folder and recipient 'Inbox'.

Experiment: With Windows XP used Thunderbird to send 'Unicode (UTF-8)' message with characters copy and pasted from Character map. Message was sent with header Content-Type: text/plain; charset=UTF-8

Result: DreamHost-provided Squirrelmail serves web page as ISO-8859-1 and converted the characters to HTML entities: • bullet É capital E acute accent é lower-case e acute accent, and the correct glyphs appear when using Bitstream Vera Serif font. Chose 'Download this as a file" and Notepad confirmed 'ANSI as UTF-8' showing the correct glyphs. This applies to both the text/plain part and the text/html parts. Expected behavior in Thunderbird as well.

Edit: Overlooked a message header.

Thunderbird sends the MIME-Version header even if the message is plain text. However DreamHost Announcement List system does not send MIME-Version for text messages, though it still indicates the charset is UTF-8.

Yeah, that's a bug, when sending UTF-8 the message needs to be MIME and have the MIME-Version header... reply to Support with a link to this post.














Customer since 2000 openvein.org Edited by Atropos7 on 09/25/09 12:15 PM (server time).

Brilliant! You're hired. Thanks for testing these scenarios out so systematically. It's a reassuring sanity check. Plus it means that the bulletin I sent out yesterday that I thought showed up in Klingon to all recipients actually went out OK, with most recipients presumably NOT experiencing the character-salad I saw in SquirrelMail.

Thanks again -- much appreciated.

Correction, forgot, wrong copy/pasta .... 3rd experiment was

Experiment: Post announcement as HTML.

Result: DreamHost-provided Squirrelmail serves web page as ISO-8859-1 and converted the characters to HTML entities: • bullet É capital E acute accent é lower-case e acute accent, and the correct glyphs appear when using Bitstream Vera Serif font. Chose 'Download this as a file" and Notepad confirmed 'ANSI as UTF-8' showing the correct glyphs. This applies to both the text/plain part and the text/html parts. Expected behavior in observed Thunderbird as well (shown as UTF-8 with no conversion)





Customer since 2000 openvein.org