tag:blogger.com,1999:blog-2030637598201068932024-03-13T07:23:54.022-04:00A Moment of ZenThe Ramblings of Jeffrey StedfastJeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.comBlogger156125tag:blogger.com,1999:blog-203063759820106893.post-91149393444063590392019-03-17T09:42:00.001-04:002019-03-17T09:43:28.111-04:00Interview with Hawk Newsome of Black Lives Matter<iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/wxqcIr56Q7k" width="480"></iframe>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-86606065798622221692017-11-07T15:18:00.001-05:002017-11-07T15:55:26.677-05:00EXCLUSIVE: Texas Massacre Hero, Stephen Willeford, Describes Stopping Gunman<iframe allowfullscreen="" frameborder="0" height="270" src="https://www.youtube.com/embed/B4HEchh0XD8" width="480"></iframe><br />
<br />
To donate to the Sutherland Springs Baptist Church to help them recover from this tragedy, check out this <a href="https://www.gofundme.com/SSFBC">GoFundMe campaign</a>.<br />
<br />
<br />Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-16808533165337611982017-04-11T11:57:00.000-04:002017-04-11T11:57:45.915-04:00Achievement Unlocked: MimeKit and MailKit in official Microsoft docs<div class="separator" style="clear: both; text-align: center;">
<a href="https://4.bp.blogspot.com/-Vovur2F233w/WOz8eI_YUdI/AAAAAAAABYI/YhU64GTdvMsYRviUntSG3IJkgw0iRjTHwCLcB/s1600/AchievementUnlockedMimeKitMailKit.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" height="272" src="https://4.bp.blogspot.com/-Vovur2F233w/WOz8eI_YUdI/AAAAAAAABYI/YhU64GTdvMsYRviUntSG3IJkgw0iRjTHwCLcB/s640/AchievementUnlockedMimeKitMailKit.png" width="640" /></a></div>
Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-37776790342932853552017-04-09T19:46:00.001-04:002017-04-10T03:10:29.637-04:00GMime 2.99.0 released<p>After a long hiatus, I am pleased to announce the release of GMime 2.99.0!</p>
<p>See below for a list of new features and bug fixes.</p>
<br>
<h3>About GMime</h3>
<p>GMime is a C library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by <a href="https://github.com/jstedfast/gmime/blob/master/RFCs.md">numerous IETF specifications</a>.</p>
<p>GMime features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated GnuPG and S/MIME v3.2 support.</p>
<p>Built on top of GObject (the object system used by the <a href="https://www.gnome.org">GNOME desktop</a>), many developers should find its API design and memory management very familiar.</p>
<br>
<h3>Noteworthy changes in version 2.99.0</h3>
<ul>
<li>Overhauled the GnuPG support to use <a href="https://www.gnupg.org/related_software/gpgme/">GPGME</a> under the hood rather than a custom wrapper.</li>
<li>Added S/MIME support, also thanks to GPGME.</li>
<li>Added International Domain Name support via GNU's <a href="https://www.gnu.org/software/libidn/">libidn</a>.</li>
<li>Improved the GMimeMessage APIs for accessing the common address headers. They now <i>all</i> return an InternetAddressList.</li>
<li>g_mime_init() no longer takes any flag arguments and the g_mime_set_user_charsets() API has also been dropped. Instead, GMimeParserOptions and GMimeFormatOptions have taken the place of these APIs to allow customization of various parser and formatting options in a much cleaner way. To facilitate this, many parsing functions and formatting functions have changed to now take these options arguments.</li>
<li>InternetAddress now has a 'charset' property that can be set to override GMime's auto-detection of the best charset to use when encoding names.</li>
<li>GMimeHeaderIter has been dropped in favor of a much simpler index-based API on GMimeHeaderList.</li>
<li>GMimeHeaderList no longer caches the raw message/mime headers in a stream. Instead, each GMimeHeader now has its own cache. This means that changing the GMimeHeaderList or any of its GMimeHeaders no longer invalidates the entire cache.</li>
<li>GMimeParser has been fixed to preserve (munged or otherwise) From-lines that sometimes appear at the start of the content of message/rfc822 parts.</li>
<li>GMimeParser now also scans for encapsulated PGP blocks within MIME parts as it is parsing them and sets a flag on each GMimePart that contains one of these blocks.</li>
<li>GMimePart now has APIs for dealing with said encapsulated PGP blocks.</li>
</ul>
<p>Developers interested in migrating to the upcoming GMime 3.0 API (of which GMime 2.99.0 is a preview) should take a look at the <a href="https://github.com/jstedfast/gmime/blob/master/PORTING">PORTING</a> document included with the source code as it contains a fairly comprehensive list of the API changes that they will need to be aware of.</p>
<br>
<h3>Getting the Source Code</h3>
<p>You can download official public release tarballs of GMime at <a href="https://download.gnome.org/sources/gmime/">https://download.gnome.org/sources/gmime/</a> or <a href="ftp://ftp.gnome.org/pub/GNOME/sources/gmime/">ftp://ftp.gnome.org/pub/GNOME/sources/gmime/</a>.</p>
<p>If you would like to contribute to the GMime project, it is recommended that you grab the source code from the official GitHub repository at <a href="https://github.com/jstedfast/gmime">https://github.com/jstedfast/gmime</a>. Cloning this repository can be done using the following command:</p>
<code>git clone https://github.com/jstedfast/gmime.git</code>
<br>
<br>
<h3>Documentation</h3>
<p>API reference documentation can be found at <a href="https://developer.gnome.org/gmime/2.99/">https://developer.gnome.org/gmime/2.99/</a>.</p>
<p>Documentation for getting started can be found in the <a href="https://github.com/jstedfast/gmime">README.md</a>.</p>
Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-91760057022232706122017-04-09T19:07:00.001-04:002017-04-09T19:07:44.473-04:00MailKit 1.14 released<p>I am pleased to announce the release of MailKit 1.14!</p>
<p>See below for a list of new features and bug fixes.</p>
<br>
<h3>About MailKit</h3>
<p>MailKit is a C# library which is built on top of <a href="https://github.com/jstedfast/MimeKit">MimeKit</a> and is intended to be used for interfacing with IMAP, POP3 and SMTP servers.</p>
<p>MailKit features incredibly robust IMAP, POP3 and SMTP clients with network APIs that are all capable of being canceled. API's that might transfer significant amounts of data between the client and server also include the ability to report progress. Asynchronous API's are also available.</p>
<p>Built on top of .NET, MailKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.</p>
<br>
<h3>Noteworthy changes in version 1.14</h3>
<ul>
<li>Improved IMAP's BODYSTRUCTURE parser to sanitize the Content-Disposition values. (issue <a href="https://github.com/jstedfast/MailKit/issues/486">#486</a>)</li>
<li>Improved robustness of IMAP's BODYSTRUCTURE parser in cases where qstring tokens have unescaped quotes. (issue <a href="https://github.com/jstedfast/MailKit/issues/485">#485</a>)</li>
<li>Fixed IMAP to properly handle NIL as a folder name in LIST, LSUB and STATUS responses. (issue <a href="https://github.com/jstedfast/MailKit/issues/482">#482</a>)</li>
<li>Added <a href="http://www.mimekit.net/docs/html/Overload_MailKit_Net_Imap_ImapFolder_GetHeaders.htm">ImapFolder.GetHeaders()</a> to allow developers to download the entire set of message headers.</li>
<li>Added SMTP support for International Domain Names in email addresses used in the MAIL FROM and RCPT TO commands.</li>
<li>Modified <a href="http://www.mimekit.net/docs/html/T_MailKit_Net_Smtp_SmtpClient.htm">SmtpClient</a> to no longer throw a NotSupportedException when trying to send messages to a recipient with a unicode local-part in the email address when the SMTP server does not support the SMTPUTF8 extension. Instead, the local-part is passed through as UTF-8, leaving it up to the server to reject either the command or the message. This seems to provide the best interoperability.</li>
</ul>
<br>
<h3>Installing via NuGet</h3>
<p>The easiest way to install MailKit is via <a href="https://www.nuget.org/packages/MailKit/">NuGet</a>.</p>
<p>In Visual Studio's <a href="http://docs.nuget.org/docs/start-here/using-the-package-manager-console">Package Manager Console</a>, simply enter the following command:</p>
<code>Install-Package MailKit</code>
<br>
<br>
<h3>Getting the Source Code</h3>
<p>First, you'll need to clone MailKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:</p>
<code>git clone --recursive https://github.com/jstedfast/MailKit.git</code>
<br>
<br>
<h3>Documentation</h3>
<p>API documentation can be found at <a href="http://mimekit.net/docs">http://mimekit.net/docs</a>.</p>
<p>A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.</p>
Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-23015385766904492042017-04-09T18:37:00.000-04:002017-04-09T18:37:25.002-04:00MimeKit 1.14 released<p>I am pleased to announce the release of MimeKit 1.14!</p>
<p>See below for a list of new features and bug fixes.</p>
<br>
<h3>About MimeKit</h3>
<p>MimeKit is a C# library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by <a href="https://github.com/jstedfast/MimeKit/blob/master/RFCs.md">numerous IETF specifications</a>.</p>
<p>MimeKit features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated DKIM-Signature, S/MIME v3.2, OpenPGP and MS-TNEF support.</p>
<p>Built on top of .NET, MimeKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.</p>
<br>
<h3>Noteworthy changes in version 1.14</h3>
<ul>
<li>Added International Domain Name support for email addresses.</li>
<li>Added a work-around for mailers that didn't provide a disposition value in a Content-Disposition header.</li>
<li>Added a work-around for mailers that quote the disposition value in a Content-Disposition header.</li>
<li>Added automatic key retrieval functionality for the GnuPG crypto context.</li>
<li>Added a virtual <a href="http://www.mimekit.net/docs/html/P_MimeKit_Cryptography_DkimSigner_DigestSigner.htm">DigestSigner</a> property to <a href="http://www.mimekit.net/docs/html/T_MimeKit_Cryptography_DkimSigner.htm">DkimSigner</a> so that consumers can hook into services such as Azure. (issue <a href="https://github.com/jstedfast/MimeKit/pull/296">#296</a>)</li>
<li>Fixed a bug in the <a href="http://www.mimekit.net/docs/html/M_MimeKit_IO_Filters_MimeFilterBase_SaveRemainingInput.htm">MimeFilterBase.SaveRemainingInput()</a> logic.</li>
<li>Preserve munged From-lines at the start of message/rfc822 parts.</li>
<li>Map code page 50220 to iso-2022-jp.</li>
<li>Format Reply-To and Sender headers as address headers when using <a href="http://www.mimekit.net/docs/html/Overload_MimeKit_Header_SetValue.htm">Header.SetValue()</a>.</li>
<li>Fixed <a href="http://www.mimekit.net/docs/html/M_MimeKit_MimeMessage_CreateFromMailMessage.htm">MimeMessage.CreateFromMailMessage()</a> to set the MIME-Version header. (issue <a href="https://github.com/jstedfast/MimeKit/issues/290">#290</a>)</li>
</ul>
<br>
<h3>Installing via NuGet</h3>
<p>The easiest way to install MimeKit is via <a href="https://www.nuget.org/packages/MimeKit/">NuGet</a>.</p>
<p>In Visual Studio's <a href="http://docs.nuget.org/docs/start-here/using-the-package-manager-console">Package Manager Console</a>, simply enter the following command:</p>
<code>Install-Package MimeKit</code>
<br>
<br>
<h3>Getting the Source Code</h3>
<p>First, you'll need to clone MimeKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:</p>
<code>git clone --recursive https://github.com/jstedfast/MimeKit.git</code>
<br>
<br>
<h3>Documentation</h3>
<p>API documentation can be found at <a href="http://mimekit.net/docs">http://mimekit.net/docs</a>.</p>
<p>A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.</p>
Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-65038790189156906432015-03-19T21:37:00.003-04:002015-03-20T07:29:11.675-04:00Code Review: Microsoft's System.Net.Mail ImplementationFor those reading my blog for the first time and don't know who I am, allow myself to introduce... myself.<br />
<div><br />
</div><div>I'm a self-proclaimed expert on the topic of email, specifically MIME, IMAP, SMTP, and POP3. I don't proclaim myself to be an expert on much, but email is something that maybe 1 or 2 dozen people in the world could probably get away with saying they know more than I do and actually back it up. I've got a lot of experience writing email software over the past 15 years and rarely do I come across mail software that does things better than I've done them. I'm also a critic of mail software design and implementation.</div><div><br />
</div><div>My latest endeavors in the email space are <a href="https://github.com/jstedfast/MimeKit">MimeKit</a> and <a href="https://github.com/jstedfast/MailKit">MailKit</a>, both of which are open source and available on GitHub for your perusal should you doubt my expertise.</div><div><br />
</div><div>My point is: I think my review carries some weight, or I wouldn't be writing this.</div><div><br />
</div><div>Is that egotistical of me? Maybe a little.</div><div><br />
</div><div>I was actually just fixing a <a href="https://github.com/jstedfast/MimeKit/issues/115">bug in MimeKit</a> earlier and when I went to go examine Mono's System.Net.Mail.MailMessage implementation in order to figure out what the problem was with my System.Net.Mail.MailMessage to MimeKit.MimeMessage conversion, I thought, "hey, wait a minute... didn't Microsoft just recently release their BCL source code?" So I ended up taking a look and pretty quickly confirmed my suspicions and was able to fix the bug.</div><div><br />
</div><div>When I begin looking at the source code for another mail library, I can't help but critique what I find.</div><div><br />
</div><h3>MailAddress and MailAddressCollection</h3><div><br />
</div><div>Parsing email addresses is probably the hardest thing to get right. It's what I would say makes or breaks a library (literally). To a casual onlooker, parsing email addresses probably seems like a trivial problem. "Just String.Split() on comma and then look for those angle bracket thingies and you're done, right?" Oh God, oh God, make the hurting stop. I need to stop here before I go into a long rant about this...</div><div><br />
</div><div>Okay, I'm back. Blood pressure has subsided.</div><div><br />
</div><div>Looking at <a href="https://github.com/jstedfast/referencesource/blob/master/System/net/System/Net/mail/MailAddressParser.cs">MailAddressParser.cs</a> (the internal parser used by <a href="https://github.com/jstedfast/referencesource/blob/master/System/net/System/Net/mail/MailAddressCollection.cs">MailAddressCollection</a>), I'm actually pleasantly surprised. It actually looks pretty decent and I can tell that a lot of thought and care went into it. They actually use a tokenizer approach. Interestingly, they parse the string in reverse which is a pretty good idea, I must say. This approach probably helps simplify the parser logic a bit because parsing forward makes it difficult to know what the tokens belong to (is it the name token? or is it the local-part of an addr-spec? hard to know until I consume a few more tokens...).</div><div><br />
</div><div>For example, consider the following BNF grammar:<br />
<br />
</div><div><pre>address = mailbox / group
mailbox = name-addr / addr-spec
name-addr = [display-name] angle-addr
angle-addr = [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
group = display-name ":" [mailbox-list / CFWS] ";"
[CFWS]
display-name = phrase
word = atom / quoted-string
phrase = 1*word / obs-phrase
addr-spec = local-part "@" domain
local-part = dot-atom / quoted-string / obs-local-part
domain = dot-atom / domain-literal / obs-domain
obs-local-part = word *("." word)
</pre></div><div><br />
Now consider the following email address: "Joe Example" <joe@example.com><br />
<br />
The first token you read will be "Joe Example" and you might think that that token indicates that it is the display name, but it doesn't. All you know is that you've got a 'quoted-string' token. A 'quoted-string' can be part of a 'phrase' or it can be (a part of) the 'local-part' of the address itself. You must read at least 1 more token before you'll be able to figure out what it actually is ('obs-local-part' makes things slightly more difficult). In this case, you'll get a '<' which indicates the start of an 'angle-addr', allowing you to assume that the 'quoted-string' you just got is indeed the 'display-name'.<br />
<br />
If, however, you parse the address in reverse, things become a little simpler because you know immediately what to expect the next token to be a part of.<br />
<br />
That's pretty cool. Kudos to the Microsoft engineers for thinking up this strategy.<br />
<br />
Unfortunately, the parser does not handle the 'group' address type. I'll let this slide, however, partly because I'm still impressed by the approach the address parser took and also because I realize that System.Net.Mail is meant for creating and sending <i>new</i> messages, not parsing existing messages from the wild.<br />
<br />
Okay, so how well does it serialize <a href="https://github.com/jstedfast/referencesource/blob/master/System/net/System/Net/mail/MailAddress.cs">MailAddress</a>?<br />
<br />
Ugh. You know that face you make when you just see a guy get kicked in the nuts? Yea, that's the face I made when I saw <a href="https://github.com/jstedfast/referencesource/blob/master/System/net/System/Net/mail/MailAddress.cs#L227">line #227</a>:<br />
<br />
</div><pre>encodedAddress = String.Format(CultureInfo.InvariantCulture, "\"{0}\"", this.displayName);
</pre><div><br />
The problem with the above code (and I'll soon be submitting a bug report about this) is that the displayName string might have embedded double quotes in it. You can't just surround it with quotes and expect it to work. This is the same mistake all those programmers make that allow SQL-injection attacks.<br />
<br />
For an example of how this should be done, see MimeKit's <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/Utils/MimeUtils.cs#L454">MimeUtils.Quote()</a> method.<br />
<br />
I had such high hopes... at least this is a fairly simple bug to fix. I'll probably just offer them a patch.<br />
<br />
<h3>ContentType and ContentDisposition</h3><br />
Their parser is decent but it doesn't handle <a href="https://tools.ietf.org/html/rfc2231">rfc2231</a> encoded parameter values, so I'm not overly impressed. It'll get the job done for simple name="value" parameter syntax, though, and it will decode the values encoded with the <a href="https://tools.ietf.org/html/rfc2047">rfc2047</a> encoding scheme (which is not the right way to encode values, but it <i>is</i> common enough that any serious parser should handle it). The code is also pretty clean and uses a tokenizer approach, so that's a plus. I guess since this isn't really meant as a full-blown MIME parser, they can get away with this and not have it be a big deal. Fair enough.<br />
<br />
Serialization, unsurprisingly, leaves a lot to be desired. Parameter values are, as I expected, encoded using <a href="https://tools.ietf.org/html/rfc2047">rfc2047</a> syntax rather than the IETF standard <a href="https://tools.ietf.org/html/rfc2231">rfc2231</a> syntax. I suppose that you could argue that this is for compatibility, but really it's just perpetuating bad practices. It also means that it can't properly fold long parameter values because the encoded value just becomes one big long encoded-word token. Yuck.<br />
<br />
<h3>Base64</h3><br />
Amusingly, Microsoft does not use their Convert.FromBase64() decoder to decode base64 in their System.Net.Mail implementation. I point this out mostly because it is the single most common problem users have with every one of the Open Source .NET mail libraries out there (other than MimeKit, of course) because Convert.FromBase64() relies on the data not having any line breaks, white space, etc in the input stream.<br />
<br />
This should serve as a big hint to you guys writing your own .NET email libraries not to use Convert.FromBase64() ;-)<br />
<br />
They use unsafe pointers, just like I do in MimeKit, but I'm not sure how their performance compares to MimeKit's yet. They do use a state machine, though, so rock on.<br />
<br />
I approve this base64 encoder/decoder implementation.<br />
<br />
<h3>SmtpClient</h3><br />
One thing they do which is pretty cool is connection pooling. This is probably a pretty decent win for the types of things developers usually use System.Net.Mail's SmtpClient for (spam, anyone?).<br />
<br />
The SASL AUTH mechanisms that they seem to support are NTLM, GSSAPI, LOGIN and WDIGEST (which apparently is some sort of IIS-specific authentication mechanism that I had never heard of until now). For those that were curious which SASL mechanisms SmtpClient supported, well, now you know.<br />
<br />
The code is a bit hard to follow for someone not familiar with the codebase (not nearly as easy reading as the address or content-type parsers, I'm afraid), but it seems fairly well designed.<br />
<br />
It does not appear to support PIPELINING or BINARYMIME like MailKit does, though. So, yay! Win for MailKit ;-)<br />
<br />
They do support SMTPUTF8, so that's good.<br />
<br />
It seems that if you set client.EnableSsl to true, it will also try STARTTLS if it isn't able to connect on the SSL port. I wasn't sure if it did that or not before, so this was something I was personally interested in knowing.<br />
<br />
Hopefully my SmtpClient implementation review isn't too disappointing. I just don't know what to say about it, really. It's a pretty straight-forward send-command-wait-for-reply implementation and SMTP is pretty dead simple.<br />
<br />
<h3>Conclusion</h3><br />
Overall the bits I was interested in were better than I expected they'd be. The parsers were pretty good (although incomplete) and the serializers were "good enough" for normal use.<br />
<br />
Of course, it's not as good as MimeKit, but let's be honest, MimeKit sets the bar pretty high ;-)<br />
<br />
</div>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-88262590689289651542014-10-16T14:32:00.000-04:002014-10-16T14:32:36.801-04:00The Wait Is Over: MimeKit and MailKit Reach 1.0After about a year in the making for <a href="https://github.com/jstedfast/MimeKit">MimeKit</a> and nearly 8 months for <a href="https://github.com/jstedfast/MailKit">MailKit</a>, they've finally reached 1.0 status.<br />
<br />
I started really working on MimeKit about a year ago wanting to give the .NET community a top-notch MIME parser that could handle anything the real world could throw at it. I wanted it to run on any platform that can run .NET (including mobile) and do it with remarkable speed and grace. I wanted to make it such that re-serializing the message would be a byte-for-byte copy of the original so that no data would ever be lost. This was also very important for my last goal, which was to support S/MIME and PGP out of the box.<br />
<br />
All of these goals for MimeKit have been reached (partly thanks to the <a href="http://www.bouncycastle.org/csharp/">BouncyCastle</a> project for the crypto support).<br />
<br />
At the start of December last year, I began working on MailKit to aid in the adoption of MimeKit. It became clear that without a way to inter-operate with the various types of mail servers, .NET developers would be unlikely to adopt it.<br />
<br />
I started off implementing an <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Smtp/index.html">SmtpClient</a> with support for <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Security/index.html">SASL</a> authentication, STARTTLS, and PIPELINING support.<br />
<br />
Soon after, I began working on a <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Pop3/index.html">Pop3Client</a> that was designed such that I could use MimeKit to parse messages on the fly, directly from the socket, without needing to read the message data line-by-line looking for a <span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">".\r\n"</span></span> sequence, concatenating the lines into a massive memory buffer before I could start to parse the message. This fact, combined with the fact that MimeKit's message parser is orders of magnitude faster than any other .NET parser I could find, makes MailKit the fastest POP3 library the world has ever seen.<br />
<br />
After a month or so of avoiding the inevitable, I finally began working on an <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Imap/index.html">ImapClient</a> which took me roughly two weeks to produce the initial prototype (compared to a single weekend for each of the other protocols). After many months of implementing dozens of the more widely used IMAP4 extensions (including the GMail extensions) and tweaking the APIs (along with bug fixing) thanks to feedback from some of the early adopters, I believe that it is finally complete enough to call 1.0.<br />
<br />
In July, at the request of someone involved with a number of the <a href="http://www.ietf.org/">IETF</a> email-related specifications, I also implemented support for the new <a href="http://en.wikipedia.org/wiki/International_email">Internationalized Email</a> standards, making MimeKit and MailKit the first - <i>and only</i> - .NET email libraries to support these standards.<br />
<br />
If you want to do anything at all related to email in .NET, take a look at MimeKit and MailKit. I guarantee that you will not be disappointed.<br />
<br />Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-10305694095431553282014-04-28T16:59:00.000-04:002014-04-28T17:19:44.124-04:00The Future of Debugging in Xamarin Studio<p>There comes a time in ever man's life when he says to himself, "there has <i>got</i> to be a better way..."</p><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-2KQzgZT5PaI/U1641S4pspI/AAAAAAAAAzc/dUj9jRQFEvk/s1600/doc-brown-clock.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://2.bp.blogspot.com/-2KQzgZT5PaI/U1641S4pspI/AAAAAAAAAzc/dUj9jRQFEvk/s400/doc-brown-clock.jpg" /></a></div><h3>Set Next Statement</h3><p>Have you ever been stepping through a method or hit a breakpoint and discovered that variables did not have the expected values? Don't you wish you could go back in time and start stepping through that method from an earlier point to see how things went so horribly wrong? Of course you do.</p><p>Last week, with the help of Zoltan Varga (who added the necessary runtime support), I implemented support in Xamarin Studio to set the next statement to execute when you resume execution of your program in the debugger. You can set the next statement to be any statement in the current method; any statement at all. This essentially allows you to jump back in time or completely step over execution of statements after the current position.</p><div class="separator" style="clear: both; text-align: left;"><a href="http://2.bp.blogspot.com/-r2UGdjV4NC8/U1641rJkyNI/AAAAAAAAAzg/XHwN4RWucNo/s1600/doc-brown-dont-worry.jpg" imageanchor="1" style="clear: left; float: left; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-r2UGdjV4NC8/U1641rJkyNI/AAAAAAAAAzg/XHwN4RWucNo/s400/doc-brown-dont-worry.jpg" /></a><p style="clear: left; float: left; margin-left: 1em; margin-right: 1em; width: 400px"><small><i>Don’t worry. As long as you hit Run To Cursor at precisely the moment the lightning strikes the tower, everything will be fine!</i></small></p></div><h3>Run to Cursor</h3><p>If you're like me, you've probably found yourself stepping through some code in the debugger and you get to a loop or something that you know is fine and you just don't feel like hitting <i>Step Over</i> the 5 bajillion times necessary to get past it, so what do you do? Hopefully you don't hit <i>Step Over</i> those 5 bajillion times. Hopefully you just set a breakpoint somewhere after that loop and then hit <i>Continue</i>.</p><p>The problem with this solution is that it's tedious.</p><p>Soon, however, you'll be able to simply right-click and select <i>Run To Cursor</i> (or just set/use a keybinding) and the debugger will resume execution until it reaches your cursor!</p><h3>Client-Side Evaluation of Simple Properties</h3><p>Assuming that you haven't disabled "Allow implicit property evaluation and method invocation" in your Xamarin Studio debugger preferences, whenever class properties are evaluated in the debugger (in the Watch pad, the Locals pad, or when you hover the mouse cursor over a property), in order to get the value, the debugger has to spin up a thread in the program being debugged in order to have it evaluate the property (or other expression) because, unlike fields, properties are really just methods that have to run arbitrary code.</p><p>For at least a year or so, now, we've mitigated this somewhat by cheating if the property has the <i>CompilerGeneratedAttribute</i> (signifying that it is an auto-property). When evaluating these properties, we would instead do a lookup of the backing field and get its value since we could do that locally without any need to round-trip to the program being debugged. While this helped a lot, there's a lot of properties out there that effectively just return a field and have no other logic (maybe an auto-property wasn't used because the setter does more than just set the field value?).</p><p>To improve performance of this, I started looking into what it would take to interpret the IL locally in the IDE. Obviously this could only really work if the property getter was "simple" enough and didn't have to take locks, etc. I started asking Zoltan some questions on the feasibility of this and he wrote a simple IL interpreter (included in Mono.Debugger.Soft) that I ended up using in Xamarin Studio to try and evaluate properties locally before falling back to having the debuggee's runtime spin up a thread to evaluate the property for us.<br />
<h3>Great Scott! When Can I Start Using These Awesome Features?</h3><p>To use these new features, you will need Mono 3.4.1 (or later) and an upcoming release of Xamarin Studio (5.0.1? It won't quite make it into 5.0).</p><div class="separator" style="clear: both; text-align: center;"><a href="http://2.bp.blogspot.com/-bzvje8dOpJQ/U16_tVpy4HI/AAAAAAAAAz0/wvfZKqSxXAM/s1600/back-to-the-future.jpg" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://2.bp.blogspot.com/-bzvje8dOpJQ/U16_tVpy4HI/AAAAAAAAAz0/wvfZKqSxXAM/s400/back-to-the-future.jpg" /></a></div><p>Well, it's time I get back... to the future! ... And implementing more new features!</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-1594734415833285892014-03-10T14:01:00.001-04:002014-03-10T14:01:55.042-04:00GMime gets a Speed BoostWith all of the <a href="http://jeffreystedfast.blogspot.com/2013/09/optimization-tips-tricks-used-by.html">performance</a> <a href="http://jeffreystedfast.blogspot.com/2013/10/optimization-tips-tricks-used-by.html">improvements</a> I've been putting into <a href="https://github.com/jstedfast/MimeKit">MimeKit</a> recently, it was about time to port some of these optimizations back to <a href="http://spruce.sourceforge.net/gmime/">GMime</a>.<br />
<br />
In addition to other fixes that were in the queue, GMime 2.6.20 includes the "SIMD" optimization hack that I blogged about doing for MimeKit and I wanted to share the results. Below is a comparison of GMime 2.6.19 and 2.6.20 parsing the same 2GB mbox file on my 2011 Core-i5 iMac with the "persistent stream" option enabled on the GMimeParser:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[fejj@localhost gmime-2.6.19]$ ./gmime-mbox-parser really-big.mbox <br />Parsed 29792 messages in 5.15 seconds.</span><br />
<span style="font-family: "Courier New",Courier,monospace;"><br /></span>
<span style="font-family: "Courier New",Courier,monospace;">[fejj@localhost gmime-2.6.20]$ ./gmime-mbox-parser really-big.mbox <br />Parsed 29792 messages in 4.70 seconds.</span><br />
<br />
That's a pretty respectable improvement. Interestingly, though, it's still not as fast as MimeKit utilizing Mono's LLVM backend:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[fejj@localhost MimeKit]$ mono --llvm ./mbox-parser.exe really-big.mbox <br />Parsed 29792 messages in 4.52 seconds.</span><br />
<br />
Of course, to be fair, without the --llvm option, MimeKit doesn't fare quite so well:<br />
<br />
<span style="font-family: "Courier New",Courier,monospace;">[fejj@localhost MimeKit]$ mono ./mbox-parser.exe really-big.mbox <br />Parsed 29792 messages in 5.54 seconds.</span><br />
<br />
I'm not sure what kind of optimizations LLVM utilizes when used from Mono vs clang (used to compile GMime via homebrew, which I suspect uses -O2), but nevertheless, it's still very impressive.<br />
<br />
After talking with Rodrigo Kumpera from the Mono runtime team, it sounds like the --llvm option is essentially the -O2 optimizations minus a few of the options that cause problems with the Mono runtime, so effectively somewhere between -O1 and -O2.<br />
<br />
I'd love to find out why MimeKit with the LLVM optimizer is faster than GMime compiled with clang (which also makes use of LLVM) with the same optimizations, but I think it'll be pretty hard to narrow down exactly because MimeKit isn't really a straight port of GMime (they are similar, but a lot of MimeKit is all-new in design and implementation).Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-64722554833305164262014-02-03T10:15:00.000-05:002014-02-03T10:15:11.720-05:00Introducing MailKit, a cross-platform .NET mail-client libraryOnce I announced <a href="https://www.blogger.com/2013/09/mimekit-coming-to-nuget-near-you.html">MimeKit</a>, I knew it would only be a matter of time before I started getting asked about SMTP, IMAP, and/or POP3 support.<br />
<br />
Let's just say,<br />
<br />
<h1>Challenge... ACCEPTED!</h1><br />
<object height="315" width="560"><param name="movie" value="//www.youtube.com/v/ekeELle5g-o?hl=en_US&version=3"></param><param name="allowFullScreen" value="true"></param><param name="allowscriptaccess" value="always"></param><embed src="//www.youtube.com/v/ekeELle5g-o?hl=en_US&version=3" type="application/x-shockwave-flash" width="560" height="315" allowscriptaccess="always" allowfullscreen="true"></embed></object><br />
<br />
I started off back in early December writing an <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Smtp/SmtpClient.html">SmtpClient</a> so that developers using MimeKit wouldn't have to <a href="http://jstedfast.github.io/MimeKit/docs/MimeKit/MimeMessage.html#M:MimeKit.MimeMessage.op_Explicit%28MimeKit.MimeMessage%29~System.Net.Mail.MailMessage" target="_blank">convert a MimeMessage to a System.Net.Mail.MailMessage</a> in order to send it using <a href="http://msdn.microsoft.com/en-us/library/system.net.mail.smtpclient%28v=vs.110%29.aspx">System.Net.Mail.SmtpClient</a>. This went pretty quickly because I've implemented several SMTP clients in the past. Implementing the various SASL authentication mechanisms probably took as much or more time than implementing the SMTP protocol.<br />
<br />
The following weekend, I ended up implementing a <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Pop3/Pop3Client.html">Pop3Client</a>. Originally, I had planned on more-or-less cloning the API we had used in Evolution, but I decided that I would take a different approach. I designed a simple <a href="http://jstedfast.github.io/MailKit/docs/MailKit/IMessageSpool.html">IMessageSpool</a> interface which more closely follows the limited functionality of POP3 and mbox spools instead of trying to map the Pop3Client to a Store/Folder paradigm like JavaMail and Evolution do (Evolution's mail library was loosely based on JavaMail). Mapping mbox and POP3 spools to Stores and Folders in Evolution was, to my recollection, rather awkward and I wanted to avoid that with MailKit.<br />
<br />
At first I was loathe to do it, but over the past 2 weeks I ended up writing an <a href="http://jstedfast.github.io/MailKit/docs/MailKit.Net.Imap/ImapClient.html">ImapClient</a> as well. I'm sure Philip van Hoof will be pleased to note that I have a very nice <a href="https://github.com/jstedfast/MailKit/blob/master/MailKit/Net/Imap/ImapUtils.cs#L588">BODYSTRUCTURE</a> parser, although that API is not publicly exported.<br />
<br />
Unlike the SmtpClient and Pop3Client, the ImapClient does not have all of its functionality on a single public class. Instead, ImapClient implements an <a href="http://jstedfast.github.io/MailKit/docs/MailKit/IMessageStore.html">IMessageStore</a> which has a limited API, mostly meant for getting <a href="http://jstedfast.github.io/MailKit/docs/MailKit/IFolder.html">IFolder</a>s. I imagine that those who are familiar with the JavaMail and/or Evolution (Camel) APIs will recognize this design.<br />
<br />
The <a href="http://jstedfast.github.io/MailKit/docs/MailKit/IFolder.html">IFolder</a> interface isn't designed to be <i>exactly</i> like the JavaMail Folder API, though. I've been designing the interface incrementally as I implement the various IMAP extensions (I've found at least 37 of them at the time of this blog post, although I don't think I'll bother with ACL, MAILBOX-REFERRAL, or LOGIN-REFERRAL), so the API may continue to evolve as I go, but I think what I've got now will likely remain - I'll probably just be including <i>additional</i> APIs for the new stuff.<br />
<br />
So far, I've implemented the following IMAP extensions: <a href="http://www.ietf.org/rfc/rfc2088.txt">LITERAL+</a>, <a href="http://www.ietf.org/rfc/rfc2342.txt">NAMESPACE</a>, <a href="http://www.ietf.org/rfc/rfc3348.txt">CHILDREN</a>, <a href="http://www.ietf.org/rfc/rfc3501.txt">LOGIN-DISABLED</a>, <a href="http://www.ietf.org/rfc/rfc3501.txt">STARTTLS</a>, <a href="http://www.ietf.org/rfc/rfc3502.txt">MULTIAPPEND</a>, <a href="http://www.ietf.org/rfc/rfc3691.txt">UNSELECT</a>, <a href="http://www.ietf.org/rfc/rfc4315.txt">UIDPLUS</a>, <a href="http://www.ietf.org/rfc/rfc4551.txt">CONDSTORE</a>, <a href="http://www.ietf.org/rfc/rfc4731.txt">ESEARCH</a>, <a href="http://www.ietf.org/rfc/rfc4959.txt">SASL-IR</a>, <a href="http://www.ietf.org/rfc/rfc5256.txt">SORT</a>, <a href="http://www.ietf.org/rfc/rfc5256.txt">THREAD</a>, <a href="http://www.ietf.org/rfc/rfc6154.txt">SPECIAL-USE</a>, <a href="http://www.ietf.org/rfc/rfc6851.txt">MOVE</a>, <a href="https://developers.google.com/gmail/imap_extensions">XLIST, and X-GM-EXT1</a>. Phew, that was exhausting listing all of those!<br />
<br />
Also news-worthy is that MimeKit is now equally as fast as GMime, which is pretty impressive considering that it is fully managed C# code.<br />
<br />
<a href="https://www.nuget.org/packages/MailKit/">Download MailKit 0.2 now</a> and let the hacking begin!Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-32869487794291895832013-10-07T07:03:00.000-04:002013-10-07T07:08:26.948-04:00Optimization Tips & Tricks used by MimeKit: Part 2<p>In my <a href="http://jeffreystedfast.blogspot.com/2013/09/optimization-tips-tricks-used-by.html">previous blog post</a>, I talked about optimizing the most critical loop in <a href="https://github.com/jstedfast/MimeKit">MimeKit</a>'s <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/MimeParser.cs">MimeParser</a> by:</p><ul><li>Extending our read buffer by an extra byte (which later became 4 extra bytes) that I could set to <font style="font-family: courier new; font-size: 85%">'\n'</font>, allowing me to do the bounds check <i>after</i> the loop as opposed to in the loop, saving us roughly half the instructions.</li>
<li>Unrolling the loop in order to check for 4 bytes at a time for that <font style="font-family: courier new; font-size: 85%">'\n'</font> by using some bit twiddling hacks (for 64-bit systems, we might gain a little more performance by checking 8 bytes at a time).</li>
</ul><p>After implementing both of those optimizations, the time taken for MimeKit's parser to parse nearly 15,000 messages in a ~1.2 gigabyte mbox file dropped from around 10s to about 6s on my iMac with Mono 3.2.3 (32-bit). That is a <i>massive</i> increase in performance.</p><p>Even after both of those optimizations, that loop is still the most critical loop in the parser and the <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/MimeParser.cs#L832">MimeParser.ScanContent()</a> method, which contains it, is still the most critical method of the parser.</p><p>While the loop itself was a huge chunk of the time spent in that method, the next largest offender was writing the content of the MIME part into a <a href="http://msdn.microsoft.com/en-us/library/system.io.memorystream.aspx">System.IO.MemoryStream</a>.</p><p>MemoryStream, for those that aren't familiar with C#, is just what it sounds like it is: a stream backed by a memory buffer (in C#, this happens to be a byte array). By default, a new MemoryStream starts with a buffer of about 256 bytes. As you write more to the MemoryStream, it resizes its internal memory buffer to either the minimum size needed to hold the its existing content plus whatever number of bytes your latest <font style="font-family: courier new; font-size: 85%">Write()</font> was called with or double the current internal buffer size, whichever is larger.</p><p>The performance problem here is that for MIME parts with large amounts of content, that buffer will be resized numerous times. Each time that buffer is resized, due to the way C# works, it will allocate a new buffer, zero the memory, and then copy the old content over to the new buffer. That's a lot of copying and creates a situation where the write operation can become exponentially worse as the internal buffer gets larger. Since MemoryStream contains a <a href="http://msdn.microsoft.com/en-us/library/system.io.memorystream.getbuffer.aspx">GetBuffer()</a> method, its internal buffer really has to be a single contiguous block of memory. This means that there's little we could do to reduce overhead of zeroing the new buffer every time it resizes beyond trying to come up with a different formula for calculating the next optimal buffer size.</p><p>At first I decided to try the simple approach of using the MemoryStream constructor that allows specifying an initial capacity. By bumping up the initial capacity to 2048 bytes, things did improve, but only by a very disappointing amount. Larger initial capacities such as 4096 and 8192 bytes also made very little difference.</p><p>After brainstorming with my coworker and Mono runtime hacker, Rodrigo Kumpera, we decided that one way to solve this performance problem would be to write a custom memory-backed stream that didn't use a single contiguous block of memory, but instead used a list of non-contiguous memory blocks. When this stream needed to grow its internal memory storage, all it would need to do is allocate a new block of memory and append it to its internal list of blocks. This would allow for minimal overhead because only the new block would need to be zeroed and no data would need to be re-copied, ever. As it turns out, this approach would also allow me to limit the amount of unused memory used by the stream.</p><p>I dubbed this new memory-backed stream <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/IO/MemoryBlockStream.cs">MimeKit.IO.MemoryBlockStream</a>. As you can see, the implementation is pretty trivial (doesn't even require scary looking bit twiddling hacks like my previous optimization), but it made quite a difference in performance. By using this new memory stream, I was able to shave a full second off of the time needed to parse that mbox file I mentioned earlier, getting the total time spent down to 5s. That's starting to get pretty respectable, performance-wise.</p><p>As a comparison, let's compare the performance of MimeKit with what seems to be the 2 most popular C# MIME parsers out there (<a href="http://sourceforge.net/projects/hpop/">OpenPOP.NET</a> and <a href="http://anmar.eu.org/projects/sharpmimetools/">SharpMimeTools</a>) and see how we do. I've been hyping up the performance of MimeKit a lot, so it had better live up to expectations, right? Let's see if it does.</p><p>Now, since none of the other C# MIME parsers I could find support parsing the Unix mbox file format, we'll write some test programs to parse the same message stream over and over (say, 20 thousand times) to compare MimeKit to OpenPOP.NET.</p><p>Here's the test program I wrote for OpenPOP.NET:</p><pre style="font-family: courier new; font-size: 85%">using System;
using System.IO;
using System.Diagnostics;
using OpenPop.Mime;
namespace OpenPopParser {
class Program
{
public static void Main (string[] args)
{
var stream = File.OpenRead (args[0]);
var stopwatch = new Stopwatch ();
stopwatch.Start ();
for (int i = 0; i < 20000; i++) {
var message = Message.Load (stream);
stream.Position = 0;
}
stopwatch.Stop ();
Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
}
}
}
</pre><p>Here's the SharpMimeTools parser I wrote for testing:</p><pre style="font-family: courier new; font-size: 85%">using System;
using System.IO;
using System.Diagnostics;
using anmar.SharpMimeTools;
namespace SharpMimeParser {
class Program
{
public static void Main (string[] args)
{
var stream = File.OpenRead (args[0]);
var stopwatch = new Stopwatch ();
stopwatch.Start ();
for (int i = 0; i < 20000; i++) {
var message = new SharpMessage (stream);
stream.Position = 0;
}
stopwatch.Stop ();
Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
}
}
}
</pre><p>And here is the test program I used for MimeKit:</p><pre style="font-family: courier new; font-size: 85%">using System;
using System.IO;
using System.Diagnostics;
using MimeKit;
namespace MimeKitParser {
class Program
{
public static void Main (string[] args)
{
var stream = File.OpenRead (args[0]);
var stopwatch = new Stopwatch ();
stopwatch.Start ();
for (int i = 0; i < 20000; i++) {
var parser = new MimeParser (stream, MimeFormat.Default);
var message = parser.ParseMessage ();
stream.Position = 0;
}
stopwatch.Stop ();
Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
}
}
}
</pre><p><small>Note: Unfortunately, OpenPOP.NET's message parser completely failed to parse the Star Trek message I pulled out of my test suite at random (first message in the jwz.mbox.txt file included in MimeKit's UnitTests project) due to the Base64 decoder not liking some byte or another in the stream, so I had to patch OpenPOP.NET to no-op its base64 decoder (which, if anything, should make it faster).</small></p><p>And here are the results running on my 2011 MacBook Air:</p><pre style="font-family: courier new; font-size: 85%">[fejj@localhost OpenPopParser]$ mono ./OpenPopParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:06:26.6825190
[fejj@localhost SharpMimeParser]$ mono ./SharpMimeParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:19:30.0402064
[fejj@localhost MimeKit]$ mono ./MimeKitParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:00:15.6159326
</pre><p>Whooooosh!</p><p>Not. Even. Close.</p><p>MimeKit is nearly <b>25x faster</b> than OpenPOP.NET even after making its base64 decoder a no-op and <b>75x faster</b> than SharpMimeTools.</p><p>Since I've been ranting against C# MIME parsers that made heavy use of regex, let me show you just how horrible regex is for parsing messages (performance-wise). There's a C# MIME parser called <a href="https://github.com/smithimage/MIMER">MIMER</a> that is nearly pure regex, so what better library to illustrate my point? I wrote a very similar loop to the other 2 that I listed above, so I'm not going to bother repeating it again. Instead, I'll just skip to the results:</p><pre style="font-family: courier new; font-size: 85%">[fejj@localhost MimerParser]$ mono ./MimerParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:16:51.4839129
</pre><p>Ouch. MimeKit is roughly <b>65x faster</b> than a fully regex-based MIME parser. It's actually rather pathetic that this regex parser beats SharpMimeTools.</p><p>This is why, as a developer, it's important to understand the limitations of the tools you decide to use. Regex is great for some things but it is a terrible choice for others. As Jamie Zawinski might say, <blockquote>Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.</blockquote></p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-7692367911620125002013-09-30T23:31:00.003-04:002013-10-11T17:31:37.254-04:00Optimization Tips & Tricks used by MimeKit: Part 1<p>One of the goals of <a href="http://github.com/jstedfast/MimeKit">MimeKit</a>, other than being the most robust MIME parser, is to be the fastest C# MIME parser this side of the Mississippi. Scratch that, fastest C# MIME parser in the <i>World</i>.</p><p>Seriously, though, I want to get MimeKit to be as fast and efficient as my C parser, <a href="http://spruce.sourceforge.net">GMime</a>, which is one of the fastest (if not <i>the</i> fastest) MIME parsers out there right now, and I don't expect that any parser is likely to smoke GMime anytime soon, so using it as a baseline to compare against means that I have a realistic goal to set for MimeKit.</p><p>Now that you know the why, let's examine the <i>how</i>.</p><p>First, I'm using one of those rarely used features of C#: unsafe pointers. While that alone is not all that interesting, it's a corner stone for one of the main techniques I've used. In C#, the <i>fixed</i> statement (which is how you get a pointer to a managed object) <i>pins</i> the object to a fixed location in memory to prevent the GC from moving that memory around while you operate on that buffer. Keep in mind, though, that telling the GC to pin a block of memory is not free, so you should not use this feature without careful consideration. If you're not careful, using pointers could actually make your code slower. Now that we've got that out of the way...</p><p>MIME is line-based, so a large part of every MIME parser is going to be searching for the next line of input. One of the reasons most MIME parsers (especially C# MIME parsers) are so slow is because they use a <i>ReadLine()</i> approach and most <i>TextReaders</i> likely use a naive algorithm for finding the end of the current line (as well as all of the extra allocating and copying into a string buffer):</p><pre style="font-family: courier new; font-size: 85%"> // scan for the end of the line
while (inptr < inend && *inptr != (byte) '\n')
inptr++;
</pre><p>The trick I used in GMime was to make sure that my read buffer was 1 byte larger than the max number of bytes I'd ever read from the underlying stream at a given time. This allowed me to set the first byte in the buffer beyond the bytes I just read from the stream to <font style="font-family: courier new; font-size: 85%">'\n'</font>, thus allowing for the ability to remove the <font style="font-family: courier new; font-size: 85%">inptr < inend</font> check, opting to do the bounds check <i>after</i> the loop has completed instead. This nearly halves the number of instructions used per loop, making it much, much faster. So, now we have:</p><pre style="font-family: courier new; font-size: 85%"> // scan for the end of the line
while (*inptr != (byte) '\n')
inptr++;
</pre><p>But is that the best we can do?</p><p>Even after using this trick, it was still the hottest loop in my parser:</p><p><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-1fIUK8R4_-Y/UkouW7qaN3I/AAAAAAAAAyQ/2earL_jA1Rk/s1600/improved-naive-loop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-1fIUK8R4_-Y/UkouW7qaN3I/AAAAAAAAAyQ/2earL_jA1Rk/s320/improved-naive-loop.png" /></a></div></p><p>We've got no choice but to use a linear scan, but that doesn't mean that we can't do it faster. If we could somehow reduce the number of loops and likewise reduce the number of pointer increments, we could eliminate a bunch of the overhead of the loop. This technique is referred to as loop <i>unrolling</i>. Here's what brianonymous (from the ##csharp irc channel on freenode) and I came up with (with a little help from Sean Eron Anderson's <a href="http://graphics.stanford.edu/~seander/bithacks.html">bit twiddling hacks</a>):</p><pre style="font-family: courier new; font-size: 85%"> uint* dword = (uint*) inptr;
uint mask;
do {
mask = *dword++ ^ 0x0A0A0A0A;
mask = ((mask - 0x01010101) & (~mask & 0x80808080));
} while (mask == 0);
</pre><p>And here are the results of that optimization:</p><p><div class="separator" style="clear: both; text-align: center;"><a href="http://1.bp.blogspot.com/-2VRQKzVSOgw/UkouabftygI/AAAAAAAAAyY/hGkRRODhTL8/s1600/unrolled-loop.png" imageanchor="1" style="margin-left: 1em; margin-right: 1em;"><img border="0" src="http://1.bp.blogspot.com/-2VRQKzVSOgw/UkouabftygI/AAAAAAAAAyY/hGkRRODhTL8/s320/unrolled-loop.png" /></a></div></p><p>Now, keep in mind that on many architectures other than x86, in order to employ the trick above, <font style="font-family: courier new; font-size: 85%">inptr</font> must first be 4-byte aligned (uint is 32bit) or it could cause a SIGBUS or worse, a crash. This is fairly easy to solve, though. All you need to do is increment <font style="font-family: courier new; font-size: 85%">inptr</font> until you know that it is 4 byte aligned and then you can switch over to reading 4 bytes at a time as in the above loop. We'll also need to figure out which of those 4 bytes contained the <font style="font-family: courier new; font-size: 85%">'\n'</font>. An easy way to solve that problem is to just linearly scan those 4 bytes using our previous single-byte-per-loop implementation starting at <font style="font-family: courier new; font-size: 85%">dword - 1</font>. Here it is, your moment of Zen:</p><pre style="font-family: courier new; font-size: 85%"> // Note: we can always depend on byte[] arrays being
// 4-byte aligned on 32bit and 64bit architectures
int alignment = (inputIndex + 3) & ~3;
byte* aligned = inptr + alignment;
byte* start = inptr;
uint mask;
while (inptr < aligned && *inptr != (byte) '\n')
inptr++;
if (inptr == aligned) {
// -funroll-loops
uint* dword = (uint*) inptr;
do {
mask = *dword++ ^ 0x0A0A0A0A;
mask = ((mask - 0x01010101) & (~mask & 0x80808080));
} while (mask == 0);
inptr = (byte*) (dword - 1);
while (*inptr != (byte) '\n')
inptr++;
}
</pre><p><small>Note: In this above code snippet, 'inputIndex' is the byte offset of 'inptr' into the byte array. Since we can safely assume that index 0 is 4-byte aligned, we can do a simple calculation to get the next multiple of 4 and add that to our 'inptr' to get the next 4-byte aligned pointer.</small></p><p>That's great, but what does all that hex mumbo jumbo do? And why does it work?</p><p>Let's go over this 1 step at a time...</p><pre style="font-family: courier new; font-size: 85%"> mask = *dword++ ^ 0x0A0A0A0A;
</pre><p>This xor's the value of <font style="font-family: courier new; font-size: 85%">dword</font> with 0x0A0A0A0A (0x0A0A0A0A is just 4 bytes of <font style="font-family: courier new; font-size: 85%">'\n'</font>). The xor sets every byte that is equal to 0x0A to 0 in <font style="font-family: courier new; font-size: 85%">mask</font>. Every other byte will be non-zero.</p><pre style="font-family: courier new; font-size: 85%"> mask - 0x01010101
</pre><p>When we subtract 0x01010101 from <font style="font-family: courier new; font-size: 85%">mask</font>, the result will be that only bytes greater than 0x80 will contain any high-order bits (and any byte that was originally 0x0A in our input will now be 0xFF).</p><pre style="font-family: courier new; font-size: 85%"> ~mask & 0x80808080
</pre><p>This inverts the value of <font style="font-family: courier new; font-size: 85%">mask</font> resulting in no bytes having the highest bit set except for those that had a 0 in that slot before (including the byte we're looking for). By then bitewise-and'ing it with 0x80808080, we get 0x80 for each byte that was originally 0x0A in our input or otherwise had the highest bit set after the bit inversion.</p><p>Because there's no way for any byte to have the highest bit set in both sides of the encompassing bitwise-and except for the character we're looking for (0x0A), the mask will always be 0 unless any of the bytes within were originally 0x0A, which would then break us out of the loop.</p><p>Well, that concludes part 1 as it is time for me to go to bed so I can wake up at a reasonable time tomorrow morning.</p><p>Good night!</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-67568866540191693032013-09-28T15:18:00.000-04:002013-10-07T07:07:17.138-04:00MimeKit: Coming to a NuGet near you.<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-gfo9JFS4wFc/Ukcg2vRp0KI/AAAAAAAAAx8/FtzoYOX7N8w/s1600/sad-mime.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" src="http://4.bp.blogspot.com/-gfo9JFS4wFc/Ukcg2vRp0KI/AAAAAAAAAx8/FtzoYOX7N8w/s1600/sad-mime.jpg" /></a></div><p>If, like me, you've been trapped in the invisible box of despair, <a href="http://jeffreystedfast.blogspot.com/2013/08/why-decoding-rfc2047-encoded-headers-is.html">bemoaning</a> the <a href="http://jeffreystedfast.blogspot.com/2013/09/time-for-rant-on-mime-parsers.html">woeful inadequacies</a> of every .NET MIME library you've ever found on the internets, cry no more: <a href="http://github.com/jstedfast/MimeKit">MimeKit</a> is here.</p><p>I've just released MimeKit v0.5 as a <a href="https://www.nuget.org/packages/MimeKit">NuGet Package</a>. There's still plenty of work left to do, mostly involving writing more API documentation, but I don't expect to change the API much between now and v1.0. For all the mobile MIME lovers out there, you'll be pleased to note that in addition to the .NET Framework 4.0 assembly, the NuGet package also includes assemblies built for <a href="http://xamarin.com/android">Xamarin.Android</a> and <a href="http://xamarin.com/ios">Xamarin.iOS</a>. It's completely open source and licensed under the <a href="http://opensource.org/licenses/MIT">MIT/X11 license</a>, so you can use it in any project you want - no restrictions. Once MimeKit goes v1.0, I plan on adding it to Xamarin's <a href="http://components.xamarin.com/">Component Store</a> as well for even easier mobile development. If that doesn't turn that frown upside down, I don't know what will.</p><p>For those that don't already know, MimeKit is a really fast MIME parser that uses a <i>real tokenizer</i> instead of regular expressions and <a href="http://msdn.microsoft.com/en-us/library/system.string.split.aspx">string.Split()</a> to parse and decode headers. Among numerous other things, it can properly handle <a href="http://www.ietf.org/rfc/rfc2047.txt">rfc2047</a> encoded-word tokens that contain quoted-printable and base64 payloads which have been improperly broken apart (i.e. a quoted-printable triplet or a base64 quartet is split between 2 or more encoded-word tokens) as well as handling cases where multibyte character sequences are split between words thanks to the state machine nature of MimeKit's rfc2047 text and phrase decoders (yes, there are 2 types of encoded-word tokens - something most other MIME parsers have failed to take notice of). With the use of <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/ParserOptions.cs">MimeKit.ParserOptions</a>, the user can specify his or her own fallback charset (in addition to UTF-8 and ISO-8859-1 that MimeKit has built in), allowing MimeKit to gracefully handle undeclared 8bit text in headers.</p><p>When constructing MIME messages, MimeKit provides the user with the ability to specify any character encoding available on the system for encoding each individual header (or, in the case of address headers: each individual email address). If none is specified, UTF-8 is used unless the characters will fit nicely into ISO-8859-1. MimeKit's rfc2047 and rfc2231 encoders do proper breaking of text (i.e it avoids breaking between surrogate pairs) <i>before</i> the actual encoding step, thus ensuring that each encoded-word token (or parameter value) is correctly self-contained.</p><p><a href="http://en.wikipedia.org/wiki/S/MIME">S/MIME</a> support is also available in the .NET Framework 4.0 assembly (not yet supported in the Android or iOS assemblies due to the System.Security assembly being unavailable on those platforms). MimeKit supports signing, encrypting, decrypting, and verifying S/MIME message parts. For signing, you can either use the preferred multipart/signed approach or the application/[x-]pkcs7-signature mime-type, whichever you prefer.</p><p>I'd love to support PGP/MIME as well, but this is a bit more complicated as I would likely need to depend on external native libraries and programs (such as <a href="http://www.gnupg.org/related_software/gpgme/">GpgME</a> and <a href="http://www.gnupg.org/">GnuPG</a>) which means that MimeKit would likely have to become 32bit-only (currently, libgpgme is only available for 32bit Windows).</p><p>I hope you enjoy using MimeKit as much as I have enjoyed implementing it!</p><p><small>Note: For those using my <a href="http://spruce.sourceforge.net/gmime">GMime</a> library, fear not! I have not forgotten about you! I plan to bring many of the API and parser improvements that I've made to MimeKit back to GMime in the near future.</small></p><p><small>For those using the C# bindings, I'd highly recommend that you consider switching to MimeKit instead. I've based MimeKit's API on my GMime API, so porting to MimeKit should be fairly straightforward.</small></p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-81614842066901174872013-09-15T16:55:00.000-04:002013-10-07T07:06:16.402-04:00Time for a rant on mime parsers...<p>Warning: Viewer discretion is advised.</p><p>Where should I begin?</p><p>I guess I should start by saying that I am obsessed with MIME and, in particular, MIME parsers. No, really. I am obsessed. Don't believe me? I've written and/or worked on several MIME parsers at this point. It started off in my college days working on <a href="http://spruce.sourceforge.net">Spruce</a> which had a horrendously bad MIME parser, and so as you read farther along in my rant about shitty MIME parsers, keep in mind: I've been there, I've written a shitty MIME parser.</p><p>As a handful of people are aware, I've recently started implementing a C# MIME parser called <a href="http://github.com/jstedfast/MimeKit">MimeKit</a>. As I work on this, I've been searching around on GitHub and Google to see what other MIME parsers exist out there to find out what sort of APIs they provide. I thought perhaps I'll find one that offers a well-designed API that will inspire me. Perhaps, by some miracle, I'd find one that was actually pretty good that I could just contribute to instead of writing my own from scratch (yea, wishful thinking). Instead, all I have found are poorly designed and implemented MIME parsers, many probably belong on the front page of the <a href="http://thedailywtf.com/">Daily WTF</a>.</p><p>I guess I'll start with some softballs.</p><p>First, there's the fact that every single one of them were written as System.String parsers. Don't be fooled by the ones claiming to be "stream parsers", because all any of those did was to slap a TextReader on top of the byte stream and start using reader.ReadLine(). What's so bad about that, you ask? For those not familiar with MIME, I'd like for you to take a look at the raw email sources in your inboxes particularly if you have correspondence with anyone outside of the US. Hopefully most of your friends and colleagues are using more-or-less MIME compliant email clients, but I guarantee you'll find at least a few emails with raw 8bit text.</p><p>Now, if the language they were using was C or C++, they <i>might</i> be able to get away with doing this because they'd technically be operating on byte arrays, but with Java and C#, a 'string' is a unicode string. Tell me: how does one get a unicode string from a raw byte array?</p><p>Bingo. You need to know the charset before you can convert those bytes into unicode characters.</p><p>To be fair, there's really no good way of handling raw 8bit text in message headers, but by using a TextReader approach, you are really limiting the possibilities.</p><p>Next up is the ReadLine() approach. One of the 2 early parsers in GMime (<a href="https://git.gnome.org/browse/gmime/tree/pan-mime-parser.c?h=GMIME_0_7_0">pan-mime-parser.c</a> back in the version 0.7 days) used a ReadLine() approach, so I understand the thinking behind this. And really, there's nothing wrong with this approach as far as correctness goes, it's more of a "this can never be fast" complaint. Of the two early parsers in GMime, the pan-mime-parser.c backend was horribly slow compared to the in-memory parser. Of course, that's not very surprising. More surprising to me at the time was that when I wrote GMime's current generation of parser (sometime between v0.7 and v1.0), it was just as fast as the in-memory parser ever was and only ever had up to 4k in a read buffer at any given time. My point is, there are far better approaches than ReadLine() if you want your parser to be reasonably performant... and why wouldn't you want that? Your users definitely want that.</p><p>Okay, now come the more serious problems that I encountered in nearly all of the mime parser libraries I found.</p><p>I think that every single mime parser I've found so far uses the "String.Split()" approach for parsing address headers and/or for parsing parameter lists on headers such as Content-Type and Content-Disposition.</p><p>Here's an example from one C# MIME parser:</p><pre style="font-family: courier new; font-size: 85%">string[] emails = addressHeader.Split(',');
</pre><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-dfAfdhqOtD0/UjYZJvgjyOI/AAAAAAAAAxU/MLfPEcIQ47A/s1600/itcrowd-facepalm.gif" imageanchor="1" style="clear: left; float: center; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-dfAfdhqOtD0/UjYZJvgjyOI/AAAAAAAAAxU/MLfPEcIQ47A/s320/itcrowd-facepalm.gif" /></a></div><p>Here's how this same parser decodes encoded-word tokens:</p><pre style="font-family: courier new; font-size: 85%">private static void DecodeHeaders(NameValueCollection headers)
{
ArrayList tmpKeys = new ArrayList(headers.Keys);
foreach (string key in headers.AllKeys)
{
//strip qp encoding information from the header if present
headers[key] = Regex.Replace(headers[key].ToString(), @"=\?.*?\?Q\?(.*?)\?=",
new MatchEvaluator(MyMatchEvaluator), RegexOptions.IgnoreCase | RegexOptions.Multiline);
headers[key] = Regex.Replace(headers[key].ToString(), @"=\?.*?\?B\?(.*?)\?=",
new MatchEvaluator(MyMatchEvaluatorBase64), RegexOptions.IgnoreCase | RegexOptions.Multiline);
}
}
private static string MyMatchEvaluator(Match m)
{
return DecodeQP(m.Groups[1].Value);
}
private static string MyMatchEvaluatorBase64(Match m)
{
System.Text.Encoding enc = System.Text.Encoding.UTF7;
return enc.GetString(Convert.FromBase64String(m.Groups[1].Value));
}
</pre><div class="separator" style="clear: both; text-align: center;"><a href="http://3.bp.blogspot.com/-FFVLLRp4Lyw/UjYZFw09wuI/AAAAAAAAAxM/W0fSZURLOok/s1600/double-facepalm.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://3.bp.blogspot.com/-FFVLLRp4Lyw/UjYZFw09wuI/AAAAAAAAAxM/W0fSZURLOok/s320/double-facepalm.jpg" /></a></div><p>Excuse my language, but what the fuck? It completely throws away the charset in each of those encoded-word tokens. In the case of quoted-printable tokens, it assumes they are all ASCII (actually, latin1 <i>may</i> work as well?) and in the case of base64 encoded-word tokens, it assumes they are all in UTF-7!?!? Where in the world did he get that idea? I can't begin to imagine his code working on <i>any</i> base64 encoded-word tokens in the real world. If anything is deserving of a double facepalm, this is it.</p><p>I'd just like to point out that this is what this project's description states:</p><blockquote>A small, efficient, and working mime parser library written in c#.<br />
...<br />
I've used several open source mime parsers before, but they all either<br />
fail on one kind of encoding or the other, or miss some crucial<br />
information. That's why I decided to finally have a go at the problem<br />
myself.<br />
</blockquote><p>I'll grant you that his MIME parser is small, but I'd have to take issue with the "efficient" and "working" adjectives. With the heavy use of string allocations and regex matching, it could hardly be considered "efficient". And as the code pointed out above illustrates, "working" is a bit of an overstatement.</p><p>Folks... this is what you get when you opt for a "lightweight" MIME parser because you think that parsers like GMime are "bloated".</p><p>On to parser #2... I like to call this the "Humpty Dumpty" approach:</p><pre style="font-family: courier new; font-size: 85%">public static StringDictionary parseHeaderFieldBody ( String field, String fieldbody ) {
if ( fieldbody==null )
return null;
// FIXME: rewrite parseHeaderFieldBody to being regexp based.
fieldbody = SharpMimeTools.uncommentString (fieldbody);
StringDictionary fieldbodycol = new StringDictionary ();
String[] words = fieldbody.Split(new Char[]{';'});
if ( words.Length>0 ) {
fieldbodycol.Add (field.ToLower(), words[0].ToLower().Trim());
for (int i=1; i<words.Length; i++ ) {
String[] param = words[i].Trim(new Char[]{' ', '\t'}).Split(new Char[]{'='}, 2);
if ( param.Length==2 ) {
param[0] = param[0].Trim(new Char[]{' ', '\t'});
param[1] = param[1].Trim(new Char[]{' ', '\t'});
if ( param[1].StartsWith("\"") && !param[1].EndsWith("\"")) {
do {
param[1] += ";" + words[++i];
} while ( !words[i].EndsWith("\"") && i<words.Length);
}
fieldbodycol.Add ( param[0], SharpMimeTools.parserfc2047Header (param[1].TrimEnd(';').Trim('\"', ' ')) );
}
}
}
return fieldbodycol;
}
</pre><p>I'll give this guy some credit, at least he saw that his String.Split() approach was flawed and so tried to compensate by piecing Humpty Dumpty back together again. Of course, with his String.Trim()ing, he just won't be able to put him back together again with any level of certainty. The white space in those quoted tokens may have significant meaning.</p><p>Many of the C# MIME parsers out there like to use Regex all over the place. Here's a snippet from one parser that is entirely written in Regex (yea, have fun maintaining that...):</p><pre style="font-family: courier new; font-size: 85%">if (m_EncodedWordPattern.RegularExpression.IsMatch(field.Body))
{
string charset = m_CharsetPattern.RegularExpression.Match(field.Body).Value;
string text = m_EncodedTextPattern.RegularExpression.Match(field.Body).Value;
string encoding = m_EncodingPattern.RegularExpression.Match(field.Body).Value;
Encoding enc = Encoding.GetEncoding(charset);
byte[] bar;
if (encoding.ToLower().Equals("q"))
{
bar = m_QPDecoder.Decode(ref text);
}
else
{
bar = m_B64decoder.Decode(ref text);
}
text = enc.GetString(bar);
field.Body = Regex.Replace(field.Body,
m_EncodedWordPattern.TextPattern, text);
field.Body = field.Body.Replace('_', ' ');
}
</pre><p>Let's pretend that the regex pattern strings are correct in their definitions (because they are god-awful to read and I can't be bothered to double-check them), the replacing of '_' with a space is wrong (it should only be done in the "q" case) and the Regex.Replace() is just evil. Not to mention that there could be multiple encoded-words per field.Body which this code utterly fails to handle.</p><p>Guys. I know you love regular expressions and that they are very very useful, but they are no substitute for writing a real tokenizer. This is especially true if you want to be lenient in what you accept (and in the case of MIME, <a href="http://jeffreystedfast.blogspot.com/2013/08/why-decoding-rfc2047-encoded-headers-is.html">you really need to be</a>).</p><div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-fmlDA2G5GKw/UjYZN79xbsI/AAAAAAAAAxc/GTCG6DQfQ_U/s1600/facepalm-cat.gif" imageanchor="1" style="clear: left; float: center; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://4.bp.blogspot.com/-fmlDA2G5GKw/UjYZN79xbsI/AAAAAAAAAxc/GTCG6DQfQ_U/s320/facepalm-cat.gif" /></a></div>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com8tag:blogger.com,1999:blog-203063759820106893.post-82221454584262191972013-08-11T09:33:00.001-04:002013-09-15T10:06:18.280-04:00Why decoding rfc2047-encoded headers is hard<p>Somewhat inspired by a recent <a href="http://notmuchmail.org/pipermail/notmuch/2013/015594.html">thread</a> on the <a href="http://notmuchmail.org/">notmuch</a> mailing-list, I thought I'd explain why decoding headers is so hard to get right. I'm sure just about every developer who has ever worked on an email client could tell you this, but I guess I'm going to be the one to do it.</p><p>Here's just a short list of the problems every developer faces when they go to implement a decoder for headers which have been (theoretically) encoded according to the <a href="http://www.ietf.org/rfc/rfc2047.txt">rfc2047</a> specification:</p><ol><li>First off, there are technically two variations of header encoding formats specified by rfc2047 - one for phrases and one for unstructured text fields. They are very similar but you can't use the same rules for tokenizing them. I mention this because it seems that most MIME parsers miss this very subtle distinction and so, as you might imagine, do most MIME generators. Hell, most MIME generators probably never even heard of specifications to begin with it seems.<br />
<p>This brings us to:</p></li>
<li>There are so many variations of how MIME headers fail to be tokenizable according to the rules of rfc2822 and rfc2047. You'll encounter fun stuff such as:<br />
<ol type="a"><li>encoded-word tokens illegally being embedded in other word tokens</li>
<li>encoded-word tokens containing illegal characters in them (such as spaces, line breaks, and more) effectively making it so that a tokenizer can no longer, well, tokenize them (at least not easily)</li>
<li>multi-byte character sequences being split between multiple encoded-word tokens which means that it's not possible to decode said encoded-word tokens individually</li>
<li>the payloads of encoded-word tokens being split up into multiple encoded-word tokens, often splitting in a location which makes it impossible to decode the payload in isolation</li>
</ol><p>You can see some examples <a href="https://git.gnome.org/browse/gmime/tree/tests/test-mime.c#n211">here</a>.</p></li>
<li>Something that many developers seem to miss is the fact that each encoded-word token is allowed to be in different character encodings (you might have one token in UTF-8, another in ISO-8859-1 and yet another in koi8-r). Normally, this would be no big deal because you'd just decode each payload, then convert from the specified charset into UTF-8 via iconv() or something. However, due to the fun brokenness that I mentioned above in (2c) and (2d), this becomes more complicated.<br />
<p>If that isn't enough to make you want to throw your hands up in the air and mutter some profanities, there's more...</p></li>
<li>Undeclared 8bit text in headers. Yep. Some mailers just didn't get the memo that they are supposed to encode non-ASCII text. So now you get to have the fun experience of mixing and matching undeclared 8bit text of God-only-knows what charset along with the content of (probably broken) encoded-words.</li>
</ol><p>That said, I was able to help the notmuch developers solve this problem by letting them know about the <span style="font-size: x-small;"><span style="font-family: "Courier New",Courier,monospace;">GMIME_ENABLE_RFC2047_WORKAROUNDS</span></span> flag that they could pass to <span style="font-family: "Courier New",Courier,monospace;"><span style="font-size: x-small;">g_mime_init(guint32 flags)</span></span>.</p><p>Any developer reading this blog post and thinking that they want to see how this is done in <a href="http://spruce.sourceforge.net/gmime/">GMime</a>, the source code for the rfc2047 decoder is located <a href="https://git.gnome.org/browse/gmime/tree/gmime/gmime-utils.c#n1664">here</a>. If the line numbers change in the future, just grep around for "rfc2047_token" and you should find it.</p><p>In other news... I cranked out a ton more code for <a href="http://github.com/jstedfast/MimeKit">MimeKit</a> (my C# MIME parser library) yesterday. Yes, I know... I've got a serious problem with masochism having already written 2 MIME parsers and now I'm working on a third. When will the hurting stop? Never!</p><p>Oh, I guess I could point people at MimeKit's rfc2047 decoders. What you'll want to look at is <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/Utils/Rfc2047.cs#L613">MimeKit.Rfc2047.DecodePhrase(byte[] phrase)</a> and <a href="https://github.com/jstedfast/MimeKit/blob/master/MimeKit/Utils/Rfc2047.cs#L682">MimeKit.Rfc2047.DecodeText(byte[] text)</a>.</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-82947225039254616702012-12-11T16:44:00.002-05:002012-12-11T17:11:02.791-05:00HOWTO: MonoTouch Enterprise Deployment<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-FZ3v85HRW88/UMeaxEKa2vI/AAAAAAAAAs0/gGkb23cYENo/s1600/picard.jpg" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" height="244" src="http://4.bp.blogspot.com/-FZ3v85HRW88/UMeaxEKa2vI/AAAAAAAAAs0/gGkb23cYENo/s320/picard.jpg" width="320" /></a></div>Okay, so you've gotten the "Engage" hand-gesture from your Captain to deploy your MonoTouch app to the rest of the crew of the Enterprise. Now all you need to know is which buttons to press on your helm...<br />
<br />
<br />
<b>Step 1.</b><br />
<br />
First, you'll need to make sure that you've created and installed your "In-House" Distribution Certificate via Apple's <a href="https://developer.apple.com/ios/manage/certificates/team/distribute.action" target="_blank">iOS Provisioning Portal</a>.<br />
<br />
<br />
<b>Step 2.</b><br />
<br />
Open your <i>Project Options</i> in MonoDevelop and navigate to the <i>iPhone Bundle Signing</i> section.<br />
<br />
If you've got MonoDevelop 3.1.0 or later, you'll be able to set your configuration to:<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><img border="0" src="https://lh6.googleusercontent.com/-dhjoGAK66k4/UMefNMkHDVI/AAAAAAAAAtg/8LG3pO8w9nM/s555/MonoDevelop-3.1.0-Enterprise.png" /></div><br />
Otherwise you'll simply have to select your <i>Provisioning Profile</i> manually.<br />
<br />
Once you've selected your signing certificate and provisioning profile, click the <i>OK</i> button to dismiss the <i>Project Options</i> dialog.<br />
<br />
<br />
<b>Step 3.</b><br />
<br />
In MonoDevelop, click on the <i>Build</i> menu and select <i>Archive</i>. This will build your project and archive it in a location that Xcode will be able to see it in its <i>Organizer</i> window.<br />
<br />
<br />
<b>Step 4.</b><br />
<br />
Launch Xcode and then click on the <i>Window</i> menu and select <i>Organizer</i>. At the top of Xcode's Organizer window, you will see an array of icons. Click on the one labeled <i>Archives</i>.<br />
<br />
Find your application in the list of archives and select it.<br />
<br />
<br />
<div class="separator" style="clear: both; text-align: center;"><a href="http://4.bp.blogspot.com/-W_P4YSyZG14/UMeaFXFdDzI/AAAAAAAAAso/tPvU5IJhRQo/s1600/StarTrekEnterprise1701A_freedesktopwallpaper_800.jpg" imageanchor="1" style="clear: right; float: right; margin-bottom: 1em; margin-left: 1em;"><img border="0" height="300" src="http://4.bp.blogspot.com/-W_P4YSyZG14/UMeaFXFdDzI/AAAAAAAAAso/tPvU5IJhRQo/s400/StarTrekEnterprise1701A_freedesktopwallpaper_800.jpg" width="400" /></a></div><b>Step 5.</b><br />
<br />
Now click on the <i>Distribute...</i> button in the top-right area of the window and select <i>Save for Enterprise or Ad-Hoc Deployment</i>.<br />
<br />
The next screen will prompt you for your code-signing certificate, providing you with a drop-down menu listing your available options.<br />
<br />
Clicking <i>Next</i> will cause an action sheet to slide into view, prompting you for the location to save the AppName.ipa package and the AppName.plist file.<br />
<br />
Important: Make sure to toggle the <i>Save for Enterprise Distribution</i> checkbox.<br />
<br />
Once you've finished filling out all of the fields, click on the <i>Save</i> button.<br />
<br />
<br />
<b>Step 6.</b><br />
<br />
You'll need to upload the saved AppName.ipa and AppName.plist files to your corporate web server in the location that you specified in the previous step. You'll also need a web page that will link to your app using a hyperlink similar to the one below:<br />
<br />
<a href="itms-services://?action=download-manifest&url=http://internal.mycompany.com/Applications/AppName.plist">Install AppName!</a><br />
<br />
That's it! You're done!<br />
<br />
Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com2tag:blogger.com,1999:blog-203063759820106893.post-26671402970274785692012-08-30T20:37:00.000-04:002012-09-02T09:35:21.624-04:00A Better Alternative to the TSA?<p>Most everyone agrees that going through airport security and being groped by the TSA is not only offensive, but also a major nuisance.</p><p>How about replacing the TSA with privately run airport security?</p><p>It sounds like San Francisco travelers much prefer their privately run airport security than the TSA at all other US airports.</p><iframe width="560" height="315" src="http://www.youtube.com/embed/h6BbowVpcFo" frameborder="0" allowfullscreen></iframe>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com5tag:blogger.com,1999:blog-203063759820106893.post-63224837743611971962012-06-17T19:12:00.002-04:002012-06-17T19:12:50.200-04:00Beware Inflation<p><blockquote>By a continuing process of inflation, government can confiscate, secretly and unobserved, an important part of the wealth of their citizens.</blockquote></p><p align="right">-- John Maynard Keynes, <a href="http://www.gutenberg.org/ebooks/15776">The Economic Consequences of the Peace</a></p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com5tag:blogger.com,1999:blog-203063759820106893.post-53666567686180532892012-05-02T19:59:00.001-04:002012-05-02T20:59:03.545-04:00Social Security vs. Private Retirement<iframe allowfullscreen="" frameborder="0" height="315" src="http://www.youtube.com/embed/PLTfOAYfbao" width="560"></iframe>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com12tag:blogger.com,1999:blog-203063759820106893.post-83245368189568890312012-03-17T16:41:00.000-04:002012-03-19T08:55:00.425-04:00Introducing MonoTouch.SQLite<p>I've been working on a personal side-project writing an app for the <a href="http://www.amazon.com/gp/product/B0047DVWZS?ie=UTF8&tag=amoofze-20&linkCode=shr&camp=213733&creative=393185&creativeASIN=B0047DVWZS&ref_=sr_1_7&qid=1332010993&sr=8-7">iPad</a> that makes use of <a href="http://code.google.com/p/sqlite-net/">SQLite-Net</a>, displaying that data in a <a href="https://developer.apple.com/library/ios/#documentation/UIKit/Reference/UITableView_Class/Reference/Reference.html#//apple_ref/doc/uid/TP40006943">UITableView</a>.</p><p>Up until this past week, I had been using <a href="http://tirania.org/blog">Miguel de Icaza</a>'s wonderful <a href="https://github.com/migueldeicaza/MonoTouch.Dialog">MonoTouch.Dialog</a> library for displaying my data. Unfortunately, I wanted search filtering to be persistent, which means that I really needed to use Apple's <a href="https://developer.apple.com/library/ios/#documentation/UIKit/Reference/UISearchDisplayController_Class/Reference/Reference.html#//apple_ref/doc/uid/TP40008225">UISearchDisplayController</a> but I couldn't find an easy way to retrofit that onto MonoTouch.Dialog's DialogViewController to replace the simpler <a href="https://developer.apple.com/library/ios/#documentation/UIKit/Reference/UISearchBar_Class/Reference/Reference.html#//apple_ref/doc/uid/TP40007529">UISearchBar</a> API that it currently uses. Since I had to look at creating an alternate solution, I figured I might as well solve the other potential problem I had with MonoTouch.Dialog, which is that my app really needed to be able to handle tables with a massive number of items. This brings us to...</p><h3>MonoTouch.SQLite</h3><p>MonoTouch.Dialog really made using UITableViews in iPhone and iPad apps trivial and I wanted to try and repeat at least some of that with MonoTouch.SQLite.</p><p>The first thing I had to do was to figure out a way of modeling the data in such a way as to allow a generic class to do most of the work for the developer. After a few sleepless nights of hacking last weekend, I figured out a fairly simple approach that seems to work pretty well. I started off thinking that I wouldn't be able to get around having to have a subclassable model, so I made most everything virtual. This is the public API that I came up with:</p><pre style="font-family: courier new; font-size: 85%">public class SQLiteTableModel<T> : IDisposable where T : new ()
{
public SQLiteTableModel (SQLiteConnection sqlitedb, int pageSize, SQLiteOrderBy orderBy, string sectionExpr);
// 2 ways of setting the search criteria
public SQLiteWhereExpression SearchExpression { get; set; }
public string SearchText { get; set; }
// Gets the total number of table rows
public int Count { get; }
// Gets the number of table sections
public int SectionCount { get; }
// Gets the section titles
public string[] SectionTitles { get; }
// Gets the row count for a particular section
public int GetRowCount (int section);
// Get the index of an item
public int IndexOf (T item, IComparer<t> comparer);
// Convert item index into a section and row
public bool IndexToSectionAndRow (int index, out int section, out int row);
// Convert section and row into an item index
public int SectionAndRowToIndex (int section, int row);
// 2 ways of getting an item
public T GetItem (int section, int row);
public T GetItem (int index);
// Reset the state of the model
public void ReloadData ();
}
</pre><p><small>You can see the full class implementation <a href="https://github.com/jstedfast/MonoTouch.SQLite/blob/master/MonoTouch.SQLite/SQLiteTableModel.cs">here</a>.</small></p><p>It turns out, though, that it really isn't necessary to subclass my model unless you want to have finer control over the specific SQL query commands that it makes (all of those methods are virtual).</p><h3>SQLiteTableViewController<T></h3><p>As I started porting my iPad app to use my SQLiteTableModel class, I started to realize that I could abstract a lot of my usage of the model into a reusable base class. What I came up with will blow your mind.</p><p>Are you ready?</p><p>In order to populate a UITableView with the contents of an SQLite table, all you have to do is subclass <a href="https://github.com/jstedfast/MonoTouch.SQLite/blob/master/MonoTouch.SQLite/SQLiteTableViewController.cs">SQLiteTableViewController<T></a> and implement 1 method:</p><pre style="font-family: courier new; font-size: 85%">protected UITableViewCell GetCell (UITableView tableView, NSIndexPath path, T item)
</pre><p>That's it.</p><p>I've written up a <a href="https://github.com/jstedfast/MonoTouch.SQLite/blob/master/Sample/SampleTableViewController.cs">sample iPhone app</a> that illustrates just how easy this is.</p><h3>But wait! There's more!</h3><p>If you order in the next 30 minutes, you can also get this free complimentary <i>Sham-Wow!</i></p><p>Okay, just kidding about that <i>Sham-Wow!</i> bit, but I wasn't kidding about there being more:</p><p>Remember when I said one of the problems I wanted to solve was persistent search filtering? Yea, well, I did it. SQLiteTableViewController handles all of that for you as well. In fact, give searching a try in that sample above.</p><p>At this point I bet you're thinking, "wow, how could this get any better?"</p><p>I'll tell you. Remember how SQLiteTableModel had 2 methods for setting the search criteria? Well, the one that takes a string parses it to create a SQLiteWhereExpression allowing the user to match against specific fields. For example, if you had the following data item:</p><pre style="font-family: courier new; font-size: 85%">public class Contact {
public string FirstName;
public string LastName;
public string PhoneNumber;
public string Address;
public string Comments;
}
</pre><p>...the user could type:</p><pre style="font-family: courier new; font-size: 85%">address:"Newton, MA"</pre><p>and SQLiteTableModel would construct a query to match "Newton, MA" against only the <i>Address</i> field.</p><p>If the user, instead, types:</p><pre style="font-family: courier new; font-size: 85%">address:"Newton, MA" firstname:Jane</pre><p>then the matches that would display in the list would be limited to contacts with a first name of "Jane" who live also in "Newton, MA".</p><p>I've also taken the liberty of implementing a SQLiteSearchAliasAttribute that allows you to specify aliases for your fields (or even the same alias to multiple fields!). For example, you could do this:</p><pre style="font-family: courier new; font-size: 85%">public class Contact {
[SQLiteSearchAlias ("first")][SQLiteSearchAlias ("name")]
public string FirstName;
[SQLiteSearchAlias ("last")][SQLiteSearchAlias ("name")]
public string LastName;
[SQLiteSearchAlias ("phone")]
public string PhoneNumber;
public string Address;
public string Comments;
}
</pre><p>This would allow your users to use "name" to match against either FirstName <i>or</i> LastName!</p>It also means they can type "first" instead of "firstname" to match against only the first name, as in the above example.</p><h3>Where Can I Find This Awesome Library?</h3><p>Glad you asked! You can find it on my GitHub page: <a href="https://github.com/jstedfast/MonoTouch.SQLite">MonoTouch.SQLite</a>.<br />
<p>Well? What are you waiting for? Get hacking!</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com9tag:blogger.com,1999:blog-203063759820106893.post-57213193785262455752012-02-12T23:16:00.002-05:002012-02-13T07:35:57.889-05:00Meet the Hackers<p>This past week, I've started to get back into photography a bit more (thanks, Nina!) and started taking my <a href="http://www.amazon.com/gp/product/B001EQ4BVI/ref=as_li_ss_il?ie=UTF8&tag=amoofze-20&linkCode=as2&camp=1789&creative=390957&creativeASIN=B001EQ4BVI">camera</a> into the office with me every day to remind myself to take photos. As a result, I've taken a bunch of photographs of my co-workers in the office.</p>
<p>Would you like to meet the hackers?</p>
<h3>The Founders</h3>
<p>Most of you would probably recognize the infamous <a href="http://tirania.org/blog">Miguel de Icaza</a>, Xamarin's CTO:</p>
<a href="http://www.flickr.com/photos/jstedfast/6854083369/" title="Miguel de Icaza by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7057/6854083369_ef0a95bee4.jpg" width="333" height="500" alt="Miguel de Icaza"></a>
<p>Next up is our very own Steve Jobs, <a href="http://nat.org">Nat Friedman</a>, our CEO and the man who reminds us to pay attention to the details:</p>
<a href="http://www.flickr.com/photos/jstedfast/6854085489/" title="Nat Friedman by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7150/6854085489_803b032c33.jpg" width="333" height="500" alt="Nat Friedman"></a>
<p>Another person many of you will recognize is our very own COO, Joseph Hill:</p>
<a href="http://www.flickr.com/photos/jstedfast/6854060617/" title="CUP<T> by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7040/6854060617_807cf98a15.jpg" width="333" height="500" alt="Joseph Hill"></a>
<p></p>
<h3>MonoDevelop Team</h3>
<p>Well, okay, I've only got a photo of the famous <a href="http://mjhutchinson.com/">Michael Hutchinson</a>, but he's a very important player in the development of MonoDevelop.</p>
<a href="http://www.flickr.com/photos/jstedfast/6854050117/" title="Michael Hutchinson by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7196/6854050117_702905f138.jpg" width="333" height="500" alt="Michael Hutchinson"></a>
<p></p>
<h3>QA Team</h3>
<p>Next up, we have the QA team. They do their best to make sure that we, the developers, didn't break anything. When they aren't testing a specific application before a launch, they hammer away at our products and try to find weak spots in our code (but we still love them anyway!)</p>
<p>This is PJ, and as you can see, he's demonstrating how to QA popcorn corn cobs:</p>
<a href="http://www.flickr.com/photos/jstedfast/6854044135/" title="PJ by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7039/6854044135_9063835a57.jpg" width="333" height="500" alt="PJ"></a>
<p>(Did it pass the test, PJ?)</p>
<p>Next up is Lindsey. She's been working on writing automated tests to make it less likely for releases to include regressions. Let's hope she's successful!</p>
<a href="http://www.flickr.com/photos/jstedfast/6854040949/" title="Lindsey by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7006/6854040949_a461ab0bba.jpg" width="333" height="500" alt="Lindsey"></a>
<p></p>
<h3>Release Team</h3>
<p><a href="https://twitter.com/#!/chknofthescene">Alex Corrado</a> is the man behind the curtain. He's our head Release Team engineer and also the brilliant mastermind that started <a href="http://tirania.org/blog/archive/2011/Dec-19.html">CXXI</a>, the Mono C++ interop project that we hope to give him time to finish someday soon.</p>
<a href="http://www.flickr.com/photos/jstedfast/6854070111/" title="Alex Corrado by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7004/6854070111_0715345702.jpg" width="333" height="500" alt="Alex Corrado"></a>
<p></p>
<h3>Web Team</h3>
<p>The newest addition to our ranks (just this week, in fact!), but long-time contributor to the Mono project, is <a href="https://twitter.com/#!/bojanrajkovic">Bojan Rajković</a>. You can see we've already put him to work (he is no doubt puzzling over some ASP.NET code on his screen).</p>
<a href="http://www.flickr.com/photos/jstedfast/6854077349/" title="Bojan Rajković by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7018/6854077349_d9a1c5c627.jpg" width="333" height="500" alt="Bojan Rajković"></a>
<p></p>
<h3>Documentation Team</h3>
<p>Nina is the only Cambridge resident on our Docs Team. Specifically, she hacks on our Documentation Portal. She's also the one who has encouraged me to get back into taking photographs, so she'll have to put up with me using her as a guinea pig the most. Here she is taunting me with her hot cup of Chaider:</p>
<a href="http://www.flickr.com/photos/jstedfast/6854052797/" title="Nina by jstedfast, on Flickr"><img src="http://farm8.staticflickr.com/7024/6854052797_de6f92016f.jpg" width="333" height="500" alt="Nina"></a>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com4tag:blogger.com,1999:blog-203063759820106893.post-74429377339068584842011-09-15T17:01:00.000-04:002011-09-15T17:01:47.436-04:00MonoTouch Tips & Tricks: Updating the Location of an MKAnnotation<p>I just spent a day figuring this out, so figured I'd share it with the world because I'm <i>sure</i> other people are going to want to know how to do this...</p>
<p>So the question is,</p>
<p><h3>How can I get my MKMapView to respond to coordinate changes in my custom MKAnnotations?</h3></p>
<p>As it turns out, this is incredibly simple. In your MKAnnotation subclass, whenever you want to change your Coordinate property value, you need to do the following:</p>
<pre style="font-family: courier new; font-size: 85%">
void UpdateCoordinate (CLLocationCoordinate2D newCoordinate)
{
this.WillChangeValue ("coordinate");
this.Coordinate = newCoordinate;
this.DidChangeValue ("coordinate");
}
</pre>
<p>That's it! It really is that simple...</p>
<p>The reason this works is because MKMapView observes changes in its list of MKAnnotations, you just need to signal to it that changes are about to happen (and did happen).</p>
<p>Happy hacking!</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-8528801042250635242011-08-14T15:40:00.002-04:002011-08-14T15:55:56.169-04:00GMime 2.5.10: A Call For Testers<div class="separator" style="clear: both; text-align: left;">
<a href="http://www.lolcats.com/images/u/07/25/lolcatsdotcom9cryvuajyxzz31eu.jpg" imageanchor="1" style="clear:right; float:right;margin-right:1em; margin-bottom:1em"><img border="0" height="175" width="174" src="http://www.lolcats.com/images/u/07/25/lolcatsdotcom9cryvuajyxzz31eu.jpg" /></a></div>
<p>I've just released <a href="http://spruce.sourceforge.net/gmime/">GMime</a> <a href="http://download.gnome.org/sources/gmime/2.5/gmime-2.5.10.tar.bz2">2.5.10</a> which I hope to be the last of the 2.5 releases before I release 2.6.0. I feel that I've stretched the development of 2.6.0 out for far too long (2.5 development began at the end of April, 2009) and even though I didn't get around to doing everything I had hoped to do, I feel that the latest 2.5.x releases are such an improvement over 2.4.x that I just want to get it out there for developers to start using. But before I make a 2.6.0 release, I'm hoping to get some feedback and some testing.</p>
<h3>What's new?</h3>
<p>New for the release of 2.5.10 is <b>GMimePartIter</b> which replaces the need for <i style="font-family: courier new; font-size: 85%">g_mime_object_foreach()</i>and its awkward callback requirement, instead allowing you to take the far nicer iterator approach that is popular in the C# and Java worlds (known as <i>IEnumerator</i> in C#). This new iterator, like the foreach function it replaces, iterates over the MIME tree structure in <a href="http://en.wikipedia.org/wiki/Depth-first">depth-first</a> order.</p>
<p>Inspired by <a href="http://www.ietf.org/rfc/rfc2060.txt">IMAP</a>'s FETCH body part-specifier syntax, I've implemented a method allowing you to jump to a part based on a part-specifier string (aka a path): <i style="font-family: courier new; font-size: 85%">g_mime_part_iter_jump_to()</i>. Also implemented is a function called <i style="font-family: courier new; font-size: 85%">g_mime_part_iter_get_path()</i>, which can be used to tell you the current part-specifier path of the iterator.</p>
<p>For example, if you had the following MIME message structure:</p>
<pre style="font-family: courier new; font-size: 85%">
multipart/related
multipart/alternative
text/plain
text/html
image/jpeg
</pre>
<p>The body part-specifier paths would be:</p>
<pre style="font-family: courier new; font-size: 85%">
1 multipart/alternative
1.1 text/plain
1.2 text/html
2 image/jpeg
</pre>
<p>This means that <i style="font-family: courier new; font-size: 85%">g_mime_part_iter_jump_to(iter, "1.2")</i> would jump to the part specified by the path "1.2" which, as we can see above, would be the text/html part. Calling <i style="font-family: courier new; font-size: 85%">g_mime_part_iter_next(iter)</i> would iterate to the next part, being the image/jpeg, while calling <i style="font-family: courier new; font-size: 85%">g_mime_part_iter_prev(iter)</i> would iterate backwards to the text/plain part and calling it again would iterate backwards to the multipart/alternative.</p>
<h3>What I Need From Testers</h3>
<p>My feeling is that developers will want to use this cool new body part-specifier path functionality for aiding them in implementing IMAP servers and/or clients. Because of this, it would be great if GMime's implementation matched IMAP's specification exactly. The problem is that I don't have the time or energy to verify that the paths work out to be identical in all cases. So... if you are one of those developers who is interested in using this functionality and need it to be identical to IMAP's syntax (or would really like it to be), I'm hoping that you could test it out and make sure that it matches. Especially worthwhile of testing, I'd imagine, is having message/rfc822 parts in the tree. I suspect that, if anywhere, this is where differences may be.</p>
<p>If body part-specifier paths aren't something you care about, don't fret; the rest of the iterator API needs testing as well and if you have no interest in the iterator API at all, perhaps you'd be willing to test the S/MIME functionality (especially since I haven't figured out <i>how</i> to test it myself, given that I don't have an S/MIME cert nor have I figured out how to generate one or add one to my gpgsm keyring).</p>
<p>Your help will be greatly appreciated.</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0tag:blogger.com,1999:blog-203063759820106893.post-82227656587980005532011-08-03T11:42:00.007-04:002011-08-03T15:39:01.829-04:00Debugging Your MonoTouch Apps: The Future<p>One of the "paper cuts" developers have been having with developing their <a href="http://ios.xamarin.com/">MonoTouch</a> 4.0.x (and earlier) applications is that for some networking setups, the IP of the developer's workstation detected by <a href="http://monodevelop.com/">MonoDevelop</a> and given to the <a href="http://is.gd/Nr2hD4">iPhone</a> or <a href="http://is.gd/DiYoT5">iPad</a> device for debugging purposes is not correct. This often happens if the WiFi is a different network than the network that the developer's machine is connected to (although there are other scenarios as well).</p>
<div class="separator" style="clear: both;">
<a href="http://3.bp.blogspot.com/-I_zA4pkHMSk/TjlsF448kKI/AAAAAAAAAq0/CIMEoSvm6D8/s1600/MonoTouchiPhoneDebugSettings.png" imageanchor="1" style="clear:right; float:left;margin-right:1em; margin-bottom:1em"><img border="0" height="400" width="206" src="http://3.bp.blogspot.com/-I_zA4pkHMSk/TjlsF448kKI/AAAAAAAAAq0/CIMEoSvm6D8/s400/MonoTouchiPhoneDebugSettings.png" /></a>
<p>Since it does not seem to be widely known about, allow me to point out that current versions of MonoTouch allow developers to modify the IP that the runtime should connect to for debugging via the iOS <b>Settings</b> app found on any iPhone or iPad (or Simulator). You can see a screenshot of this per-App Settings page in the screenshot to the left. Each of these fields are editable, allowing you to override the defaults filled-in by MonoDevelop.</p>
</p>For our upcoming 4.1 release, <a href="https://twitter.com/#!/rolfkvinge">Rolf Kvinge</a> and I (but mostly Rolf) have been working on improving this. Rolf has modified the code to check the value of the IP provided in the per-App Settings and if it is set to nil or "automatic", the debugger falls back to checking for a file bundled with the app called MonoTouchDebugConfiguration.txt which can list any number of IP's to try and connect to, each one being on a separate line prefixed with "IP: ". For example:</p>
<pre>
IP: 10.0.1.31
IP: 192.168.1.31
IP: 204.11.102.79
</pre>
<p>The runtime will then attempt to connect to each of these IPs asynchronously until it establishes a connection to one of them (at which point it aborts the other waiting connections). This config file solution will hopefully help simplify things for developers a bit by allowing them to pre-configure which IPs to try for their local network configuration w/o having to manually override the iPhone debug settings on the device or simulator.
</div>
<p>For <i>Phase 2</i> of our plan for World Domination, Rolf is hard at work adding support to MonoDevelop and the runtime to allow for USB debugging which will obsolete the above functionality in future versions where the developer has a MonoDevelop which supports USB debugging. For developers stuck on an older MonoDevelop (like 2.4), the solution illustrated above requires no changes to MonoDevelop and so will be available for use.</p>Jeffrey Stedfasthttp://www.blogger.com/profile/12271561115384429651noreply@blogger.com0