Achievement Unlocked: MimeKit and MailKit in official Microsoft docs

MailKit 1.14 released

I am pleased to announce the release of MailKit 1.14!

See below for a list of new features and bug fixes.

About MailKit

MailKit is a C# library which is built on top of MimeKit and is intended to be used for interfacing with IMAP, POP3 and SMTP servers.

MailKit features incredibly robust IMAP, POP3 and SMTP clients with network APIs that are all capable of being canceled. API's that might transfer significant amounts of data between the client and server also include the ability to report progress. Asynchronous API's are also available.

Built on top of .NET, MailKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.

Noteworthy changes in version 1.14

• Improved IMAP's BODYSTRUCTURE parser to sanitize the Content-Disposition values. (issue #486)
• Improved robustness of IMAP's BODYSTRUCTURE parser in cases where qstring tokens have unescaped quotes. (issue #485)
• Fixed IMAP to properly handle NIL as a folder name in LIST, LSUB and STATUS responses. (issue #482)
• Added ImapFolder.GetHeaders() to allow developers to download the entire set of message headers.
• Added SMTP support for International Domain Names in email addresses used in the MAIL FROM and RCPT TO commands.
• Modified SmtpClient to no longer throw a NotSupportedException when trying to send messages to a recipient with a unicode local-part in the email address when the SMTP server does not support the SMTPUTF8 extension. Instead, the local-part is passed through as UTF-8, leaving it up to the server to reject either the command or the message. This seems to provide the best interoperability.

Installing via NuGet

The easiest way to install MailKit is via NuGet.

In Visual Studio's Package Manager Console, simply enter the following command:

Install-Package MailKit

Getting the Source Code

First, you'll need to clone MailKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:

git clone --recursive https://github.com/jstedfast/MailKit.git

Documentation

API documentation can be found at http://mimekit.net/docs.

A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.

Code Review: Microsoft's System.Net.Mail Implementation

For those reading my blog for the first time and don't know who I am, allow myself to introduce... myself.

I'm a self-proclaimed expert on the topic of email, specifically MIME, IMAP, SMTP, and POP3. I don't proclaim myself to be an expert on much, but email is something that maybe 1 or 2 dozen people in the world could probably get away with saying they know more than I do and actually back it up. I've got a lot of experience writing email software over the past 15 years and rarely do I come across mail software that does things better than I've done them. I'm also a critic of mail software design and implementation.

My latest endeavors in the email space are MimeKit and MailKit, both of which are open source and available on GitHub for your perusal should you doubt my expertise.

My point is: I think my review carries some weight, or I wouldn't be writing this.

Is that egotistical of me? Maybe a little.

I was actually just fixing a bug in MimeKit earlier and when I went to go examine Mono's System.Net.Mail.MailMessage implementation in order to figure out what the problem was with my System.Net.Mail.MailMessage to MimeKit.MimeMessage conversion, I thought, "hey, wait a minute... didn't Microsoft just recently release their BCL source code?" So I ended up taking a look and pretty quickly confirmed my suspicions and was able to fix the bug.

When I begin looking at the source code for another mail library, I can't help but critique what I find.

MailAddress and MailAddressCollection

Parsing email addresses is probably the hardest thing to get right. It's what I would say makes or breaks a library (literally). To a casual onlooker, parsing email addresses probably seems like a trivial problem. "Just String.Split() on comma and then look for those angle bracket thingies and you're done, right?" Oh God, oh God, make the hurting stop. I need to stop here before I go into a long rant about this...

Okay, I'm back. Blood pressure has subsided.

Looking at MailAddressParser.cs (the internal parser used by MailAddressCollection), I'm actually pleasantly surprised. It actually looks pretty decent and I can tell that a lot of thought and care went into it. They actually use a tokenizer approach. Interestingly, they parse the string in reverse which is a pretty good idea, I must say. This approach probably helps simplify the parser logic a bit because parsing forward makes it difficult to know what the tokens belong to (is it the name token? or is it the local-part of an addr-spec? hard to know until I consume a few more tokens...).

For example, consider the following BNF grammar:

address         =       mailbox / group
mailbox         =       name-addr / addr-spec
name-addr       =       [display-name] angle-addr
angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]
display-name    =       phrase
word            =       atom / quoted-string
phrase          =       1*word / obs-phrase
addr-spec       =       local-part "@" domain
local-part      =       dot-atom / quoted-string / obs-local-part
domain          =       dot-atom / domain-literal / obs-domain
obs-local-part  =       word *("." word)

Now consider the following email address: "Joe Example" <joe@example.com>

The first token you read will be "Joe Example" and you might think that that token indicates that it is the display name, but it doesn't. All you know is that you've got a 'quoted-string' token. A 'quoted-string' can be part of a 'phrase' or it can be (a part of) the 'local-part' of the address itself. You must read at least 1 more token before you'll be able to figure out what it actually is ('obs-local-part' makes things slightly more difficult). In this case, you'll get a '<' which indicates the start of an 'angle-addr', allowing you to assume that the 'quoted-string' you just got is indeed the 'display-name'.

If, however, you parse the address in reverse, things become a little simpler because you know immediately what to expect the next token to be a part of.

That's pretty cool. Kudos to the Microsoft engineers for thinking up this strategy.

Unfortunately, the parser does not handle the 'group' address type. I'll let this slide, however, partly because I'm still impressed by the approach the address parser took and also because I realize that System.Net.Mail is meant for creating and sending new messages, not parsing existing messages from the wild.

Okay, so how well does it serialize MailAddress?

Ugh. You know that face you make when you just see a guy get kicked in the nuts? Yea, that's the face I made when I saw line #227:

encodedAddress = String.Format(CultureInfo.InvariantCulture, "\"{0}\"", this.displayName);

The problem with the above code (and I'll soon be submitting a bug report about this) is that the displayName string might have embedded double quotes in it. You can't just surround it with quotes and expect it to work. This is the same mistake all those programmers make that allow SQL-injection attacks.

For an example of how this should be done, see MimeKit's MimeUtils.Quote() method.

I had such high hopes... at least this is a fairly simple bug to fix. I'll probably just offer them a patch.

ContentType and ContentDisposition

Their parser is decent but it doesn't handle rfc2231 encoded parameter values, so I'm not overly impressed. It'll get the job done for simple name="value" parameter syntax, though, and it will decode the values encoded with the rfc2047 encoding scheme (which is not the right way to encode values, but it is common enough that any serious parser should handle it). The code is also pretty clean and uses a tokenizer approach, so that's a plus. I guess since this isn't really meant as a full-blown MIME parser, they can get away with this and not have it be a big deal. Fair enough.

Serialization, unsurprisingly, leaves a lot to be desired. Parameter values are, as I expected, encoded using rfc2047 syntax rather than the IETF standard rfc2231 syntax. I suppose that you could argue that this is for compatibility, but really it's just perpetuating bad practices. It also means that it can't properly fold long parameter values because the encoded value just becomes one big long encoded-word token. Yuck.

Base64

Amusingly, Microsoft does not use their Convert.FromBase64() decoder to decode base64 in their System.Net.Mail implementation. I point this out mostly because it is the single most common problem users have with every one of the Open Source .NET mail libraries out there (other than MimeKit, of course) because Convert.FromBase64() relies on the data not having any line breaks, white space, etc in the input stream.

This should serve as a big hint to you guys writing your own .NET email libraries not to use Convert.FromBase64() ;-)

They use unsafe pointers, just like I do in MimeKit, but I'm not sure how their performance compares to MimeKit's yet. They do use a state machine, though, so rock on.

I approve this base64 encoder/decoder implementation.

SmtpClient

One thing they do which is pretty cool is connection pooling. This is probably a pretty decent win for the types of things developers usually use System.Net.Mail's SmtpClient for (spam, anyone?).

The SASL AUTH mechanisms that they seem to support are NTLM, GSSAPI, LOGIN and WDIGEST (which apparently is some sort of IIS-specific authentication mechanism that I had never heard of until now). For those that were curious which SASL mechanisms SmtpClient supported, well, now you know.

The code is a bit hard to follow for someone not familiar with the codebase (not nearly as easy reading as the address or content-type parsers, I'm afraid), but it seems fairly well designed.

It does not appear to support PIPELINING or BINARYMIME like MailKit does, though. So, yay! Win for MailKit ;-)

They do support SMTPUTF8, so that's good.

It seems that if you set client.EnableSsl to true, it will also try STARTTLS if it isn't able to connect on the SSL port. I wasn't sure if it did that or not before, so this was something I was personally interested in knowing.

Hopefully my SmtpClient implementation review isn't too disappointing. I just don't know what to say about it, really. It's a pretty straight-forward send-command-wait-for-reply implementation and SMTP is pretty dead simple.

Conclusion

Overall the bits I was interested in were better than I expected they'd be. The parsers were pretty good (although incomplete) and the serializers were "good enough" for normal use.

Of course, it's not as good as MimeKit, but let's be honest, MimeKit sets the bar pretty high ;-)

The Wait Is Over: MimeKit and MailKit Reach 1.0

After about a year in the making for MimeKit and nearly 8 months for MailKit, they've finally reached 1.0 status.

I started really working on MimeKit about a year ago wanting to give the .NET community a top-notch MIME parser that could handle anything the real world could throw at it. I wanted it to run on any platform that can run .NET (including mobile) and do it with remarkable speed and grace. I wanted to make it such that re-serializing the message would be a byte-for-byte copy of the original so that no data would ever be lost. This was also very important for my last goal, which was to support S/MIME and PGP out of the box.

All of these goals for MimeKit have been reached (partly thanks to the BouncyCastle project for the crypto support).

At the start of December last year, I began working on MailKit to aid in the adoption of MimeKit. It became clear that without a way to inter-operate with the various types of mail servers, .NET developers would be unlikely to adopt it.

I started off implementing an SmtpClient with support for SASL authentication, STARTTLS, and PIPELINING support.

Soon after, I began working on a Pop3Client that was designed such that I could use MimeKit to parse messages on the fly, directly from the socket, without needing to read the message data line-by-line looking for a ".\r\n" sequence, concatenating the lines into a massive memory buffer before I could start to parse the message. This fact, combined with the fact that MimeKit's message parser is orders of magnitude faster than any other .NET parser I could find, makes MailKit the fastest POP3 library the world has ever seen.

After a month or so of avoiding the inevitable, I finally began working on an ImapClient which took me roughly two weeks to produce the initial prototype (compared to a single weekend for each of the other protocols). After many months of implementing dozens of the more widely used IMAP4 extensions (including the GMail extensions) and tweaking the APIs (along with bug fixing) thanks to feedback from some of the early adopters, I believe that it is finally complete enough to call 1.0.

In July, at the request of someone involved with a number of the IETF email-related specifications, I also implemented support for the new Internationalized Email standards, making MimeKit and MailKit the first - and only - .NET email libraries to support these standards.

If you want to do anything at all related to email in .NET, take a look at MimeKit and MailKit. I guarantee that you will not be disappointed.

Introducing MailKit, a cross-platform .NET mail-client library

Once I announced MimeKit, I knew it would only be a matter of time before I started getting asked about SMTP, IMAP, and/or POP3 support.

Let's just say,

Challenge... ACCEPTED!

I started off back in early December writing an SmtpClient so that developers using MimeKit wouldn't have to convert a MimeMessage to a System.Net.Mail.MailMessage in order to send it using System.Net.Mail.SmtpClient. This went pretty quickly because I've implemented several SMTP clients in the past. Implementing the various SASL authentication mechanisms probably took as much or more time than implementing the SMTP protocol.

The following weekend, I ended up implementing a Pop3Client. Originally, I had planned on more-or-less cloning the API we had used in Evolution, but I decided that I would take a different approach. I designed a simple IMessageSpool interface which more closely follows the limited functionality of POP3 and mbox spools instead of trying to map the Pop3Client to a Store/Folder paradigm like JavaMail and Evolution do (Evolution's mail library was loosely based on JavaMail). Mapping mbox and POP3 spools to Stores and Folders in Evolution was, to my recollection, rather awkward and I wanted to avoid that with MailKit.

At first I was loathe to do it, but over the past 2 weeks I ended up writing an ImapClient as well. I'm sure Philip van Hoof will be pleased to note that I have a very nice BODYSTRUCTURE parser, although that API is not publicly exported.

Unlike the SmtpClient and Pop3Client, the ImapClient does not have all of its functionality on a single public class. Instead, ImapClient implements an IMessageStore which has a limited API, mostly meant for getting IFolders. I imagine that those who are familiar with the JavaMail and/or Evolution (Camel) APIs will recognize this design.

The IFolder interface isn't designed to be exactly like the JavaMail Folder API, though. I've been designing the interface incrementally as I implement the various IMAP extensions (I've found at least 37 of them at the time of this blog post, although I don't think I'll bother with ACL, MAILBOX-REFERRAL, or LOGIN-REFERRAL), so the API may continue to evolve as I go, but I think what I've got now will likely remain - I'll probably just be including additional APIs for the new stuff.

So far, I've implemented the following IMAP extensions: LITERAL+, NAMESPACE, CHILDREN, LOGIN-DISABLED, STARTTLS, MULTIAPPEND, UNSELECT, UIDPLUS, CONDSTORE, ESEARCH, SASL-IR, SORT, THREAD, SPECIAL-USE, MOVE, XLIST, and X-GM-EXT1. Phew, that was exhausting listing all of those!

Also news-worthy is that MimeKit is now equally as fast as GMime, which is pretty impressive considering that it is fully managed C# code.

Download MailKit 0.2 now and let the hacking begin!