Interview with Hawk Newsome of Black Lives Matter

EXCLUSIVE: Texas Massacre Hero, Stephen Willeford, Describes Stopping Gunman

To donate to the Sutherland Springs Baptist Church to help them recover from this tragedy, check out this GoFundMe campaign.

Achievement Unlocked: MimeKit and MailKit in official Microsoft docs

GMime 2.99.0 released

After a long hiatus, I am pleased to announce the release of GMime 2.99.0!

See below for a list of new features and bug fixes.

About GMime

GMime is a C library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by numerous IETF specifications.

GMime features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated GnuPG and S/MIME v3.2 support.

Built on top of GObject (the object system used by the GNOME desktop), many developers should find its API design and memory management very familiar.

Noteworthy changes in version 2.99.0

• Overhauled the GnuPG support to use GPGME under the hood rather than a custom wrapper.
• Added S/MIME support, also thanks to GPGME.
• Added International Domain Name support via GNU's libidn.
• Improved the GMimeMessage APIs for accessing the common address headers. They now all return an InternetAddressList.
• g_mime_init() no longer takes any flag arguments and the g_mime_set_user_charsets() API has also been dropped. Instead, GMimeParserOptions and GMimeFormatOptions have taken the place of these APIs to allow customization of various parser and formatting options in a much cleaner way. To facilitate this, many parsing functions and formatting functions have changed to now take these options arguments.
• InternetAddress now has a 'charset' property that can be set to override GMime's auto-detection of the best charset to use when encoding names.
• GMimeHeaderIter has been dropped in favor of a much simpler index-based API on GMimeHeaderList.
• GMimeHeaderList no longer caches the raw message/mime headers in a stream. Instead, each GMimeHeader now has its own cache. This means that changing the GMimeHeaderList or any of its GMimeHeaders no longer invalidates the entire cache.
• GMimeParser has been fixed to preserve (munged or otherwise) From-lines that sometimes appear at the start of the content of message/rfc822 parts.
• GMimeParser now also scans for encapsulated PGP blocks within MIME parts as it is parsing them and sets a flag on each GMimePart that contains one of these blocks.
• GMimePart now has APIs for dealing with said encapsulated PGP blocks.

Developers interested in migrating to the upcoming GMime 3.0 API (of which GMime 2.99.0 is a preview) should take a look at the PORTING document included with the source code as it contains a fairly comprehensive list of the API changes that they will need to be aware of.

Getting the Source Code

You can download official public release tarballs of GMime at https://download.gnome.org/sources/gmime/ or ftp://ftp.gnome.org/pub/GNOME/sources/gmime/.

If you would like to contribute to the GMime project, it is recommended that you grab the source code from the official GitHub repository at https://github.com/jstedfast/gmime. Cloning this repository can be done using the following command:

git clone https://github.com/jstedfast/gmime.git

Documentation

API reference documentation can be found at https://developer.gnome.org/gmime/2.99/.

Documentation for getting started can be found in the README.md.

MailKit 1.14 released

I am pleased to announce the release of MailKit 1.14!

See below for a list of new features and bug fixes.

About MailKit

MailKit is a C# library which is built on top of MimeKit and is intended to be used for interfacing with IMAP, POP3 and SMTP servers.

MailKit features incredibly robust IMAP, POP3 and SMTP clients with network APIs that are all capable of being canceled. API's that might transfer significant amounts of data between the client and server also include the ability to report progress. Asynchronous API's are also available.

Built on top of .NET, MailKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.

Noteworthy changes in version 1.14

• Improved IMAP's BODYSTRUCTURE parser to sanitize the Content-Disposition values. (issue #486)
• Improved robustness of IMAP's BODYSTRUCTURE parser in cases where qstring tokens have unescaped quotes. (issue #485)
• Fixed IMAP to properly handle NIL as a folder name in LIST, LSUB and STATUS responses. (issue #482)
• Added ImapFolder.GetHeaders() to allow developers to download the entire set of message headers.
• Added SMTP support for International Domain Names in email addresses used in the MAIL FROM and RCPT TO commands.
• Modified SmtpClient to no longer throw a NotSupportedException when trying to send messages to a recipient with a unicode local-part in the email address when the SMTP server does not support the SMTPUTF8 extension. Instead, the local-part is passed through as UTF-8, leaving it up to the server to reject either the command or the message. This seems to provide the best interoperability.

Installing via NuGet

The easiest way to install MailKit is via NuGet.

In Visual Studio's Package Manager Console, simply enter the following command:

Install-Package MailKit

Getting the Source Code

First, you'll need to clone MailKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:

git clone --recursive https://github.com/jstedfast/MailKit.git

Documentation

API documentation can be found at http://mimekit.net/docs.

A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.

MimeKit 1.14 released

I am pleased to announce the release of MimeKit 1.14!

See below for a list of new features and bug fixes.

About MimeKit

MimeKit is a C# library which may be used for the creation and parsing of messages using the Multipurpose Internet Mail Extension (MIME), as defined by numerous IETF specifications.

MimeKit features an extremely robust high-performance parser designed to be able to preserve byte-for-byte information allowing developers to re-seralize the parsed messages back to a stream exactly as the parser found them. It also features integrated DKIM-Signature, S/MIME v3.2, OpenPGP and MS-TNEF support.

Built on top of .NET, MimeKit can be used with any of the .NET languages including C#, VB.NET, F#, and more. It will also run on any platform that Mono or the new .NET Core runtime have been ported to including Windows, Linux, Mac OS, Windows Phone, Apple TV, Apple Watch, iPhone/iPad, Xbox, PlayStation, and Android devices.

Noteworthy changes in version 1.14

• Added International Domain Name support for email addresses.
• Added a work-around for mailers that didn't provide a disposition value in a Content-Disposition header.
• Added a work-around for mailers that quote the disposition value in a Content-Disposition header.
• Added automatic key retrieval functionality for the GnuPG crypto context.
• Added a virtual DigestSigner property to DkimSigner so that consumers can hook into services such as Azure. (issue #296)
• Fixed a bug in the MimeFilterBase.SaveRemainingInput() logic.
• Preserve munged From-lines at the start of message/rfc822 parts.
• Map code page 50220 to iso-2022-jp.
• Format Reply-To and Sender headers as address headers when using Header.SetValue().
• Fixed MimeMessage.CreateFromMailMessage() to set the MIME-Version header. (issue #290)

Installing via NuGet

The easiest way to install MimeKit is via NuGet.

In Visual Studio's Package Manager Console, simply enter the following command:

Install-Package MimeKit

Getting the Source Code

First, you'll need to clone MimeKit from my GitHub repository. To do this using the command-line version of Git, you'll need to issue the following command in your terminal:

git clone --recursive https://github.com/jstedfast/MimeKit.git

Documentation

API documentation can be found at http://mimekit.net/docs.

A copy of the xml formatted API documentation is also included in the NuGet and/or Xamarin Component package.

Code Review: Microsoft's System.Net.Mail Implementation

For those reading my blog for the first time and don't know who I am, allow myself to introduce... myself.

I'm a self-proclaimed expert on the topic of email, specifically MIME, IMAP, SMTP, and POP3. I don't proclaim myself to be an expert on much, but email is something that maybe 1 or 2 dozen people in the world could probably get away with saying they know more than I do and actually back it up. I've got a lot of experience writing email software over the past 15 years and rarely do I come across mail software that does things better than I've done them. I'm also a critic of mail software design and implementation.

My latest endeavors in the email space are MimeKit and MailKit, both of which are open source and available on GitHub for your perusal should you doubt my expertise.

My point is: I think my review carries some weight, or I wouldn't be writing this.

Is that egotistical of me? Maybe a little.

I was actually just fixing a bug in MimeKit earlier and when I went to go examine Mono's System.Net.Mail.MailMessage implementation in order to figure out what the problem was with my System.Net.Mail.MailMessage to MimeKit.MimeMessage conversion, I thought, "hey, wait a minute... didn't Microsoft just recently release their BCL source code?" So I ended up taking a look and pretty quickly confirmed my suspicions and was able to fix the bug.

When I begin looking at the source code for another mail library, I can't help but critique what I find.

MailAddress and MailAddressCollection

Parsing email addresses is probably the hardest thing to get right. It's what I would say makes or breaks a library (literally). To a casual onlooker, parsing email addresses probably seems like a trivial problem. "Just String.Split() on comma and then look for those angle bracket thingies and you're done, right?" Oh God, oh God, make the hurting stop. I need to stop here before I go into a long rant about this...

Okay, I'm back. Blood pressure has subsided.

Looking at MailAddressParser.cs (the internal parser used by MailAddressCollection), I'm actually pleasantly surprised. It actually looks pretty decent and I can tell that a lot of thought and care went into it. They actually use a tokenizer approach. Interestingly, they parse the string in reverse which is a pretty good idea, I must say. This approach probably helps simplify the parser logic a bit because parsing forward makes it difficult to know what the tokens belong to (is it the name token? or is it the local-part of an addr-spec? hard to know until I consume a few more tokens...).

For example, consider the following BNF grammar:

address         =       mailbox / group
mailbox         =       name-addr / addr-spec
name-addr       =       [display-name] angle-addr
angle-addr      =       [CFWS] "<" addr-spec ">" [CFWS] / obs-angle-addr
group           =       display-name ":" [mailbox-list / CFWS] ";"
                        [CFWS]
display-name    =       phrase
word            =       atom / quoted-string
phrase          =       1*word / obs-phrase
addr-spec       =       local-part "@" domain
local-part      =       dot-atom / quoted-string / obs-local-part
domain          =       dot-atom / domain-literal / obs-domain
obs-local-part  =       word *("." word)

Now consider the following email address: "Joe Example" <joe@example.com>

The first token you read will be "Joe Example" and you might think that that token indicates that it is the display name, but it doesn't. All you know is that you've got a 'quoted-string' token. A 'quoted-string' can be part of a 'phrase' or it can be (a part of) the 'local-part' of the address itself. You must read at least 1 more token before you'll be able to figure out what it actually is ('obs-local-part' makes things slightly more difficult). In this case, you'll get a '<' which indicates the start of an 'angle-addr', allowing you to assume that the 'quoted-string' you just got is indeed the 'display-name'.

If, however, you parse the address in reverse, things become a little simpler because you know immediately what to expect the next token to be a part of.

That's pretty cool. Kudos to the Microsoft engineers for thinking up this strategy.

Unfortunately, the parser does not handle the 'group' address type. I'll let this slide, however, partly because I'm still impressed by the approach the address parser took and also because I realize that System.Net.Mail is meant for creating and sending new messages, not parsing existing messages from the wild.

Okay, so how well does it serialize MailAddress?

Ugh. You know that face you make when you just see a guy get kicked in the nuts? Yea, that's the face I made when I saw line #227:

encodedAddress = String.Format(CultureInfo.InvariantCulture, "\"{0}\"", this.displayName);

The problem with the above code (and I'll soon be submitting a bug report about this) is that the displayName string might have embedded double quotes in it. You can't just surround it with quotes and expect it to work. This is the same mistake all those programmers make that allow SQL-injection attacks.

For an example of how this should be done, see MimeKit's MimeUtils.Quote() method.

I had such high hopes... at least this is a fairly simple bug to fix. I'll probably just offer them a patch.

ContentType and ContentDisposition

Their parser is decent but it doesn't handle rfc2231 encoded parameter values, so I'm not overly impressed. It'll get the job done for simple name="value" parameter syntax, though, and it will decode the values encoded with the rfc2047 encoding scheme (which is not the right way to encode values, but it is common enough that any serious parser should handle it). The code is also pretty clean and uses a tokenizer approach, so that's a plus. I guess since this isn't really meant as a full-blown MIME parser, they can get away with this and not have it be a big deal. Fair enough.

Serialization, unsurprisingly, leaves a lot to be desired. Parameter values are, as I expected, encoded using rfc2047 syntax rather than the IETF standard rfc2231 syntax. I suppose that you could argue that this is for compatibility, but really it's just perpetuating bad practices. It also means that it can't properly fold long parameter values because the encoded value just becomes one big long encoded-word token. Yuck.

Base64

Amusingly, Microsoft does not use their Convert.FromBase64() decoder to decode base64 in their System.Net.Mail implementation. I point this out mostly because it is the single most common problem users have with every one of the Open Source .NET mail libraries out there (other than MimeKit, of course) because Convert.FromBase64() relies on the data not having any line breaks, white space, etc in the input stream.

This should serve as a big hint to you guys writing your own .NET email libraries not to use Convert.FromBase64() ;-)

They use unsafe pointers, just like I do in MimeKit, but I'm not sure how their performance compares to MimeKit's yet. They do use a state machine, though, so rock on.

I approve this base64 encoder/decoder implementation.

SmtpClient

One thing they do which is pretty cool is connection pooling. This is probably a pretty decent win for the types of things developers usually use System.Net.Mail's SmtpClient for (spam, anyone?).

The SASL AUTH mechanisms that they seem to support are NTLM, GSSAPI, LOGIN and WDIGEST (which apparently is some sort of IIS-specific authentication mechanism that I had never heard of until now). For those that were curious which SASL mechanisms SmtpClient supported, well, now you know.

The code is a bit hard to follow for someone not familiar with the codebase (not nearly as easy reading as the address or content-type parsers, I'm afraid), but it seems fairly well designed.

It does not appear to support PIPELINING or BINARYMIME like MailKit does, though. So, yay! Win for MailKit ;-)

They do support SMTPUTF8, so that's good.

It seems that if you set client.EnableSsl to true, it will also try STARTTLS if it isn't able to connect on the SSL port. I wasn't sure if it did that or not before, so this was something I was personally interested in knowing.

Hopefully my SmtpClient implementation review isn't too disappointing. I just don't know what to say about it, really. It's a pretty straight-forward send-command-wait-for-reply implementation and SMTP is pretty dead simple.

Conclusion

Overall the bits I was interested in were better than I expected they'd be. The parsers were pretty good (although incomplete) and the serializers were "good enough" for normal use.

Of course, it's not as good as MimeKit, but let's be honest, MimeKit sets the bar pretty high ;-)