The Wait Is Over: MimeKit and MailKit Reach 1.0

After about a year in the making for MimeKit and nearly 8 months for MailKit, they've finally reached 1.0 status.

I started really working on MimeKit about a year ago wanting to give the .NET community a top-notch MIME parser that could handle anything the real world could throw at it. I wanted it to run on any platform that can run .NET (including mobile) and do it with remarkable speed and grace. I wanted to make it such that re-serializing the message would be a byte-for-byte copy of the original so that no data would ever be lost. This was also very important for my last goal, which was to support S/MIME and PGP out of the box.

All of these goals for MimeKit have been reached (partly thanks to the BouncyCastle project for the crypto support).

At the start of December last year, I began working on MailKit to aid in the adoption of MimeKit. It became clear that without a way to inter-operate with the various types of mail servers, .NET developers would be unlikely to adopt it.

I started off implementing an SmtpClient with support for SASL authentication, STARTTLS, and PIPELINING support.

Soon after, I began working on a Pop3Client that was designed such that I could use MimeKit to parse messages on the fly, directly from the socket, without needing to read the message data line-by-line looking for a ".\r\n" sequence, concatenating the lines into a massive memory buffer before I could start to parse the message. This fact, combined with the fact that MimeKit's message parser is orders of magnitude faster than any other .NET parser I could find, makes MailKit the fastest POP3 library the world has ever seen.

After a month or so of avoiding the inevitable, I finally began working on an ImapClient which took me roughly two weeks to produce the initial prototype (compared to a single weekend for each of the other protocols). After many months of implementing dozens of the more widely used IMAP4 extensions (including the GMail extensions) and tweaking the APIs (along with bug fixing) thanks to feedback from some of the early adopters, I believe that it is finally complete enough to call 1.0.

In July, at the request of someone involved with a number of the IETF email-related specifications, I also implemented support for the new Internationalized Email standards, making MimeKit and MailKit the first - and only - .NET email libraries to support these standards.

If you want to do anything at all related to email in .NET, take a look at MimeKit and MailKit. I guarantee that you will not be disappointed.

The Future of Debugging in Xamarin Studio

There comes a time in ever man's life when he says to himself, "there has got to be a better way..."

Set Next Statement

Have you ever been stepping through a method or hit a breakpoint and discovered that variables did not have the expected values? Don't you wish you could go back in time and start stepping through that method from an earlier point to see how things went so horribly wrong? Of course you do.

Last week, with the help of Zoltan Varga (who added the necessary runtime support), I implemented support in Xamarin Studio to set the next statement to execute when you resume execution of your program in the debugger. You can set the next statement to be any statement in the current method; any statement at all. This essentially allows you to jump back in time or completely step over execution of statements after the current position.

Don’t worry. As long as you hit Run To Cursor at precisely the moment the lightning strikes the tower, everything will be fine!

Run to Cursor

If you're like me, you've probably found yourself stepping through some code in the debugger and you get to a loop or something that you know is fine and you just don't feel like hitting Step Over the 5 bajillion times necessary to get past it, so what do you do? Hopefully you don't hit Step Over those 5 bajillion times. Hopefully you just set a breakpoint somewhere after that loop and then hit Continue.

The problem with this solution is that it's tedious.

Soon, however, you'll be able to simply right-click and select Run To Cursor (or just set/use a keybinding) and the debugger will resume execution until it reaches your cursor!

Client-Side Evaluation of Simple Properties

Assuming that you haven't disabled "Allow implicit property evaluation and method invocation" in your Xamarin Studio debugger preferences, whenever class properties are evaluated in the debugger (in the Watch pad, the Locals pad, or when you hover the mouse cursor over a property), in order to get the value, the debugger has to spin up a thread in the program being debugged in order to have it evaluate the property (or other expression) because, unlike fields, properties are really just methods that have to run arbitrary code.

For at least a year or so, now, we've mitigated this somewhat by cheating if the property has the CompilerGeneratedAttribute (signifying that it is an auto-property). When evaluating these properties, we would instead do a lookup of the backing field and get its value since we could do that locally without any need to round-trip to the program being debugged. While this helped a lot, there's a lot of properties out there that effectively just return a field and have no other logic (maybe an auto-property wasn't used because the setter does more than just set the field value?).

To improve performance of this, I started looking into what it would take to interpret the IL locally in the IDE. Obviously this could only really work if the property getter was "simple" enough and didn't have to take locks, etc. I started asking Zoltan some questions on the feasibility of this and he wrote a simple IL interpreter (included in Mono.Debugger.Soft) that I ended up using in Xamarin Studio to try and evaluate properties locally before falling back to having the debuggee's runtime spin up a thread to evaluate the property for us.

Great Scott! When Can I Start Using These Awesome Features?

To use these new features, you will need Mono 3.4.1 (or later) and an upcoming release of Xamarin Studio (5.0.1? It won't quite make it into 5.0).

Well, it's time I get back... to the future! ... And implementing more new features!

GMime gets a Speed Boost

With all of the performance improvements I've been putting into MimeKit recently, it was about time to port some of these optimizations back to GMime.

In addition to other fixes that were in the queue, GMime 2.6.20 includes the "SIMD" optimization hack that I blogged about doing for MimeKit and I wanted to share the results. Below is a comparison of GMime 2.6.19 and 2.6.20 parsing the same 2GB mbox file on my 2011 Core-i5 iMac with the "persistent stream" option enabled on the GMimeParser:

[fejj@localhost gmime-2.6.19]$ ./gmime-mbox-parser really-big.mbox
Parsed 29792 messages in 5.15 seconds.

[fejj@localhost gmime-2.6.20]$ ./gmime-mbox-parser really-big.mbox
Parsed 29792 messages in 4.70 seconds.

That's a pretty respectable improvement. Interestingly, though, it's still not as fast as MimeKit utilizing Mono's LLVM backend:

[fejj@localhost MimeKit]$ mono --llvm ./mbox-parser.exe really-big.mbox
Parsed 29792 messages in 4.52 seconds.

Of course, to be fair, without the --llvm option, MimeKit doesn't fare quite so well:

[fejj@localhost MimeKit]$ mono ./mbox-parser.exe really-big.mbox
Parsed 29792 messages in 5.54 seconds.

I'm not sure what kind of optimizations LLVM utilizes when used from Mono vs clang (used to compile GMime via homebrew, which I suspect uses -O2), but nevertheless, it's still very impressive.

After talking with Rodrigo Kumpera from the Mono runtime team, it sounds like the --llvm option is essentially the -O2 optimizations minus a few of the options that cause problems with the Mono runtime, so effectively somewhere between -O1 and -O2.

I'd love to find out why MimeKit with the LLVM optimizer is faster than GMime compiled with clang (which also makes use of LLVM) with the same optimizations, but I think it'll be pretty hard to narrow down exactly because MimeKit isn't really a straight port of GMime (they are similar, but a lot of MimeKit is all-new in design and implementation).

Introducing MailKit, a cross-platform .NET mail-client library

Once I announced MimeKit, I knew it would only be a matter of time before I started getting asked about SMTP, IMAP, and/or POP3 support.

Let's just say,

Challenge... ACCEPTED!

I started off back in early December writing an SmtpClient so that developers using MimeKit wouldn't have to convert a MimeMessage to a System.Net.Mail.MailMessage in order to send it using System.Net.Mail.SmtpClient. This went pretty quickly because I've implemented several SMTP clients in the past. Implementing the various SASL authentication mechanisms probably took as much or more time than implementing the SMTP protocol.

The following weekend, I ended up implementing a Pop3Client. Originally, I had planned on more-or-less cloning the API we had used in Evolution, but I decided that I would take a different approach. I designed a simple IMessageSpool interface which more closely follows the limited functionality of POP3 and mbox spools instead of trying to map the Pop3Client to a Store/Folder paradigm like JavaMail and Evolution do (Evolution's mail library was loosely based on JavaMail). Mapping mbox and POP3 spools to Stores and Folders in Evolution was, to my recollection, rather awkward and I wanted to avoid that with MailKit.

At first I was loathe to do it, but over the past 2 weeks I ended up writing an ImapClient as well. I'm sure Philip van Hoof will be pleased to note that I have a very nice BODYSTRUCTURE parser, although that API is not publicly exported.

Unlike the SmtpClient and Pop3Client, the ImapClient does not have all of its functionality on a single public class. Instead, ImapClient implements an IMessageStore which has a limited API, mostly meant for getting IFolders. I imagine that those who are familiar with the JavaMail and/or Evolution (Camel) APIs will recognize this design.

The IFolder interface isn't designed to be exactly like the JavaMail Folder API, though. I've been designing the interface incrementally as I implement the various IMAP extensions (I've found at least 37 of them at the time of this blog post, although I don't think I'll bother with ACL, MAILBOX-REFERRAL, or LOGIN-REFERRAL), so the API may continue to evolve as I go, but I think what I've got now will likely remain - I'll probably just be including additional APIs for the new stuff.

So far, I've implemented the following IMAP extensions: LITERAL+, NAMESPACE, CHILDREN, LOGIN-DISABLED, STARTTLS, MULTIAPPEND, UNSELECT, UIDPLUS, CONDSTORE, ESEARCH, SASL-IR, SORT, THREAD, SPECIAL-USE, MOVE, XLIST, and X-GM-EXT1. Phew, that was exhausting listing all of those!

Also news-worthy is that MimeKit is now equally as fast as GMime, which is pretty impressive considering that it is fully managed C# code.

Download MailKit 0.2 now and let the hacking begin!

Optimization Tips & Tricks used by MimeKit: Part 2

In my previous blog post, I talked about optimizing the most critical loop in MimeKit's MimeParser by:

• Extending our read buffer by an extra byte (which later became 4 extra bytes) that I could set to '\n', allowing me to do the bounds check after the loop as opposed to in the loop, saving us roughly half the instructions.
• Unrolling the loop in order to check for 4 bytes at a time for that '\n' by using some bit twiddling hacks (for 64-bit systems, we might gain a little more performance by checking 8 bytes at a time).

After implementing both of those optimizations, the time taken for MimeKit's parser to parse nearly 15,000 messages in a ~1.2 gigabyte mbox file dropped from around 10s to about 6s on my iMac with Mono 3.2.3 (32-bit). That is a massive increase in performance.

Even after both of those optimizations, that loop is still the most critical loop in the parser and the MimeParser.ScanContent() method, which contains it, is still the most critical method of the parser.

While the loop itself was a huge chunk of the time spent in that method, the next largest offender was writing the content of the MIME part into a System.IO.MemoryStream.

MemoryStream, for those that aren't familiar with C#, is just what it sounds like it is: a stream backed by a memory buffer (in C#, this happens to be a byte array). By default, a new MemoryStream starts with a buffer of about 256 bytes. As you write more to the MemoryStream, it resizes its internal memory buffer to either the minimum size needed to hold the its existing content plus whatever number of bytes your latest Write() was called with or double the current internal buffer size, whichever is larger.

The performance problem here is that for MIME parts with large amounts of content, that buffer will be resized numerous times. Each time that buffer is resized, due to the way C# works, it will allocate a new buffer, zero the memory, and then copy the old content over to the new buffer. That's a lot of copying and creates a situation where the write operation can become exponentially worse as the internal buffer gets larger. Since MemoryStream contains a GetBuffer() method, its internal buffer really has to be a single contiguous block of memory. This means that there's little we could do to reduce overhead of zeroing the new buffer every time it resizes beyond trying to come up with a different formula for calculating the next optimal buffer size.

At first I decided to try the simple approach of using the MemoryStream constructor that allows specifying an initial capacity. By bumping up the initial capacity to 2048 bytes, things did improve, but only by a very disappointing amount. Larger initial capacities such as 4096 and 8192 bytes also made very little difference.

After brainstorming with my coworker and Mono runtime hacker, Rodrigo Kumpera, we decided that one way to solve this performance problem would be to write a custom memory-backed stream that didn't use a single contiguous block of memory, but instead used a list of non-contiguous memory blocks. When this stream needed to grow its internal memory storage, all it would need to do is allocate a new block of memory and append it to its internal list of blocks. This would allow for minimal overhead because only the new block would need to be zeroed and no data would need to be re-copied, ever. As it turns out, this approach would also allow me to limit the amount of unused memory used by the stream.

I dubbed this new memory-backed stream MimeKit.IO.MemoryBlockStream. As you can see, the implementation is pretty trivial (doesn't even require scary looking bit twiddling hacks like my previous optimization), but it made quite a difference in performance. By using this new memory stream, I was able to shave a full second off of the time needed to parse that mbox file I mentioned earlier, getting the total time spent down to 5s. That's starting to get pretty respectable, performance-wise.

As a comparison, let's compare the performance of MimeKit with what seems to be the 2 most popular C# MIME parsers out there (OpenPOP.NET and SharpMimeTools) and see how we do. I've been hyping up the performance of MimeKit a lot, so it had better live up to expectations, right? Let's see if it does.

Now, since none of the other C# MIME parsers I could find support parsing the Unix mbox file format, we'll write some test programs to parse the same message stream over and over (say, 20 thousand times) to compare MimeKit to OpenPOP.NET.

Here's the test program I wrote for OpenPOP.NET:

using System;
using System.IO;
using System.Diagnostics;
using OpenPop.Mime;

namespace OpenPopParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var message = Message.Load (stream);
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

Here's the SharpMimeTools parser I wrote for testing:

using System;
using System.IO;
using System.Diagnostics;
using anmar.SharpMimeTools;

namespace SharpMimeParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var message = new SharpMessage (stream);
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

And here is the test program I used for MimeKit:

using System;
using System.IO;
using System.Diagnostics;
using MimeKit;

namespace MimeKitParser {
    class Program
    {
        public static void Main (string[] args)
        {
            var stream = File.OpenRead (args[0]);
            var stopwatch = new Stopwatch ();

            stopwatch.Start ();
            for (int i = 0; i < 20000; i++) {
                var parser = new MimeParser (stream, MimeFormat.Default);
                var message = parser.ParseMessage ();
                stream.Position = 0;
            }
            stopwatch.Stop ();

            Console.WriteLine ("Parsed 20,000 messages in {0}", stopwatch.Elapsed);
        }
    }
}

Note: Unfortunately, OpenPOP.NET's message parser completely failed to parse the Star Trek message I pulled out of my test suite at random (first message in the jwz.mbox.txt file included in MimeKit's UnitTests project) due to the Base64 decoder not liking some byte or another in the stream, so I had to patch OpenPOP.NET to no-op its base64 decoder (which, if anything, should make it faster).

And here are the results running on my 2011 MacBook Air:

[fejj@localhost OpenPopParser]$ mono ./OpenPopParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:06:26.6825190

[fejj@localhost SharpMimeParser]$ mono ./SharpMimeParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:19:30.0402064

[fejj@localhost MimeKit]$ mono ./MimeKitParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:00:15.6159326

Whooooosh!

Not. Even. Close.

MimeKit is nearly 25x faster than OpenPOP.NET even after making its base64 decoder a no-op and 75x faster than SharpMimeTools.

Since I've been ranting against C# MIME parsers that made heavy use of regex, let me show you just how horrible regex is for parsing messages (performance-wise). There's a C# MIME parser called MIMER that is nearly pure regex, so what better library to illustrate my point? I wrote a very similar loop to the other 2 that I listed above, so I'm not going to bother repeating it again. Instead, I'll just skip to the results:

[fejj@localhost MimerParser]$ mono ./MimerParser.exe ~/Projects/MimeKit/startrek.msg
Parsed 20,000 messages in 00:16:51.4839129

Ouch. MimeKit is roughly 65x faster than a fully regex-based MIME parser. It's actually rather pathetic that this regex parser beats SharpMimeTools.

This is why, as a developer, it's important to understand the limitations of the tools you decide to use. Regex is great for some things but it is a terrible choice for others. As Jamie Zawinski might say,

Some people, when confronted with a problem, think “I know, I'll use regular expressions.” Now they have two problems.

Optimization Tips & Tricks used by MimeKit: Part 1

One of the goals of MimeKit, other than being the most robust MIME parser, is to be the fastest C# MIME parser this side of the Mississippi. Scratch that, fastest C# MIME parser in the World.

Seriously, though, I want to get MimeKit to be as fast and efficient as my C parser, GMime, which is one of the fastest (if not the fastest) MIME parsers out there right now, and I don't expect that any parser is likely to smoke GMime anytime soon, so using it as a baseline to compare against means that I have a realistic goal to set for MimeKit.

Now that you know the why, let's examine the how.

First, I'm using one of those rarely used features of C#: unsafe pointers. While that alone is not all that interesting, it's a corner stone for one of the main techniques I've used. In C#, the fixed statement (which is how you get a pointer to a managed object) pins the object to a fixed location in memory to prevent the GC from moving that memory around while you operate on that buffer. Keep in mind, though, that telling the GC to pin a block of memory is not free, so you should not use this feature without careful consideration. If you're not careful, using pointers could actually make your code slower. Now that we've got that out of the way...

MIME is line-based, so a large part of every MIME parser is going to be searching for the next line of input. One of the reasons most MIME parsers (especially C# MIME parsers) are so slow is because they use a ReadLine() approach and most TextReaders likely use a naive algorithm for finding the end of the current line (as well as all of the extra allocating and copying into a string buffer):

    // scan for the end of the line
    while (inptr < inend && *inptr != (byte) '\n')
        inptr++;

The trick I used in GMime was to make sure that my read buffer was 1 byte larger than the max number of bytes I'd ever read from the underlying stream at a given time. This allowed me to set the first byte in the buffer beyond the bytes I just read from the stream to '\n', thus allowing for the ability to remove the inptr < inend check, opting to do the bounds check after the loop has completed instead. This nearly halves the number of instructions used per loop, making it much, much faster. So, now we have:

    // scan for the end of the line
    while (*inptr != (byte) '\n')
        inptr++;

But is that the best we can do?

Even after using this trick, it was still the hottest loop in my parser:

We've got no choice but to use a linear scan, but that doesn't mean that we can't do it faster. If we could somehow reduce the number of loops and likewise reduce the number of pointer increments, we could eliminate a bunch of the overhead of the loop. This technique is referred to as loop unrolling. Here's what brianonymous (from the ##csharp irc channel on freenode) and I came up with (with a little help from Sean Eron Anderson's bit twiddling hacks):

    uint* dword = (uint*) inptr;
    uint mask;

    do {
        mask = *dword++ ^ 0x0A0A0A0A;
        mask = ((mask - 0x01010101) & (~mask & 0x80808080));
    } while (mask == 0);

And here are the results of that optimization:

Now, keep in mind that on many architectures other than x86, in order to employ the trick above, inptr must first be 4-byte aligned (uint is 32bit) or it could cause a SIGBUS or worse, a crash. This is fairly easy to solve, though. All you need to do is increment inptr until you know that it is 4 byte aligned and then you can switch over to reading 4 bytes at a time as in the above loop. We'll also need to figure out which of those 4 bytes contained the '\n'. An easy way to solve that problem is to just linearly scan those 4 bytes using our previous single-byte-per-loop implementation starting at dword - 1. Here it is, your moment of Zen:

    // Note: we can always depend on byte[] arrays being
    // 4-byte aligned on 32bit and 64bit architectures
    int alignment = (inputIndex + 3) & ~3;
    byte* aligned = inptr + alignment;
    byte* start = inptr;
    uint mask;

    while (inptr < aligned && *inptr != (byte) '\n')
        inptr++;

    if (inptr == aligned) {
        // -funroll-loops
        uint* dword = (uint*) inptr;

        do {
            mask = *dword++ ^ 0x0A0A0A0A;
            mask = ((mask - 0x01010101) & (~mask & 0x80808080));
        } while (mask == 0);

        inptr = (byte*) (dword - 1);
        while (*inptr != (byte) '\n')
            inptr++;
    }

Note: In this above code snippet, 'inputIndex' is the byte offset of 'inptr' into the byte array. Since we can safely assume that index 0 is 4-byte aligned, we can do a simple calculation to get the next multiple of 4 and add that to our 'inptr' to get the next 4-byte aligned pointer.

That's great, but what does all that hex mumbo jumbo do? And why does it work?

Let's go over this 1 step at a time...

    mask = *dword++ ^ 0x0A0A0A0A;

This xor's the value of dword with 0x0A0A0A0A (0x0A0A0A0A is just 4 bytes of '\n'). The xor sets every byte that is equal to 0x0A to 0 in mask. Every other byte will be non-zero.

    mask - 0x01010101

When we subtract 0x01010101 from mask, the result will be that only bytes greater than 0x80 will contain any high-order bits (and any byte that was originally 0x0A in our input will now be 0xFF).

    ~mask & 0x80808080

This inverts the value of mask resulting in no bytes having the highest bit set except for those that had a 0 in that slot before (including the byte we're looking for). By then bitewise-and'ing it with 0x80808080, we get 0x80 for each byte that was originally 0x0A in our input or otherwise had the highest bit set after the bit inversion.

Because there's no way for any byte to have the highest bit set in both sides of the encompassing bitwise-and except for the character we're looking for (0x0A), the mask will always be 0 unless any of the bytes within were originally 0x0A, which would then break us out of the loop.

Well, that concludes part 1 as it is time for me to go to bed so I can wake up at a reasonable time tomorrow morning.

Good night!

MimeKit: Coming to a NuGet near you.

If, like me, you've been trapped in the invisible box of despair, bemoaning the woeful inadequacies of every .NET MIME library you've ever found on the internets, cry no more: MimeKit is here.

I've just released MimeKit v0.5 as a NuGet Package. There's still plenty of work left to do, mostly involving writing more API documentation, but I don't expect to change the API much between now and v1.0. For all the mobile MIME lovers out there, you'll be pleased to note that in addition to the .NET Framework 4.0 assembly, the NuGet package also includes assemblies built for Xamarin.Android and Xamarin.iOS. It's completely open source and licensed under the MIT/X11 license, so you can use it in any project you want - no restrictions. Once MimeKit goes v1.0, I plan on adding it to Xamarin's Component Store as well for even easier mobile development. If that doesn't turn that frown upside down, I don't know what will.

For those that don't already know, MimeKit is a really fast MIME parser that uses a real tokenizer instead of regular expressions and string.Split() to parse and decode headers. Among numerous other things, it can properly handle rfc2047 encoded-word tokens that contain quoted-printable and base64 payloads which have been improperly broken apart (i.e. a quoted-printable triplet or a base64 quartet is split between 2 or more encoded-word tokens) as well as handling cases where multibyte character sequences are split between words thanks to the state machine nature of MimeKit's rfc2047 text and phrase decoders (yes, there are 2 types of encoded-word tokens - something most other MIME parsers have failed to take notice of). With the use of MimeKit.ParserOptions, the user can specify his or her own fallback charset (in addition to UTF-8 and ISO-8859-1 that MimeKit has built in), allowing MimeKit to gracefully handle undeclared 8bit text in headers.

When constructing MIME messages, MimeKit provides the user with the ability to specify any character encoding available on the system for encoding each individual header (or, in the case of address headers: each individual email address). If none is specified, UTF-8 is used unless the characters will fit nicely into ISO-8859-1. MimeKit's rfc2047 and rfc2231 encoders do proper breaking of text (i.e it avoids breaking between surrogate pairs) before the actual encoding step, thus ensuring that each encoded-word token (or parameter value) is correctly self-contained.

S/MIME support is also available in the .NET Framework 4.0 assembly (not yet supported in the Android or iOS assemblies due to the System.Security assembly being unavailable on those platforms). MimeKit supports signing, encrypting, decrypting, and verifying S/MIME message parts. For signing, you can either use the preferred multipart/signed approach or the application/[x-]pkcs7-signature mime-type, whichever you prefer.

I'd love to support PGP/MIME as well, but this is a bit more complicated as I would likely need to depend on external native libraries and programs (such as GpgME and GnuPG) which means that MimeKit would likely have to become 32bit-only (currently, libgpgme is only available for 32bit Windows).

I hope you enjoy using MimeKit as much as I have enjoyed implementing it!

Note: For those using my GMime library, fear not! I have not forgotten about you! I plan to bring many of the API and parser improvements that I've made to MimeKit back to GMime in the near future.

For those using the C# bindings, I'd highly recommend that you consider switching to MimeKit instead. I've based MimeKit's API on my GMime API, so porting to MimeKit should be fairly straightforward.

Introducing MonoTouch.SQLite

I've been working on a personal side-project writing an app for the iPad that makes use of SQLite-Net, displaying that data in a UITableView.

Up until this past week, I had been using Miguel de Icaza's wonderful MonoTouch.Dialog library for displaying my data. Unfortunately, I wanted search filtering to be persistent, which means that I really needed to use Apple's UISearchDisplayController but I couldn't find an easy way to retrofit that onto MonoTouch.Dialog's DialogViewController to replace the simpler UISearchBar API that it currently uses. Since I had to look at creating an alternate solution, I figured I might as well solve the other potential problem I had with MonoTouch.Dialog, which is that my app really needed to be able to handle tables with a massive number of items. This brings us to...

MonoTouch.SQLite

MonoTouch.Dialog really made using UITableViews in iPhone and iPad apps trivial and I wanted to try and repeat at least some of that with MonoTouch.SQLite.

The first thing I had to do was to figure out a way of modeling the data in such a way as to allow a generic class to do most of the work for the developer. After a few sleepless nights of hacking last weekend, I figured out a fairly simple approach that seems to work pretty well. I started off thinking that I wouldn't be able to get around having to have a subclassable model, so I made most everything virtual. This is the public API that I came up with:

public class SQLiteTableModel<T> : IDisposable where T : new ()
{
    public SQLiteTableModel (SQLiteConnection sqlitedb, int pageSize, SQLiteOrderBy orderBy, string sectionExpr);

    // 2 ways of setting the search criteria
    public SQLiteWhereExpression SearchExpression { get; set; }
    public string SearchText { get; set; }

    // Gets the total number of table rows
    public int Count { get; }

    // Gets the number of table sections
    public int SectionCount { get; }

    // Gets the section titles
    public string[] SectionTitles { get; }

    // Gets the row count for a particular section
    public int GetRowCount (int section);

    // Get the index of an item
    public int IndexOf (T item, IComparer<t> comparer);

    // Convert item index into a section and row
    public bool IndexToSectionAndRow (int index, out int section, out int row);

    // Convert section and row into an item index
    public int SectionAndRowToIndex (int section, int row);

    // 2 ways of getting an item
    public T GetItem (int section, int row);
    public T GetItem (int index);

    // Reset the state of the model
    public void ReloadData ();
}

You can see the full class implementation here.

It turns out, though, that it really isn't necessary to subclass my model unless you want to have finer control over the specific SQL query commands that it makes (all of those methods are virtual).

SQLiteTableViewController<T>

As I started porting my iPad app to use my SQLiteTableModel class, I started to realize that I could abstract a lot of my usage of the model into a reusable base class. What I came up with will blow your mind.

Are you ready?

In order to populate a UITableView with the contents of an SQLite table, all you have to do is subclass SQLiteTableViewController<T> and implement 1 method:

protected UITableViewCell GetCell (UITableView tableView, NSIndexPath path, T item)

That's it.

I've written up a sample iPhone app that illustrates just how easy this is.

But wait! There's more!

If you order in the next 30 minutes, you can also get this free complimentary Sham-Wow!

Okay, just kidding about that Sham-Wow! bit, but I wasn't kidding about there being more:

Remember when I said one of the problems I wanted to solve was persistent search filtering? Yea, well, I did it. SQLiteTableViewController handles all of that for you as well. In fact, give searching a try in that sample above.

At this point I bet you're thinking, "wow, how could this get any better?"

I'll tell you. Remember how SQLiteTableModel had 2 methods for setting the search criteria? Well, the one that takes a string parses it to create a SQLiteWhereExpression allowing the user to match against specific fields. For example, if you had the following data item:

public class Contact {
    public string FirstName;
    public string LastName;
    public string PhoneNumber;
    public string Address;
    public string Comments;
}

...the user could type:

address:"Newton, MA"

and SQLiteTableModel would construct a query to match "Newton, MA" against only the Address field.

If the user, instead, types:

address:"Newton, MA" firstname:Jane

then the matches that would display in the list would be limited to contacts with a first name of "Jane" who live also in "Newton, MA".

I've also taken the liberty of implementing a SQLiteSearchAliasAttribute that allows you to specify aliases for your fields (or even the same alias to multiple fields!). For example, you could do this:

public class Contact {
    [SQLiteSearchAlias ("first")][SQLiteSearchAlias ("name")]
    public string FirstName;
    [SQLiteSearchAlias ("last")][SQLiteSearchAlias ("name")]
    public string LastName;
    [SQLiteSearchAlias ("phone")]
    public string PhoneNumber;
    public string Address;
    public string Comments;
}

This would allow your users to use "name" to match against either FirstName or LastName!

It also means they can type "first" instead of "firstname" to match against only the first name, as in the above example.

Where Can I Find This Awesome Library?

Glad you asked! You can find it on my GitHub page: MonoTouch.SQLite.

Well? What are you waiting for? Get hacking!

Meet the Hackers

This past week, I've started to get back into photography a bit more (thanks, Nina!) and started taking my camera into the office with me every day to remind myself to take photos. As a result, I've taken a bunch of photographs of my co-workers in the office.

Would you like to meet the hackers?

The Founders

Most of you would probably recognize the infamous Miguel de Icaza, Xamarin's CTO:

Next up is our very own Steve Jobs, Nat Friedman, our CEO and the man who reminds us to pay attention to the details:

Another person many of you will recognize is our very own COO, Joseph Hill:

MonoDevelop Team

Well, okay, I've only got a photo of the famous Michael Hutchinson, but he's a very important player in the development of MonoDevelop.

QA Team

Next up, we have the QA team. They do their best to make sure that we, the developers, didn't break anything. When they aren't testing a specific application before a launch, they hammer away at our products and try to find weak spots in our code (but we still love them anyway!)

This is PJ, and as you can see, he's demonstrating how to QA popcorn corn cobs:

(Did it pass the test, PJ?)

Next up is Lindsey. She's been working on writing automated tests to make it less likely for releases to include regressions. Let's hope she's successful!

Release Team

Alex Corrado is the man behind the curtain. He's our head Release Team engineer and also the brilliant mastermind that started CXXI, the Mono C++ interop project that we hope to give him time to finish someday soon.

Web Team

The newest addition to our ranks (just this week, in fact!), but long-time contributor to the Mono project, is Bojan Rajković. You can see we've already put him to work (he is no doubt puzzling over some ASP.NET code on his screen).

Documentation Team

Nina is the only Cambridge resident on our Docs Team. Specifically, she hacks on our Documentation Portal. She's also the one who has encouraged me to get back into taking photographs, so she'll have to put up with me using her as a guinea pig the most. Here she is taunting me with her hot cup of Chaider:

Debugging Your MonoTouch Apps: The Future

One of the "paper cuts" developers have been having with developing their MonoTouch 4.0.x (and earlier) applications is that for some networking setups, the IP of the developer's workstation detected by MonoDevelop and given to the iPhone or iPad device for debugging purposes is not correct. This often happens if the WiFi is a different network than the network that the developer's machine is connected to (although there are other scenarios as well).

Since it does not seem to be widely known about, allow me to point out that current versions of MonoTouch allow developers to modify the IP that the runtime should connect to for debugging via the iOS Settings app found on any iPhone or iPad (or Simulator). You can see a screenshot of this per-App Settings page in the screenshot to the left. Each of these fields are editable, allowing you to override the defaults filled-in by MonoDevelop.

For our upcoming 4.1 release, Rolf Kvinge and I (but mostly Rolf) have been working on improving this. Rolf has modified the code to check the value of the IP provided in the per-App Settings and if it is set to nil or "automatic", the debugger falls back to checking for a file bundled with the app called MonoTouchDebugConfiguration.txt which can list any number of IP's to try and connect to, each one being on a separate line prefixed with "IP: ". For example:

IP: 10.0.1.31
IP: 192.168.1.31
IP: 204.11.102.79

The runtime will then attempt to connect to each of these IPs asynchronously until it establishes a connection to one of them (at which point it aborts the other waiting connections). This config file solution will hopefully help simplify things for developers a bit by allowing them to pre-configure which IPs to try for their local network configuration w/o having to manually override the iPhone debug settings on the device or simulator.

For Phase 2 of our plan for World Domination, Rolf is hard at work adding support to MonoDevelop and the runtime to allow for USB debugging which will obsolete the above functionality in future versions where the developer has a MonoDevelop which supports USB debugging. For developers stuck on an older MonoDevelop (like 2.4), the solution illustrated above requires no changes to MonoDevelop and so will be available for use.

Moonlight on Android

For the past week, the Moonlight team has been busy porting Moonlight to Android devices and today, showed it off at Mix 11.

The video shows Moonlight running on both a Motorola Xoom tablet and a Nexus S phone.

Keep in mind that we're still in the early phases of porting and there's still a lot of work left to do before we can ship a product, but it's still exciting!

Update: For those of you reading my blog from Planet GNOME (or some other planet that doesn't show the video above), you can find it here, on YouTube.

Update: Now you can see Moonlight rendering video with 3D transforms, too!

Using Git

Now that Mono has moved to GitHub, a number of contributors (including myself) have been somewhat lost, not knowing how to properly use Git. So today, I'm recommending a book to help all the lost souls out there get comfortable with using Git.

As many long-time programmers probably know, O'Reilly books are top notch. Most probably also would not be surprised that O'Reilly has a book covering Git: Version Control with Git: Powerful Tools and Techniques for Collaborative Software Development.

Don't let the bat scare you.

New Phone (Android)

Ever since the first Android phones came out back in October 2008, I've been keeping an eye on their progress. I didn't care for the original HTC Hero, but the newest phones based on Android 2.1/2.2 look very impressive. Now that my iPhone 3G contract is up, I decided I'd get one. Since I'm on AT&T and wanted to stay with that provider, I went with the Samsung Captivate and a screen protector for it. My friend/co-worker, Michael Hutchinson, recently bought a Samsung Vibrant (he's on T-Mobile) and has been very happy with it. Another friend/co-worker, Gonzalo has also been happy with his switch to the Captivate on AT&T, so I expect that I'll be pleased as well.

One of the reasons I decided to get an Android phone is that the Mono team is working on MonoDroid, a port of Mono to Android phones. While I'm not on the team working on the port, I am interested as a potential Android developer in using MonoDroid to write some of my own applications for Android phones.

10 years ago, when I first started working for Helix Code on the GNOME desktop for Linux to help make Linux usable for average Joes, I never expected that I'd ever have a phone that ran Linux! Especially a Linux-based phone that is taking the smart-phone market by storm! These are very exciting times for me, and I'm sure for the Google folks working on the Android project!

After I get my feet wet for a week or so with using my new Captivate phone, I'll try to write up a review of what I think, so stay tuned!

Trolls

Thanks go out to Stormy Peters and Jo Shields for tweeting a link to Seth Godin's blog post entitled: Trolls. Had they not tweeted it, I likely would have missed these marvelous words of wisdom:

Lots of things about work are hard. Dealing with trolls is one of them. Trolls are critics who gain perverse pleasure in relentlessly tearing you and your ideas down. Here's the thing(s):

1. trolls will always be trolling
2. critics rarely create
3. they live in a tiny echo chamber, ignored by everyone except the trolled and the other trolls
4. professionals (that's you) get paid to ignore them. It's part of your job.

"Can't please everyone," isn't just an aphorism, it's the secret of being remarkable.

Thanks, Seth. I needed to be reminded of this fact.

And yea, Team Mono is remarkable ;-)

Looking Back: 10 Years of Ximian

Nat and Miguel both reflected on the incorporation of Ximian which has made me a bit nostalgic myself. I've only been with Ximian for about 9 and a half years, but they have been pretty awesome years! I'm very thankful for having had the chance to be a part of something great and for having been able to work with both Nat and Miguel, 2 of my heroes.

The things I respect most about both Nat and Miguel are their positive attitudes and energy. They both have an amazing ability to inspire the people around them to do great things, and we did. From Ximian Desktop and Evolution to Mono, Moonlight and SuSE Studio. Nat and Miguel have set out to change the Linux Desktop for the better, and I believe they (and we, together with the rest of the GNOME and Mono communities) have accomplished that. So congratulations Nat & Miguel on 10 years of Awesome(tm)!

Mono not Chasing Tail Lights

I've been on vacation and just got back and heard about this whole Richard Stallman spouting nonsense about Miguel being a traitor thing and saw that OSNews had an article criticizing Stallman for such ridiculous statements, so figured I'd skim the comments. Unsurprisingly someone attacked Miguel and Mono for "always chasing tail lights" which is a common logical fallacy that the anti-Mono folks love to argue, but in response there was, what I thought, a pretty good post by elanthis that rings fairly accurately as far as I remember events happening back in 2001 while I was at Ximian so figured I would quote it in order to share.

This "Mono is just chasing tail lights" crap is getting silly.

If you take Mono to mean "a project whose sole goal is to enable running Windows .NET apps on Linux" then yes, Mono will always be behind.

If you take Mono for what it actually is meant to be and what it is actually freaking used for, which is "a project that provides a clean, modern language on which to more efficiently develop and deliver Linux applications" then you realize that Mono isn't chasing anything, it's actually 20 steps _ahead_ of anything else on the Linux front.

If Microsoft came along and said, "we're going to assert some mythical patents and Mono can't be compatible with the ECMA CLR or class library anymore," Linux and Mono would not be devastated in the least by it. Mono would drop the offending bits and it would _still_ be a clean, modern language with a helluva-efficient runtime library providing a ton of brand-new non-Microsoft-derived Linux-focused APIs and libraries (from POSIX to dbus to GTK to OpenGL and so on) that real applications like Tomboy or Banshee or iFolder or MonoDevelop or Unity3D or SunUO or Landell and countless others can continue to be built on top of, taking full advantage of the features of C# (or even a "very similar to C#" language, if some change were forced on Mono by MS), the high quality Mono runtime implementation, and the quality APIs and tools developed by the Mono community.

There's no difference in theory behind a project like Mono and a project like Python, other than that Mono leisurely follows a language specification for the sake of being compatible __as a nice side benefit__ while Python just makes shit up as it goes. Oh, and Mono is way faster, has significantly better documentation, has better tools, and is already used in large-scale commercial apps (that have NO dependency on being Microsoft .NET compatible) despite having been around for a far shorter amount of time.

Honestly look around and see how many people try to use or even _care_ about using Mono as a way to run apps developed for Microsoft .NET. You just don't see it. Mainsoft makes a business out of making it easy to get ASP.NET apps running on Linux servers and that's about the end of it. All of the really cool users of Mono are people that are using it either because it provides a top-notch embeddable VM that blows away every other FOSS scripting engine (e.g., Unity3D) or because it provides some of the best APIs around for creating brand-new Linux-specific applications (e.g. Beagle).

The only people who are even remotely hung up on chasing Microsoft's taillights are the people who don't understand what the purpose of Mono is and use their misconception as a strawman argument against the Mono project.

Mono was conceived by the Ximian folks because of their experience with Evolution. They were writing big Linux desktop apps. Writing them in C sucked. Writing them in C++ sucked slightly less. Writing them in Python sucked far less in some ways and far more in other ways. Writing them in any FOSS-friendly language available was a nightmare. They saw C#. They saw it was like Java, except it fixed many of the things that sucked about Java. They thought, "it would be sweet if we had a language like that to develop in." They thought, "we could write a language like that to develop in." They thought, "we could write a totally new language, but then we have to solve a bunch of problems that have already been solved over and over and over" and they knew that good engineers don't waste time resolving old problems. They thought, "if we make a C# compiler and runtime for Linux, not only do we get what we want, but what the heck maybe it'll help bring more ex-Windows developers over to the Linux side and make the world a better place." And, instead of sitting around bitching about all the hard work other people are doing, they went ahead and made the project. Their lives got easier, the lives of quite a few Linux developers got easier, the lives of even a number of Windows developers got easier, and the regular users started seeing some cool software that none of the Linux developers were making pre-Mono because it was such a time-consuming pain in the ass to do it before, and Windows users even started seeing more cool software because now there was this awesome tool for embedding high-performance reliable easy-to-write scripts into larger applications which made the applications more awesome.

Everyone has won because of Mono, and because Miguel and the many other awesome contributors to Mono have done something FOSS never did before. The only thing the FOSS world has that even comes close to Mono is Java, and that was only just recently Open Sourced, and Sun's Java is actually already way behind Mono despite being far older.

So, seriously, quit with the taillight chasing crap. The Microsoft specification is largely irrelevant to just about all users of Mono, it's not even a priority for the Mono developers, and Mono has far surpassed anything else the FOSS community has in the same field.

C/C++ are a pain in the ass to work with and are light years behind modern languages in ease of use and features, and modern VM technology is already more or less on par with C/C++, and upcoming VM technology is quite likely to exceed the performance of C/C++. Python, Ruby, Perl, and so on are all great languages for some tasks but are total whores to work in for a great deal of other tasks, and they'll never be able to match Mono in speed (simple fact -- even if you implement the same VM technology in something like Parrot as you have in Mono, dynamic typing will always be slower; even if you use the same tracing optimizing JIT to compiler both C# and Python to machine code, the C# result will need far less type guards than the Python result would... which you can even prove by just using IronPython on Mono).

The point of Mono is NOT CLR compatibility. It's not even C# the language. The point of Mono is having a runtime and multiple languages that make developing real-world applications easy and making those applications more reliable, which frees up more developer time to work on actual features and performance rather than working on endless layers of complicated framework code or debugging stupid bugs that only exist because of poor language design.

As always, you have the typical know-it-alls who attacked the response saying that C# doesn't have many advantages over C++/Qt, but as usual, they neglect to consider the fact that not everyone wants to program using C++ or Qt.

As one of the developers involved in the development of Evolution, neither C++ nor Qt were tools that would have helped to solve our problems. First off, Evolution was a GNOME program (thus making Qt useless) and C++ was plagued with problems in those days. I'm also not sure that the C++ bindings for Gtk+ were even mature enough to use at the time. Even had these not been problems, none of us wanted to use C++ anyway.

Even today, with the surviving members of the original Evolution team now working on Moonlight (Chris Toshok, Larry Ewing and myself) which is partially written in C++, we have stuck with a very minimal subset and have gotten frustrated with a number of idiosyncrasies in C++ that were not a problem in C.

As a contributor to the Mono project (along with MonoDevelop and Moonlight), I feel that the Mono detractors are woefully uninformed and seem unwilling to educate themselves, but I guess that's their loss. The Mono Project will continue whether they call us traitors or not, whether they approve of the project or not, and whether they go on making fools of themselves across the internet or not.

We're not interested in childish popularity contests, we're interested in making great cross-platform development tools and making the Linux Desktop more inviting to a wider audience of both users and developers.

And we are succeeding.

QuakeLight in Moonlight

As we approach Moonlight 2.0 beta, more and more cool websites are starting to work under Moonlight. This week the team got QuakeLight working!

Here is a screenshot of QuakeLight under Moonlight in action:

Please note that there appears to be a bug in the version of Cairo that we embed into Moonlight which prevents the graphics from displaying on some NVidia and ATI cards (at least depending on which drivers you use?). It works fine on my Intel i965 graphics chipset, though.

There's also a bug where it runs out of file descriptors after a while (which I'll be looking into shortly since it crops up for DrPopper as well).

Stay tuned for more exciting updates!

Note: More information about QuakeLight (and GPL source code!) can be found at http://www.innoveware.com/quakelight.html

I am not afraid

More Exciting Updates on Mono/Android

Looks like Koushik Dutta has been kicking butt getting Mono onto Google's Android platform:

• Compiling Mono under the Android Build Environment
• Microsoft's DLR and Mono bring Python and Ruby to Android
• Mono/Dalvik Interop

Keep up the awesome hacking, Koushik!

Also of interest is his article on why JNI sucks compared to doing the same thing in .NET: JNI in Android (and a foreword about why JNI sucks)

I never actually used JNI back when I was doing Java development in school, so when people told me it sucked I just took their word for it, but always wondered just how bad it was. Now I know. All I can say is: Ugh.

Hopefully the Java community replaces JNI with something as clean and elegant as what we find in .NET.

Mono 2.0 Wins Developer.com's Product of the Year Award

I'd just like to congratulate the Mono team on their Product of the Year Award and for a job well done!

Congratulations guys, you deserved this award! You guys kicked ass this past year and we all get to benefit from your awesomeness.