Moonlight on Android

For the past week, the Moonlight team has been busy porting Moonlight to Android devices and today, showed it off at Mix 11.

The video shows Moonlight running on both a Motorola Xoom tablet and a Nexus S phone.

Keep in mind that we're still in the early phases of porting and there's still a lot of work left to do before we can ship a product, but it's still exciting!

Update: For those of you reading my blog from Planet GNOME (or some other planet that doesn't show the video above), you can find it here, on YouTube.

Update: Now you can see Moonlight rendering video with 3D transforms, too!

QuakeLight in Moonlight

As we approach Moonlight 2.0 beta, more and more cool websites are starting to work under Moonlight. This week the team got QuakeLight working!

Here is a screenshot of QuakeLight under Moonlight in action:

Please note that there appears to be a bug in the version of Cairo that we embed into Moonlight which prevents the graphics from displaying on some NVidia and ATI cards (at least depending on which drivers you use?). It works fine on my Intel i965 graphics chipset, though.

There's also a bug where it runs out of file descriptors after a while (which I'll be looking into shortly since it crops up for DrPopper as well).

Stay tuned for more exciting updates!

Note: More information about QuakeLight (and GPL source code!) can be found at http://www.innoveware.com/quakelight.html

Moonlight TextBox

This past week I've been furiously hacking away on my reimplementation of the Silverlight TextBox control for Moonlight. I never realized just how much work goes into writing such a simple text-entry widget before this past week, even after having written Moonlight's text layout and rendering engine for the 1.0 release.

Keyboard input, keyboard navigation, keyboard selection, mouse selection & cursor positioning, key repeat, cursor blink, etc. The list goes on.

Most of it isn't hard, it's just time consuming.

At the same time, it's also fun in the sense that it's a new challenge for me to overcome (I love a good challenge). It makes you think a lot about designing for performance because you just don't know how much text you'll be rendering. It could be a short sentence or it could be an entire document, and the way you design your layout/rendering engine could mean the difference between taking 5 minutes to render or a fraction of a second.

Optimizing Moonlight's InkPresenter

This past week I started out staring at endless amounts of javascript trying to figure out what it was supposed to do so that I could figure out what Moonlight was breaking on in order to fix some bugs. As you can imagine, this is a slow and boring process where one's eyes go dry and it feels like you are getting nowhere fast.

As occasionally happens, I give up and move onto the next bug hoping that the next bug will be easier to fix and/or give me some insight into the previous bug. As it turned out, I moved onto bug #409793 which was a performance bug on sites like Ink Journal and Ink Tattoo Studio.

What was happening on these sites was that as more points got added to the stroke, rendering would get slower and slower (thus causing the line to lag behind the mouse cursor) because Moonlight was invalidating the entire InkPresenter canvas on each frame and so having to render the entire thing even though it was unnecessary.

To optimize this, I added a 'dirty' rectangle that kept track of the actual regions we needed to redraw. As points got added to the collection between frames, I added the bounds of the new point plus the region between the new point and the previous/next points (the 'next' point is obviously only needed if the newly added point was an insertion). The result for sites like InkJournal and InkTattooStudio was that we only invalidated the newly appended points at each frame render, vastly improving our performance which now matches Microsoft's Silverlight performance for these sites afaict.

QuakeLight: A Silverlight Port of Quake

Looks like Channel9 has just put up a video screen cast of QuakeLight, the Silverlight port of Quake - probably the game I lost the most amount of productivity to back in the late 90's.

I'd love to play with this under Moonlight to see how well we fare...

Update: Here's a handy link to an interview with the developer.

Moonlight Text Rendering

For those who haven't been following Moonlight development, I'm the guy that has been implementing all the text layout and rendering.

Let me start off by saying that text is hard. Really hard. Especially layout.

Font Hinting

The aspect of text rendering that I've been thinking a lot about lately is hinting. For an example of the difference that hinting makes with text rendering, see the following screenshot:

In the screenshot above, I've posted side-by-side text renderings produced by both Moonlight and Pango (courtesy of Michael Dominic K.) for Arial, Verdana and Times New Roman fonts. You'll notice that especially at the smaller font sizes, Moonlight's text rendering gets a bit fuzzy and hard to read. On the contrary, Pango's rendering, using hinting, has improved readability over Moonlight.

Why don't I just use hinting, you ask?

The problem is that hinting changes the font metrics in such a way that as you scale to larger sizes, the metrics do not scale uniformly (you'll notice that as the font sizes increase, the width of the line gets uniformly wider using Moonlight, but not so with Pango). This causes a bit of a problem for Moonlight (and Silverlight) because:

• with hinting enabled, scaling the font size in an animation would look jumpy
• this is made even worse when you consider line wrapping and how changing font metrics would mean that line wrapping would also change with the changing font size

In most applications, the text in each UI element is unlikely to change size (unless the user manually changes the font settings), and so enabling the use of hinting has no adverse side effects. Silverlight (and thus Moonlight), however, are meant to do all sorts of animations which may apply matrix transforms (such as scaling) to any item in the canvas (even text!).

Since scaling has to look smooth, it's hard to use hinting because of the way it changes the metrics.

Hopefully I can find a way to improve rendering quality at the smaller sizes without getting the side effects I mentioned above, or at least have them be far less noticeable.

Line Breaking

While not thinking about the render quality of the text, the bane of my existence has been implementing layout algorithms for the 3 different TextWrapping modes used in Silverlight (Wrap, NoWrap, and WrapWithOverflow).

I've mostly been focusing on TextWrapping=Wrap lately as it's probably the hardest one to implement, especially if I want to emulate the Silverlight wrapping behavior (from what I can gather by reading the Pango source code, there are ambiguities in the Unicode specification for line-breaking, so I've been having to familiarize myself with the rules for line-breaking lately).

Going Forward

In the future, I hope to replace my text layout/rendering code with the use of Pango but in order for that to happen, Pango will need a few new features:

• The ability to specify fonts outside of the configured font directories (e.g. a way to bypass FontConfig). From what I understand, this is already happening so I expect to be able to cross this off the list soon ;-)
• The ability to mark the beginning/end of a sub-run of text. Silverlight's <TextBlock> element supports child <Run> elements. This normally wouldn't be a big deal, but Silverlight uses the joint between two runs as a possible break opportunity. For example, <Run Text="abcdefg"/><Run Text="hijklmn"/> might line-wrap differently than <Run Text="abcdefghijklmn"/> even though there are no spaces between the two runs. Currently, when rendering a string of text using Pango, you combine all runs into a single string to pass to a PangoLayout, so the layout engine doesn't have a way of knowing that a position mid-word should be treated as a break opportunity.
• Pango will need a way to specify alternative text wrapping modes (so that it would be possible to emulate Silverlight's TextWrapping behavior). Apparently Owen and Behdad have already discussed the desire to have this sort of feature, so I'm looking forward to working with them on designing a suitable API.
• A way to specify that font metrics should scale uniformly as the font size changes. Update: Behdad has notified me that this feature is actively being worked on and that I can enable it via CAIRO_HINT_METRICS_OFF.
• The last thing that I can think of at the moment that Pango will need is a way to reset the Foreground brush for each run and each time each of those runs line-breaks. For example, the following screenshot consists of a single TextBlock with two Run elements which share their parent's font and foreground brush properties. This means that the prepare_run() virtual method for the subclassed PangoRenderer will need to know the extents of the next sub-run of text to be rendered so that it can properly setup the custom foreground brush.

In the meantime, having to write my own layout/rendering engine for text has taught me a lot more about text and fonts than I ever wanted to know and I'm continuing to learn more every day.

Update: With all of these custom changes that I'll need in order to move to Pango, I'm beginning to wonder if it's really worth it? The only thing that it could get me is the Unicode line-breaking logic, but the thing is... if I have to implement my own pluggable TextWrapping modes, then wouldn't I still essentially be implementing my own line-breaking logic? This makes me very sad :(

Text Rendering

Moonlight has some fairly unique text rendering requirements that I've not seen done anywhere else in Linux, Gtk+ application or no. Most applications stick to rendering text horizontally or vertically, they rarely, if ever, perform unusual matrix transformations on text (e.g. rotations).

When I first implemented text rendering for Moonlight, I used the obvious choice: Pango, using the Cairo backend (since we're using cairo in Moonlight for 2D graphics rendering anyway).

Unfortunately, we ran into some problems...

The first major problem is that as we applied rotation transforms and called pango_cairo_show_layout(), we'd get rendering glitches in that each glyph seemed to have its own independent baseline and so each frame, glyphs appeared to jitter.

The second major problem was that calling pango_cairo_show_layout() each frame had some major performance problems.

Thirdly, there appears to be no way to tell pango to load a font face from a specific file path.

As far as the rendering performance issue, we considered caching the layout path but my discussion with Owen suggested that it would gain us nothing. That, plus the perceived difficulty of doing this (since we may have to change brushes mid-stroke), shied me away from bothering to try.

These problems led me to consider implementing our own text layout/rendering engine to see if we could solve the above problems since the pango maintainers didn't seem to know what the problems could be offhand and thus had no suggestions for us.

At first, my text layout/rendering engine only handled rendering of glyphs via bitmaps, but even so, the result was that this new layout/rendering engine was quite a bit faster than pango.

Seeing this, Chris Toshok, Larry Ewing and I started digging into pango text rendering performance problems a bit more, not quite willing to give up on pango.

Toshok noticed that pango was loading the same font over and over again each frame, so started digging into that aspect a bit and came up with a patch to pango to fix a bug where it used the entire transform matrix as part of the hash instead of just the scaling component (which is all that was needed for uniqueness).

For one of the very simple text rendering test cases we had (the text "Moonlight in 21 Days" spinning and resizing via cairo matrix transforms), Toshok's patch nearly doubled the speed of pango rendering from something like 20 to 40fps (40fps was our cap, so it may have even rendered faster).

Meanwhile, I began looking into that cairo path caching idea and discovered it wasn't nearly as complicated to implement as I had originally feared. The results were just as amazing, again doubling the performance or better, altho this was before I had applied Toshok's patch (so don't get the idea that my patch + Toshok's patch = 4x speed improvement).

Not only did my patch make a huge performance improvement, it also got rid of the glyph jittering.

Unfortunately, this still left us with problem #3 as well as a few other problems regarding layout dissimilarities between pango and Microsoft's text layout in Silverlight, so for now, it seems I needed to go back to my own text layout/rendering engine.

Once I had finished adding support for rendering glyph paths, I implemented a similar cairo_path_t caching hack for my own text rendering engine and made it possible to choose which text layout/rendering engine to use at runtime via an environment variable.

Out of curiosity, I decided to compare performance of my own text layout/rendering engine vs pango on a test case I had of several "Hello" strings each having different combinations of matrix transforms applied to them in an ongoing animation. One of the "Hello" strings was simply undergoing FontSize changes which cause each of the text layout engines to have to recalculate the layout (wrapping, etc).

The performance difference was shocking... the pango implementation (which doesn't even render the underline for one of the text strings due to a bug in my cairo_path_t caching hack? If anyone has any suggestions on how to fix this in mango.cpp, don't hesitate to poke me) only gets about 23fps while my home-rolled implementation gets 45fps.

It might be possible to improve the performance of my home-rolled implementation if I were to fix my code to use a true FT_Face LRU cache... right now it simply keeps a hash of loaded FT_Faces with a ref_count, when that ref_count hits 0, it gets removed from the hash table. This means that each frame it has to load a new FT_Face from FontConfig because the FontSize attribute changes and since it was had the only ref on that particular FT_Face, it goes away and has to be reloaded again next time it changes back to that size. Oh, and the 45 fps was with debug spew turned on showing me whenever a new font got loaded - so turning off that printf() would probably bump me up to 50fps (which is the new fps cap on my machine).

As a further test, I removed the "Hello" TextBlock that had the FontSize attribute changes each frame. The result was that both pango and my own text layout/rendering engines jumped to ~50fps.

This suggests pango's layout calculation is where the performance bottleneck is.

I guess I'll have to dig into this problem some more later... Or, even better, maybe one of the pango developers can take a peek at Moonlight's font.cpp and see if they can maybe glean some ideas from there that they can apply to pango :)

Implementing Silverlight in 21 Days

For those of you that want to know what we're all about, it's like this ya'll: this is 10% luck, 20% skill, 15% concentrated power of will, 5% pleasure, 50% pain and 100% reason to remember the name...
  -- Fort Minor

Miguel de Icaza, Chris Toshok, Jackson Harper, Sebastien Pouliot, Everaldo Canuto, Rolf Bjarne Kvinge, Atsushi Enomoto, myself and a few other guys from the kick-ass Mono Team, here at Novell, managed to do the impossible: Implement Silverlight in just 21 days.

3 Weeks ago, Miguel sent out a Call-To-Arms email to our internal Mono Team mailing list explaining that he'd been invited to participate in Microsoft's conference in France where he'd he able to demo whatever he had accomplished as far as reimplementing a version of Silverlight for Linux. I would say that within minutes of the team reading that call to arms, I and a bunch of others replied, volunteering for the mission...

I think I speak for all of us when I say that it was a blast working on this project. I know that, for myself, even after heading home from work at around 8 or 9pm every night, I'd walk in the door, grab a quick bite to eat and immediately sit down in front of my computer and begin hacking on it again and I'd almost always find at least Jackson and Toshok on IRC hacking on it too.

I haven't been this enthusiastic about hacking in a long time (and that says a lot seeing as how I said the same thing just a few months ago when I joined the Mono Team to work on MonoDevelop). Even now, I've got ideas I want to experiment with to improve Moonlight even further. I'll probably get back to hacking on it (even though the 21 day hackathon is over) as soon as I finish posting this.

To all the people out there that congratulated us in response to reading Miguel's blog, thanks - we appreciate it. It feels good to both have accomplished something this amazing and be cheered at the same time :-)

To those of you who think this is a waste of time because similar things exist, I have this to say:

nothing ventured = nothing gained.

We, as the Free Software Community, wouldn't have all the great and wonderful things that we have today if the developers didn't "waste their time implementing alternatives." I urge you to take a moment to reflect on that.

If, after that reflection, you still wish to resist change: no one is forcing you to use Mono - you're free to remain in the stone age, living in a cave, trying to write software by scraping coal along the cave walls ;-)

That said, Rock On fellow Mono lovers!

DemoLib

Started hacking on a new project this weekend... just a little something to break the monotony.

The idea of DemoLib is to make my life simpler in those rare moments when I get inspired enough to write a 3d graphics demo. Just so I don't ever again have the excuse of not having all the proper utility classes and/or framework for writing a demo, I've gone and written those foundation classes/interfaces. I now have reasonable Vertex, Vector and Camera classes... along with a "Scene" interface and a default "Demo" class which plays the scenes, trying to keep a reasonable frame rate by slowing down or dropping frames, depending on the speed of rendering on the system.

I've also implemented 2 example Scene subclasses (neither take advantage of my Camera class yet... but that's because they were written before I had implemented it... they were both also precursors to my Vertex and Vector classes, but have been mostly ported to using those classes now).

Here are some token screenshots:

It's all in c#, because, well, why not? For those interested in watching my progress (or for laughing at the 3d graphics programming noob), I've checked DemoLib into Mono's subversion repository, so... `svn co` away.