Recent changes in comm-central’s test mechanisms

There’s been a few changes around comm-central’s automated testing recently that I’ve been meaning to post about.

The most important changes to note are:

  • make check now only runs the compiled-code unit tests (we only have two of these in mailnews).
  • make xpcshell-tests now runs the xpcshell tests that were previously run as part of make check. I have just checked in the patch which means you can now do make -C objdir/mailnews xpcshell-tests like you could with the check command before.
  • The check-one and check-interactive commands remain the same.

I have also just updated the wiki page explaining the numbers on the Tinderbox pages.

The xpcshell-tests also now have shutdown leak statistics enabled. Unfortunately for mailnews we currently have a lot of leaks due to our test harness (which provides a consistent profile directory that isn’t the build directory and a few other things). We’re currently not cleaning the test harness leaks up as this would break check-interactive – a useful tool to developers. Hopefully this will be fixed soon (patch in review) and then we can tidy up our leak logs.

>100 in-testsuite+ bugs in MailNews Core & Thunderbird

A discussion at our status meeting last week, prompted me to take a look at how many bugs MailNews Core and Thunderbird have marked as in-testsuite+. I was pleased to see it was just reaching 100.

(For those that don’t know, we mark a bug as in-testsuite+ when we have an automated test that covers that bug).

I committed the first in-testsuite+ bug for MailNews in November 2007, this covered some very basic address book functions. Since then we’ve grown the tests to cover address book (more extensively), POP3, IMAP, SMTP, NNTP, bayesian spam filter, aspects of message composition and sending, gloda (the new search database), text import of address books, searching messages, folder management, and a few other things.

Although our actual code coverage of these areas is not very big, we do cover a lot of different areas of MailNews with these tests. Our major protocols are covered (to a limited extent) with the use of fake servers – as a result of the password manager switch, we also have basic authentication tests for these protocols. So if someone breaks something significant, we’re much more likely to know about it before it even hits the nightly builds.

We’ve never (yet) had a policy of all patches requiring unit tests, because for Thunderbird we haven’t got the same testsuite possibilities as Firefox – mainly due to the lack of a browser interface. Various solutions have been proposed and partially investigated, I expect we’ll be looking at the chrome/UI area of testing more after we release Thunderbird 3.

We’re going to be looking at how to encourage writing more unit tests, and what we can do to make writing them easier, so keep watching.

I’d like to say thank you to all those that have written unit tests for MailNews so far. Keep going, unit tests have, and will make Thunderbird & SeaMonkey better products.

Tinderstatus 0.3 is here!

With the addition of new tinderbox trees for various branches happening in the few months before Christmas, and Thunderbird’s setting up of a Thunderbird3.0 tree, I finally decided to get round to doing some more hacking on Tinderstatus.

As a result, we now have the capability to monitor different trees on different icons (up to 10):

Having been using these for a couple of weeks now, I can say that I find it much easier to monitor the trees I’m interested in and get quick status for them all and I’m glad to say the update is now live on amo.