I should’ve written this up last month sometime, as it mostly results from conversations I had with JonBob and jvandyk over above average but not exactly stellar Thai food in Vancouver.

Drupal’s growth rate has reached the point where individuals testing individual bits is no longer keeping up with the other problems that the type of patching we do can cause: the problem, of course, being that one change here can break something seemingly unrelated over there. This kind of problem exists on almost every type of software, but previously I believe the methodical nature of patch committing by Dries and Steven has helped to mollify it somewhat, and the general smallish nature of patches being committed have made it easier to see what might happen.

Drupal’s growth is skyrocketing, though, and major changes have been put in and are coming in the future.

During the google Summer of Code, which is almost but not quite before my time with Drupal, a student wrote a simpletest module, and populated it with some tests. Moshe, at the very least, still works with this somewhat, but it’s been some months before the project’s been touched. And for simpletest to be useful, it needs to be very, very important to the development process.

Now, I don’t feel the need to address the actual tests here. This isn’t to say that it isn’t a huge, major part of the project, but I believe the actual tests are a different problem that needs to be solved in part by getting someone dedicated to maintaining and writing tests. But that’s not the focus of this article. My focus is getting us to the point where we use the module, and I believe that requires automation.

The ideal we discussed — and by the way, this was mostly JonBob’s idea and I am basically documenting and embellishing it here — is a tiered process that can be started with the minimum functionality and expanded upon until everything works well.

To start with, we need to be able to automate a Drupal install. We don’t have to automate it on every environment; in fact, we only really need to automate it on the testbed, which we can hope would run on drupal.org’s hardware. This would involve:

• Start with a chrooted environment, for safety.
• Creating a directory under test.drupal.org that is uniquely named; based on a timestamp or a hash or somesuch. This directory is empty.
• Doing a fresh CVS checkout of Drupal of the appropriate version. This will usually be HEAD but it can be used to test point releases of older versions as well.
• Apply any patches that we are going to test. If the patch fails, stop.
• Creating a database. This requires some security because it will require the mysql root password. Alternatively, we use one test database and use prefixes. But to test properly we may need to have both prefixed and non-prefixed databases available. I think that’s actually not a huge concern.
• Run the mysql script to create all the databases.
• Run any additional queries necessary. This includes, at the very least, creating an administrative user the easy way, and ensuring the simpletest module (and any modules that require testing) are turned on. It might require simply activating all core modules.
• Generate some content. I can imagine many tests will require some existing content in the database. devel.module includes scripts to generate basic content.
• Run the simpletest batch.

Stage two builds upon this with project module’s help. It requires adding 3 states to project issues: patch (needs automated test), patch (passed automated tests), and patch (failed automated tests). A cron job runs incrementally, and it collects all issues attached to core that are set that they need automated testing. The above is run and the output collected. If all of the tests past, the patch is marked as passed, otherwise it is marked as failed. If a patch fails tests, the patch author must determine if the test broke because his code was broken, or because it changed the expected output. If it’s the latter, the patch can be rerolled to include patches to the test.

A test that is marked as passed is now ready for human review. Once it passes human review, it can then be committed. A process also needs to clean up no longer used test-beds, so they must be carefully tracked for garbage collection purposes. The process could do it right away, but it might be nice to leave the testbed for a few hours so that a tester could check out the instance and perhaps see what’s going on.

Security is an important issue here. We need ways to ensure that random PHP isn’t run via the patches. Perhaps only vetted users are allowed to submit patches for automated testing. Perhaps chrooting the testbed is sufficient security.

As near as I can tell, the actual implementation of this procedure — even without any actual simpletests running — isn’t that hard. I’ve already managed to write a short script that does a Drupal install on my system, just to make it easier for me to do the testing process, and I can expand it a little bit to be closer to the process I’ve outlined above. Most of this is straight-forward, brute force beating on Drupal. Then there’s the matter of tests. But they can come later. As we build a library of tests, this can only get better and better. All tests are maintained by force because patches that break existing tests simply won’t be committed until the tests are updated. If the patch was important, updating the test will be important too.