Setting up continuous integration for PHP using Hudson and Phing

In this, my first post, I’m going to write about the benefits of Unit Testing and how Continuous Integration (CI) can be used to get the best out of Unit Testing. This will include details of how I setup a CI system using Hudson CI server, Phing build tool combined with various other analysis tools (including PHP Unit).

One of the best explanations of Unit Testing I’ve read was posted by benzado on Stack Overflow.

Unit testing is a lot like going to the gym. You know it is good for you, all the arguments make sense, so you start working out. There’s an initial rush, which is great, but after a few days you start to wonder if it is worth the trouble.

The difficulty with Unit Testing is keeping it up. It is very easy to slip into poor habits and before you know it there’s a huge chunk of code with no tests. Possibly a huge, badly designed chunk of code, that didn’t benefit from having tests written before it was coded. Before you know what’s going on, you end up with a project that you really can’t write tests for, because retrofitting the tests is near impossible.

For me, there are two critical reasons for Unit Testing:

  1. Enforcing good design
    To be able to write tests, you need to be able to zero in on a “unit” of code, isolating it from all the rest of your 1,000,000 lines of web application. Writing Unit Tests forces you to design systems that have loose coupling because otherwise it is impossible to test.
  2. Allowing changes to be made in confidence
    Without Unit Tests, you get to the point where no one really wants to make any changes to the code. This is especially true in a commercial environment, where many people have worked on the code, including some key team member who has since left. Unit Tests allow you to make changes to one part of the code and be pretty convinced you haven’t messed up something else.

Continuous integration

Martin Fowler describes the process of Continuation Integration in detail. He suggests:

Continuous Integration is a software development practice where members of a team integrate their work frequently, usually each person integrates at least daily – leading to multiple integrations per day. Each integration is verified by an automated build (including test) to detect integration errors as quickly as possible. Many teams find that this approach leads to significantly reduced integration problems and allows a team to develop cohesive software more rapidly. This article is a quick overview of Continuous Integration summarizing the technique and its current usage.

The key idea behind CI is to do what is most painful often, namely “building” everyone’s code from source and making sure it all works.

A CI system usually consists of the following key elements:

Continuous integration

Continuous integration

  • Developers commit code
  • CI server detects changes
  • CI server checksout code, runs tests, analyses code
  • CI server feeds back to development team

If you want to find out more about CI, I recommend the excellent book Continuous Integration: Improving Software Quality and Reducing Risk. There is an excerpt published on JavaWorld which covers a lot of the key advantages. In particular, it highlights:

1. Reduce risks
2. Reduce repetitive manual processes
3. Generate deployable software at any time and at any place
4. Enable better project visibility
5. Establish greater confidence in the software product from the development team

CI gets the most out of Unit Tests by forcing them to be run after every change. Not only that, but with a good CI setup, developers instantly know if they haven’t written enough tests. If avoids the situtation where Joe Bloggs has added in a huge chunk of code with zero tests.

Setting up CI for a PHP project

To get my environment setup, I consulted the following blog posts which are worth a read:

  1. http://blog.jepamedia.org/2009/10/28/continuous-integration-for-php-with-hudson/
  2. http://toptopic.wordpress.com/2009/02/26/php-and-hudson/

I’m assuming you’re using a CentOS 5 server (or I guess RHEL5). If not, you may still find various parts of this useful.

1. Install JDK

EPEL provide a set of CentOS packages, including a package for openJDK. This is the easiest way of installing Java.

Firstly, setup EPEL:

wget -O /etc/yum.repos.d/hudson.repo http://hudson-ci.org/redhat/hudson.repo

Next install OpenJDK:

yum install java-1.6.0-openjdk

2. Install Hudson

Download and install the CentOS RPM for Hudson:

wget -O /etc/yum.repos.d/hudson.repo http://hudson-ci.org/redhat/hudson.repo
rpm --import http://hudson-ci.org/redhat/hudson-ci.org.key
yum install hudson

Now Hudson is installed, we can start using the standard CentOS “service” command.

service hudson start

We can check Hudson is working by pointing the browser at port 8080 (the default Hudson port). Hudson will work “out of the box” and give you a web interface immediately. This is the primary reason I decided to go with Hudson over the other possibilities, eg: CruiseControl and phpUnderControl. Although I didn’t do an exhaustive analysis before I decided on Hudson, it just seemed right to me.

To get the graphing engine working for Hudson, you may need to install x.

yum groupinstall base-x

3. Install phing

Phing is a PHP project build system or build tool based on Apache Ant. A build tool ensures that the process of creating your working web application from source code happens in a structured and repeatable way. This helps reduce the possibility of errors caused by simply uploading files via FTP or some other simple method.

Make sure PEAR is installed for PHP (this is the easiest way of installing phing):

yum install php-pear

Then install the PEAR phing package:

pear channel-discover pear.phing.info
pear install phing/phing

4. Setup SVN

If you haven’t got a Subversion repository, you’re going to need one (or some other SCM tool like CVS, GIT or Mercurial).

yum install mod_dav_svn

The simplest setup involves creating a repo in /var/www/svn/<my repo>

mkdir -v /var/www/svn/test
svnadmin create --fs-type fsfs /var/www/svn/test
chown –R apache:apache /var/www/svn/test

Setup Apache by pretty much uncommenting the lines in /etc/httpd/conf.d/subversion.conf. Once Apache restarted, you’ll be able to get to it via /repos/test, assuming you’re using the default settings (sets up SVN on /repos). I haven’t gone into the details of getting SVN up and running; there are lots of resources out there that will help you do this.

5. Install PHP tools

PHPDocumentor – to generate documentation automatically from code
pear install PhpDocumentor
PHP CPD – “copy and paste detector” for PHP

This requires PHP 5.2. At time of writing, this wasn’t standard with CentOS 5, but is part of the CentOS “test” repo. This can be setup by creating a yum repo file, eg: /etc/yum.repos.d/centos-test.repo and populating with:

[c5-testing]
name=CentOS-5 Testing
baseurl=http://dev.centos.org/centos/5/testing/$basearch/
enabled=1
gpgcheck=1
gpgkey=http://dev.centos.org/centos/RPM-GPG-KEY-CentOS-testing

Then you can do:

yum update php

You may also need to upgrade pear; if the install of phpcpd fails (below). To do this, try:

pear upgrade pear

or, if this wants to be forced, and you think it’s a good idea (I did):

pear upgrade --force pear

Finally we can install phpcpd!

pear channel-discover pear.phpunit.de
pear install phpunit/phpcpd
PHP Depend – help analyse quality of codebase

Note you may have update PHP to include the DOM module (first line below).

yum install php-dom
pear channel-discover pear.pdepend.org
pear install pdepend/PHP_Depend-beta
PHP Code Sniffer – analyse code for adherence to style/standards
pear install PHP_CodeSniffer-1.2.0
PHP Unit – unit test framework for PHP
pear channel-discover pear.phpunit.de
pear install phpunit/PHPUnit

To make PHP Unit work, we need XDebug installed, the PHP profiler.

yum install php-devel gcc
pecl install xdebug

6. Install Hudson plugins

Use the web interface to install the following plugins (Manage Hudson -> Plugins).

  • Checkstyle
  • Clover
  • DRY
  • Green Balls (handy because it shows successful builds as green circles rather than blue)
  • JDepend
  • xUnit (will handle the output of PHPUnit test results XML)

7. Setup the phing build script

The Phing build script defines what steps will be taken to “build” the application.

Hudson itself works by placing our code into a project workspace. It will checkout the code from subversion and place it into the following location, where “Test” is the name of our project.

/var/lib/hudson/jobs/Test/workspace/

We can then use the Phing build script to carry out a number of processes on this code. When we talk about “building”, what we will actually do is place the code where we need it so it can actually run the website (we’ll keep this within the workspace) plus we run tests etc…

We’ll keep the build script in the subversion repository, so effectively it will be updated from SVN each build. For this approach to work, the following XML needs to be stored in a file named build.xml, stored in the project root folder (within trunk), eg: /trunk/build.xml

<?xml version="1.0" encoding="UTF-8"?>
 <project name="test" basedir="." default="app">
    <property name="builddir" value="${ws}/build" />

    <target name="clean">
        <echo msg="Clean..." />
        <delete dir="${builddir}" />
    </target>

    <target name="prepare">
        <echo msg="Prepare..." />
        <mkdir dir="${builddir}" />
        <mkdir dir="${builddir}/logs" />
        <mkdir dir="${builddir}/logs/coverage" />
        <mkdir dir="${builddir}/docs" />
        <mkdir dir="${builddir}/app" />
    </target>

    <!-- Deploy app -->
    <target name="app">
        <echo msg="We do nothing yet!" />
    </target>

    <!-- PHP API Documentation -->
    <target name="phpdoc">
        <echo msg="PHP Documentor..." />
        <phpdoc title="API Documentation"
                destdir="${builddir}/docs"
                sourcecode="yes"
                defaultpackagename="MHTest"
                output="HTML:Smarty:PHP">
            <fileset dir="./app">
                <include name="**/*.php" />
            </fileset>
        </phpdoc>
    </target>

    <!-- PHP copy/paste analysis -->
    <target name="phpcpd">
        <echo msg="PHP Copy/Paste..." />
        <exec command="phpcpd --log-pmd=${builddir}/logs/pmd.xml source" escape="false" />
    </target>

    <!-- PHP dependency checker -->
    <target name="pdepend">
        <echo msg="PHP Depend..." />
        <exec command="pdepend --jdepend-xml=${builddir}/logs/jdepend.xml ${ws}/source" escape="false" />
    </target>

    <!-- PHP CodeSniffer -->
    <target name="phpcs">
        <echo msg="PHP CodeSniffer..." />
        <exec command="phpcs --standard=ZEND --report=checkstyle ${ws}/source > ${builddir}/logs/checkstyle.xml" escape="false" />
    </target>

    <!-- Unit Tests & coverage analysis -->
    <target name="phpunit">
        <echo msg="PHP Unit..." />
        <exec command="phpunit --log-junit ${builddir}/logs/phpunit.xml --log-pmd ${builddir}/logs/phpunit.pmd.xml --coverage-clover ${builddir}/logs/coverage/clover.xml --coverage-html ${builddir}/logs/coverage/ ${ws}/source/tests"/>
    </target>
</project>

8. Setup Hudson

The first step is to create a new job.

  • From the Hudson homepage, click New Job.
  • Enter a Job name, for example “Dave’s Product Build” and choose “Build a free-style software project”. Click OK.

Now you need to configure the job; the configuration form should be displayed immidiately after adding.

Under Source Code Management choose Subversion and enter:

  • Repository URL: http://www.myrepo.com/path/to/repo
  • Local module directory: source
  • Check “Use update” which speeds up checkout

Under Build Triggers select Poll SCM and enter the following schedule:

5 * * * *
10 * * * *
15 * * * *
20 * * * *
25 * * * *
30 * * * *
35 * * * *
40 * * * *
45 * * * *
50 * * * *
55 * * * *

Note that this will poll for changes to the repository every 5 minutes and rebuild if any changes are detected.

Under Build click the button to Add build step and choose Execute shell, enter the command:

phing -f $WORKSPACE/source/build.xml prepare app phpdoc phpcs phpunit -Dws=$WORKSPACE

Under Post-build Actions choose:

  • Check Publish Javadoc and then enter:
    Javadoc directory = build/docs/
  • Check Publish testing tools result report and then click Add and pick PHP Unit, enter:
    + PHPUnit Pattern = build/logs/phpunit.xml
  • Check Publish Clover Coverage Report and enter:
    + Clover report directory = build/logs/coverage
    + Clover report file name = clover.xml
  • Check Publish duplicate code analysis results and enter:
    + Duplicate code results = build/logs/phpunit.pmd-cpd.xml
  • Check Publish Checkstyle analysis results and enter:
    + Checkstyle results = build/logs/checkstyle.xml

Finally, click Build Now to test it all works.

Tags: , , , , ,

36 Responses to “Setting up continuous integration for PHP using Hudson and Phing”

  1. Social comments and analytics for this post…

    This post was mentioned on Twitter by davegardnerisme: Setting up Continuous Integration for PHP with Hudson and Phing. http://is.gd/4QUTo #php (my first blog post!)…

  2. ramon says:

    man, that is so cool

  3. Rehan says:

    Here is a platform neutral way of installing the required JRE (I don’t think you need the JDK…). I also don’t like using the Linux package managers because of 1) outdated packages and 2) it installs files everywhere.

    Goto http://www.java.com/en/download/manual.jsp, find your relevant download. It’ll be one of the “self-extracting” files.

    Use wget to download the binary file, in my case its: wget http://javadl.sun.com/webapps/download/AutoDL?BundleId=35679

    Move the newly downloaded file to a new location. I have my JREs in /opt/java

    Make the binary file executable: chmod +x name_of_file.bin

    Execute the binary file: ./name_of_file.bin

    Answer yes to all the questions and your JRE should be extracted and ready to go where ever you moved it to. To use the new JRE you can refer to java by the absolute path: /opt/java/jre6_u10/bin/java. Since PHP developers don’t have much of a need to use a JRE you don’t have to worry about setting the java environment variables, but if you do, look here: http://www.cyberciti.biz/faq/linux-unix-set-java_home-path-variable/

  4. Dave says:

    Thanks for the suggestion Rehan.

    I understand your problems with package managers. I quite like them because I think it makes my life simpler.

  5. hoschi says:

    your poll schedule is simple: */5 * * * *
    ;)

  6. Dave says:

    Good point hoschi!

  7. Nick says:

    Thanks for this tutorial! I did run into a small issue, though. Your example uses the Hudson project name “Dave’s Product Build.” I used a similar name and am now have trouble with both the $WORKSPACE and ${ws} variables as the job directly has spaces in it and Hudson’s command line executor and Phing are both have trouble because the workspace name is not escaped.

    I just wondered if you had any suggestions on how to handle this.

    Thanks!

  8. Dave says:

    Hi Nick. Interestingly I didn’t actually use this name in my “working” example, I used a one-word project name, so I guess I avoided the problem that way! I didn’t think about the impact of changing it to something else when writing up the post.

    I think my solution would just be to use a simpler one-word project name! Let me know if you do manage to find a way round it.

  9. In step eight you provide configuration for the Poll SCM section so that Hudson checks SVN every 5 minutes. Instead of listing out:

    5 * * * *
    10 * * * *
    15 * * * *
    20 * * * *
    25 * * * *
    30 * * * *
    35 * * * *
    40 * * * *
    45 * * * *
    50 * * * *
    55 * * * *

    you can simply use */5 * * * *

  10. Dave says:

    A good point; also noted earlier by hoschi.

  11. Zachary Young says:

    Hi Dave,

    I am a Config Eng. and am just getting started using PHP (I am not a PHP developer). I know that our PHPUnit tests will require Zend, which I am currently waiting for help to install on our CentOS box which is hosting Hudson.

    Do you have any experience with Zend? If so, would integrating it with Hudson change any of the above?

    Thank you,
    Zach

  12. Dave says:

    If your application uses the Zend Framework (I assume that’s what you mean) then that’s fine – nothing in this post needs to be changed. You just need to make sure your unit tests have included the ZF bootstrap/autoloader. Hudson doesn’t care what kind of PHP framework / technologies you use.

  13. drahreg80 says:

    Did you forgot to choose “Report JDepend” under Post-build Actions? Value should be like “build/logs/jdepend.xml”.

    Is there an way to show the graph of phpdepend in hudson?

  14. Dave says:

    @drahreg80 – Yes you’re right about reporting on PHPDepend.

    Not sure about the graph. Let me know if you find a way of getting this displayed.

  15. EllisGL says:

    The build command should be:
    phing -f $WORKSPACE/source/build.xml prepare app phpdoc phpcpd pdepend phpcs phpunit -Dws=$WORKSPACE

  16. Dave says:

    @EllisGL
    Yep – you’re right about the build command. This amended version will make sure every Phing “target” gets executed. Another interesting point here is that you can setup different Hudson jobs to execute different targets. This is handy if the “full” build takes too long – in this situation you can setup a build that only executes a subset of tests (for example).

  17. ff78 says:

    Hi,
    I don’t understand how it can be used especially the build.xml

    What’s the role of the “app” target ? The app folder is empty so there’s no php files for phpdoc…

    Regards.

  18. Dave says:

    The build script in my example is fairly light-weight. You’re right – the “app” target is pointless. However it really depends on what you want your build process to “do”. New build scripts that I’ve got do things like crunch config scripts to generate code and look at code annotations to work out dependency injection container “wiring” config. These all lie in the “app” target. Further more I’ve added in some other targets to do things like deploy to a staging environment… there’s loads you can do here.

    I guess that “build” as a concept is more obviously useful in the context of compiled apps where you actually have to build them! Things like PHPDoc and/or CodeSniffr etc.. can really just run on the raw “source” code (as checked out of SCM).

    That said it all depends what you do! I’d urge you to have a play with a build.xml file. Automating things is a very powerful thing and I find it very useful. That said, if you wanted you could probably not use Phing and just use Hudson to do the heavy lifting.

  19. Ben Waine says:

    Dave,

    I found this guide really useful, thanks very much for taking the time to blog it!

    Have you got any insight into automatic deployment to staging and also did you learn anything about getting phpdpend graph on the main dash?

    Ben

  20. Dave says:

    It depends on how you want the deployment to work; what you’re requirements are. Previously, I have tweaked the “build” script to create a “deploytoalpha” Phing target. This was pretty simple stuff, using rsync to get the data over.

    Right now, I’m working somewhere where things are more sophisticated and actually less “automatic”. A developer must submit a ticket to deploy staging release, and this must involve a tagged version of the code (in SCM). This system doesn’t integrate (currently) with our Hudson setup.

  21. Ben Waine says:

    Hi Dave,

    I followed your advice regards the build being rsync’d to an ‘alpha’ staging area.

    Works a treat, though it was a little fiddly setting up the public / private SSH keys!

    Thanks,

    Ben

  22. I just discovered this template by Sebastian Bergmann: http://github.com/sebastianbergmann/php-hudson-template . It can be used to setup PHP jobs quickly. There’s also a Phing plugin for Hudson, it can be installed via the webinterface and the documentation can be found at http://wiki.hudson-ci.org/display/HUDSON/Phing+Plugin .

  23. Kwasi says:

    I can’t seem to set PHING_HOME I get message directory doesn’t seem like phing home. Though I’m point to location phing was installed too.

    Can an one help me with this issue:

    I’m running hudson 1.374

  24. Olle Jonsson says:

    Depending on one’s requirements, phpUnderControl could be interesting as a more “shrink-wrapped” alternative.

    http://phpundercontrol.org/

    “phpUnderControl is an addon application for the continuous integration tool CruiseControl, which integrates some of the best PHP development tools. This project aims to make your first steps with CruiseControl and PHP as easy as possible. Therefore phpUnderControl comes with a command line tool that performs all modifications to an existing CruiseControl installation.”

  25. Chris says:

    I got everything setup, but I’m having one problem. I am unable to execute my PHPUnit Phing target via Hudson. It works fine via command line, but gives this error when running in Hudson (same build.xml):

    strpos(): Empty delimiter

    It seems to happen when the $this->dispatch(…) method is executed.

    Anyone have any ideas? I’ve been working on this for two days and it’s driving me crazy!

  26. Dave says:

    Which version of PHPUnit? I know there was some issue recently with PHPUnit 3.5. However it sounds like you have a different issue.

  27. Laoneo says:

    It is called now Jenkins and not hudson anymore

  28. Dave says:

    Not strictly true. Jenkins is forked from Hudson after a dispute with Oracle, however both projects will continue to exist.
    http://en.wikipedia.org/wiki/Jenkins_%28software%29

  29. A.J. Brown says:

    @Dave,

    If you install using your instructions within this article on CentOS, it will be Jenkins that’s installed, not Hudson.

    You will have to do “service start jenkins”.

  30. Pura says:

    Thanks. It’s a very helpful article. But I’ve a question.
    I’ve an application build on Zend Framework and file structure of tests is pretty much like application structure.

    tests
    —-modules
    ——–default
    ————controllers
    —————-aControllerTest.php
    ————models
    —————-aModelTest.php
    —————-a2ModelTest.php
    —————-resources
    ———————aResourceModelTest.php
    ——–anothermodule
    ————controllers
    and same……

    So, in the script above you’ve got a line as follows under the target phpunit.

    My questions are:
    1. How would I change the line above to adopt my file structure? In other words what i replace $(ws}/source/tests with?

    2. Where in the script I specify, these are the tests and these are the fileset tested?

    Your help would complete my CI integration and appreciated a lot.

    TIA.
    P

  31. Dave says:

    I should update the post really, now that we have the Jenkins/Hudson split! Thanks for pointing this out.

  32. Pura says:

    Hi Dave

    Any suggestion on my question above ? (I should have saide Phing Script rather than just script, in the previous post)

    Thanks
    Pura

  33. Stewart says:

    You can just do */5 * * * * for the SCM polling …

  34. Dave, indeed things have come a long way since you wrote this post. I for one believe that there’s a new age of easier CI in front of us. Jenkins/Hudson, CruiseControl, etc, are all mammoths, designed to accomodate the 20% more devastatingly complex web projects. But for the simpler 80%, it’s just too much hassle… I’ve been developing an open source project, called Cintient, which is precisely my solution to this. Cintient aims to ease in CI practices in those 80% simpler/smaller web projects, and make everyone use CI practices. It is a PHP-based, self-hosted, dead-simple CI server, and it’s up on GitHub (with some screenshots, BTW). It’s still pretty much a one-man-show, but I’m trying to get traction and people’s attention to it, because I really think it rocks. Go check it out, maybe you can take it into account when you rewrite this post in the future. :-)
    https://github.com/matamouros/cintient

    Cheers.
    Pedro.

  35. Dave says:

    I think whilst Jenkins is complex, it is also very easy to mould it to your needs, plus many people know how to use it. Therefore I’m not entirely convinced by your arguments! But if you want to get more people using CI, I guess I can’t complain.

  36. Dave says:

    /me really should update the content.

Leave a Reply