Continuous Integration with Maestro

If you are not already familiar with it, continuous integration enables automated builds of your project on a regular interval, ensuring that conflicts are detected earlier in a project’s release life cycle, rather than close to a release. More than just nightly builds, continuous integration can enable a better development culture where team members can make smaller, iterative changes that can more easily support concurrent development processes. As such, continuous integration is a key element of effective collaboration.

Continuum1 is Maven’s continuous integration and build server. It is also a component of Exist’s Maestro which is referred to as Build Management. In this chapter, you will pick up the Proficio example from earlier in the book, and learn how to use Maestro Build Management to build this project on a regular basis. The examples discussed are based on Maestro 1.3, however newer versions should be similar. The examples assume you have Subversion installed, which you can obtain for your operating system from http://subversion.tigris.org/.

First, you will need to install the Maestro Project Server. This is very simple – once you have downloaded it from www.exist.com and unpacked it, you can run it. For example on Windows, use the following command:

[maestro_home]/project-server/bin/windows-x86-32/run.bat

For Linux, use

./[maestro_home]/project-server/bin/linux-x86-32/run.sh start

or

./[maestro_home]/project-server/bin/plexus/plexus.sh

You need to make the file executable in Linux by running the command from the directory where the file is located. For example:

chmod +x run.sh

Refer to the Maestro Getting Started Guide for more instructions on starting the Maestro Project Server. This document can be found in the Documentation link of Maestro user interface.

There are scripts for most major platforms, as well as the generic bin/plexus/plexus.sh for use on other Unix-based platforms. Starting up the Maestro Project Server will also start a http server and servlet engine.

You can verify the installation by viewing the web site at http://localhost:8080/continuum/.

The first screen to appear will be the one-time setup page shown in the figure below. The configuration on the screen is straight forward – all you should need to enter are the details of the administration account you’d like to use.

Figure 7-1: The Administrator account screen

For most installations, the figure below shows all the configuration that’s required when Build Management is accessed for the first time. If you are running Maestro on your desktop and want to try the examples in this section, some additional steps are required. As of Maestro 1.3, these additional configuration requirements can be set only after the previous step has been completed, and you must stop the server to make the changes (to stop the server, press Ctrl-C in the window that is running Maestro).

Figure 7-2: Build Management general configuration screen

To complete the Build Management setup page, you can cut and paste field values from the following list:

In the following examples, POM files will be read from the local hard disk where the server is running. By default, this is disabled as a security measure, since paths can be entered from the Web interface. To enable this setting, edit the application.xml found in the [maestro_home]/project-server/apps/continuum/webapp/WEB-INF/classes/META-INF/plexus/ directory and verify that the following lines are present and are not commented out:

[...]
<implementation>org.codehaus.plexus.formica.validation.UrlValidator</implementation>
<configuration>
    <allowedSchemes>
        [...]
        <allowedScheme>file</allowedScheme>
    </allowedSchemes>
</configuration>
[...]

To have Build Management send you e-mail notifications, you will also need an SMTP server to which to send the messages. The default is to use localhost:25. If you do not have this set up on your machine, edit the file above to change the smtp-host setting. For instructions, refer to Configuring mail, servers, ports, and directories section of Maestro Project Server User Guide found in the Documentation link of the Maestro user interface.

After these steps are completed, you can start Maestro again.

The next step is to set up the Subversion repository for the examples. This requires obtaining the Code_Ch07.zip archive and unpacking it in your environment. You can then check out Proficio from that location, for example if it was unzipped in C:mvnbooksvn, execute the following:

C:mvnbookproficio> svn co 

file://localhost/C:/mvnbook/svn/proficio/trunk

file://localhost/[path_to_svn_code]/proficio/trunk

Edit proficio/trunk/pom.xml to correct the e-mail address to which notifications will be sent, and edit the location of the Subversion repository, by uncommenting and modifying the following lines:

[...]
<ciManagement>
    <system>continuum</system>
    <url>http://localhost:8080/continuum
    <notifiers>
        <notifier>
            <type>mail</type>
            <configuration>
                <address>youremail@yourdomain.com</address>
            </configuration>
        </notifier>
    </notifiers>
</ciManagement>
[...]
<scm>
    <connection>
        scm:svn:file://localhost/C:/mvnbook/svn/proficio/trunk
    </connection>
    <developerConnection>
        scm:svn:file://localhost/C:/mvnbook/svn/proficio/trunk
    </developerConnection>
</scm>
[...]
<distributionManagement>
    <site>
        <id>website</id>
        <url>
            file://localhost/C:/mvnbook/repository/sites/proficio/reference/${project.version}
        </url>
    </site>
</distributionManagement>
[...]

The ciManagement section is where the project’s continuous integration is defined and in the above example has been configured to use Maestro Build Management locally on port 8080.

The distributionManagement setting will be used in a later example to deploy the site from your continuous integration environment. This assumes that you are still running the repository Web server on localhost:8081, from the directory C:mvnbookrepository. If you haven’t done so already, refer to section 7.3 for information on how to set this up.

Once these settings have been edited to reflect your setup, commit the file with the following command:

C:mvnbookproficiotrunk> svn ci -m "my settings" pom.xml

You should build all these modules to ensure everything is in order, with the following command:

C:mvnbookproficiotrunk> mvn install

You are now ready to start using Continuum or Maestro Build Management.

If you return to the http://localhost:8080/continuum/ location that was set up previously, you will see an empty project list. Before you can add a project to the list, or perform other tasks, you must either log in with the administrator account you created during installation, or with another account you have since created with appropriate permissions. The login link is at the top-left of the screen, under the Maestro logo.

Once you have logged in, you can now select Maven 2.0+ Project from the Add Project menu. This will present the screen shown in figure 7-3. You have two options: you can provide the URL for a POM, or upload from your local drive. While uploading is a convenient way to configure from your existing check out, in Maestro 1.3 and newer versions, this does not work when the POM contains modules, as in the Proficio example. Instead, enter the file:// URL as shown. When you set up your own system later, you will enter either a HTTP URL to a POM in the repository, a ViewCVS installation, or a Subversion HTTP server.

Figure 7-3: Add project screen shot

When using the file:// protocol for the URL, the File protocol permission should be selected. To configure:

1. Go to the Project Server page.
2. Under Configure > Build Management, check the box before FILE.
3. To make the settings take effect, click the Save button.

This is all that is required to add a Maven 2 project to Build Management. After submitting the URL, Maestro Build Management will return to the project summary page, and each of the modules will be added to the list of projects. Initially, the builds will be marked as New and their checkouts will be queued. The result is shown in the figure below.

Figure 7-4: Summary page after projects have built

Build Management will now build the project hourly, and send an e-mail notification if there are any problems. If you want to put this to the test, go to your earlier checkout and introduce an error into Proficio.java, for example, remove the interface keyword:

[...]
    public Proficio
    [...]

Now, check the file in:

C:mvnbookproficiotrunkproficio-api> svn ci -m "introduce error"src/main/java/com/exist/mvnbook/proficio/Proficio.java

Finally, press the Build Now icon on the Build Management user interface next to the Proficio API module. First, the build will show an “In progress” status, and then fail, marking the left column with an “!” to indicate a failed build (you will need to refresh the page using the Show Projects link in the navigation to see these changes). In addition, you should receive an e-mail at the address you configured earlier. The Build History link can be used to identify the failed build and to obtain a full output log.

To avoid receiving this error every hour, restore the file above to its previous state and commit it again. The build in Build Management will return to the successful state.

This chapter will not discuss all of the features available in Maestro Build Management, but you may wish to go ahead and try them. For example, you might want to set up a notification to your favorite instant messenger – IRC, Jabber, MSN and Google Talk are all supported.

Regardless of which continuous integration server you use, there are a few tips for getting the most out of the system:

• Commit early, commit often. Continuous integration is most effective when developers commit regularly. This doesn’t mean committing incomplete code, but rather keeping changes small and well tested. This will make it much easier to detect the source of an error when the build does break.
• Run builds as often as possible. This will be constrained by the length of the build and the available resources on the build machine, but it is best to detect a failure as soon as possible, before the developer moves on or loses focus. Build Management can be configured to trigger a build whenever a commit occurs, if the source control repository supports post-commit hooks. This also means that builds should be fast – long integration and performance tests should be reserved for periodic builds.
• Fix builds as soon as possible. While this seems obvious, it is often ignored. Continuous integration will be pointless if developers repetitively ignore or delete broken build notifications, and your team will become desensitized to the notifications in the future.
• Establish a stable environment. Avoid customizing the JDK, or local settings, if it isn’t something already in use in other development, test and production environments. When a failure occurs in the continuous integration environment, it is important that it can be isolated to the change that caused it, and independent of the environment being used.
• Run a copy of the application continuously. If the application is a web application, for example, run a servlet container to which the application can be deployed from the continuous integration environment. This can be helpful for non-developers who need visibility into the state of the application, separate from QA and production releases.

In addition to the above best practices, there are two additional topics that deserve special attention: automated updates to the developer web site, and profile usage.

In Chapter 6, you learned how to create an effective site containing project information and reports about the project’s health and vitality. For these reports to be of value, they need to be kept up-to-date. This is another way continuous integration can help with project collaboration and communication. Though it would be overkill to regenerate the site on every commit, it is recommend that a separate, but regular schedule is established for site generation.

Verify that you are still logged into your Maestro instance. Next, from the Administration menu on the left-hand side, select Schedules. You will see that currently, only the default schedule is available. Click the Add button to add a new schedule, which will be configured to run every hour during business hours (8am – 4pm weekdays).

The appropriate configuration is shown below:

Figure 7-5: Schedule configuration
To complete the schedule configuration, you can cut and paste field values from the following list:

The cron expression entered here is much like the one entered for a Unix crontab and is further described at http://www.opensymphony.com/quartz/api/org/quartz/CronTrigger.html. The example above runs at 8:00:00, 9:00:00,…, 16:00:00 from Monday to Friday.

The “quiet period” is a setting that delays the build if there has been a commit in the defined number of seconds prior. This is useful when using CVS, since commits are not atomic and a developer might be committing midway through an update. It is not typically needed if using Subversion.

Once you add this schedule, return to the project list, and select the top-most project, Maven Proficio. The project information shows just one build on the default schedule that installs the parent POM, but does not recurse into the modules (the -N or --non-recursive argument). Since this is the root of the multi-module build – and it will also detect changes to any of the modules – this is the best place from which to build the site. In addition to building the sites for each module, it can aggregate changes into the top-level site as required.

The downside to this approach is that Build Management (Continuum) will build any unchanged modules, as well – if this is a concern, use the non-recursive mode instead, and add the same build definition to all of the modules.

In this example you will add a new build definition to run the site deployment for the entirety of the multi-module build, on the business hours schedule. To add a new build definition, click the Add button below the default build definition. The Add Build Definition screen is shown below.

Figure 7-6: Adding a build definition for site deployment

To complete the Add Build Definition screen, you can cut and paste field values from the following list:

The goals to run are clean and site-deploy. The site will be deployed to the file system location you specified in the POM, when you first set up the Subversion repository earlier in this chapter, which will be visible from http://localhost:8081/sites/proficio/.

The arguments provided are ‑‑batch-mode, which is essential for all builds to ensure they don’t block for user input, and -Pci, which sets the given system property. The meaning of this system property will be explained shortly. The --non-recursive option is omitted.

You can see also that the schedule is set to use the site generation schedule created earlier, and that it is not the default build definition, which means that Build Now from the project summary page will not trigger this build. However, each build definition on the project information page (to which you would have been returned after adding the build definition) has a Build Now icon. Click this for the site generation build definition, and view the generated site from http://localhost:8081/sites/proficio/.

It is rare that the site build will fail, since most reports continue under failure conditions. However, if you want to fail the build based on these checks as well, you can add the test, verify or integration-test goal to the list of goals, to ensure these checks are run.

In the previous example, a system property called ci was set to true. In Chapter 6, a number of plugins were set up to fail the build if certain project health checks failed, such as the percentage of code covered in the unit tests dropping below a certain value. However, these checks delayed the build for all developers, which can be a discouragement to using them.

If you compare the example proficio/trunk/pom.xml file in your Subversion checkout to that used in Chapter 6, you’ll see that these checks have now been moved to a profile. Profiles are a means for selectively enabling portions of the build. If you haven’t previously encountered profiles, please refer to Chapter 3. In this particular case, the profile is enabled only when the ci system property is set to true.

[...]
<profiles>
    <profile>
        <id>ci</id>
        <activation>
            <property>
                <name>ci</name>
                <value>true</value>
            </property>
        </activation>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-pmd-plugin</artifactId>
                <executions>
                    [...]

You’ll find that when you run the build from the command line (as was done in Continuum originally), none of the checks added in the previous chapter are executed. The checks will be run when you enable the ci using mvn -Pci.

There are two ways to ensure that all of the builds added in Maestro Build Management use this profile. The first is to adjust the default build definition for each module, by going to the module information page, and clicking Edit next to the default build definition. As you saw before, it is necessary to do this for each module individually, at least in the version of Continuum current at the time of writing.

The other alternative is to set this profile globally, for all projects in Build Management. As Maven 2 is still executed as normal, it reads the ${user.home}/.m2/settings.xml file for the user under which it is running, as well as the settings in the Maven installation. To enable this profile by default from these settings, add the following configuration to the settings.xml file in <user_home>/.m2/settings.xml:

[...]
<activeProfiles>
    [...]
    <activeProfile>ci</activeProfile>
</activeProfiles>
[...]

In this case the identifier of the profile itself, rather than the property used to enable it, indicates that the profile is always active when these settings are read.

How you configure your continuous integration depends on the culture of your development team and other environmental factors such as the size of your projects and the time it takes to build and test them. The guidelines discussed in this chapter will help point your team in the right direction, but the timing and configuration can be changed depending upon your circumstances. For example, if the additional checks take too much time for frequent continuous integration builds, it may be necessary to schedule them separately for each module, or for the entire multi-module project to run the additional checks after the site has been generated, the verify goal may need to be added to the site deployment build definition, as discussed previously.