Saturday, November 27, 2010

How to Set Up Pentaho Community Build Framework

How to Set Up Pentaho Community Build Framework 


Please note that I am still writing on this article. I will update some parts later on. For some parts I have open questions that still have to be clarified. If you have any ideas/suggestions, please let me know and I happily will add it to this article.

Some sections were also copied from other sources, some of which are mentioned in the "Sources" section.

In the following chapters we will have a look at how to set up an easily maintainable environment. This article is not an introduction to Pentaho. I assume that you are familiar with the Pentaho products.

We will look at how to set up the Pentaho BI Server with the Community Build Framework (CBF). 

Why would you use CBF?
  • Easily upgrade to new version of the BI server without going through a long setup process
  • Easily create environment specific versions of your BI server without going through a long setup process

Pentaho BI Server Setup with CBF


Introducing Community Build Framework

We are in the lucky position that Pedro Alves from Webdetails offers the freely available CBF (Community Build Framework), which makes setting up and upgrading the BI Server for various environments a relative easy task. I try to talk you through the setup for the most popular operating sytems.

The main folder/file structure of CBF looks like this (simplified):

  • CBF
    • Pentaho BI Server source code folder(s): Each version of the source code will be stored in one folder. These folders are left completely untouched. Changes to the original source can be made via patches.
    • Project folder(s): For each project you can create a separate folder (i.e. project-nike, project-puma, ...) which holds the project specific configurations and files.
      • config
        • file, which tells ant what and where we want to build. A minimal config contains the path to your Java compiler, solutions and web server (among others). 
        • (optional) where env is a placeholder for any environment. So you could name it, etc. This is very neat as you can have global settings in the and environment specific ones in the specific When we are using Ant to build our BI server, ant will load the and then use the specific
      • patches: all files that need some kind of modification are copied here from the source code (maintaining nearly the same folder structure) and replacing constant values with tokens in form @tokenname@. The tokens are specified in the
      • solution: This folder holds your reports, xactions etc. (same as Pentaho solutions folder on standard BI Server)
    • Target-Build: All source code files will be copied to this folder to build your BI server. All patches will be applied to this BI server.
    • Target-Dist: The BI Server will be copied to this folder after everything has been built
    • build.xml: Note that any build.xml doesn't work with all versions of the BI server. When downloading make sure you choose the version of build.xml that accepts the version of your BI server. 

Note that the Pentaho BI Server source code folder(s), target-build and target-dist are shared among all projects. target-build and traget-dist are handled by ant, so you don't have to do anything about them.

Note: Currently the main folder structure has to be created mostly manually.

Create following folder structure to start with. This is common for all operating systems. We will discuss soon OS specific steps.

  • Pentaho
    • CBF
      • bi-server-source
      • project-tutorial
        • config
        • patches
        • solution
      • target-build
      • target-dist

The folder for the source code will be created automatically once we download the code with SVN.

In general, the below is how a folder structure could look like after completion of the setup. Ours will look slightly different.

|-- build.xml
|-- pentaho
|   |-- bi-platform-api
|   |-- bi-platform-appserver
|   |-- bi-platform-assembly
|   |-- bi-platform-assembly-open
|   |-- bi-platform-build
|   |-- bi-platform-engine-core
|   |-- bi-platform-engine-security
|   |-- bi-platform-engine-services
|   |-- bi-platform-legacy
|   |-- bi-platform-plugin-actions
|   |-- bi-platform-plugin-services
|   |-- bi-platform-repository
|   |-- bi-platform-sample-data
|   |-- bi-platform-sample-solution
|   |-- bi-platform-scheduler
|   |-- bi-platform-test-foundation
|   |-- bi-platform-ui-foundation
|   |-- bi-platform-util
|   |-- bi-platform-web
|   |-- bi-platform-web-portlet
|   |-- bi-platform-web-servlet
|   |-- dummy-jre
|   |-- mantle
|   `-- test-solutions
|-- project-client
|   |-- patches
|   |   |-- pentaho
|   |   `-- target-preconfiguredinstall
|   `-- solution
|-- target-build
`-- target-dist

The pentaho directory is where we will copy the a recent build of Pentaho BI Server. The idea is to leave this directory untouched and do all changes in the project-client directory. The below is an example structure, which depends on your specific project needs:

|-- config
|   |--
|   `--
|-- patches
|   |-- target-build
|   |   `--
|   `-- target-dist
|       `-- server
|            |-- conf
|            |   |--
|            |   `-- jboss-service.xml
|            |-- webapps
|            |   |-- pentaho
|            |   |   `-- WEB-INF
|            |   |       `-- web.xml
|            `-- lib
|                `-- postgresql-8.2-505.jdbc3.jar
`-- solution
    |-- Portal.url
    |-- admin
    .... etc

Pedro Alves explains
"The idea is very simple: 
  • All changes that would normally go to pentaho/* are placed under "patches" directory (project-client/patches/). 
  • The CBF ant script will pick up the files in the project-client/patches/ directory, scan for tokens and replace the tokens with the variables defined inside the project-client/config/ files, and copy the files to the top level directory of the entire project (In this example MyProjectDir [remark: here we change it to CBF]). 
  • It's not recommended to patch anything under pentaho/*; sources changes are patched in to target-build/* and all other changes are made by patching the final directory, target-dist."

Windows setup

Please note that I set up everything in the directory D:\Pentaho. You might want to choose a more convenient folder for your own project.

Java, Tomcat and Ant

If you don't have Java, Tomcat and Ant running already, install them:
  • Java: JDK [Java Standard Edition; short Java SE] (JRE is not enough as Ant needs JDK to work properly). Read installation instructions. Install it in D:\Pentaho
  • Tomcat: At the time of writing, v6 was the most recent one.  The location of the tomcat directory is not important; it can be placed almost any where. The location of the tomcat will be set inside of the CBF file. I installed Tomcat in D:\Pentaho
  • Ant: Download Apache Ant 1.8.1 and place it in D:\Pentaho. Installation notes: here.

Set environment variables

Using the GUI
Windows 7: Control Panel > System > Advanced System Settings > Click on the Environment Variables button

Set the JAVA_HOME environment variable to the directory where you installed JDK: 
  1. Click on New under the System variables section.
  2. Type JAVA_HOME in the variable name field.
  3. Type D:\Program Files\Java\jdk1.6.0_21 in the variable value field

Set the ANT_HOME environment variable to the directory where you installed Ant: 
  1. Click on New under the System variables section.
  2. Type ANT_HOME in the variable name field.
  3. Type C:\ant in the variable value field.

Set the PATH environment variable to include the directory where you installed the Ant bin directory: 
  1. Find the PATH environment variable in the list. If PATH is not listed, click on New under the System variables section.
  2. Type %ANT_HOME%\bin;%JAVA_HOME%\bin;
Important: If there are other variables listed, create a new variable separated by a semicolon. Ensure there are no spaces before or after the semicolon.

Windows Note:
The ant.bat script makes use of three environment variables - ANT_HOME, CLASSPATH and JAVA_HOME. Ensure that ANT_HOME and JAVA_HOME variables are set, and that they do not have quotes (either ' or ") and they do not end with \ or with /. CLASSPATH should be unset or empty.
Using Command Line
Assume Ant is installed in c:\ant\. The following sets up the environment:

set ANT_HOME=c:\ant
set JAVA_HOME=c:\jdk-
set PATH=%PATH%;%ANT_HOME%\bin;%JAVA_HOME%\bin;

Check Ant and Java are working

You can check the basic installation by opening a new shell and typing ant. You should get a message like this

Buildfile: build.xml does not exist!
Build failed

or something like this:

Buildfile: D:\Pentaho\CBF\build.xml

D:\Pentaho\CBF\build.xml:7: FATAL: 'project' property not set. Please provide it
 the ant command, eg: ant -Dproject=myproject -Denv=dev

So Ant works. This message is there because you need to write an individual buildfile for your project. With a ant -version you should get an output like

D:\Pentaho\CBF>ant -version
Apache Ant version 1.8.1 compiled on April 30 2010

Type in Java, hit Enter and you should get same info back:

Usage: java [-options] class [args...]
           (to execute a class)
   or  java [-options] -jar jarfile [args...]
           (to execute a jar file)

where options include:
    -client       to select the "client" VM
    -server       to select the "server" VM
    -hotspot      is a synonym for the "client" VM  [deprecate
                  The default VM is client.

    -cp <class search path of directories and zip/jar files>

If you have problems with either Ant or Java, check that your environment variables are properly set.

Creating the .ant directory

For our work we will need an optional ant task that is not available with the default set up. In your ant directory you find a file called build.xml.

This is an Ant script that you can run to install almost all the dependencies the optional Ant tasks need.

To do so, change to the ANT_HOME directory and execute the command:

ant -f fetch.xml -Ddest=user

You should see then something like this (extract):

D:\Pentaho\apache-ant-1.8.1>ant -f fetch.xml -Ddest=user
Buildfile: D:\Pentaho\apache-ant-1.8.1\fetch.xml

     [echo] Downloading to C:\Documents and Settings\diethard.steiner\.ant\lib


     [echo] Downloading to C:\Documents and Settings\diethard.steiner\.ant\lib
    [mkdir] Created dir: C:\Documents and Settings\diethard.steiner\.ant\lib
      [get] Getting:
      [get] To: C:\Documents and Settings\diethard.steiner\.ant\lib\maven-artifa

In general, options for Ddest are:
  • system - store in Ant's lib directory 
  • user - store in the user's home directory
  • optional - store in Ant's source code lib/optional directory, used if building Ant source code

If you go to your user directory now, you should see the .ant folder. 


Download XMLTask.jar from here and move the jar file to the .ant\lib\ folder.


Install on of the distributions. I provide here a list of options, choose one that suits you:


Let's get started with Hypersonic DB, let on you can change to your own choice of DB.


Create the directory D:\Pentaho\CBF. Download CBF's build.xml from the Wiki Page and rename it to build.xml. Save it in Create the directory D:\Pentaho\CBF

Download a BI Server Build

Open the command prompt (DOS Client) and go to this folder. Change to D:\Pentaho\CBF\bi-server-source\ and issue following command to download (check out) BI Server 4:

svn co svn://

If you are under a proxy you can use http:// instead of svn:// 
svn co

You will see if the command is working once you don't get an error message and it starts adding a lot of files (A at the beginning of each line).
If you are not sure which version is the latest BI Server, just type this URL in your web browser:


An example of a file you can find here. Make sure that for Windows you use double backslashes in your paths!


javac.path = D:\\Program Files\\Java\\jdk1.6.0_21\\bin\\

# Solution path. Required for compile time
solution.path = D:\\Pentaho\\CBF\\project-tutorial\\solution\\

# Solution path required for runtime. Defaults to the above value but can be different if we're
# deploying to another place. Should be used in WEB-INF/web.xml in the solution-path parameter
solution.deploy.path = D:\\Pentaho\\CBF\\project-tutorial\\solution\\

#accepted values: tomcat or jboss = tomcat

# Tomcat 5.5 path:
tomcat.path = D:\\Pentaho\\apache-tomcat-6.0.29\\

# Copy the sample solutions to our project directory? true | false
copy.pentaho.samples = true

BASE_URL = put your URL here

# Java options for the run command
java.opts = -Xmx512m -XX:MaxPermSize=512m -Xrunjdwp:transport=dt_socket,address=8765,server=y,suspend=n

## PROJECT DIRECTORIES - The defaults usually work fine
pentaho.dir = bi-server-source\\3.5\\ = target-build\\

deploy.mode = rsync
deploy.args = -avz --exclude '.svn/' --exclude '*.log' --exclude 'work/' --exclude  'temp_user/' --exclude 'temp/' --exclude 'tmp/'
deploy.dest.server = user@host:path
deploy.dest.solution = user@host:path

Create patches
We will only go through one example here. Please prepare all other files that need changes in a similar fashion.


Copy web.xml from pentaho/bi-platform-appserver/webapps/pentaho/WEB-INF/web.xml to project-client/patches/target-dist/server/webapps/pentaho/WEB-INF/web.xml
Inside of this file, tokens can be placed that will be set by the CBF ant script.  See the example snippet of web.xml bellow.


So basically, the files stays exactly the same, you only replace some values by tokens (highlighted in red).

The tokens @solution.deploy.path@ and @BASE_URL@ are defined in the project-client/config/ or project-client/config/ files and will be replaced by the CBF ant script and the new revised web.xml with the replaced tokens will be copied to the top level directory (In this example CBF).

Start Hypersonic DB

Go to D:\Pentaho\hsqldb-2.0.0\hsqldb\bin and start runServer.bat.

Create the build

In your command line tool go to the CBF folder and issue ant -Dproject=tutorial -p, which will show you all the parameters that you can pass to the build.xml. Find an extract below:

D:\Pentaho\CBF>ant -Dproject=tutorial -p
Buildfile: D:\Pentaho\CBF\build.xml
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo] ------       CBF - Community Build Framework           -------
     [echo] ------                 Version: 3.5.0                  -------
     [echo] ------ Author: Pedro Alves ( -------
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
     [echo]                          SETTINGS:
     [echo]    Project:         ukint
     [echo]    Environment:
     [echo]    Config File:     project-tutorial/config/
     [echo]    Server:          tomcat
     [echo] --------------------------------------------------------------
     [echo] --------------------------------------------------------------
Pentaho BI Platform build helper..
Main targets:

 all              Compiles and builds the entire project
 clean            Cleans the solution
 copy-finish      Copy target files
 copy-init        Copy project files

Now issue the following:

ant -Dproject=tutorial -Denv=dev dist-clean all run

The command will first load and then because of the -Denv=dev, the command will load As mentioned, you don't have to have a file, it's optional. If you don't you don't have this file, you just run the following:

ant -Dproject=tutorial dist-clean all run

dist-clean will delete any previous buids.

Ubuntu (Linux) Setup

  1. Open terminal (Applications > Accessories > Terminal).
  2. Issue javac -version to see if you have a recent JDK installed.

The next section describes the setup for users who want to avoid working with command line as much as possible. Users familiar with the Terminal, please jump to "Using Terminal". 

Using Applications > Ubuntu Software Center

For users that are not familiar with the command line interface, this is the way to go ... (although we still have to use the Terminal a bit)
  1. In Ubuntu Software Center search for "openjdk-6-jdk" and click to install it [OPEN]
  2. search for Ant, "Java based built tool like Make, Ant" will show up. Click the install button.
  3. Search for subversion. "Advanced version control system, subversion" will shop up. Click the install button. This package includes the subversion client (svn), tools to create a Subversion repository (svnadmin) and to make a repository available over a network (svnserve). The fastest way now to progress is to use the Terminal. Follow these instructions, skip the first step as we already installed subversion.  Your svn repository should then reside in /usr/local/svn/repos.
  4. Search for subversion and choose one of the clients, like Subcommander
  5. Search for Tomcat. At the time of this writing, Tomcat 6 was the current version. Install it.
  6. Download a hypersonic database from here. Go to Place > Home Folder, click File > Create Folder and name it "Pentaho". Unzip the HSQLDB file in the Downloads folder and move the unzipped folder to the recently created "Pentaho" folder. [/home/diethardsteiner/Pentaho]
  7. You can also download MySQL if you want (some install info you can find here). We will not cover setting up the environment with MySQL here, but you can later on progress to include MySQL in your environment.
  8. Still being in the "Pentaho" folder, go to File > Create Folder and name it "CBF". Download CBF's build.xml form the CBF Wiki page, extract it and move it to the recently created "CBF" folder. Mark the file, hit F2 and rename it to build.xml.  [/home/diethardsteiner/Pentaho/CBF]

Using Terminal

Note: If you followed the instructions of "Using Applications > Ubuntu Software Center", then you can ignore this section.
  1. Install Java: sudo apt-get install openjdk-6-jdk
  1. Install ant: sudo apt-get install ant 
  2. Install subversion: sudo apt-get install subversion. A good documentation on how to set up subversion on Ubuntu can be found here. Follow these instructions. Your svn repository should then reside in /usr/local/svn/repos. Additional info can be found here.
  3. Install Tomcat: sudo apt-get install tomcat6
  4. Download a hypersonic database from here. Create a folder: mkdir $Home/diethardsteiner/Pentaho. Move the folder in this directory. 

Create the .ant Directory

Ant is located in usr/share/ant. Follow the instructions mentioned in the Windows section: click here.


Download XMLTask.jar from here and move the jar file to /home/<username>/.ant/lib/. ".ant" is a hidden folder, hence you normally wont see it if you go to Places > Home Folder. Press "CTRL+H" to temporarily see the hidden folders or go to Edit > Preferences and in the "View" tab tag "Show hidden and backup files" to permanently see the hidden files.


See here

Download a BI Server Build

See here


See here

Create patches

See here

Mac OS X Setup

I won't go through the procedure here again, just some special notes:

Subversion is included in Mac OS X Leopard and Snow Leopard. Have a look at this tutorial on how to get it running.

Mac specific SVN Clients: Cornerstone and SCPluging


  1. Hi Diethard,

    I found I had unresolved dependencies when I tried to build. This forum thread helped:

    I hope it's useful!

    1. Thanks a lot for sharing! Best regards, Diethard