Continuous Integration

Bamboo is a continuous integration (CI) tool that is part of the Atlassian tool suite. The Axom project uses two Bamboo projects, one on the LC Collaboration Zone (CZ) and one on the LC Restricted Zone (RZ).

The basic mechanics for managing test plans described in Bamboo Test Plans is the same for both Bamboo instances.

CZ Bamboo Project

We use the Axom CZ Bamboo project primarily for manually running Bamboo plans on specific branches. For example, if you are working on a branch and you want to build the branch code and run tests on several LC platforms on the CZ with a variety of compilers, you can do this easily. The plans for such branch testing are configured so that when a new Axom git branch is created and pushed to our Bitbucket project, the branch will appear in the list of branches on which a plan can be run.

There are Bamboo plan options for this branch testing. One is a “smoke test” that will build and run tests on a single platform with our default GNU compiler. The other is a more comprehensive plan that will build and run tests on each of the platforms we need to support using the compilers and versions our users require.

To run a plan on a specific branch, do the following:

  1. Log into the CZ Bamboo tool.
  2. Click on the ‘Axom’ project.
  3. Select the test plan you want to run.
  4. Click on the branch you want to build and test.
  5. Click ‘Run plan’ via the ‘Run’ pulldown menu on the upper right.

Depending on resource availability and what the plan does, the plan may take a while to run. When the plan completes, you will receive an email indicating this and whether everything passed or if there were any failures. If you need to look at the detailed results (e.g., if a test failed), you can do the following:

  1. Click the branch name in the plan.
  2. Click the run number in the summary.
  3. Click the ‘artifacts’ tab.
  4. Click the artifact that you want to see. The test plan artifacts contain the detailed output generated by Google Test and FRUIT.

We also have one plan that is scheduled to automatically build and run tests on the master branch on a select set of platforms and compilers once per week. This plan may be run manually at any time by selecting the plan and clicking on ‘Run plan’ as described above. Each member of the team receives an email notification indicating passes and failures of builds and tests run by this plan.

RZ Bamboo Project

We use the Axom RZ Bamboo project for the bulk of Axom automated build and test operations. This includes: nightly scheduled builds and tests of the develop branch on each of the platform types we support with a variety of compilers, nightly builds of user documentation installation of this documentation on our LC web pages, and weekly third-party library builds.

Similar to our CZ Bamboo plans, each of these plans may be run manually at any time by selecting the plan and clicking on ‘Run plan’.

Bamboo Test Plans

A Bamboo plan consists of stages, each of which represents a step in a build and/or test process. Each stage may contain one or more jobs that can be run in parallel by Bamboo. For example, you may have a stage that compiles a code and multiple testing stages which follow that can run in parallel. Each stage contains specific tasks to run. Currently, Axom plans don’t exercise parallel execution in Bamboo. A typical Axom test plan consists of a set of stages, one to build the code and run tests for each platform and compiler combination. The tasks comprising a stage are shell commands that navigate workspace directories on LC systems, check out the axom code, and run scripts (that we archive in our source code repo) that encode all of the code compilation and test execution operations. A final stage parses CTest output.

Note

All of our scripts that are run in Bamboo can be found in the source directory axom/scripts/bamboo.

To see the specific contents of any particular plan, click on the Axom project in one of our Bamboo projects and click on a plan. Then, in the ‘Actions’ pull-down menu at the upper right, click on ‘Configure plan’. Clicking on the ‘Stages’ tab will reveal the stages in the plan and clicking on a stage will show the tasks in the stage, including shell commands, scripts that are run, etc.

Creating and configuring a Bamboo plan

Here are the steps to create and set up a Bamboo plan once you are in the appropriate Axom Bamboo project (either CZ or RZ):

  1. Click the ‘Create’ button at the top of the page and choose either ‘Create a new plan’ or ‘Clone an existing plan’.

    • When you choose to create a new plan, you can define everything about what the plan does.
    • When you clone an existing plan, you make a copy of the plan and its entire configuration.

  2. Enter information in all required fields (i.e., required fields are denoted with a red asterisk).

    • If you clone an existing plan, make sure you choose the correct destination project for the plan.

    • Make sure you link the correct repository and branch to the plan.

      Note

      This only applies to the CZ Bamboo project since our repo lives on the CZ Bitbucket server. We could use the Bamboo plugin to clone the repo as a plan task; however, we typically do not do this and use the ‘script’ interface to clone via the git clone command. Nevertheless, linking a repo to a plan allows us to set up the plan to run on any branch in the repo.

      A CZ repo cannot be linked to a plan on the RZ Bamboo project due to LC security constraints. To set up an RZ Bamboo plan that is not associated with a repository choose ‘Link new repository’ and select ‘None’ from the pulldown menu.

  3. Click ‘Configure plan’. Bamboo will ask you to configure plan tasks.

  4. Click ‘Add task’ for each task you wish to add to the plan. Note that a typical Axom plan includes script tasks (either inline or run from files).

    • Start typing ‘script’ in the search field and select the script icon when it appears.

  5. Configure each task. For a ‘script’ task, this means:

    • Enter a short script description.
    • Choose ‘inline’ or ‘file’. If inline, type in the script commands. If file, choose the file containing the script to run.
    • Fill in arguments, environment variables, etc. as needed.

  6. Check the box under ‘Enable this plan?’

  7. Click the ‘Create’ button.

Important

After a plan is created, many of its configuration options can be set or modified using the appropriate option tabs that appear across the top of the web page when ‘Configure plan’ is selected from the ‘Actions’ pulldown menu.

Associating an agent to a Bamboo plan

Every Bamboo project has agents the run Bamboo jobs on LC platforms. When Bamboo executes a plan, it communicates with an agent associated with it for the desired platform that runs tasks on the machine. Here are the steps to associate an agent with a plan:

  1. Log on to the MyLC portal for the appropriate network (CZ or RZ).
  2. Switch to the Axom shared user account which is the account under which all Axom plans run. First, click the ‘change user’ link at the upper right. Enter ‘atk’ as the user in the field at the upper left, choose the account associated with the ‘atk’ LC username, and check the ‘su to user’ box.
  3. Go into the ‘bamboo agent management’ portlet. You will see a list of agents and machines that we have. Select the machine you want to create the agent on and enter a descriptive name (e.g., axom-rzalastor), and click ‘create’.
  4. Finally, to attach the agent to a plan, you must send an email to the LC Atlassian email list lc-atlassian-admin@llnl.gov requesting that the named agent you created be attached to the plans you want.

Restarting agents

Occasionally, agents must be restarted. When an agent dies, Bamboo jobs will be queued and stalled until the agent that should run them is restarted.

We have ‘cron’ jobs running on the CZ and RZ to check our agents and restart those that need it. The crontab files are located in the directory axom/scripts/bamboo in our source code repo.

Disabling or deleting a Bamboo plan

Any Bamboo plan may be disabled by selecting the plan, clicking the ‘Actions’ pulldown menu and then clicking ‘Disable plan’. When a plan is disabled, its configuration and history is preserved, but it will not run.

Deleting a plan removes its configuration, history, artifacts, labels, etc. If the plan is ever needed again after it is deleted, it must be completely reconstructed from scratch. Therefore, a plan should only be deleted if it is clear that it will not be needed in the future. A couple of other important points to note:

  • A plan that is running cannot be deleted – the plan must be stopped first.
  • A record of a plans results can be preserved, if necessary, before it is deleted. See Exporting data for backup for details.

To delete a Bamboo plan, select it, click the ‘Actions’ pulldown menu and click on ‘Configure plan’. Then, click the ‘Actions’ pulldown menu and click on ‘Delete plan’.

Other Bamboo Things…

  • Run plan from command line. I believe this can be done using the queue_build.py script in axom/scripts/bamboo but am not sure how…