11. Advanced Configuration

11.1. Inca Component Options

Each inca component has a set of options that can be set in either the $INCA_DIST/etc/common/inca.properties file or from the command line. The inca.properties file has a list of name value pairs of the format "inca.component.property=value". For example, to start the agent on port 5323 instead of 6323 and enter the password on the command line rather than get it from standard in, you could:

1. edit $INCA_DIST/etc/common/inca.properties and replace:

   • "inca.agent.port=6323" with "inca.agent.port=5323"

   • "inca.agent.password=stdin:password>" with "inca.agent.password=pass:<password>" (where <password> is the password set with the createauth command)

   execute:

   % cd $INCA_DIST; ./bin/inca start agent

2. OR execute the following command:

   % cd $INCA_DIST; ./bin/inca start agent -p 5323 -P pass:<password>

Man pages with component options are described in Section 12.

Note: To change the port of the consumer, see Section 6.10.

Note: if you have more than 5 reporter managers running, increase the number of agent and depot threads in the inca.properties file to be 10 more than the number of reporter managers. For example, if running 15 reporter managers edit the inca.properties file as follows:

31 # Maximum number of threads running on the agent
32 inca.agent.numthreads=25
...
82 # Maximum number of threads running on the depot
83 inca.depot.numthreads=25

11.2. Customizing Series Notification

To customize the notification you receive whenever the result of a series comparison changes, you can either modify the default Inca notification scripts (found in the sbin/ subdirectory of your Inca Depot installation) or write your own. Set inca.incat.notifiers in your inca.property file to add a new script to the options presented in the incat series dialog. For example, to add a script named "MyNotifier" to the options, you would add the following line to inca.properties.

  inca.incat.notifiers=EmailNotifier,LogNotifier,MyNotifier
  

The Inca Depot sets environment variables that provide information to the notification script about the series and comparison, and the script can incorporate these variables into its notification. For example, EmailNotifier uses several of these variables in constructing the email body. These are the environment variable names that the Depot defines:

Values that are specified as the "Script Arguments" in incat are passed as arguments to the script.

11.3. Report Filtering

The Inca Depot allows you to filtering incoming reports before information about them is placed in the Inca database. To do so, you need to write a class that extends edu.sdsc.inca.depot.util.ReportFilter and set the property inca.depot.reportFilter to the name of the class.

An Inca report consists of five elements, all strings: the execution context (reporter name and arguments), the name of the resource where the reporter ran, the reporter stdout (i.e., report), the reporter stderr, and a report of the system resources used by the reporter execution. Of these, stderr may be null; the other four are required.

The ReportFilter class provides set and get methods for each of these five elements. If the inca.depot.reportFilter property is set, the Inca Depot creates an instance of the named class, then calls each of its set and get methods in turn to allow it to make changes to any of the elements. Changes made by the filter are incorporated into the information stored in the Depot database. If a ReportFilter get method for any of the elements other than stderr returns null, the Depot discards the report.

For example, this class directs the Depot to ignore reports that arrive from blue.ufo.edu and modifies report stderr values that contain particular messages.

public class MyReportFilter extends edu.sdsc.inca.depot.util.ReportFilter {

  public String getResource() {
    String resource = super.getResource();
    return resource.equals("blue.ufo.edu") ? null : resource;
  }

  public String getStderr() {
    String stderr = super.getStderr();
    stderr = stderr.replaceAll("Try again.*\n", "");
    return stderr;
  }

}

To install your filter, first compile the class.

% javac -classpath lib/inca-depot.jar MyReportFilter.java

Copy the compiled class to your $INCA_DIST/lib directory. For example,

% cp MyReportFilter.class $INCA_DIST/lib

Then add the inca.depot.reportFilter property to your $INCA_DIST/etc/common/inca.properties file.

...
inca.depot.reportFilter=MyReportFilter
...

Finally, restart the depot.

% ./bin/inca restart depot

package edu.sdsc.inca.depot.util;

import org.apache.log4j.Logger;
import java.net.URL;
import java.util.Properties;
import java.io.InputStream;
import java.io.IOException;

/**
 * Prefixes error messages in depot reports with "DOWNTIME: +optionalString+: "
 * if the resource the report ran on is in downtime.   Resources are determined
 * to be in downtime if they are listed in a downtime properties file.  In order
 * to reduce overhead, the downtime properties file is retrieved and cached at
 * a refresh interval in the getDowntimes() method instead of being retrieved
 * for each filter instance.
 */
public class DowntimeFilter extends edu.sdsc.inca.depot.util.ReportFilter {
  private static Logger logger = Logger.getLogger(DowntimeFilter.class);
  private static Properties downtimes = new Properties();
  private static long lastRefresh = 0;

  /**
   * Returns cached property list of resources in downtime.  Gets and caches
   * property list from file in classpath (downtime.properties) if cache has
   * expired according to refreshMins.
   *
   * The property list file contents can be:
   *
   *  downResource1=optionalErrorMessagePrefixStringForResource1
   *  downResource2=optionalErrorMessagePrefixStringForResource2
   *
   * OR
   *
   *  downResource1
   *  downResource2
   *
   */
  synchronized static Properties getDowntimes()  {
    String downtimePropFile = System.getProperty("inca.depot.downtimeFile");
    if(downtimePropFile == null) {
      downtimePropFile  = "downtime.properties";
    }
    String downtimeRefresh = System.getProperty("inca.depot.downtimeRefresh");
    if(downtimeRefresh == null) {
      downtimeRefresh  = "15";
    }
    Integer refreshMins = Integer.parseInt(downtimeRefresh);
    long minSinceLastRefresh = (System.currentTimeMillis()-lastRefresh)/60000;
    if (minSinceLastRefresh >= refreshMins){
      URL url = ClassLoader.getSystemClassLoader().getResource(downtimePropFile);
      if(url == null) {
        logger.error( downtimePropFile + " not found in classpath" );
      }
      logger.debug( "Located file " + url.getFile() );
      downtimes.clear();
      try {
        InputStream is = url.openStream();
        downtimes.load(is);
        is.close();
      } catch (IOException e){
        logger.error( "Can't load properties file" );
      }
      lastRefresh = System.currentTimeMillis();
    }
    return downtimes;
  }

  /**
   * Writes new report with modified error message to depot if resource is down
   *
   * @return  string with depot report (reporter Stdout)
   */
  public String getStdout() {
    String resourceProp  = getDowntimes().getProperty(super.getResource());
    if (resourceProp != null){
      logger.debug( super.getResource() + " is down " + resourceProp );
      return super.getStdout().replaceFirst(
          "<errorMessage>", "<errorMessage>DOWNTIME:"+ resourceProp +": ");
    } else{
      return super.getStdout();
    }
  }

}

...
inca.depot.downtimePropFile=MyDowntimeFilename
...

...
inca.depot.downtimeRefresh=5
...

downResource1=optionalErrorMessagePrefixStringForResource1
downResource2=optionalErrorMessagePrefixStringForResource2

downResource1
downResource2

...
inca.depot.reportFilter=edu.sdsc.inca.depot.util.DowntimeFilter
...

% ./bin/inca restart depot

...
inca.depot.reportFilter=edu.sdsc.inca.depot.util.All2AllFilter
...

% ./bin/inca restart depot

inca.depot.reportFilter=edu.sdsc.inca.depot.util.DowntimeFilter, edu.sdsc.inca.depot.util.All2AllFilter

11.4. Depot Database Configuration

The Inca depot uses Hibernate to interface to a relational database backend for storing reports and incat configuration. By default, the Inca depot uses Hibernate's HSQL database but can be configured to use any Hibernate supported database. We have tested the Inca depot with PostgreSQL and Oracle.

Steps for using a depot database other than HSQL are as follows:

1. Stop the depot

   % cd $INCA_DIST; ./bin/inca stop depot

2. Edit $INCA_DIST/etc/hibernate.properties

   • Comment out the first 5 lines which specifies for hibernate to use hsql as its backend database:

     1 #hibernate.dialect=org.hibernate.dialect.HSQLDialect 2 #hibernate.connection.driver_class=org.hsqldb.jdbcDriver 3 #hibernate.connection.url=jdbc:hsqldb:test 4 #hibernate.connection.username=sa 5 #hibernate.connection.password=

   • Uncomment the block which specifies for hibernate to use your database (i.e., for PostgreSQL uncomment 8-13, for MySQL uncomment 17-21, for Oracle uncomment 24-28).

   • Change the uncommented hibernate.connection.url, hibernate.connection.username and hibernate.connection.password property values to be the host/db name, login username and password for your database.

3. Put JDBC drivers for your database in the $INCA_DIST/lib directory. Driver download locations: PostgreSQL, MySQL, Oracle

4. Initialize the depot (set up the Inca tables):

   % cd $INCA_DIST; ./bin/inca depot -d

   You should see something like:

   Initializing c3p0 pool... ... Database Initialization Completed

5. Start the depot

   % ./bin/inca start depot

11.5. Manual Access Method

A resource administrator may be unable to start a reporter manager using one of the automated methods (ssh, globus2, or local). In this case, an Inca administrator can add the resource using the access method 'manual'. The following steps will need to be taken by the Inca administrator and resource administrator:

Inca Administrator

Resource Administrator

1. RESOURCE ADMIN: install the reporter manager distribution on your resource using the following steps.

   1. Create an installation directory for the reporter manager (e.g., $RM_INSTALL_DIR). Download the reporter manager tarball and build script:

      % cd $RM_INSTALL_DIR; \ wget http://inca.sdsc.edu/releases/2.5/Inca-ReporterManager.tar.gz; \ wget http://inca.sdsc.edu/releases/2.5/buildRM.sh

      At this point the directory on the remote machine should look something like this:

      % ls Inca-ReporterManager.tar.gz buildRM.sh

   2. Install the reporter manager and list directories to verify files unpacked correctly:

      % bash buildRM.sh $RM_INSTALL_DIR Inca-ReporterManager.tar.gz % ls $RM_INSTALL_DIR Inca-ReporterManager-9.6764 build.log lib share Inca-ReporterManager.tar buildRM.sh man var bin etc sbin

      Install the default set of reporters and their libraries:

      % cd $RM_INSTALL_DIR; wget http://inca.sdsc.edu/releases/2.5/Inca-Reporter.tar.gz % cd var; tar zxvf ../Inca-Reporter.tar.gz; cd Inca-Reporter-* % perl Makefile.PL PREFIX=${RM_INSTALL_DIR}/var/reporter-packages \ INSTALLDIRS=perl LIB=${RM_INSTALL_DIR}/var/reporter-packages/lib/perl \ INSTALLSCRIPT=${RM_INSTALL_DIR}/var/reporter-packages/bin \ INSTALLMAN1DIR=${RM_INSTALL_DIR}/var/reporter-packages/man/man1 \ INSTALLMAN3DIR=${RM_INSTALL_DIR}/var/reporter-packages/man/man3 % make % make install

2. RESOURCE ADMIN: create a set of credentials for the reporter manager (i.e., private key and certificate request) using the command below.

   % cd $RM_INSTALL_DIR; ./sbin/inca createRmCertRequest -P stdin:password:

   Enter a password for your key (to use when you start up the reporter manager). Two files will be created in $RM_INSTALL_DIR/etc: an encrypted private key called rmkey.pem and a certificate request called rmreq.pem. Email rmreq.pem to your Inca administrator and they will generate a certificate for your reporter manager.

3. INCA ADMIN: upon receiving a rmreq.pem file, generate a certificate for a reporter manager using the command below. Replace "rmreq.pem" with to the path to the rmreq.pem file that you received from the resource administrator and "rmcert-resource.pem" with the path to the reporter manager certificate that will be generated by the command.

   % cd $INCA_DIST; ./bin/inca createRmCert -P stdin:password: rmreq.pem rmcert-resource.pem

   Enter the password for the inca distribution (i.e., created in Step 4 during the initial installation process). Email the reporter manager certificate, "rmcert-resource.pem", and trusted certificate to the resource administrator. The trusted certificate is the file ending with the .0 extension in your $INCA_DIST/etc/trusted directory. For example f73fee74.0 is the trusted certificate in the following directory:

   % ls etc/trusted/ agentcert.pem f73fee74.0 rocks-101.sdsc.educert.pem

4. INCA ADMIN: add the specified resource within incat and choose 'manual' as below:

   Figure 20. Add Resource Screen

   

   Make sure the "Equivalent" box is checked, otherwise the depot may discard reports with "unattached to any DB config" warnings. The new "manualResource" will also need to be added to the "defaultGrid" resource in order to run the default sampleSuite. Select "Agent->Commit" from the menu to commit the changes.

5. RESOURCE ADMIN: install the certificate and trusted certificate from the Inca admin in your reporter manager installation. Replace "rmcert-resource.pem" and "trusted.0" with the names of the files received from your Inca administrator.

   % cd $RM_INSTALL_DIR % cp rmcert-resource.pem $RM_INSTALL_DIR/etc/rmcert.pem % mkdir $RM_INSTALL_DIR/etc/trusted % cp trusted.0 $RM_INSTALL_DIR/etc/trusted

6. RESOURCE ADMIN: Finally, you can start up the reporter manager using the commands below. Replace "depotHost" with the hostname where the depot is running and replace "manualResource" with the manual resource group name added in step 4:

   % cd $RM_INSTALL_DIR % ./sbin/inca reporter-manager \ -a incas://agentHost:6323 \ -d incas://depotHost:6324 \ -c etc/rmcert.pem \ -k etc/rmkey.pem -t etc/trusted \ -e bin/inca-null-reporter \ -r var/reporter-packages \ -R sbin/reporter-instance-manager \ -v var \ -w 1 \ -i manualResource \ -s suite.xml \ -L DEBUG \ -l var/reporter-manager.log \ -P true <enter your password>

   Command will hang until the password for the reporter manager key is entered. If the private key is not password protected, don't use the -P option in the command above. Check to make sure the reporter manager is running by doing a "ps | grep reporter-manager" and make sure there aren't errors by doing a "grep ERROR $RM_INSTALL_DIR/var/*".

   To stop the reporter manager at any time, type

   % ./sbin/inca stop reporter-manager

   Make sure all reporter-manager ps are stopped

   % ps | grep manager

11.6. Batch Systems

The cluster.batch.wrapper reporter can be useful when running Inca on batch systems. Without an --exec argument, this reporter submits a trivial program to the batch scheduler and reporters the amount of time it spends in the queue before executing. If given an --exec argument, cluster.batch.wrapper instead submits the argument value (which should contain a reporter invocation) and collects and reports it output. This use allows reporters to be executed on batch nodes, rather than on the submission host. For example, this command will report the version of gcc that is installed on the batch nodes of a PBS cluster.

% cluster.batch.wrapper --scheduler=pbs --exec=cluster.compiler.gcc.version
  

The --scheduler argument is the only one required; valid values are cobalt, dqs, loadleveler, lsf, pbs, and sge. Additional recognized arguments are as follows.

Here are some additional examples of using cluster.batch.wrapper to submit Inca reporters to a batch queue.

% cluster.batch.wrapper --scheduler=pbs --account=alf63 \
  --exec=cluster.compiler.gcc.version

% cluster.batch.wrapper --scheduler=loadleveler --queue=normal --type=parallel \
  --exec='network.ping.unit --host=ufo.edu'

% cluster.batch.wrapper --scheduler=sge --submitparam='-js 1' \
  --submitparam='-l h_vmem=600' \
  --exec='cluster.ps.unit --process=init --psargs=-x'
  

In incat, you can specify that cluster.batch.wrapper should be used to submit a series by including its name and arguments in the series context string. Inca stores installed reporters in the directory $INSTALL_DIR/bin, so the path to cluster.batch.wrapper will be $INSTALL_DIR/bin/cluster.batch.wrapper. The figure below shows the context string for the network.ping.unit series mentioned above.

11.7. Manual Run Now

Oftentimes resource or system administrators will want to show that a problem has been resolved by independently executing inca tests before they are scheduled to run so that their results appear on status pages. Rather than granting resource administrators full privileges just to use incat's "run now" button, the inca administrator can provide resource administrators with a "manual run now" option - a command line script to execute tests and send results to the depot. A "run now" can also be invoked via the Inca web pages if configured (by default "run now" is not enabled). See Section 6.5 for more information.

% ./bin/admin-run-now -h

11.8. Source Distributions

Source distributions of the Inca components are also available. The following table lists the Inca component source distributions and shows how to build each of them. Note, that Apache Ant is needed for the Inca components implemented in Java.