Inca 2.5 User's Guide

5.1. Repositories

Incat begins by showing the panel for Inca reporter repositories (as shown in Figure 10). The default Inca configuration retrieves reporters from the $INCA_DIST/Inca-Reporter-* repository on the agent machine. To add this repository to your list, press the Add button in the Repositories section, enter the file:/ location of the Packages.gz file in the pop-up window that appears, then press the OK button. Within a few seconds you should see the repository appear in the Repositories list, a set of reporters in the Reporters list, and properties for the first reporter in the Reporter Properties list. Reporter source code is viewable by double clicking on a reporter name or by selecting the reporter name and pressing the "Show" button.

5.2. Resource Configuration

The default Inca configuration defines three resource groups -- a group called "localResource" that specifies the host where the Inca Agent will launch a new Reporter Manager (localhost) and two container groups called "localSite" (contains "localResource") and "defaultGrid" (contains "localSite") that can be extended to include any other hosts running Reporter Managers. The default resource configuration is shown is Figure 11.

The configuration above can be duplicated in an empty Inca installation by pressing the Resource Configuration tab near the top of the incat window and then entering information about the hosts on which you want Inca to run reporters. To define the default groups above in incat, press the Add button in the Resource Groups section to open the resource group edit dialog shown in Figure 12.

In the Group Name text box, enter the name "localResource" as a nickname for the machine the Reporter Manager client runs on. Enter "localhost" in the Members text box and select "local" as the access method (for a description of access methods see Section 5.5). Press OK to complete entry of this resource group. Incat will close the resource group edit dialog and will display localResource in the Resource Groups section of the Resource Configuration panel.

Press the Add button again to add a second resource group. Give this one the name "defaultGrid" and enter "localResource" in the Members text box. This tells incat that any hosts in the "localResource" group (localhost, in this case) are also part of the defaultGrid group. If you defined other groups, "siteB", "siteC", etc., you could include these in defaultGrid by listing them in the Members text box, separated by spaces. Press OK to complete entry of this resource group. In the Resource Groups list, select each group and notice that localhost is listed in the the Members panel.

5.3. Suites

Pressing the Suites tab near the top of the incat window takes you to incat's suite/series specification panel. Here you specify the reporters you want to run, the resource groups to run them on, how frequently to run them, and the arguments to use when running them.

The default Inca configuration defines a single suite named sampleSuite that contains eight series as shown in Figure 13.

In a new Inca installation, you can add the default suite above by pressing Add button in the Suites section of the panel, entering "sampleSuite" in the pop-up window, then pressing the OK button. To add series to the new suite, press the Add button in the Series section to open the incat series dialog and configure each like the gcc_hello_world series in Figure 14.

The bottommost box in the series dialog allows you to test the output of the reporter and send email or take other actions if the test fails. This is covered in Section 5.9 below; for this series leave this box blank. Press the OK button, and incat will close the series dialog box and add the series to the Series list in the suite/series specification panel.

The other nine series in the default Inca configuration are composed similarly to the first one. Press the Add button in the series section for each of them, then set the values in the series dialog as specified in the table below. Set the log argument for each series to 3 and the frequency to 10 minutes.

5.4. Committing/Saving

Your Inca deployment configuration is now complete. At this point, it's a good idea to use the "Save" option in incat's File menu to write the configuration to a file (Figure 15). That way, you have a local copy of the configuration that you can later modify. The file is formatted XML; if you're curious, you can read through it to see how incat represents the information you've entered.

Although your configuration is complete, it's not yet active. To tell Inca to begin running reporters, you need to have incat send your configuration to your Inca Agent. If you started incat with the -A argument, then you're already connected to your Agent. Otherwise, use the "Connect" option in incat's Agent menu to establish a connection. Once you're connected, you can use the "Commit" option in the Agent menu to send the configuration to the Agent (Figure 16). In response, the Agent will install the Inca Reporter Manager code on the host specified in incat's "Resource Configuration" panel and begin running the reporters you specified in the "Suite Series" panel.

5.5. Resource Access Methods

The Inca Agent can use the following access methods to stage and start Reporter Managers on resources:

• ssh:

  The most common access method. The Agent starts a Reporter Manager on a remote machine using ssh to access the remote machine. The Agent must have ssh key access to the remote machine. Reporter Manager files are copied from the Agent to the remote machine using sftp. For ssh resource groups, incat provides text boxes for you to enter the login id, password, and path to the ssh key file on the Agent machine. For security purposes, incat displays asterisks for the password and encrypts it when you save the configuration to a file.

• globus2:

  The Agent starts a Reporter Manager on a remote machine using Globus PreWS to access the remote machine. Reporter Manager files are copied from the Agent to the remote machine using GridFTP. When you select globus2 in the Access Method pull-down, incat provides text boxes for you to enter contact information for the resource's Globus GRAM and GridFTP servers. If you leave these boxes blank, Inca defaults to ports 2119 and 2811, respectively, for the first host in the resource group's member list. Access to Globus hosts requires an active Globus proxy on the Agent's host. You can either create a manual proxy on the Agent machine before starting, or you can store a proxy on a myproxy server and complete the four incat proxy dialog boxes (see Section 5.7) so that the Agent can obtain one as needed.

• globus4:

  The Agent starts a Reporter Manager on a remote machine using Globus WS to access the remote machine. Reporter Manager files are copied from the Agent to the remote machine using GridFTP. When you select globus4 in the Access Method pull-down, incat provides text boxes for you to enter contact information for the resource's Globus WS GRAM and GridFTP servers. If you leave these boxes blank, Inca defaults to ports 8443 and 2811, respectively, for the first host in the resource group's member list. Access to Globus hosts requires an active Globus proxy on the Agent's host. You can either create a manual proxy on the Agent machine before starting, or you can store a proxy on a myproxy server and complete the four incat proxy dialog boxes (see Section 5.7) so that the Agent can obtain one as needed.

• local:

  The Agent starts a Reporter Manager on the same machine where the Agent is running (localhost).

• manual:

  Entering a manual resource group indicates that you want complete, direct control over Inca execution on the group. For a manual resource group you must start the Reporter Manager on the command line and restart it any time you want to change the reporter series configuration for the group. The Agent will NOT automatically start a Reporter Manager for a manual resource as it will for local, ssh, or globus2 resources. See Section 11.5 for more details.

5.6. Requiring Resource Administrator Approval

By default, Inca installation changes take effect automatically once you commit them to the Agent. In circumstances where Inca changes must be approved before they take effect, enter the approver's email address in the "Approval Email" text box of the resource edit dialog. When changes to the Inca installation on this resource are committed, Inca will send approval instructions to the email specified in the resource dialog. Note that a run now request as described in Section 5.3 is not considered a change and will be forwarded automatically if that series has already been approved.

The emailed approval instructions will look like the image below by default and will describe how changes can be approved using Inca's approveChanges command. To customize the approval instruction email edit $INCA_DIST/etc/approveEmail.txt.

The approveChanges command allows viewing and approval of the individual changes committed using incat. Use the --help flag with the approveChanges command to see all possible options. After approveChanges is invoked, the approver will see a text menu like the one in the screen below with a list of proposed changes. More information about each change is viewable by hitting ctrl-y. By default all changes are checked. If any proposed changes are incorrect for their system, the approver should uncheck them by hitting space. After checking only correct changes for their resource, the approver should hit ctrl-s to approve the selected changes. The Agent will only transmit approved changes to the resource. The proposed changes not approved will remain in the queue and indicate that the Inca administrator has incorrect information about the resource. The approver should notify their Inca administrator of these errors to correct their proposed changes list.

The image below shows an example of more information about a change (viewable by hitting ctrl-y). In this example a change to the series logging is represented by the before and after values separated by the "=>" symbol and the green color highlighting the new log and context values. If colors aren't viewable in your terminal, try setting the TERM environment variable to "xterm-color".

Here the approveChanges command was invoked using the --color=blue,red,green flag to show unchanged attributes in blue, changed attributes in red and new values in green. Edit the $INCA_DIST/etc/approveEmail.txt file to add your own color preferences.

5.7. Proxies

For convenience, the Inca framework can be used to retrieve a proxy for the globus2 access method or for reporters that require an active proxy. Reporters that require a proxy should use the Inca::Reporter::GridProxy module described in Section 7.2.3.

Before configuring Inca to retrieve proxies, first store a proxy on a myproxy server. For information about setting up a myproxy server or storing proxies on a server, please see the official myproxy documentation.

The Agent can automatically retrieve a proxy from a myproxy server if the proxy information is defined in incat as follows:

The dialog boxes are the hostname of the myproxy server, the username and password used to store the proxy, and the lifetime in hours that the agent should retrieve a proxy for (the default is 12 hours).

Once proxy information is committed to the agent, it can be retrieved by reporter managers. Each time a reporter manager is ready to run a reporter that needs a proxy it:

• requests the MyProxy passphrase from the agent

• uses the MyProxy command-line client to retrieve proxy credentials from a MyProxy server

• clears the MyProxy passphrase from memory

The MyProxy passphrase is stored on the agent in the $INCA_DIST/var/resources.xml file and is encrypted with the same passphrase as the agent's private key. The MyProxy passphrase passes between the reporter manager and agent over their SSL connection.

5.8. Using Resource Macros

Resource macros provide a shorthand for defining multiple, similar series. For example, suppose you wanted to add three series to the configuration defined above to measure the ping time to three different hosts, named blue.ufo.edu, green.ufo.edu, and red.ufo.edu. One approach would be to define a series for blue, use the Clone button in the Series section of the Suites panel to make two copies, then modify them to ping green and red.

A better approach is to use a macro for the host names and let Inca replicate the series for you. In the Resource Configuration panel, click on defaultGrid in the Resource Group section. Next, press the Add button beneath the Macros section. This opens a dialog box that allows you to enter the name and value(s) of a macro associated with the current resource group.

Enter "targets" in the Macro Name text box and "blue" in the macro value edit box, then use the "Add" button or hit "enter/return" to add "green" and "red" as the second and third values of the macro. Afterwards, press OK. The definition of the targets macro now appears in the Macros section of the Resource Configuration panel. You may also edit values in the list by selecting them, changing the value in the edit value box, and hitting "enter/return". To delete macro values, select the value in the list and press the "Delete" button.

The targets macro is also defined for the other resource groups since the defaultGrid contains all other groups. As shown in Figure 17, macros appear grey if they were defined in a resource group other than the one selected. You can override an inherited macro value by selecting the macro in the Macros panel and pressing the Edit button to open the macro edit dialog. After you change the macro value and press OK, the updated macro definition will show in black in the Macros panel, indicating that the resource is no longer using the inherited value.

To make use of the macro you've defined, click the Suites tab, then press the Add button underneath the Series section to open the series edit dialog. In the dialog, set the reporter to grid.benchmark.performance.ping and the resource group to localResource.

In the host text box in the Arguments section of the dialog, enter "@targets@.ufo.edu". Macro references in incat are indicated by placing a "@" before and after the macro name. When the Inca Agent encounters a macro reference in a series, it makes one copy of the series for each value of the macro. Since the targets macro has three values--blue, green, and red--the Inca Agent will make three copies of this series, substituting a single value for the macro reference in each. In this case that means that you'll have one series with a host argument of "blue.ufo.edu", one with a host argument of "green.ufo.edu", and one with a host argument of "red.ufo.edu".

The inca web pages use the series nickname when displaying series results. If you leave the series nickname with its default value, the name of the reporter, then all three series will have the same nickname. Instead, you can enter "ping_to_@targets@" in the nickname text box. The Inca Agent will expand this reference in parallel with the reference in the host argument, so your three series will have the nicknames ping_to_blue, ping_to_green, and ping_to_red, respectively.

5.9. Series Comparison and Notification

For a particular series, the Inca system by default reports only whether or not the series reporter was able to execute successfully--whether a version reporter was able to determine a package version, a unit reporter was able to run a program, etc. Using Inca's comparison and notification feature, you can refine a series to define success more precisely and to receive notification from Inca when a series reporter detects a problem. The bottom text boxes of the series edit dialog provide access to Inca's comparison and notification feature.

The comparison expression can test the content of the report body, the content of the report error message, or the value of any symbols defined in the report body by <ID> tags. The expression may use any of the boolean operators <, <=, >, >=, ==, and !=, plus perl's pattern match (=~) and mismatch (!~) operators. One simple test would be "body =~ /./", which would test whether the report body contained any characters. Tests can be joined together by the && and || operators. Using these, you could ignore an expected, minor error with the test "body =~ /./ || errorMessage == 'Try again later'".

As mentioned above, you can include symbols defined in the report body in your tests. The Inca system uses the content of any subsequent tag as the symbol value. For example, the body of the output of the gcc version reporter might be

  <body>
    <package>
      <ID>gcc</ID>
      <version>3.1</version>
    </package>
  </body>
  

Here, Inca will use "3.1" as the value of the symbol "gcc". With this output, the comparison test "gcc >= 3.0" would succeed, while the comparison "gcc == 3.0" would fail. If the report body contains an <ID> tag with no subsequent tag, the value of the symbol is defined to be "".

The image below illustrates how a comparison for subpackage versions would be configured.

Inca supports the ability to run a script that notifies you of changes in a series comparison. When you specify a series comparison in the incat series dialog, two additional components become visible. The Notification Script pull-down menu allows you to select the script to run, and the Script Parameters text box allows you to enter parameters to pass to the script. By default, the Inca installation provides two notification scripts, EmailNotifier and LogNotifier. The first sends email to each address specified by the script parameters; the second writes a message to the specified log file. See Section 11.2 for directions on customizing the notifications you receive.

6.1. Current Data

The default page header navigation contains links to tabular and map view result summaries for the sample suite. To add other current data pages under this heading (e.g., result summary tables for additional suites), customize the header.xsl file as described in Section 6.7.3.

      <context-param>
        <param-name>restId</param-name>
        <param-value>rest</param-value>
      </context-param>
      

      <context-param>
        <param-name>restId</param-name>
        <param-value>kit-status-v1</param-value>
      </context-param>
      

6.2. Reports

The items in the REPORTS menu display graphic reports of current Inca data and series histories. These graphs are generated using Cewolf, a JSP tag library for graphing based on JFreeChart. Some reports require the "incaQueryStatus" query to have run previously; information about queries can be found in Section 6.3.3.

6.3. Query

The query pages allow users to generate graphs and cache queries to improve data display.

6.4. Admin

The ADMIN menu item offers two views of the running reporters in an Inca deployment and a configuration page. To link other informational pages under this heading, customize the header.xsl file as described in Section 6.7.3.

6.5. Report Details

The Current Data, Reports, and Query views described in previous sections provide links to a view that displays the individual report details as shown in the figure below.

6.6. Description of Data Consumer Files

The following table describes the main JSPs contained in the default consumer. The JSPs generally retrieve XML from depot and agent functions and apply XSL stylesheets to display HTML. If the "debug=1" parameter for a JSP is used, the JSP displays the XML. Parameters marked with an asterisk (*) are optional.

The default JSPs use the XSL stylesheets in $INCA_DIST/webapps/xsl to transform the XML into HTML. The following stylesheets are installed with the default consumer:

6.7. Changing the Look

.header {
          background-color: #D07651;

188  <xsl:variable name="cellText">
            189    <xsl:choose>
            190      <xsl:when test="string($instance)=''">
            191        <xsl:value-of select="''" />
            192      </xsl:when>
            193      <xsl:when test="string($result/body)!=''
            194             and string($result/errorMessage)=''
            195             and ($comparitor='Success' or count($comparitor)=0)">
            196               passed: 
            197        <!-- get yyyy-mm-dd from gmt timestamp -->
            198        <xsl:value-of select="substring($result/gmt, 1, 10)" /> 
            199        <!-- get HH:MM from gmt timestamp -->
            200        <xsl:value-of select="substring($result/gmt, 12, 5)" />
            201      </xsl:when>
            202      <xsl:otherwise>
            203               error:
            204        <xsl:value-of select="substring($result/errorMessage, 1, 30)" />
            205      </xsl:otherwise>
            206    </xsl:choose>
            207  </xsl:variable>
            208  <xsl:choose>
            209    <xsl:when test="$exit!=''">
            210      <td class="{$exit}">
            211        <a href="{$href}"><xsl:value-of select="$cellText"/></a>
            212        <xsl:if test="$url[matches(., 'markOld')]">

57  <li><h2>Current Data</h2>
58    <ul>
59      <li>
60        <a href="'status.jsp?xsl=default.xsl&amp;suiteNames=newSuite&amp;resourceIds=newResource'">
61           table of newSuite results</a>
62      </li>

13  <xsl:include href="custom-header.xsl"/>
          ...
          24  <xsl:template name="printBodyTitle">
          25    <xsl:param name="title"/>
          26    <xsl:call-template name="custom-header"/>
          27    <xsl:variable name="datenow" select="date:new()" />
          28    <xsl:variable name="dateformat" select="sdf:new('MM-dd-yyyy hh:mm a (z)')"/>
          29    <table width="100%" border="0">
          30      <tr align="left">

<?xml version="1.0" encoding="UTF-8"?>

          <!-- ================================================ -->
          <!-- Prints out custom header for Inca status pages   -->
          <!-- ================================================ -->
          <xsl:stylesheet version="1.0"
          xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
          xmlns="http://www.w3.org/1999/xhtml">

          <xsl:template name="custom-header">
          <table class="header" width="100%">
          <tr>
          <td bgcolor="#003366">
          <img class="logo" src="img/header.jpg"/>
          </td>
          </tr>
          </table>
          <table class="menu" width="100%">
          ... custom navigation ...
          </table>
          </xsl:template>

          </xsl:stylesheet>

6.8. Displaying Errors Neutrally

You may wish to display certain errors neutrally in historical reports. For example, if an error message indicates that a machine was down when the report was collected, you may want to display that error neutrally (i.e. neither as a pass nor a fail). By default, Inca marks error messages that start with "DOWNTIME" or "NOT_AT_FAULT" neutrally because they are generally the result of report filtering (see Section 11.3). Errors that contain "Inca error" or "Unable to fetch proxy for reporter execution" are also marked neutrally since the error may be due to the Inca framework instead of the machine being tested. To change the errors that are marked neutrally, edit the regular expression that is matched as follows:

• Open the $INCA_DIST/etc/common/inca.properties local file and edit the inca.consumer.ignoreErrors property:

  inca.consumer.ignoreErrors=(^DOWNTIME:.*|^NOT_AT_FAULT.*|.*Inca error.*|.*Unable to fetch proxy for reporter execution.*)

• Restart the consumer:

  % cd $INCA_DIST; ./bin/inca restart consumer

Now error messages matching the pattern you chose will be marked neutrally. For example, the four reports that have error messages that begin with "NOT_AT_FAULT" are marked neutrally as "unknown" in the graph below:

6.9. Changing the Installation Location

You can install the data consumer in a non-default location (e.g., on a machine where the depot and agent are not running) if you:

• Copy the incaInstall.sh script to the machine where the consumer will run.

  % wget http://inca.sdsc.edu/releases/2.5/incaInstall.sh

• Install consumer on the new machine:

  % ./incaInstall.sh $INCA_DIST consumers

• Copy the consumer key, certificate, and trusted directory from the machine where the agent/depot are running (orig.machine) to the new machine:

  % scp orig.machine:$ORIG_INCA_DIST/etc/consumerkey.pem $INCA_DIST/etc/; \ scp orig.machine:$ORIG_INCA_DIST/etc/consumercert.pem $INCA_DIST/etc/; scp "orig.machine:$ORIG_INCA_DIST/etc/trusted/*" $INCA_DIST/etc/trusted/;

• Edit the $INCA_DIST/etc/common/inca.properties local file and specify the full hostname of the machine where the agent and depot are running:

  114 inca.consumer.agent=incas://agent.hostname:6323 ... 128 inca.consumer.depot=incas://depot.hostname:6324

• Start the consumer component on the new machine:

  % cd $INCA_DIST; ./bin/inca start consumer

6.10. Changing Ports

By default, the consumer is started on port 8080 (HTTP) and 8443 (HTTPS). To change the port numbers, edit $INCA_DIST/etc/jetty.xml. For example to change the HTTP port to 9080, search for 'SelectChannelConnector' and change the following line:

      <Set name="port">8080</Set>
    

to:

      <Set name="port">9080</Set>
    

Likewise to change the HTTPS port to 9443, search for 'SslSocketConnector' and change the following line:

      <Set name="port">8443</Set>
    

to:

      <Set name="port">9443</Set>
    

If you also have the HTTP port enabled, change the confidentialPort tag under its configuration from:

      <Set name="confidentialPort">8443</Set>
    

to:

      <Set name="confidentialPort">9443</Set>
    

6.11. Disabling HTTP

By default, the consumer is configured as both an HTTP and HTTPS server. To disable HTTP, edit $INCA_DIST/etc/jetty.xml and comment out the section "Add a HTTP listener on port 8080"

      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <!-- Add a HTTP listener on port 8080                                -->
      <!-- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -->
      <!--
      <Call name="addConnector">
        <Arg>
            <New class="org.mortbay.jetty.nio.SelectChannelConnector">
              <Set name="host"><SystemProperty name="jetty.host" /></Set>
              <Set name="port"><SystemProperty name="jetty.port" default="8080"/></Set>
              <Set name="maxIdleTime">30000</Set>
              <Set name="Acceptors">2</Set>
              <Set name="statsOn">false</Set>
              <Set name="confidentialPort">8443</Set>
        <Set name="lowResourcesConnections">5000</Set>
        <Set name="lowResourcesMaxIdleTime">5000</Set>
            </New>
        </Arg>
      </Call>
      -->
    

Then restart the consumer.

      % ./bin/inca restart consumer
    

6.12. Changing HTTPS credential

By default, the HTTPS server will use the credential stored in $INCA_DIST/etc/consumerKeystore. Its DN is "cn=Inca Consumer SSL, o=SDSC, l=San Diego, st=California, c=US". If you'd like to generate a certificate with a different DN, run the keytool command as follows:

% rm -f etc/consumerKeystore
% keytool -keystore etc/consumerKeystore -alias jetty -genkey -keyalg RSA -dname your_DN
    

keytool will prompt you for a keystore password and a key password. You can either make them different or use the same one. If you use the password "consumer", no further changes are needed. Otherwise, you will have to modify <Set name="keyPassword">, <Set name="password">, and <Set name="trustPassword"> in the "Add a HTTPS SSL listener on port 8443" section in $INCA_DIST/etc/jetty.xml. You can either put the password in there in plain text or obfuscate it using Jetty's password utility as follows:

% java -classpath lib/jetty-6.1.7.jar:lib/jetty-util-6.1.7.jar org.mortbay.jetty.security.Password your_password
      

It will output two lines such as follows:

    OBF:1t3b1uh61ugk1t2v
    MD5:482b9f833150f58964b78ddbfa5ab23d
    

Edit $INCA_DIST/etc/jetty.xml and replace the string beginning with OBF in both <Set name="KeyPassword"> and <Set name="Password"> with the string provided by Jetty's password utility:

      <Set name="password">OBF:1v8w1v2h1wg01z0d1z0h1wfy1v1x1v9q</Set>
      <Set name="keyPassword">OBF:1v8w1v2h1wg01z0d1z0h1wfy1v1x1v9q</Set>
      <Set name="trustPassword">OBF:1v8w1v2h1wg01z0d1z0h1wfy1v1x1v9q</Set>
    

6.13. Adding Password Protection

By default, a password is only required on the query.jsp and status-auth.jsp pages. To expand or remove password protection, edit the section 'require authentication on specific status pages' in $INCA_DIST/webapps/inca/WEB-INF/web.xml.

    <security-constraint>
      <web-resource-collection>
        <web-resource-name>Inca Status Pages</web-resource-name>
        <url-pattern>/jsp/admin.jsp</url-pattern>
        <url-pattern>/jsp/query.jsp</url-pattern>
        <url-pattern>/jsp/runNow.jsp</url-pattern>
        <url-pattern>/jsp/status-auth.jsp</url-pattern>
      </web-resource-collection>
      <auth-constraint>
        <role-name>*</role-name>
      </auth-constraint>
      <user-data-constraint>
        <transport-guarantee>CONFIDENTIAL</transport-guarantee>
      </user-data-constraint>
    </security-constraint>
    

Modify or add url-pattern tags to modify password access. For example to apply password access to all Inca status pages, modify the url-pattern tag to below:

        ...
        <url-pattern>/*</url-pattern>
        ...
    

Or to add password protection to another JSP page like config.jsp, add another url-pattern tag like below:

        ...
        <url-pattern>/jsp/query.jsp</url-pattern>
        <url-pattern>/jsp/config.jsp</url-pattern>
        ...
    

By default, the username and password for the pages will be "inca". To change this, edit $INCA_DIST/etc/realm.properties and customize the username and password for your installation. If you do not want to store the password in plain text, use Jetty's password utility described in Section 6.12.

Then restart the consumer.

% ./bin/inca restart  consumer

The next time you view the status pages, you should see a login like:

7.1. Executing Reporters

Because they are executables, Inca reporters are independent of the rest of the Inca system. Reporters can be executed manually from a command line or automatically as part of an Inca installation. Incorporating your own reporters into a running Inca installation requires only writing the reporters (Section 7.2), including them in a repository (Section 7.3), and configuring the repository's series using incat (Section 5). Most developers will execute reporters from the command line before adding them to their Inca installation. After installing Inca, you can try executing some of the reporters that come with the distribution from the command line:

% cd $INCA_DIST/Inca-Reporter-*/bin
% setenv PERL5LIB ../lib/perl
% setenv PYTHONPATH ../lib/python
% ./cluster.compiler.gcc.version 

All Inca reporters must support the command line arguments listed in the table below. In addition, a reporter may support additional command line arguments specific to that reporter's task.

Executing a reporter using different arguments:

% ./cluster.compiler.gcc.version -log=3 
% ./cluster.compiler.gcc.version -help=yes -verbose=0
% ./cluster.compiler.gcc.version -version=yes

7.2. Writing Reporters

Reporters can be written in any language as long as they output XML according to the schema described in Section 7.2.1. New reporter developers may choose to write reporters in Perl or Python since the Inca distribution includes sample reporters and API modules in those languages (Section 7.2.3) for printing XML according to our schema.

NOTE: Because some databases won't allow queries greater than 4kB, the XML portion of the report for logging/debugging and the error message must each be smaller than 4kB in size. The body XML can be 12kB in size because it is queried in three parts. If the report XML is greater than its limit, the depot truncates the oversized section from the beginning until it is the right size.

<?xml version='1.0'?>
<inca:report xmlns:inca='http://inca.sdsc.edu/dataModel/report_2.1'>
  <gmt>2006-11-17T17:35:40Z</gmt>
  <hostname>jhayes-Computer.local</hostname>
  <name>cluster.compiler.gcc.version</name>
  <version>2</version>
  <workingDir>/Users/jhayes/Inca/subversion/inca/trunk/devel/reporters/bin</workingDir>
  <reporterPath>cluster.compiler.gcc.version</reporterPath>
  <args>
    <arg>
      <name>help</name>
      <value>no</value>
    </arg>
    <arg>
      <name>log</name>
      <value>0</value>
    </arg>
    <arg>
      <name>verbose</name>
      <value>1</value>
    </arg>
    <arg>
      <name>version</name>
      <value>no</value>
    </arg>
  </args>
  <body>
    <package>
      <ID>gcc</ID>
      <version>3.3</version>
    </package>
  </body>
  <exitStatus>
    <completed>true</completed>
  </exitStatus>
</inca:report>

#!/usr/bin/env perl

use strict;
use warnings;
use Inca::Reporter::Version;

my $reporter = new Inca::Reporter::Version(
  name => 'cluster.compiler.gcc.version',
  version => 2,
  description => 'Reports the version of gcc',
  url => 'http://gcc.gnu.org',
  package_name => 'gcc'
);
$reporter->processArgv(@ARGV);

$reporter->setVersionByExecutable('gcc -dumpversion');
$reporter->print();

#!/usr/bin/env perl

use strict;
use warnings;
use Inca::Reporter::SimpleUnit;
use Date::Parse;
use Cwd;

my $reporter = new Inca::Reporter::SimpleUnit(
  name => 'security.ca.unit',
  version => 9,
  description => 'Checks whether the CA certificates or CRLs have expired',
  unit_name => 'caCertNCrlExpire'
);

 ...

my $scratchDir = "/tmp/security.ca.unit.$$";
if ( ! mkdir($scratchDir) ) {
  failClean("Cannot mkdir scratch dir $scratchDir"); 
}
$reporter->tempFile( $scratchDir );
if ( ! chdir($scratchDir) ) {
  failClean("Cannot change to scratch dir $scratchDir"); 
}

 ...

if ($err ne ""){
  failClean($err);
} else {
  $reporter->unitSuccess();
  $reporter->print();
}

sub failClean {
  my $err = shift;
  $err =~ s/--\d{2}:\d{2}:\d{2}--/--xx:xx:xx--/g;
  $err =~ s/$$/PID/g;
  $reporter->failPrintAndExit($err);
}

7.3. Reporter Repositories

The Inca system retrieves reporters from external collections called repositories. A reporter repository is simply a file directory, accessed via a file: or http: URL, that contains a catalog file named Packages.gz. This gzipped file includes a sequence of name:value attribute pairs for every reporter and support package in the repository; blank lines separate the attributes for different reporters. For example, here is a portion of the Packages.gz file for the Inca standard reporter repository.

arguments: help no|yes no;log [012345]|debug|error|info|system|warn 0;verbose [0
12] 1;version no|yes no
dependencies: Inca::Reporter;Inca::Reporter::Version
description: Reports the version of tgusage
file: cluster.accounting.tgusage.version
name: cluster.accounting.tgusage.version
url: http://www.teragrid.org
version: 2

arguments: help no|yes no;log [012345]|debug|error|info|system|warn 0;verbose [0
12] 1;version no|yes no
dependencies: Inca::Reporter;Inca::Reporter::SimpleUnit
description: ant hello world test
file: cluster.admin.ant.unit
name: cluster.admin.ant.unit
version: 3

arguments: help no|yes no;log [012345]|debug|error|info|system|warn 0;verbose [0
12] 1;version no|yes no
dependencies: Inca::Reporter;Inca::Reporter::Version
description: Reports the version of Apache Ant
file: cluster.admin.ant.version
name: cluster.admin.ant.version
version: 2

Of the attributes shown, only file and name are required. The file attribute gives the relative path to the reporter file, and the name attribute specifies the unique package name of the reporter. If the reporter requires support packages to execute, it should include a dependencies attribute with a semicolon-separated list of package names. For more information about reporter package dependencies see Section 7.3.1. The incat administration tool uses the Packages.gz file's arguments and description attributes as part of its series edit dialog. The value of the arguments attribute is a semicolon-separated list giving the name, value pattern, and default value, if any, for each supported command-line argument.

To create a local repository for your own reporters, you only need to collect them into a directory and create a Packages.gz in that directory. The default Inca installation has a Packages.gz file in $INCA_DIST/Inca-Reporter-* that can be added in incat. Inca also supplies a web accessible repository that can be added in incat as "http://inca.sdsc.edu/repository/latest/".

The Inca distribution includes a perl script, incpack, that can create Packages.gz for you. Simply run incpack with a list of reporters that you want to include in Packages.gz, e.g.,

% perl incpack jade.version f77.unit vim.version

incpack runs each of the listed reporters with --help=yes --verbose=1 to extract a standard set of attributes. If your reporters use the Inca reporter APIs, you might need to run incpack with -I switches to specify the location of the Inca libraries, like this.

% perl incpack -I ${INCA_DIST}/lib/perl -I ${INCA_DIST}/lib/python jade.version f77.unit vim.version

For more information about incpack usage, click here.

% ./configure --prefix=$RM_INSTALL_DIR/var/reporter-packages
% [g]make
% [g]make install

% [g]make INSTALL_DIR=$RM_INSTALL_DIR/var/reporter-packages 
% [g]make INSTALL_DIR=$RM_INSTALL_DIR=/var/reporter-packages

% perl -I$RM_INSTALL_DIR/var/reporter-packages/lib/perl  Makefile.PL \
  PREFIX=$RM_INSTALL_DIR/var/reporter-packages/lib/perl \
  LIB=$RM_INSTALL_DIR/var/reporter-packages/lib/perl \
  INSTALLDIRS=perl \
  INSTALLSCRIPT=$RM_INSTALL_DIR/var/reporter-packages/bin \
  INSTALLMAN1DIR=$RM_INSTALL_DIR/var/reporter-packages/man/man1 \
  INSTALLMAN3DIR=$RM_INSTALL_DIR/var/reporter-packages/man/man3
% [g]make
% [g]make install

% perl -I$RM_INSTALL_DIR/var/reporter-packages/lib/perl Build.PL \
  --install_path lib=$RM_INSTALL_DIR/var/reporter-packages/lib/perl \
  --install_path libdoc=$RM_INSTALL_DIR/var/reporter-packages/man/man3 \
  --install_path bindoc=$RM_INSTALL_DIR/var/reporter-packages/man/man1 \
  --install_path bin=$RM_INSTALL_DIR/var/reporter-packages/bin \
  --install_path script=$RM_INSTALL_DIR/var/reporter-packages/bin \
  --install_path arch=$RM_INSTALL_DIR/var/reporter-packages/lib/perl
% perl -I$RM_INSTALL_DIR/var/reporter-packages/lib/perl Build
% perl -I$RM_INSTALL_DIR/var/reporter-packages/lib/perl Build install

11.1. Inca Component Options

Each inca component has a set of options that can be set in either the $INCA_DIST/etc/common/inca.properties file or from the command line. The inca.properties file has a list of name value pairs of the format "inca.component.property=value". For example, to start the agent on port 5323 instead of 6323 and enter the password on the command line rather than get it from standard in, you could:

1. edit $INCA_DIST/etc/common/inca.properties and replace:

   • "inca.agent.port=6323" with "inca.agent.port=5323"

   • "inca.agent.password=stdin:password>" with "inca.agent.password=pass:<password>" (where <password> is the password set with the createauth command)

   execute:

   % cd $INCA_DIST; ./bin/inca start agent

2. OR execute the following command:

   % cd $INCA_DIST; ./bin/inca start agent -p 5323 -P pass:<password>

Man pages with component options are described in Section 12.

Note: To change the port of the consumer, see Section 6.10.

Note: if you have more than 5 reporter managers running, increase the number of agent and depot threads in the inca.properties file to be 10 more than the number of reporter managers. For example, if running 15 reporter managers edit the inca.properties file as follows:

31 # Maximum number of threads running on the agent
32 inca.agent.numthreads=25
...
82 # Maximum number of threads running on the depot
83 inca.depot.numthreads=25

11.2. Customizing Series Notification

To customize the notification you receive whenever the result of a series comparison changes, you can either modify the default Inca notification scripts (found in the sbin/ subdirectory of your Inca Depot installation) or write your own. Set inca.incat.notifiers in your inca.property file to add a new script to the options presented in the incat series dialog. For example, to add a script named "MyNotifier" to the options, you would add the following line to inca.properties.

  inca.incat.notifiers=EmailNotifier,LogNotifier,MyNotifier
  

The Inca Depot sets environment variables that provide information to the notification script about the series and comparison, and the script can incorporate these variables into its notification. For example, EmailNotifier uses several of these variables in constructing the email body. These are the environment variable names that the Depot defines:

Values that are specified as the "Script Arguments" in incat are passed as arguments to the script.

11.3. Report Filtering

The Inca Depot allows you to filtering incoming reports before information about them is placed in the Inca database. To do so, you need to write a class that extends edu.sdsc.inca.depot.util.ReportFilter and set the property inca.depot.reportFilter to the name of the class.

An Inca report consists of five elements, all strings: the execution context (reporter name and arguments), the name of the resource where the reporter ran, the reporter stdout (i.e., report), the reporter stderr, and a report of the system resources used by the reporter execution. Of these, stderr may be null; the other four are required.

The ReportFilter class provides set and get methods for each of these five elements. If the inca.depot.reportFilter property is set, the Inca Depot creates an instance of the named class, then calls each of its set and get methods in turn to allow it to make changes to any of the elements. Changes made by the filter are incorporated into the information stored in the Depot database. If a ReportFilter get method for any of the elements other than stderr returns null, the Depot discards the report.

For example, this class directs the Depot to ignore reports that arrive from blue.ufo.edu and modifies report stderr values that contain particular messages.

public class MyReportFilter extends edu.sdsc.inca.depot.util.ReportFilter {

  public String getResource() {
    String resource = super.getResource();
    return resource.equals("blue.ufo.edu") ? null : resource;
  }

  public String getStderr() {
    String stderr = super.getStderr();
    stderr = stderr.replaceAll("Try again.*\n", "");
    return stderr;
  }

}

To install your filter, first compile the class.

% javac -classpath lib/inca-depot.jar MyReportFilter.java

Copy the compiled class to your $INCA_DIST/lib directory. For example,

% cp MyReportFilter.class $INCA_DIST/lib

Then add the inca.depot.reportFilter property to your $INCA_DIST/etc/common/inca.properties file.

...
inca.depot.reportFilter=MyReportFilter
...

Finally, restart the depot.

% ./bin/inca restart depot

package edu.sdsc.inca.depot.util;

import org.apache.log4j.Logger;
import java.net.URL;
import java.util.Properties;
import java.io.InputStream;
import java.io.IOException;

/**
 * Prefixes error messages in depot reports with "DOWNTIME: +optionalString+: "
 * if the resource the report ran on is in downtime.   Resources are determined
 * to be in downtime if they are listed in a downtime properties file.  In order
 * to reduce overhead, the downtime properties file is retrieved and cached at
 * a refresh interval in the getDowntimes() method instead of being retrieved
 * for each filter instance.
 */
public class DowntimeFilter extends edu.sdsc.inca.depot.util.ReportFilter {
  private static Logger logger = Logger.getLogger(DowntimeFilter.class);
  private static Properties downtimes = new Properties();
  private static long lastRefresh = 0;

  /**
   * Returns cached property list of resources in downtime.  Gets and caches
   * property list from file in classpath (downtime.properties) if cache has
   * expired according to refreshMins.
   *
   * The property list file contents can be:
   *
   *  downResource1=optionalErrorMessagePrefixStringForResource1
   *  downResource2=optionalErrorMessagePrefixStringForResource2
   *
   * OR
   *
   *  downResource1
   *  downResource2
   *
   */
  synchronized static Properties getDowntimes()  {
    String downtimePropFile = System.getProperty("inca.depot.downtimeFile");
    if(downtimePropFile == null) {
      downtimePropFile  = "downtime.properties";
    }
    String downtimeRefresh = System.getProperty("inca.depot.downtimeRefresh");
    if(downtimeRefresh == null) {
      downtimeRefresh  = "15";
    }
    Integer refreshMins = Integer.parseInt(downtimeRefresh);
    long minSinceLastRefresh = (System.currentTimeMillis()-lastRefresh)/60000;
    if (minSinceLastRefresh >= refreshMins){
      URL url = ClassLoader.getSystemClassLoader().getResource(downtimePropFile);
      if(url == null) {
        logger.error( downtimePropFile + " not found in classpath" );
      }
      logger.debug( "Located file " + url.getFile() );
      downtimes.clear();
      try {
        InputStream is = url.openStream();
        downtimes.load(is);
        is.close();
      } catch (IOException e){
        logger.error( "Can't load properties file" );
      }
      lastRefresh = System.currentTimeMillis();
    }
    return downtimes;
  }

  /**
   * Writes new report with modified error message to depot if resource is down
   *
   * @return  string with depot report (reporter Stdout)
   */
  public String getStdout() {
    String resourceProp  = getDowntimes().getProperty(super.getResource());
    if (resourceProp != null){
      logger.debug( super.getResource() + " is down " + resourceProp );
      return super.getStdout().replaceFirst(
          "<errorMessage>", "<errorMessage>DOWNTIME:"+ resourceProp +": ");
    } else{
      return super.getStdout();
    }
  }

}

...
inca.depot.downtimePropFile=MyDowntimeFilename
...

...
inca.depot.downtimeRefresh=5
...

downResource1=optionalErrorMessagePrefixStringForResource1
downResource2=optionalErrorMessagePrefixStringForResource2

downResource1
downResource2

...
inca.depot.reportFilter=edu.sdsc.inca.depot.util.DowntimeFilter
...

% ./bin/inca restart depot

...
inca.depot.reportFilter=edu.sdsc.inca.depot.util.All2AllFilter
...

% ./bin/inca restart depot

inca.depot.reportFilter=edu.sdsc.inca.depot.util.DowntimeFilter, edu.sdsc.inca.depot.util.All2AllFilter

11.4. Depot Database Configuration

The Inca depot uses Hibernate to interface to a relational database backend for storing reports and incat configuration. By default, the Inca depot uses Hibernate's HSQL database but can be configured to use any Hibernate supported database. We have tested the Inca depot with PostgreSQL and Oracle.

Steps for using a depot database other than HSQL are as follows:

1. Stop the depot

   % cd $INCA_DIST; ./bin/inca stop depot

2. Edit $INCA_DIST/etc/hibernate.properties

   • Comment out the first 5 lines which specifies for hibernate to use hsql as its backend database:

     1 #hibernate.dialect=org.hibernate.dialect.HSQLDialect 2 #hibernate.connection.driver_class=org.hsqldb.jdbcDriver 3 #hibernate.connection.url=jdbc:hsqldb:test 4 #hibernate.connection.username=sa 5 #hibernate.connection.password=

   • Uncomment the block which specifies for hibernate to use your database (i.e., for PostgreSQL uncomment 8-13, for MySQL uncomment 17-21, for Oracle uncomment 24-28).

   • Change the uncommented hibernate.connection.url, hibernate.connection.username and hibernate.connection.password property values to be the host/db name, login username and password for your database.

3. Put JDBC drivers for your database in the $INCA_DIST/lib directory. Driver download locations: PostgreSQL, MySQL, Oracle

4. Initialize the depot (set up the Inca tables):

   % cd $INCA_DIST; ./bin/inca depot -d

   You should see something like:

   Initializing c3p0 pool... ... Database Initialization Completed

5. Start the depot

   % ./bin/inca start depot

11.5. Manual Access Method

A resource administrator may be unable to start a reporter manager using one of the automated methods (ssh, globus2, or local). In this case, an Inca administrator can add the resource using the access method 'manual'. The following steps will need to be taken by the Inca administrator and resource administrator:

Inca Administrator

Resource Administrator

1. RESOURCE ADMIN: install the reporter manager distribution on your resource using the following steps.

   1. Create an installation directory for the reporter manager (e.g., $RM_INSTALL_DIR). Download the reporter manager tarball and build script:

      % cd $RM_INSTALL_DIR; \ wget http://inca.sdsc.edu/releases/2.5/Inca-ReporterManager.tar.gz; \ wget http://inca.sdsc.edu/releases/2.5/buildRM.sh

      At this point the directory on the remote machine should look something like this:

      % ls Inca-ReporterManager.tar.gz buildRM.sh

   2. Install the reporter manager and list directories to verify files unpacked correctly:

      % bash buildRM.sh $RM_INSTALL_DIR Inca-ReporterManager.tar.gz % ls $RM_INSTALL_DIR Inca-ReporterManager-9.6764 build.log lib share Inca-ReporterManager.tar buildRM.sh man var bin etc sbin

      Install the default set of reporters and their libraries:

      % cd $RM_INSTALL_DIR; wget http://inca.sdsc.edu/releases/2.5/Inca-Reporter.tar.gz % cd var; tar zxvf ../Inca-Reporter.tar.gz; cd Inca-Reporter-* % perl Makefile.PL PREFIX=${RM_INSTALL_DIR}/var/reporter-packages \ INSTALLDIRS=perl LIB=${RM_INSTALL_DIR}/var/reporter-packages/lib/perl \ INSTALLSCRIPT=${RM_INSTALL_DIR}/var/reporter-packages/bin \ INSTALLMAN1DIR=${RM_INSTALL_DIR}/var/reporter-packages/man/man1 \ INSTALLMAN3DIR=${RM_INSTALL_DIR}/var/reporter-packages/man/man3 % make % make install

2. RESOURCE ADMIN: create a set of credentials for the reporter manager (i.e., private key and certificate request) using the command below.

   % cd $RM_INSTALL_DIR; ./sbin/inca createRmCertRequest -P stdin:password:

   Enter a password for your key (to use when you start up the reporter manager). Two files will be created in $RM_INSTALL_DIR/etc: an encrypted private key called rmkey.pem and a certificate request called rmreq.pem. Email rmreq.pem to your Inca administrator and they will generate a certificate for your reporter manager.

3. INCA ADMIN: upon receiving a rmreq.pem file, generate a certificate for a reporter manager using the command below. Replace "rmreq.pem" with to the path to the rmreq.pem file that you received from the resource administrator and "rmcert-resource.pem" with the path to the reporter manager certificate that will be generated by the command.

   % cd $INCA_DIST; ./bin/inca createRmCert -P stdin:password: rmreq.pem rmcert-resource.pem

   Enter the password for the inca distribution (i.e., created in Step 4 during the initial installation process). Email the reporter manager certificate, "rmcert-resource.pem", and trusted certificate to the resource administrator. The trusted certificate is the file ending with the .0 extension in your $INCA_DIST/etc/trusted directory. For example f73fee74.0 is the trusted certificate in the following directory:

   % ls etc/trusted/ agentcert.pem f73fee74.0 rocks-101.sdsc.educert.pem

4. INCA ADMIN: add the specified resource within incat and choose 'manual' as below:

   Figure 20. Add Resource Screen

   

   Make sure the "Equivalent" box is checked, otherwise the depot may discard reports with "unattached to any DB config" warnings. The new "manualResource" will also need to be added to the "defaultGrid" resource in order to run the default sampleSuite. Select "Agent->Commit" from the menu to commit the changes.

5. RESOURCE ADMIN: install the certificate and trusted certificate from the Inca admin in your reporter manager installation. Replace "rmcert-resource.pem" and "trusted.0" with the names of the files received from your Inca administrator.

   % cd $RM_INSTALL_DIR % cp rmcert-resource.pem $RM_INSTALL_DIR/etc/rmcert.pem % mkdir $RM_INSTALL_DIR/etc/trusted % cp trusted.0 $RM_INSTALL_DIR/etc/trusted

6. RESOURCE ADMIN: Finally, you can start up the reporter manager using the commands below. Replace "depotHost" with the hostname where the depot is running and replace "manualResource" with the manual resource group name added in step 4:

   % cd $RM_INSTALL_DIR % ./sbin/inca reporter-manager \ -a incas://agentHost:6323 \ -d incas://depotHost:6324 \ -c etc/rmcert.pem \ -k etc/rmkey.pem -t etc/trusted \ -e bin/inca-null-reporter \ -r var/reporter-packages \ -R sbin/reporter-instance-manager \ -v var \ -w 1 \ -i manualResource \ -s suite.xml \ -L DEBUG \ -l var/reporter-manager.log \ -P true <enter your password>

   Command will hang until the password for the reporter manager key is entered. If the private key is not password protected, don't use the -P option in the command above. Check to make sure the reporter manager is running by doing a "ps | grep reporter-manager" and make sure there aren't errors by doing a "grep ERROR $RM_INSTALL_DIR/var/*".

   To stop the reporter manager at any time, type

   % ./sbin/inca stop reporter-manager

   Make sure all reporter-manager ps are stopped

   % ps | grep manager

11.6. Batch Systems

The cluster.batch.wrapper reporter can be useful when running Inca on batch systems. Without an --exec argument, this reporter submits a trivial program to the batch scheduler and reporters the amount of time it spends in the queue before executing. If given an --exec argument, cluster.batch.wrapper instead submits the argument value (which should contain a reporter invocation) and collects and reports it output. This use allows reporters to be executed on batch nodes, rather than on the submission host. For example, this command will report the version of gcc that is installed on the batch nodes of a PBS cluster.

% cluster.batch.wrapper --scheduler=pbs --exec=cluster.compiler.gcc.version
  

The --scheduler argument is the only one required; valid values are cobalt, dqs, loadleveler, lsf, pbs, and sge. Additional recognized arguments are as follows.

Here are some additional examples of using cluster.batch.wrapper to submit Inca reporters to a batch queue.

% cluster.batch.wrapper --scheduler=pbs --account=alf63 \
  --exec=cluster.compiler.gcc.version

% cluster.batch.wrapper --scheduler=loadleveler --queue=normal --type=parallel \
  --exec='network.ping.unit --host=ufo.edu'

% cluster.batch.wrapper --scheduler=sge --submitparam='-js 1' \
  --submitparam='-l h_vmem=600' \
  --exec='cluster.ps.unit --process=init --psargs=-x'
  

In incat, you can specify that cluster.batch.wrapper should be used to submit a series by including its name and arguments in the series context string. Inca stores installed reporters in the directory $INSTALL_DIR/bin, so the path to cluster.batch.wrapper will be $INSTALL_DIR/bin/cluster.batch.wrapper. The figure below shows the context string for the network.ping.unit series mentioned above.

11.7. Manual Run Now

Oftentimes resource or system administrators will want to show that a problem has been resolved by independently executing inca tests before they are scheduled to run so that their results appear on status pages. Rather than granting resource administrators full privileges just to use incat's "run now" button, the inca administrator can provide resource administrators with a "manual run now" option - a command line script to execute tests and send results to the depot. A "run now" can also be invoked via the Inca web pages if configured (by default "run now" is not enabled). See Section 6.5 for more information.

% ./bin/admin-run-now -h

11.8. Source Distributions

Source distributions of the Inca components are also available. The following table lists the Inca component source distributions and shows how to build each of them. Note, that Apache Ant is needed for the Inca components implemented in Java.

12.1. Depot

% bin/inca help depot
java edu.sdsc.inca.Depot
  P|password   str     Specify how to obtain encryption password
  V|version    null    Display program version
  a|auth       boolean Authenticated (secure) connection?
  c|cert       path    Path to the authentication certificate
  d|dbinit     null    init depot DB tables
  h|help       null    Print help/usage
  h|hostname   str     Hostname the server should provide to clients
  i|init       path    Path to properties file
  k|key        path    Path to the authentication key
  l|logfile    str     Route log messages to a file
  n|numthreads int     # threads in worker pool
  p|port       int     Server listening port
  r|remove     null    remove depot DB tables
  t|trusted    path    Path to authentication trusted certificate dir
  v|var        path    Absolute path to server temp dir

12.2. Agent

% bin/inca help agent
java edu.sdsc.inca.Agent
  C|check            str     check the reporter manager on resources
  D|depot            str     Depot specification; host:port
  H|hostname         str     Hostname where the server is running
  P|password         str     Specify how to obtain encryption password
  R|refreshPkgs      int     repository check period for package updates
  S|server           str     Server specification; host:port
  S|startAttempt     int     re-start attempt period fpr the manager
  U|upgradeTargets   str     makefile targets to execute during upgrade
  V|version          null    Display program version
  a|auth             boolean Authenticated (secure) connection?
  b|buildscript      str     path to reporter manager build script
  c|cert             path    Path to the authentication certificate
  e|email            str     email to send notices of manager restarts
  h|help             null    Print help/usage
  i|init             path    Path to properties file
  k|key              path    Path to the authentication key
  l|logfile          str     Route log messages to a file
  n|numthreads       int     # threads in worker pool
  p|port             int     Server listening port
  r|rmdist           str     path to reporter manager tarball distribution
  s|stayAlive        int     stay alive ping period for the manager
  t|trusted          path    Path to authentication trusted certificate dir
  u|upgradeResources str     upgrade managers in specified resource group
  v|var              path    Absolute path to server temp dir

12.3. Reporter Manager

% sbin/inca help reporter-manager
Usage:
    reporter-manager [-a|-s] [options]

Options:
    a|--agent
         A string containing the URI to the Reporter Agent process that will
         be responsible for the reporter manager. Either this option or -s
         must be specified. Currently accepted URIs include:

         incas://host:port inca://host:port

    -c|--cert
         A path to a valid certificate file [default: none]

    -d|--depot
         A string containing the URI of a depot to send its reporter data
         to. Currently accepted URIs include:

         incas://host:port inca://host:port file://path

         This option can be specified more than once. The report will be
         sent to the first specified depot. If the first depot is
         unreacheable, the next depots in the list will be tried.

    -e|--error-reporter
         A string containing a path to the error reporter. E.g.,
         inca-null-reporter

    -h|--help
         Print help/usage information

    -i|-id
         The resource identifier supplied by the reporter agent that the
         reporter manager will use to identify itself back to the reporter
         agent.

    -k|--key
         A path to a valid key file [default: none]

    -l|--logfile
         A string containing a path to the file where the log messages can
         be stored. If not specified, log messages will be printed to the
         console.

    -L|--level
         A string containing the log message level (i.e., print statements
         of this level and higher). [default: INFO]

    -P|--passphrase
         Read a passphrase for key from stdin

    -r|--reporter-cache
         A string containing the path to the local cache of reporters.

    -R|--rim
         A string containing a path to the reporter-instance-manager script.
         If not specified, this script will look into the directory where
         itself is located.

    -s|--suite
         A string containing a path to the Inca suite file containing the
         reporters to be executed.

    -t|--trusted
         A path to either a directory or file of trusted certificates
         [default: none]

    -v|--var
         A string containing a path to a temporary file space that Inca can
         use while executing reporters

    -w|--wait
         A positive integer indicating the period in seconds of which to
         check the reporter for a timeout [default: 2]

12.4. Consumer

% bin/inca help consumer
java edu.sdsc.inca.Consumer
  P|password str     Specify how to obtain encryption password
  V|version  null    Display program version
  a|agent    str     URI to the Inca agent
  a|auth     boolean Authenticated (secure) connection?
  c|cert     path    Path to the authentication certificate
  d|depot    str     URI to the Inca depot
  h|help     null    Print help/usage
  i|init     path    Path to properties file
  k|key      path    Path to the authentication key
  l|logfile  str     Route log messages to a file
  m|maxWait  int     Max wait time a JSP tag should wait on a cached item
  r|reload   int     Reload period for cached objects (e.g., suites)
  t|trusted  path    Path to authentication trusted certificate dir
  v|var      path    Path to temporary directory

12.5. Incat

% bin/inca help incat
java incat
  A|agent    str     Agent specification; host:port
  P|password str     Specify how to obtain encryption password
  V|version  null    Display program version
  a|auth     boolean Authenticated (secure) connection?
  c|cert     path    Path to the authentication certificate
  f|file     path    Inca installation configuration file path
  h|help     null    Print help/usage
  i|init     path    Path to properties file
  k|key      path    Path to the authentication key
  l|logfile  str     Route log messages to a file
  t|trusted  path    Path to authentication trusted certificate dir

12.6. Inca Web Services

% ./bin/inca help incaws
incaws [opts] depot agent
  --auth=yes/no
  --cert=path
  --help=yes/no
  --init=path
  --key=path
  --logfile=path
  --password=str
  --port=int
  --trusted=path
  --version=yes/no

13.1. Log Files

The agent, depot and consumer logs are located in the $INCA_DIST/var directory. Reporter manager logs are located in each manager's install directory under the var directory (e.g. ~/incaReporterManager/var).

Logging is informational by default, but can be adjusted to be more verbose ('info' to 'debug') or less verbose ('info' to 'error') by editing the $INCA_DIST/etc/common/log4j.properties file and then restarting inca components. Note that passwords are logged when 'debug' logging is turned on. Logging for the inca components can be adjusted by editing lines 26 and 27 ("log4j.rootLogger=info, stdout" and "log4j.logger.edu.sdsc.inca=info"). To log the most verbose globus error messages change line 33 in log4j.properties from "log4j.logger.org.globus=error" to "log4j.logger.org.globus=debug".

If a reporter manager is not started on a resource after you have scheduled reporters to run there, it is likely the build on that resource failed. You can confirm by looking for "Unable to stage reporter manager to " in $INCA_DIST/var/agent.log. If found, look for errors on the resource in the 2 build log files from the reporter manager build attempt: ~/incaReporterManager/build.log and ~/incaReporterManager/Inca-ReporterManager-*/build.log. The most common build failure is a bad build of the dependency Net::SSLeay which is required for secure communication; the build for Net::SSLeay will fail if it is unable to find OpenSSL on the resource. The Perl SSL modules call the same functions as "openssl verify".

13.2. Authentication Problems

If you observe that a reporter manager on a resource is having trouble connecting to either the agent or depot, there could be a problem with either the installed SSL libraries or certificates. To test, use the pingClient command as in the example below. Replace $RM_INSTALL_DIR with the path to the directory where the reporter manager is installed. Supply the appropriate hostname and port for either the depot or agent. After pressing return, type the password for the certificates.

% cd $RM_INSTALL_DIR
% sbin/inca pingClient \
  -c etc/rmcert.pem \
  -k etc/rmkey.pem -P true -t etc/trusted \
  -uri incas://<agent or depot hostname>:<port>
<your password>

If there are no problems contacting either the agent or depot, the exit code will be 0 and nothing will be printed to the screen. If there is an authentication problem, the exit code will be non-zero and a message such as the following will be printed to stderr:

ERROR - Unable to create Inca::IO socket: : IO::Socket::INET configuration failed
Error contacting Inca server 'incas://rocks-101:6325' at bin/inca-ping-client line 137, <STDIN> line 1.

One possible cause of connection problems may occur when the agent does a local lookup of 'localhost' for its hostname but Java doesn't find the fully qualified hostname. An example is when the logged agent hostname is something like 'agent-machine:6323' but should be 'agent-machine.sdsc.edu:6323'. You can override this by adding the fully qualified hostname to your $INCA_DIST/etc/common/inca.properties file on the agent (add 'inca.agent.hostname=agent-machine.sdsc.edu' to inca.properties in this case). Change other properties with 'localhost' values to your fully qualified hostname, e.g.:

inca.agent.depot=incas://depot-machine.sdsc.edu:6324
...
details at http://consumer-machine.sdsc.edu:8080/inca/xslt.jsp?xsl=instance.xsl&instanceID=@instanceId@&configID=@configId@&resourceName=@resource@\n\n
...
inca.consumer.agent=incas://agent-machine.sdsc.edu:6323
...
inca.consumer.depot=incas://depot-machine.sdsc.edu:6324

After you do this, you may need to remove the reporter manager directories on your Grid machines and re-initialize the configuration with the "bin/inca default" command so that new suite names are created with the correct hostname. If you've already invested a lot in your configuration and don't want to re-initialize, email inca@sdsc.edu for help.

Please email inca@sdsc.edu if you are unable to determine the cause of the authentication problem.