7. Writing Custom Data Consumers

7.1. Retrieving XML

7.1.1. REST URLs

Inca also provides the ability to fetch data in XML or HTML format using REST urls. By default, the consumer recognizes a REST url using the following format:

http://localhost:8080/inca/XML|HTML/rest/<suiteName>[/<resourceId>[/<seriesNickname>[/week|month|quarter|year]]]

For example, the url

http://localhost:8080/inca/HTML/rest/sampleSuite

is equivalent to

http://localhost:8080/inca/jsp/status.jsp?suiteNames=sampleSuite.

Likewise, the url

http://localhost:8080/inca/HTML/rest/sampleSuite/defaultGrid

is equivalent to

http://localhost:8080/inca/jsp/status.jsp?suiteNames=sampleSuite&resourceIds=defaultGrid.

Specific resources can also be specified as resourceId,

http://localhost:8080/inca/HTML/rest/sampleSuite/localResource

The latest instance of a test can also be retrieved as follows:

http://localhost:8080/inca/HTML/rest/sampleSuite/localResource/ant_version

Historical results for a test can also be retrieved for a week, month, quarter or year. For example:

http://localhost:8080/inca/HTML/rest/sampleSuite/localResource/ant_version/week

If you want to fetch the data in XML, just replace HTML as below:

http://localhost:8080/inca/XML/rest/sampleSuite/localResource

If you would like to change the id 'rest' to a more transparent id such as 'kit-status-v1', edit <context-param> in $INCA_DIST/webapps/inca/WEB-INF/web.xml and restart the consumer. For example, change

      <context-param>
        <param-name>restId</param-name>
        <param-value>rest</param-value>
      </context-param>
      

to

      <context-param>
        <param-name>restId</param-name>
        <param-value>kit-status-v1</param-value>
      </context-param>
      

7.1.2. Inca Client APIs

Currently, we provide Perl and Java client APIs to the Inca agent and depot.

7.1.3. Web Services

To access Inca from a Web services API, you will need to install the Inca web services component, incaws.

% wget http://inca.sdsc.edu/releases/2.6/incaInstall.sh
% sh incaInstall.sh $INCA_DIST incaws

The results should look similar to:

Retrieving http://inca.sdsc.edu/releases/latest/Inca-WS.tar.gz
--12:59:23--  http://inca.sdsc.edu/releases/latest/Inca-WS.tar.gz
           => `Inca-WS.tar.gz'
Resolving inca.sdsc.edu... 198.202.75.28
Connecting to inca.sdsc.edu|198.202.75.28|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1,226,347 (1.2M) [application/x-tar]

100%[====================================>] 1,226,347     --.--K/s             

12:59:23 (81.68 MB/s) - `Inca-WS.tar.gz' saved [1226347/1226347]

Unpacking http://inca.sdsc.edu/releases/latest/Inca-WS.tar.gz
Inca-WS-1.6421/
Inca-WS-1.6421/lib/
...
Inca-WS-1.6421/etc/IncaWS.wsdl
Inca-WS-1.6421/version.svn
Will install Inca prerequisite Net::SSLeay
Will install Inca prerequisite IO::Socket::SSL
Will install Inca prerequisite Expat
Will install Inca prerequisite LWP::UserAgent
Will install Inca prerequisite MIME::Base64
Will install Inca prerequisite SOAP::Lite
Writing Makefile.perl.inc for Inca-WS
Inca-WS installed

To start incaws, specify the port, credentials, and hostname/port for the Inca agent and depot as below. Replace "origHost", "agentHost" and "depotHost" with the correct names for your installation.

% cd $INCA_DIST
% ./bin/inca incaws \
 --auth=yes \
 --cert=etc/agentcert.pem \
 --key=etc/agentkey.pem \
 --trusted=etc/trusted/origHostcert.pem \
 --port=8001 \
 --password=yes \
 depotHost:6324 \
 agentHost:6323
enter password (no prompt displayed)

Check to make sure the incaws is running on port 8001:

% netstat -an | grep 8001
tcp4       0      0  *.8001                 *.*                    LISTEN
Error logs are in $INCA_DIST/var.

The WSDL file for the incaws component is in $INCA_DIST/etc/IncaWS.wsdl. The following table summarizes the available functions.

Table 5. Web services functions

Function

Description

getCatalog( [$url] )

Asks the agent to retrieve and return the package catalog from the reporter repository accessed via $url. An undefined $url indicates that the agent should return a merged catalog for all known repositories.

getConfig()

Asks the agent to return XML for the Inca deployment configuration.

pingAgent( $string )

Check that the Inca agent is accessible.

pingDepot( $string )

Check that the Inca depot is accessible.

queryGuids()

Asks the depot to return a space-separated list of known suite guids.

queryHql($hql)

Asks the depot use the HQL select statement $hql to extract and return information from the DB. On success, returns a reference to an array that contains the objects selected by the select statement.

queryInstance($instanceId, $configId)

Asks the depot to report details about one particular invocation of a reporter. $instanceId is the DB id of the instance for the invocation; $configId the related series configuration DB id. On success, returns a reference to a single-element array that contains a ReportDetails document (described in Section 7.2.1) for the instance.

querySeries($configId)

Asks the depot to retrieve information about all instances related to the series configuration identified by $configId. On success, returns a reference to an array that contains a set of ReportDetail documents (described in Section 7.2.1) related to the series.

querySuite($guid)

Asks the depot to retrieve information about all the series of the suite identified by $guid. On success, returns a reference to an array that contains a set of ReportSummary documents (described in Section 6.6) related to the series configurations of the suite.

Below shows an example of how to access the Inca web services from Perl using SOAP::Lite.

use SOAP::Lite;
use Cwd;

my $cwd = getcwd();
my $ws = SOAP::Lite->service("file:$cwd/etc/IncaWS.wsdl");

# check agent and depot are available
print $ws->pingAgent('hello agent'), "\n";
print $ws->pingDepot('hello depot'), "\n";

# get the Inca configuration
print $ws->getConfig(), "\n";
my $guid = $ws->queryGuids();

# get the latest instances of a suite
my $results = $ws->querySuite( $guid );
for my $result ( @{$results} ) {
  print $result;
}

Place the above code in a file called $INCA_DIST/sampleWS.pl and set the environment variable PERL5LIB to $INCA_DIST/lib/perl. Then type,

% perl sampleWS.pl

When run against the default installation, the results should look similar to below.

hello agent
hello depot
<inca:inca xmlns:inca="http://inca.sdsc.edu/dataModel/inca_2.0">
<repositories>
  <repository>http://inca.sdsc.edu/repository/latest</repository>
</repositories>
<resourceConfig>
  <resources>
    <resource>
        <name>defaultGrid</name>
        <xpath>//resource[matches(name, "localSite")]</xpath>
      <macros>
 ...
  </resources>
</resourceConfig>
<suites>
  <suite>
    <seriesConfigs>
      <seriesConfig>
        <series>
          <name>cluster.math.atlas.version</name>
          <uri>http:// ... cluster.math.atlas.version</uri>
          <args>
            <arg>
              <name>cc</name>
              <value>cc</value>
            </arg>
            <arg>
              <name>dir</name>
              <value/></arg>
            <arg>
              <name>help</name>
              <value>no</value>
            </arg>
            <arg>
              <name>log</name>
              <value>3</value>
            </arg>
            <arg>
              <name>verbose</name>
              <value>1</value>
            </arg>
 ...
        <action>add</action>
      </seriesConfig>
    </seriesConfigs>
    <name>sampleSuite</name>
    <guid>incas://rocks-101.sdsc.edu:6323/sampleSuite</guid>
    <description/>
    <version>1</version>
  </suite>
</suites>
</inca:inca>

<reportSummary xmlns="http://inca.sdsc.edu/queryResult/reportSummary_2.0">
  <hostname xmlns="">localResource</hostname>
  <uri xmlns="">http:// ... cluster.math.atlas.version</uri>
  <nickname xmlns="">atlas_version</nickname>
  <seriesConfigId xmlns="">1</seriesConfigId>
  <instanceId xmlns="">24</instanceId>
  <gmt xmlns="">2007-02-01T13:21:01.000-08:00</gmt>
  <body xmlns:rep="http://inca.sdsc.edu/dataModel/report_2.1" xmlns=""/>
  <errorMessage xmlns="">Cannot locate ATLAS installation; use
-dir</errorMessage>                    
</reportSummary>
...

7.2. Understanding XML Schemas

7.2.1. Report Details

The individual report details are generated by a depot query that returns XML formatted like the following (tags are described below):

<reportDetails xmlns="http://inca.sdsc.edu/dataModel/reportDetails_2.1">
  <suiteId xmlns="">8140012</suiteId>
  <seriesConfigId xmlns="">8156370</seriesConfigId>
  <seriesId xmlns="">1712066</seriesId>
  <reportId xmlns="">28430963</reportId>
  <instanceId xmlns="">30977056</instanceId>
  <seriesConfig xmlns="">
    <series>
      <name>cluster.compiler.gcc.version</name>
      <version>2</version>
      <uri>http://inca.sdsc.edu/2.0/ctssv3/bin/cluster.compiler.gcc.version</uri>
      <args>
        <arg>
          <name>log</name>
          <value>5</value>
        </arg>
        <arg>
          <name>version</name>
          <value>no</value>
        </arg>
        <arg>
          <name>help</name>
          <value>no</value>
        </arg>
        <arg>
          <name>verbose</name>
          <value>1</value>
        </arg>
      </args>
      <limits>
        <wallClockTime>600.0</wallClockTime>
        <memory>-1.0</memory>
        <cpuTime>-1.0</cpuTime>
      </limits>
      <context><![CDATA[bash -l -c 'set -a; cd /usr/users/9/inca/inca2install; cp ~/.soft.v3 ~/.soft.v3.$$ && soft-msc ~/.soft.v3.$$ && source ~/.soft.v3.$$.cache.sh && export PERL5LIB=/usr/users/9/inca/inca2install/var/reporter-packages/lib/perl:${HOME}/inca/install/lib/perl &&cluster.compiler.gcc.version -help="no" -log="5" -verbose="1" -version="no"; rm -f ~/.soft.v3.$$*';]]></context>
      <nice>false</nice>
    </series>
    <nickname>compiler-gnu-version-as-4.0.1</nickname>
    <resourceHostname>psc-bigben</resourceHostname>
    <schedule>
      <cron>
        <min>2</min>
        <hour>17</hour>
        <mday>*</mday>
        <wday>*</wday>
        <month>*</month>
      </cron>
      <numOccurs>-1</numOccurs>
      <suspended>false</suspended>
    </schedule>
    <acceptedOutput>
      <comparitor>ExprComparitor</comparitor>
      <comparison>gcc=~".*"</comparison>
      <notifications>
        <notification>
          <notifier>EmailNotifier</notifier>
          <target>FailTo:inca@sdsc.edu</target>
        </notification>
      </notifications>
    </acceptedOutput>
    <action>add</action>
  </seriesConfig>
  <report xmlns="">...</report>
  <comparisonResult xmlns="">Success</comparisonResult>
  <sysusage xmlns="">
    <wallClockTime>0.929562</wallClockTime>
    <memory>0.0</memory>
    <cpuTime>0.556034</cpuTime>
  </sysusage>
  <stderr xmlns=""/>
</reportDetails>

Report detail output is surrounded by <reportDetails> tags. A prefix with a tag name that references http://inca.sdsc.edu/dataModel/reportDetails_2.1, which is the namespace that defines the report schema, can also be used.

The following tags are defined within a <reportDetails>:

suiteId

(internal) the database identifier for the suite id number this report series belongs to (used in further queries)

seriesConfigId

(internal) the database identifier for the series configuration information for this report series (used in futher queries)

seriesId

(internal) the database identifier for the series information for this report series (used in further queries)

reportId

(internal) the database identifier for the report information for the particular result from this report series (used in further queries)

instanceId

(internal) the database identifier for the instance information for the particular time this report series executed (used in futher queries)

seriesConfig

all of the configuration options for this report series: name (of reporter), version (of reporter), uri (for reporter), args, limits (for consumption of wall clock time, memory, and cpu time), context (command to execute series), nickname (of series), resourceHostname (where series will execute), schedule (cron for executing series), acceptedOutput (can include "comparison" string to match in the report and "notification" actions to take if the comparison fails)

report

report XML like that described in Section 8.2.2

comparisonResult

if series was configured with a comparison, the result of the comparison for this particular report series execution

sysusage

amount of wall clock time, memory and cpu time this particular report series execution consumed

stderr

standard error, if any, for this particular report series execution