NGS Accounting Client Design, Architecture, Installation, and Configuration

The overall work flow is simple:

• Various accounting logs on a clusters head node are parsed directly or converted into an intermediate form for the purposes of speed and provenance (intermediate form is often a Berkeley database for fast look ups and maintaining history when logs are rotated).
• The batch system accounting log is the primary key, so to speak, of the various accounting sources (logs and configuration files). As each accounting record from the batch system is parsed supplemental information from other accounting logs is added.
• The outcome of all this parsing of accounting information is to generate an OGF Usage Record (an XML specification for recording compute usage).
• These Usage Records are then uploaded to a central database. Another OGF specification, the Resource Usage Service (RUS) describes how the upload should be done. Specifically it defines the interface front-ending the database and the transport mechanisms for the client use. Note: the term RUS is used to represent the database and the web-service front-end.

From the work-flow there are three main stages for the system administrator to consider:

1. Establishing which accounting logs should be considered as a "source" of information and for those accounting logs that need it, converted into an intermediate form for speed and history purposes.
2. Generating Usage Records from the set of accounting data.
3. Uploading the Usage Records to the central database.

Much of the technology used is proscribed by the use of the OGF UR and RUS specifications: XML, SOAP, and WSDL i.e., Web Services.

Where we do have a choice is in the client implementation. Perl is used as this the most likely language to be known and used by system administrators (allow customisation to their specific environment).

For portability reasons and the extremely low maintenance required, the Berkeley database (which can be thought of as a giant Perl hash) was chosen as the intermediate form for log data when required.

The BATCH2UR module contains a parser for each of the various accounting sources. Each of these parsers extract information out of the logs and into a Perl hash. The various hashes are combined into one large hash that is used to populate a Usage Record XML template. It is the system administrators job to configure the information sources to select the correct parsers and templates for their system.

The RUS client perl module main use is in providing Perl functions that either extract or upload a Usage Record to a RUS. We are mainly concerned with upload as alternate mechanisms exist for accessing the RUS database. It's single purpose means that this module is much simpler than BATCH2UR and configuration easier.

The system administrator should note that authentication between the client and the RUS server is via X509 certificates. As such it is necessary to provide your RUS administrators with the following information:

• DN of the uploading certificate
• Machinenames for which that certificate is authorised to upload for (typically this is the same as the machine name in the Usage Record).

This allows the RUS administrators to setup the appropriate authorisation server side.

If you have all ready deployed the rus and batch2ur client and are upgrading then there are a few things you need to bear in mind. Otherwise skip this section.

• An RPM install will not over-write the existing source installation of a module. The best course of action is to manually delete the source installed modules (typically located in /usr/lib/perl5/site_perl). If not this version may be found before the newer version. You can check with:

   perl -MBATCH2UR::Generate -e 'print "$BATCH2UR::Generate::VERSION\n"';
  

  The version should be >= 0.6
• The batch2ur and rus client perl modules have not had any change in API so all scripts should work as is.
• The batch2ur.conf and rus.conf options have slightly changed names to allow more flexibility and more accurately reflect what options do. In particular options regarding the location of Globus logs have changed - check this carefully.
• The createjbmdb script now also depends on the batch2ur.conf file for the location of Globus job logs and the place to save the generated database. This allows it to be installed in a sensible path e.g., /usr/bin (before it had to be in the Globus log directory - please delete the old version).

In order to provide the necessary Perl modules we use some packages from the Redhat EPEL directory (a source of good extra packages for Redhat Enterprise supported by Fedora (indirectly Redhat)). See information about EPEL here.

Install the EPEL repo like so:

 wget http://download.fedora.redhat.com/pub/epel/4/i386/epel-release-4-9.noarch.rpm
 rpm -ivh epel-release-4-9.noarch.rpm
 rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL

The rest of the required packages are provided by the RCS Manchester NGS Yum repository. See the package listings here.

Install the RCS Man Yum repo like so:

 wget http://storage008.vidar.ngs.manchester.ac.uk/yum/rcs-man-ngs/4/x86_64/rcs-man-ngs-release-4-7.noarch.rpm
 rpm -ivh rcs-man-ngs-release-4.7.noarch.rpm
 rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-RCS-MAN-NGS

To install the clients use yum like so:

 yum install batch2ur-scripts rus-scripts

All libraries and dependencies should be installed automatically.

[Note:1] NGS ClusterVision sites, if you have the CVOS yum repositories installed you may need to temporarily disable it (yum option –disablerepo). The CVOS repository has a version of the perl module soap-lite that conflicts with the core-utils package. Best to install soap-lite package from EPEL as this doesn't conflict.

[Note:2] If you wish to build RPMs for your own system the source packages are here. Please contribute back for inclusion.

1. Using the table from the Accounting Source table form the Software Architecture table establish which sources of information you have available.
2. If either the Globus or LCAS VO logs are to be used then we need to setup the pre-processing. Both createjbmdb and createlcasdb are designed to be run from a logrotate script and work on the post-rotated file i.e., the log ending in .1.

   For a VDT installation the file to edit is

    $VDT_LOCATION/vdt/etc/vdt.logrotate 
   

   and it should have entries as follows.

   /usr/local/NGS/vdt-1.8.1/globus/var/accounting.log {
   nocompress
   postrotate
      export BATCH2UR=/etc/batch2ur/batch2ur.conf
      /usr/bin/createjbmdb
   endscript
   }
   /usr/local/NGS/vdt-1.8.1/glite/var/voms_accounting.log {
   nocompress
   postrotate
     export BATCH2UR=/etc/batch2ur/batch2ur.conf
     /usr/bin/createlcasdb
   endscript
   }
   

   [note:1] The export lines provide the location of the configuration file used by each of the create scripts to locate the accounting log files and decide where to put the generated databases. Also ensure the paths to the create scripts is correct for your installation.

   [note:2] The client needs the accounting log to be directly readable so the nocompress option is necessary to prevent the rotated logs from being zipped.

   [note:3] To test the generation scripts are working you need to force log-rotation (normally a logrotate state file will prevent this). Do this like so:

    $VDT_LOCATION/logrotate/sbin/logrotate -f \
    -s $VDT_LOCATION/vdt/var/logrotate.state \
    $VDT_LOCATION/vdt/etc/vdt.logrotate
   

3. Next configure /etc/batch2ur/batch2ur.conf to your own particular sites setup. The configuration file is fully commented and all setting self-explanatory. The example below is the live configuration file for vidar.ngs.manchester.ac.uk and uses all information sources.

   # Configuration file for BATCH2UR
   
   # Each of the various data-sources used to generate a usage record is
   # specified here.  If a data-source is commented out then it will not
   # be used by the client when generating a record.  Ideally as many
   # data-sources as possible are used to generate the fullest accounting
   # information.
   
   # First we specify the save location of the generated Usage Records
   # (it is expecting to have two sub-dirs: invalid and upload)
   # e.g:
   UrFileDir=/var/spool/batch2ur
   
   # Next, specify the MachineName - ideally the head node of a batch
   # system.  If not set records will fail to upload to the NGS RUS.
   # e.g:
   MachineName=vidar.ngs.manchester.ac.uk
   
   # The BatchSystem backend, used to specify which log parsers to try.
   # e.g:
   # BatchSystem=LSF
   # BatchSystem=Torque
   # BatchSystem=PBS
   BatchSystem=PBS
   
   # Specify the location of the Batch accounting log directory
   BatchAccountingDir=/var/spool/PBS/server_priv/accounting
   
   # Specify the location of the Globus accountings logs, used by createjbmdb
   # to create batch system specific databases from jobmanager strings
   # e.g:
   GlobusAccountingDir=/usr/local/NGS/vdt-1.8.1/globus/var
   
   # Specify the location of the batch databases to be saved by
   # createjbmdb and read by the parsers.  If not set then no jobmanager
   # databases will be used.
   # e.g:
   BatchDatabaseDir=/usr/local/NGS/vdt-1.8.1/globus/var
   
   # Specify the location of the LCAS (or VOMS) accounting logs to be used by
   # createlcasdb in generating VOMS databases for the various services
   # on the system.
   # e.g:
   LcasAccountingDir=/usr/local/NGS/vdt-1.8.1/glite/var
   
   # Specify the location of the LCAS databases to be saved by createlcasdb
   # and used by the parsers.
   # e.g:
   LcasDatabaseDir=/usr/local/NGS/vdt-1.8.1/glite/var
   
   # Specify the location of the edg-mkgridmap.conf file.  This is used
   # as a fail-over for VOMS info or to work out non-globus-job VO info.
   # e.g:
   MakeGridMapFile=/usr/local/NGS/vdt-1.8.1/edg/etc/edg-mkgridmap.conf
   
   # Specify the location of the GridMapDir - used to work out the DN for
   # local use or as a fail-over, if the DN has not been found in the
   # data-sources above.
   # e.g:
   GridMapDir=/etc/grid-security/gridmapdir
   
   # Specify the location of the Gridmap-file - used as a last attempt to
   # work out the DN if all data-sources above have failed to provide it.
   # e.g:
   GridMapFile=/etc/grid-security/grid-mapfile
   

4. Test the generation of records by

    export BATCH2UR=/etc/batch2ur/batch2ur.conf
    /usr/bin/batch2ur
   

   Check the /var/spool/batch2ur/upload directory and confirm that all the expected information in the xml records is present.

1. Configure /etc/rus/rus.conf with appropriate information from your site. The example is the live configuration for vidar.ngs.manchester.ac.uk.

   # Configuration file for RUS Client
   
   # If you are only interested in extraction then you only need to set
   # the RusServer and Certificate/Key values.  For uploading of data
   # also set the UrFileDir and UploadDir.
   
   # The location of the RUS server to upload to or download from
   RusServer=https://rus.vidar.ngs.manchester.ac.uk:8443/rus/service
   
   # The locations of the certificate and key to use for authentication
   # e.g:
   Certificate=/etc/grid-security/hostcert.pem
   Key=/etc/grid-security/hostkey.pem
   
   # The location of the RUS spool files directory (it should have the
   # following sub-dirs: invalid, duplicate, and denied). Here the logs
   # and rejected records are kept for analysis e.g:
   UrFileDir=/var/spool/rus 
   
   # The location of the directory that contains the Usage Records to
   # upload These are typically generated from a client using the perl
   # BATCH2UR module.
   # e.g:
   UploadDir=/var/spool/batch2ur/upload
   

As we wish the upload of a sites records to happen daily and be automatic we use the sites host certificate and key (as these are typically not passphrase protected). This does however mean that any uploads have to be performed by root.

[note:1] The location of the RUS server is vital. The current NGS production RUS server is hosted at:

 https://rus.vidar.ngs.manchester.ac.uk:8443/rus/service

1. If you have not done so all ready ensure that RUS server administrators have configured the authorisation so that your certificate and machine name are allowed to upload records (for the NGS please email the helpdesk).
2. Test an upload of a file generated in the previous BATCH2UR section.

    export BATCH2UR=/etc/batch2ur/batch2ur.conf
    /usr/bin/rusuploadfile 34567.xml
   

3. With generation and upload fully configured and confirmed working it is necessary to customise the rusbulkupload script to your particular site. The provided one should work fine for PBSPro and Torque. It records the last processed PBS accounting file and then when ran processes all PBS accounting files from then up until yesterday. It is run daily from a cron job that looks like this:

    BATCH2UR=/etc/batch2ur/batch2ur.conf
    RUS=/etc/rus/rus.conf
    export BATCH2UR RUS
    /usr/bin/rusbulkupload
   

   [note:1] It is important that both environment variables are set as rusbulkupload used both perl libraries.