Setting Up HermesLite

Check out the code

Check out the code from SourceForge, see the Link to source repository page for details.

Build OSUtil

Build the osutil library and install the jar into your maven repository

mvn package
mvn install 

Setup your database

Generate the database setup script from the osds code

mvn hibernate3:hbm2ddl

The database setup script will have been dumped into the target/OSDS-CreateDatabase.sql file. You can use this file to setup your PostgreSQL database.

Prepare the 'send' component

Your external server receives data from an internal component that retrieves and distributes the data to the external server.

Build the osds internal component

mvn assembly:assembly

You now find a file named osds-0.1-distribution.zip in the target directory.

Copy the zip file to your internal server and extract it. Run the InstallHermesLite-NT.bat script to install the HermesLite service on your internal machine.

Configure the database(s) that you want to export

The first step in distributing an internal database is to setup a configuration file that describes the data source. An example configuration is

# Task 
# This is the Java class task that is configured by the following properties.
# For most cases, using this example task is sufficient. It handles the case of package up
# data and sending it via HTTP/HTTPS to the external server.
task.class=au.com.gaiaresources.osds.task.remote.example.BaseRemoteTask

# Schedule
# The export can run on any CRON schedule, for example, at 9pm every Saturday
schedule.cron=0 0 21 ? * SAT
# Schedule Test 
# Set this property to Yes of you would like the task to invoke when the service starts, 
# useful for testing.
schedule.testonstartup=Yes

# Get
# Source database
get.db.server=localhost
# The database type is those supported by OSUtil, this can be 
# SQLServer, MySQL, PostgreSQL, Oracle. (OSUtil is easily extended to allow for 
# other vendors if you need).
get.db.type=SQLServer
get.db.username=data_base_user_name
get.db.password=data_base_password
get.db.port=data_base_port_number
get.db.name=DatabaseName
get.db.table.name=DatabaseTableName
get.db.table.primarykeycolumn=PRIMARY_KEY_COLUMN_OF_DatabaseTableName
# Optionally define a query to filter data, e.g. select * from TABLE where CHECK_COL = 1
#get.db.query=select * from DatabaseTableName where X = 10 and Y = 'Yes'

# Compress the package - indicates that the file should be compressed before sending.
package.file.compress=Yes

# Send
# Send to
send.targettask=your.target.class
# The http target is the server that you will be serving data from
# it can be HTTP or HTTPS as your preference / setup
send.http.target=https://your.external.server/Hermes/filereceiver
# If you are running HTTPS and you have a self signed certificate then set
# the acceptselfsigned property to Yes
send.http.acceptselfsigned=Yes
# If the connection from the internal server to the external server
# goes through a proxy the detail that here
send.http.proxy.server=proxy.server.address
send.http.proxy.port=8080

Create the receiving task

In the properties file we set the send.target.task to your.target.class. This needs to be replaced by a class that you write that will drop the values from the received data into the database, for example:

public class MyReceiveTask extends au.com.gaiaresources.osds.wam.task.receive.BaseReceivingTask {
    private Date receivedDateTime;
    
    /**
     * Default constructor.
     */
    public MyReceiveTask() {
        super();
        // Indicate that the file received will be compressed
        setFileIsCompressed(true);
        receivedDateTime = new Date();
    }
    
    /**
     * Process a row.
     * 
     * @param metaData The row meta data, example:<br/>
     * MetaData [name: Imported, cols: [0: [REGNO, 4], 1: [OLDNO, 12], 2: [FLDNO, 12], 3: [ACCDT, 12], 4: [PHYLUM, 12], 
     * 5: [SUBPHYLUM, 12], 6: [CLASS, 12], 7: [SUBCLASS, 12], 8: [SUPERORDER, 12], 9: [ORDER, 12], 10: [SUBORDER, 12], 
     * 11: [INFRAORDER, 12], 12: [COHORT, 12], 13: [SUPERFAMILY, 12], 14: [FAMILY, 12], 15: [SUBFAMILY, 12], 16: [TRIBE, 12], 
     * 17: [GENUS, 12], 18: [SUBGENUS, 12], 19: [SPECIES, 12], 20: [SUBSPECIES, 12], 21: [AUTHORITY, 12], 22: [DTMNDBY, 12], 
     * 23: [DTMNDDT, 12], 24: [RELIABLE, 12], 25: [COUNTRY, 12], 26: [STATE, 12], 27: [DISTANCE, 12], 28: [DIST_UNIT, 12], 
     * 29: [DISTANCEWITHUNIT, 12], 30: [DIRECTION, 12], 31: [NEAREST, 12], 32: [SITE, 12], 33: [LATITUDE, 12], 34: [LONGITUDE, 12], 
     * 35: [LATDEC, 12], 36: [LONGDEC, 12], 37: [RESOLTN, 12], 38: [ALTITUDE, 12], 39: [DTFR, 12], 40: [DTTO, 12], 41: [COLLTOR, 12], 
     * 42: [COLLMETH, 12], 43: [HABIT, 12], 44: [SEX, 12], 45: [MALE, 12], 46: [FEMALE, 12], 47: [JUVENILE, 12], 48: [SPECNUM, 12], 
     * 49: [AGE, 12], 50: [SOURCE, 12], 51: [SNAME, 12], 52: [SYEAR, 12], 53: [SPCMAS, 12], 54: [DISPOSE, 12], 55: [SPCMTYPE, 12], 
     * 56: [CATBY, 12], 57: [REMARKS, 12], 58: [LOAN_NO, 12], 59: [LOAN_TO, 12], 60: [DATENT, 12], 61: [DATMOD, 12], 62: [FLAG, 12], 
     * 63: [GDA_LAT, 4], 64: [GDA_LONG, 4], 65: [ZONE, 4], 66: [AMG_NORTH, 4], 67: [AMG_EAST, 4], 68: [DATCAT, 12], 69: [EGAZ, 12], 
     * 70: [CHECK, 12], ]] 
     * @param rowData The content of the row.
     */
    @Override
    protected void processRow(MetaData metaData, RowData rowData) {
        InboundRecord record = new InboundRecord();
        
        Object catalogueNumber = extractColumn("REGNO", metaData, rowData);
        String gui = "URN:LSID:my.organisation.org:" 
                              + "MyInstitutionCode".toLowerCase() + "MyCollectionCode".toLowerCase() 
                              + ":" + catalogueNumber.toString();
        
        record.addValue(DWCConcepts.GLOBAL_UNIQUE_IDENTIFER, gui);
        record.addValue(DWCConcepts.DATE_LAST_MODIFIED, receivedDateTime);
        record.addValue(DWCConcepts.BASIS_OF_RECORD, DWCValues.BASIS_OF_RECORD.PreservedSpecimen.name());
        record.addValue(DWCConcepts.CATALOGUE_NUMBER, catalogueNumber.toString());
        record.addValue(DWCConcepts.INSTITUTION_CODE, "MyInstitutionCode");
        record.addValue(DWCConcepts.COLLECTION_CODE, "MyCollectionCode");
        
        String genus = extractColumnAsString("GENUS", metaData, rowData);
        String species = extractColumnAsString("SPECIES", metaData, rowData);
        String subSpecies = extractColumnAsString("SUBSPECIES", metaData, rowData);
        String scientificName = StringUtils.buildDelimitedConcatenation(new String[] {genus, species, subSpecies}, " ", false);
        record.addValue(DWCConcepts.SCIENTIFIC_NAME, scientificName);
        record.addValue(DWCConcepts.GENUS, genus);
        record.addValue(DWCConcepts.SPECIFIC_EPITHET, species);
        if (StringUtils.notEmpty(subSpecies)) {
            record.addValue(DWCConcepts.INFRASPECIFIC_RANK, "subsp.");
            record.addValue(DWCConcepts.INFRASPECIFIC_EPITHET, subSpecies);
        }
        
        record.addValue(DWCConcepts.DECIMAL_LATITUDE, extractColumnAsDouble("LATDEC", metaData, rowData));
        record.addValue(DWCConcepts.DECIMAL_LONGITUDE, extractColumnAsDouble("LONGDEC", metaData, rowData));
        
        try {
            getDWCService().importRecord(record);
        } catch (DataValidationException dve) {
            getLogger().error("Row does not validate", dve);
        }
    }
}

Install the configuration file and recieve task

Create JAR file with your receive task in it. Put the JAR file in the lib directory of the osds folder on your internal server. Create a sibling folder to lib called conf and put your properties file into there. Name the file something_meaningful.properties.

Build the web application

Create a maven profile in your settings.xml to configure the web application, for example,

<settings>
    <profiles>
        <profile>
            <id>HermesExternalServer</id>
            <properties>
                <!-- The database user on your external server -->
                <osds.server.database.user>external_server_database_user</osds.server.database.user>
                <!-- The database password on your external server -->
                <osds.server.database.password>external_server_database_password</osds.server.database.password>
                <!-- The JDBC driver for your external database. -->
                <osds.server.database.jdbc.driver>org.postgresql.Driver</osds.server.database.jdbc.driver>
                <!-- The JDBC url for your external database -->
                <osds.server.database.jdbc.url>jdbc:postgresql://localhost:5432/database_name</osds.server.database.jdbc.url>
                <!-- The Hibernate dialect for database access, we are using spatial data so use the PostgisDialect -->
                <osds.server.hibernate.dialect>org.hibernatespatial.postgis.PostgisDialect</osds.server.hibernate.dialect>
                
                <!-- The following properties are only if you want to generate maps from your results -->
                <!-- Where map requests are received from -->
                <osds.mapserver.url>/Hermes/map</osds.mapserver.url>
                <!-- The MapServer map file file on the external server -->
                <osds.mapserver.mapfile>/usr/local/osds/maps/primary.map</osds.mapserver.mapfile>
                <!-- The names of the layers in the map file that are used as the base -->
                <osds.mapserver.baselayer>GlobalImage Cities</osds.mapserver.baselayer>
                <!-- The name of the layer in the map file that contains the record data -->
                <osds.mapserver.wmslayer>OSDS</osds.mapserver.wmslayer>
                <!-- The URL of the mapserv cgi executatable. -->
                <osds.mapserver.local.url>http://localhost/cgi-bin/mapserv</osds.mapserver.local.url>
            </properties>
        </profile>
    </profiles>
</settings>

Now you can build the web application

mvn -P HermesExternalServer war:war

You can now deploy the generated war file on your external app server within your servlet container of choice, for example, Tomcat.

Put your jar file with the receive task code into the WEB-INF lib folder of the web application.

Run the process!

Start the HermesLite service that you installed at the beginning of the process. If you have the schedule.testonstartup option set in any of your configuration files then the data will be extracted and send immediately.

Improve the process!

This is a looooooooong process to get set up. We'd like improve it. Any suggestions welcome.