Data Normalization Framework 2.0 Install By Download

From SHARP Project Wiki
Jump to navigationJump to search

THIS PAGE IS UNDER CONSTRUCTION

Overview

The Data Normalization pipeline can be invoked on its own, that is without Mirth Connect. However, this presumes that you have the appropriate input data and that you know how to invoke UIMA collection processing. While Mirth Connect is technically optional you will find it indispensable for feeding the pipeline and for transforming the resulting output.

The pipeline install and UIMA-AS represents step 3 in the overview. Everything within the Data Normalization box, from the Initializer to Generate CEM. Mirth Connect and CouchDB represents steps 1, 2, 4, and 5 as seen in the overview. We have chosen Mirth Connect as the application to move original data to the right place and in the right format for the Data Normalization pipeline. Mirth Connect serves as a hub. Incoming data is recognized, potentially transformed, and sent to a processing location. Channels are set up to serve this purpose. Mirth Connect software also serves to take CEM objects off the end of the Data Normalization pipeline and place them in to various other formats, for example, a DBMS.

(Optional) Getting data to arrive at the system where the Data Normalization pipeline can process it is left to the user. If you require a Gateway to Gateway exchange from one location to another, NwHIN could be used. This was tested successfully using a non-secure gateway by the SHARPn group. If you choose NwHIN Connect, then it must be setup prior to the installation and configuration of Mirth Connect given that you are using the standard SHARP deployment, configuration artifacts, and installation instructions.

Data Normalization pipeline

Prerequisites for Data Normalization

  1. Some software developer skills
  2. Minimal familiarity with Subversion code repositories
  3. A way to checkout files from Subversion
  4. Oracle's Sun Java 1.6 or greater runtime. Mirth Connect will be used and this version of Java is required by Mirth Connect.

Install Data Normalization

Command line install:

  • Install an SVN client.
  • Checkout the Data Normalization code from the repository location to a directory on your local machine, for example /usr/local/sharpdn.
  • :svn co http://svn.code.sf.net/p/sharpn/datan/code/trunk/SharpnNormalization_XMLTree /usr/local/sharpdn <-- Location to be validated once we move code from internal to external SVN
    svn co http://sourceforge.net/p/sharpn/datan/code/HEAD/tree/trunk/SharpnNormalization_XMLTree2 /usr/local/sharpdn <-- Location to be validated once we move code from internal to external SVN
  • Set environment variable SHARPN_FRAMEWORK_HOME, for example
    set SHARPN_FRAMEWORK_HOME=/usr/local/sharpdn
  • Set environment variable SHARPN_LOG_DIR, for example
    set SHARPN_LOG_DIR=/usr/local/sharpdn/logs

Eclipse IDE install:

  • Start Eclipse
  • (Optional) Make a new workspace
  • File -> New -> Project -> Checkout projects from SVN

*Create a new repository location: svn://svn.code.sf.net/p/sharpn/datan/code/trunk/SharpnNormalization_XMLTree <-- Location to be validated once we move code from internal to external SVN

  • Create a new repository location: http://sourceforge.net/p/sharpn/datan/code/HEAD/tree/trunk/SharpnNormalization_XMLTree2 <-- Location to be validated once we move code from internal to external SVN
  • If not the default, change the project JRE to 1.6
  • Set environment variable SHARPN_FRAMEWORK_HOME, for example
    set SHARPN_FRAMEWORK_HOME=<Eclipse root location for this project where SharpnNormalization_XMLTree is now located>
  • Set environment variable SHARPN_LOG_DIR, for example
    set SHARPN_LOG_DIR=/usr/local/sharpdn/logs

Test sample data with default pipeline configuration

The UIMA Collection Processing Engine (CPE) is used to run the pipeline. The CPE is configured by a file called a descriptor. This descriptor defines things like the location of input and output data, the mechanism by which to process the collection of input, and what documents are being passed in and processed. There are more. It is left to the reader to study the descriptor in the GUI or in its XML format.

  1. Start the CPE:
    • Command line: Change to the SHARPN_FRAMEWORK_HOME directory and execute runCPE.sh|bat
    • In Eclipse: Run -> Run Configurations. You will see a Java Application configured: Launch UIMA_CPE_GUI--sharpnnormalizationXMLTree
  2. Load the descriptor
    • File -> Open CPE Descriptor
    • SharpNormalization_XMLTree -> desc -> collectionprocessingengine -> MayoLabsHL7CPE.xml
      • The descriptor file names and directory names contain a designation of what kind of data being dealt with. This is a good practice to use as it may not be easy to discern just by looking at the input file names. In this example we can tell these should be HL7 messages.
  3. Input data
    • Notice the incoming data location (Input Directory in the Collection Reader section)
    • Use any editor to view the files in the input directory.
      • As you might expect, these HL7 messages have already been converted to XML files since that is the only thing the pipeline can process. A step you will need to take using Mirth Connect if you have this kind of input.
  4. Press the Play button (looks like a green arrow near the bottom of the interface).
  5. Output data
    • Check out the data that is now in the output directory (Output Directory in the Analysis Engines section); each file is a CEM in XML format. The results are not displayed in the GUI. The GUI is simply a means to run the pipeline. Use any editor to view the files by navigating through the system directories.

The output results for labs have a special naming convention. You will notice that lab results output file names do not have the HL7 lab categories: coded, narrative, ordinal, quantitative, quantitativeInterval.

The pipeline has an Analysis Engine defined with a set of special parameters. You can see these parameters in the middle of the Collection Processing Engine Configurator GUI launched previously. When we talk about configuring the mapping files, the MODEL_MAPPING and SEMANTIC_MAPPING files are the ones you will be changing. When you create your own pipeline, you will use the sample configuration files as the basis for your configuration. Once you have created your own configuration files, they would be pointed to from here and saved to your own CPE descriptor. After setting up Mirth Connect, you will be creating this configuration.

  • MODEL_MAPPING - Path and name of a configuration file mapping out how to transform from one model to another.
  • SEMANTIC_MAPPING - Path and name of a configuration file mapping out how to transform terms used in the data.
  • Element property - A static property file. Use unchanged. A mechanism to move attributes from the source to the target.
  • LOINC2CEM MAPPING - A static property file. Use unchanged for lab processing. This file is used to tell the pipeline which of the six CEM lab types to use based on the LOINC codes that end up being involved.
  • DOCTYPE - This is the type of the document or data being sent into the pipeline. There are these types of documents: HL7, CCD, CDA, TABLE.
  • DEID - Reserved for future use.

Mirth Connect

Prerequisites for Mirth Connect

  1. Installing Mirth Connect requires an existing pipeline (steps 3 in the overview).
  2. Oracle's Sun Java 1.6 or greater runtime. Mirth Connect FAQ.
  3. Root level authority on your system.

These instructions require the following artifacts: NEED TO MOVE THESE FILES TO SVN RATHER THAN THE SHARP WEB SITE

Install Mirth Connect

  1. In order to provide a cohesive install for Data Normalization a specific folder structure is required.
    1. Create the following directories:
      • /SHARPnDataNormalization
    2. Change permissions on /SHARPnDataNormalization
      • chmod 774 /SHARPnDataNormalization
  2. One of the Mirth Connect Channels which is always installed requires its own set of directories
    1. Create the following directories:
      • /SHARPnDataNormalization/hl7_msg
      • /SHARPnDataNormalization/hl7_xml
      • /SHARPnDataNormalization/hl7_pipe_processed
      • /SHARPnDataNormalization/hl7_error_processed
    2. Change permissions
      • chmod 774 /SHARPnDataNormalization/hl7_msg
      • chmod 774 /SHARPnDataNormalization/hl7_xml
      • chmod 774 /SHARPnDataNormalization/hl7_pipe_processed
      • chmod 774 /SHARPnDataNormalization/hl7_error_processed
  3. Install Mirth Connect 2.1.0 or greater
    1. Follow any guidelines from Mirth Connect Downloads for install
    2. We find that you either use a GUI installer download or simply unzip a file to install.
      • For example: tar -xzf mirthconnect-2.1.0.5374.b644-unix.tar.gz -C /SHARPnDataNormalization
      • This would make your MIRTH_HOME be /SHARPnDataNormalization/Mirth\ Connect
    3. (Optional) Install the Mirth Connect Command Line Interface

Configure Mirth Connect

  1. Log-in as the root user (or Administrator if Windows).
  2. Ports must be accessible through any firewall on the system. The default ports for Mirth Connect are listed here. Firewall settings may need to be made now. If these are in use then you will need to change that here and in the Mirth Connect properties (subsequent step).
    • 8080
    • 8443
    • 1099
  3. (Optional) Change Mirth Connect to use port 7080 if your system already uses port 8080 (the default).
    1. edit /SHARPnDataNormalization/Mirth\ Connect/conf/mirth.properties (Note the space in the name "Mirth Connect" where the "\" escapes the space character.)
    2. change field http.port to be 7080
  4. Install JAXWS libraries so that MirthConnect can use them.
    1. URL to retrieve the JAXWS libraries: http://jax-ws.java.net/ - Version 2.2.1
      1. This file has already been included in the artifacts above.
      2. Unzip JAXWS2.2.1-20100617.zip and copy all of its contents to /SHARPnDataNormalization/Mirth\ Connect/custom-lib
  5. Start MirthConnect
    1. Navigate to /SHARPnDataNormalization/Mirth\ Connect
    2. Run mcserver
      1. sudo ./mcserver
    3. Note: This will run MirthConnect.
  6. Run MirthConnect Administrator
    1. Run FireFox on the host server where Data Normalization is installed not from a remote machine.
    2. Go to URL: http://localhost:8080 Modify if you changed the default ports.
    3. Click on "Launch Mirth Connect Administrator"
      1. Select "Open With", browse to <JAVA_HOME>/bin/javaws, and select Open.
      2. Select Do this automatically for files like this and click OK.
      3. Select Run to launch Mirth Connect Administrator.
        1. NOTE: On a Linux Ubuntu system the package libxtst6 was also required to be installed.
    4. Login
      1. User: admin
      2. Password: admin
      3. On the first time it may give you a Welcome to Mirth Connect screen. If so:
        1. Keep the user name and password the same.
        2. Fill in required fields (optional fields if desired).
        3. Uncheck the Register and Submit usage statistics checkboxes.
        4. Select Finish.
  7. Import the HL7 text message (pipe delimited) to XML converter: (HL7toXML.xml)
    1. From the MirthConnectAdministrator, click on "Channels".
    2. Click on the "Import Channel" link under "Channel Tasks".
    3. Navigate to where you placed the artifacts, perhaps /SHARPnDataNormalization.
    4. Select the "HL7toXML.xml" file, followed by "Open".
    5. You will be in edit mode. Click on the "Save Changes" link under "Channel Tasks".
    6. You should now see a channel called "HL7 to XML" in the list of Channels within the MirthConnect Administrator.
  8. Deploy the HL7 to XML channel
    1. Click on the "Channels" link to show the channels.
    2. Right-click on the "HL7 to XML" and select "Deploy Channel".
    3. You will be switched to the Dashboard where you can see the results of any processing by the channels.

CouchDB

Prerequisites for CouchDB

These instructions require the following artifacts: NEED TO MOVE THESE FILES TO SVN RATHER THAN THE SHARP WEB SITE

Install CouchDB

Note that another DBMS could be used but you would need to write Mirth Connect channels that would take the CEMs and put them into that DBMS.

  1. Install CouchDB 1.0.1 or greater
    • See the CouchDB documentation for help downloading and installing.
    • CouchDB comes with certain versions of Linux already. Check which one you might already have.

Configure CouchDB

  1. Ports must be accessible through any firewall on the system. The default ports for CouchDB are listed here. Firewall settings may need to be made now. This would only be if you want to administer this DB from a remote system. The Data Normalization pipeline does not need this open as it runs together on the local system.
    • 5984
  2. Add the following directories to Mirth Connect home:
    • /SHARPnDataNormalization/secUseCemInputMessage
    • /SHARPnDataNormalization/secUseCemProcessedMessage
    • /SHARPnDataNormalization/secUseCemErrorMessage
  3. Change permissions on these directories
    • chmod 774 /SHARPnDataNormalization/secUseCemInputMessage
    • chmod 774 /SHARPnDataNormalization/secUseCemProcessedMessage
    • chmod 774 /SHARPnDataNormalization/secUseCemErrorMessage
  4. Edit the CemSecUseCemToCouchDB.properties file
    1. If your CouchDB is installed on a remote server
      • Change the property SECUSECEM_COUCHDB_IP so it contains the IP address of the other server.
  5. Add the Channel Configuration(s) for converting CEM results to another form. In order for Mirth Connect to work with CouchDB a channel and properties file are provided.
    1. Run the Mirth Connect Administrator, as when you installed Mirth, and log-in as admin.
    2. Import the CEM to Database channel
      1. From the MirthConnectAdministrator, click on "Channels".
      2. Click on the "Import Channel" link under "Channel Tasks".
      3. Navigate to where you placed the artifacts, perhaps /SHARPnDataNormalization.
      4. Select CemSecUseCemToCouchDBV2.xml, followed by "Open".
      5. You will be in edit mode. Click on the "Save Changes" link under "Channel Tasks".
      6. You should now see all the CEM to Database channels within the MirthConnect Administrator.

UIMA-AS

Prerequisites for UIMA-AS

These instructions require the following artifacts: NEED TO MOVE THESE FILES TO SVN RATHER THAN THE SHARP WEB SITE

Install UIMA-AS

Unstructured Information Management Architecture - Asynchronous Scaleout is used as a required component of the Data Normalization pipeline. Mirth Connect will be configured to include a plug-in and connector that takes advantage of UIMA-AS. UIMA-AS includes UIMA and the Apache ActiveMQ implementation of JMS.

  1. Install UIMA-AS 2.4.0 or greater
    1. Follow any guidelines from Apache UIMA
    2. You can also use the UIMA README for help
    3. Make sure you select the files for UIMA-AS not UIMA

Configure UIMA-AS

  1. The UIMA instructions are not explicit about what environment variables need to be set
    1. Set JAVA_HOME to the directory of your JRE installation you would like to use for UIMA.
    2. Set UIMA_HOME to the apache-uima directory of your unpacked Apache UIMA distribution
      • For example, export UIMA_HOME=/usr/local/uima-as/apache-uima-as-2.4.0
    3. Append UIMA_HOME/bin to your PATH
      • For example, export PATH=$PATH:/usr/local/uima-as/apache-uima-as-2.4.0/bin
  2. Copy the UIMA JAR files to the Mirth Connect lib for its use as a Java library
    1. From UIMA_HOME/lib to MIRTH_HOME/lib
      • uimaj-as-core.jar
      • uimaj-as-jms.jar
      • uima-core.jar
      • uimaj-as-activemq.jar
  3. Copy the Active MQ JAR files to the Mirth Connect lib for its use as a Java library
    1. From UIMA_HOME/apache-activemq-5.4.1/lib to MIRTH_HOME/lib
      • activemq-core-5.4.2.jar
  4. Starting the ActiveMQ Broker
    1. Change to the UIMA_HOME/bin directory
    2. Start the Active MQ broker, for example:
      • sudo ./startBroker.sh
  5. Add UIMA capability to Mirth Connect (plugin and connector)
    1. (Optional) Examples of UIMA and Mirth Integration
    2. Steps are base on these Mirth UIMA install instructions.
    3. Install the UIMA Plugin Mirth Connect extension
      • Click on the "Extensions" link to show the extensions.
      • Click the Browse... button
      • Navigate to where you placed the artifacts, perhaps /SHARPnDataNormalization.
      • Select the "Uimaservice-2.1.0.b662.zip" file, followed by "Open" and "Install".
      • Restart the Mirth Connect server which also means the Mirth Connect Administrator.
    4. Install the UIMA Sender Mirth Connect Connector
      • Click on the "Extensions" link to show the extensions.
      • Navigate to where you placed the artifacts, perhaps /SHARPnDataNormalization.
      • Select the "uima-2.1.0.b662_modified.zip" file, followed by "Open" and "Install".
      • Restart the Mirth Connect server which also means the Mirth Connect Administrator must be stopped and restarted.
    5. After restart, in the Mirth Connect Administrator click Extensions
      • The connector should show as "UIMA Sender"
      • The plug-in should show as "UIMA Plugin"

Starting Services

Everything is installed now. Before you go to specific configurations for your enterprise, you'll want to test out the pipeline. In order to do this you will need to start the services.

  1. Make sure Mirth Connect is started as described in its section
  2. Deploy the UIMA-AS data normalization annotators. Run this as root or administrator.
    • SHARPN_FRAMEWORK_HOME/deploy/script/deploy.sh

Testing

Prerequisites for Testing

These instructions require the following artifacts: NEED TO MOVE THESE FILES TO SVN RATHER THAN THE SHARP WEB SITE

Run Tests

  1. Use the file TestHL7Med1.txt to test the pipeline.
  2. The name of the file must end in ".txt". When the channel was configured, it was configured to look for files with that ending.
  3. Copy the file to: /SHARPnDataNormalization/HL7OutputMessage
    • cp /SHARPnDataNormalization/TestHL7Med1.txt /SHARPnDataNormalization/hl7_msg/
  4. Watch the Mirth Connect dashboard for a report of the file being processed.
  5. Look in the output directory for a new file representing the result
  6. Look into the database for the result

Next Steps

Configuration of this framework for your enterprise is the next step.

Data Normalization Framework 2.0 Configuration Files