Monday, December 24, 2012

Message Reliability in OSB

Message reliability is one of the features which customer look for. OSB too provides options to preserve the message. Consider a typical scenario where in proxy service reads a message from a source queue and send that message to a target queue. Once proxy service picks up the message it will try to post the message to the target queue. If for any reason the target queue is not accessible or any failure happens during posting there is a chance to loose the original message if proper steps not taken.

To ensure message reliability OSB provides transactional support while configuring  the proxy service.

As the first step create a source queue(TestQueue) and a corresponding error queue(TestErrorQueue) for the source queue


Set the delivery failure option for the source queue - TestQueue

Create a proxy service to consume the message from the TestQueue and enable the transaction parameters as follows

Create a business service to post the message to target queue( To generate error I have configured  a fictitious server and queue)

Create a message flow to route the message to business service


Now post a message to the TestQueue. Set the Redelivery limit to 2



Now on the OSB console we can see message is getting retried 2 times








Each time when the business process try to post the message to the target queue on the remote server, it will fail and the transaction will roll back the message back to the source queue.


After the retry limit is reached the message will be removed from the TestQueue and will move to the TestErrorQueue





We can see the message content in the ErrorQueue. So in any case the original message is not lost . It is preserved for further action which can be automated or manual






Friday, December 21, 2012

JMS Message Retry in BPEL

Reliability is one of the key features in integration scenario. Customer never wants to hear that application failed to process the message and it is even worst when application lost the message.

 Consider a typical scenario where client posts a message to queue and BPEL is consuming the message from the queue. If any failure happens in the BPEL the message should be rolled back to the original JMS Queue and should retry after some delay. If the retry limit is reached then the message should be moved to the ErrorQueue. Later the failed messages can be retrieved from the error queue and can be reprocessed. Ultimately this helps to preserve the customer data and shows the applications reliability.

Lets create a queue, error queue and connection factory as shown below


 For the MyDistributedQueue configure the error queue as follows


 Goto Deployments->jmsAdapter->configuration->OutboundConnectionPool

 Create a new connection pool  and specifying the connection factory which we created in the first step


Set the acknowledgement mode as client_ack

 

Create a BPEL process with a JMSAdapter for consuming the message from the queue




Also its important to set the oneWayDeliveryPolicy to sync in the composite.xml


Now in the BPEL process add an throw activity with rollback exception/system exception





Deploy the process and post a message in the MyDistributedQueue

Set the ReDeliveryLimit to 5 while posting the message






Now watch the emconsole , we can see 6 instances of the process getting created as the redelivery limit value was set as 5



After 5 retries the message will be redirected to the ErrorQueue.

Now if we watch the MyDistributeQueue, the message wont be available there.


Now check the error queue, we can see the message over there


Select the queue and go to show message. we can see the original message


Thursday, December 20, 2012

Sending message to Distributed Queue from OSB in a clustered environment

Sending a message from a OSB business service to JMS queue is very straight forward. This will work fine in any regular environment. But the code is moved to production which is clustered , then there are few points to consider.

The normal approach to set the EndPoint URI in OSB business service is using the below format
jms://hostname:port/connectionfactoryJNDI/resourceJNDI.

But in a clustered environment there will be multiple nodes. so to which node OSB will send the message?. To which node the queue is associated? What will happen if that node is down?

To address such challenges weblogic provides the Distributed Destinations
A distributed destination is a set of destinations (queues or topics) that are accessible as a single, logical destination to a client

Applications that use distributed destinations are more highly available than applications that use simple destinations because WebLogic JMS provides load balancing and failover for member destinations of a distributed destination within a cluster. Once properly configured, your producers and consumers are able to send and receive messages through the distributed destination. WebLogic JMS then balances the messaging load across all available members of the distributed destination. When one member becomes unavailable due a server failure, traffic is then redirected toward other available destination members in the set

Creating a distributed queue is the same as normal queue and it should target on all nodes of the server

If we monitor the distributed queue it will  look as shown below.


so now a distributed destination is set, how do we send the message to this distributed destination.

From OSB business service for the end point URI, provide all the nodes in the cluster as coma separated values


This will ensure it will post the message to the first node and if that failed it will try to post to the second node and so on. Complete fail over can be achieved using this approach in a clustered environment

TF_GENERAL_TRANSLATION_FAILURE. Please correct the NXSD schema.

I had business requirement to move the FTP polling job to OSB instead of using the FTP Adapter directly from BPEL. Earlier implementation was FTPAdapter will get the DAT file from remote location, translate the file content based on the native schema defined and then initiate the RequesterABCS component.

As part of the new requirement I used OSB FTP Protocol approach to get the file from ftp server and then from OSB the file content to a JMS Queue.

Then a JMSAdapter will consume the message from the queue and it will initiate the same requester component.

The new approach was working fine for some payloads. But for some payloads it was throwing "TF_GENERAL_TRANSLATION_FAILURE,Please correct the NXSD schema". The same payload were working fine with the initial approach.

I compared the DAT file content and the message posted in the queue by OSB. The message size in the queue were slightly higher than the original DAT file. This lead me to think that OSB file transfer might be adding some characters like carriage return or line feed, which might be the cause of this increase size of the message.

In the OSB proxy service, the transfer mode was ASCII, I changed this to Binary. Retested, the original file size and the message size matched . Also the JMSAdapter started processing the messages without any errors.

Thursday, December 13, 2012

xsl transformation remains in the pending state

Sometimes in BPEL the xsl transformation remain in the "pending" state and logs show errors like "javax.xml.transform.TransformerException: XML-22900: (Fatal Error) An internal error condition occurred".

Its often difficult to debug the problem. In such situations verify and ensure the name space associated with each element in the xsl mapping is declared in the beginning of the xsl file.

If a namespace is used without declaring it, the transformation will be shown as in pending state in the BPEL

Wednesday, December 5, 2012

OSB Scripted modification of customization file

While migrating OSB artifacts from one environment to another, we use customization file to apply changes specific to the environment. But to changing the customization file for each environment still demands a manual work.

This manual change is error prone and might create unexpected results.

Here is an alternative for that. Instead of changing the customization file manually, the following ANT script will do the job by reading the target environment specific values from a property file.

Step 1:  Create a base folder called automation and create a sub folder called source under it

Step 2 :  In the customization file identify the properties that might change from environment to envrironment and tokenize them in braces { }

For eg. the below excerpts from the customization file , in which a proxy service picks a file from FTP server and a business service will post that file content to a JMS queue

Proxy Service
=============

   <cus:envValueAssignments>
      <xt:envValueType>Service URI</xt:envValueType>
      <xt:location xsi:nil="true"/>
      <xt:owner>
        <xt:type>ProxyService</xt:type>
        <xt:path>InitiateOrderProcessing/ProxyServices/OrderProcessingInitiateFTP_PS</xt:path>
      </xt:owner>
      <xt:value xsi:type="xs:string" xmlns:xs="http://www.w3.org/2001/XMLSchema">ftp://{FTPHOST}:{FTPPORT}/import</xt:value>
    </cus:envValueAssignments>

Business Service
================


 <tran:tableElement xmlns:tran="http://www.bea.com/wli/sb/transports">
          <tran:URI>jms://{JMSHOST}:{JMSPORT}/weblogic.jms.XAConnectionFactory/Jms.nOrderProcessing</tran:URI>
          <tran:weight>1</tran:weight>
        </tran:tableElement>



Step 3 : Save the customization file as .seed file and keep that in the source folder

Step 4: Create target specific properties file and save it in the automation folder

==========build.dev.properties===================

JMSHOST=localhost
JMSPORT=8001
FTPHOST=testftp
FTPPORT=24
ARCHIVE_DIR=/archive/test
ERROR_DIR=/error/test
DOWNLOAD_DIR=/download/test
=============end of build.dev.properties==============

==============build.qa.properties===================

JMSHOST=myhost
JMSPORT=8011
FTPHOST=mytftp
FTPPORT=21
ARCHIVE_DIR=/archive/testmy
ERROR_DIR=/error/testmy
DOWNLOAD_DIR=/download/testmy
=============end of build.qa.properties================

Step 5 : Create and store the following build.xml file in the base directory(automation)
================build.xml============================

<project name="SOA_OSB_Deployment" basedir="." default="init" xmlns:ac="antlib:net.sf.antcontrib">
    <input message="Enter the environment to deploy:" addproperty="env"/>
    <property file="${basedir}/build.${env}.properties"/>
<property name="sa.source.dir" value="${basedir}/source"/>
<property name="sa.target.dir" value="${basedir}/target"/>
 <taskdef resource="net/sf/antcontrib/antlib.xml">
  <classpath>
    <pathelement location="/automation/ant-contrib-1.0b3.jar"/>
  </classpath>
</taskdef>
<target name="init">
    <delete dir="${sa.target.dir}" verbose="${OSBVerbose}" failonerror="false" includeemptydirs="true"/>
    <mkdir dir="${sa.target.dir}"/>
<antcall target="updateCustomizationFile" inheritAll="Yes"/>
 </target>
<target name="replaceTokens">
    <!-- replace ${} tokens into <param1>.new file -->
    <echo message="Replace Tokens in ${param1}"/>
<echo message="properties file in ${param2}"/>
    <copy file="${param1}" tofile="${param1}.new" overwrite="true">
      <filterchain>
        <replaceregex pattern="\$\{" replace="{"/>
        <filterreader classname="org.apache.tools.ant.filters.ReplaceTokens">
          <param type="propertiesfile" value="${param2}"/>
          <param type="tokenchar" name="begintoken" value="{"/>
          <param type="tokenchar" name="endtoken" value="}"/>
        </filterreader>
      </filterchain>
    </copy>
  </target>
    <target name="updateCustomizationFile" description="Updates the property values for the target environment">
    <tstamp prefix="Start updateCustomizationFile"/>
    <echo message="Parse service accounts"/>
 <for param="seed.file">
<path>
<fileset dir="${sa.source.dir}" includes="**/*.seed"/>
      </path>
      <sequential>
        <echo message="Parse service accounts file: @{seed.file}"/>
         <antcall target="replaceTokens" inheritAll="No">
          <param name="param1" value="@{seed.file}"/>
          <param name="param2" value="build.${env}.properties"/>
        </antcall>
        <copy file="@{seed.file}.new" tofile="customization.xml"/>
<copy file="customization.xml" todir="${sa.target.dir}"/>
<delete file="@{seed.file}.new" quiet="true"/>
           
     </sequential>
    </for>
    <tstamp prefix="finished updateCustomizationFile"/>
  </target>
  </project>
================ end of build.xml=========================




Step 6 : open command prompt and go to the base folder automation . Execute the run command
automation> run

It will prompt for you to give the target environment name.

Step7 :Based on the environment name given it will read the property from the respective properties file and update the customization file.

Result : The modified customization file will be stored in the target folder. Now just apply this customization file from sbconsole or using script as mentioned in the previous listing




Monday, December 3, 2012

Updating Service Account Values in OSB using script

OSB customization file does not support the update of Service Account values.

If we move from one environment to another there is a possibility of the username and password change.

For e.g during QA testing the FTP server may be a sample server and while moving the code to PROD we might to target to a another FTP server which has totally different set of credentials.

Its a hard job to do this change via sb-console after the deployment. Instead of that before the deployment we can change these values for the target environment

Step1 : In the Service Account configuration , instead of hard coding the username and password , provide some tokens as below
================ FTPUser.ServiceAccount ===================

<?xml version="1.0" encoding="UTF-8"?>
<ser:service-account xsi:type="ser:StaticServiceAccount" xmlns:ser="http://www.bea.com/wli/sb/services" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:con="http://www.bea.com/wli/sb/resources/config">
    <ser:description/>
    <ser:static-account>
        <con:username>{ServiceAccount.FTPUsername.value}</con:username>
        <con:password>{ServiceAccount.FTPPassword.value}</con:password>
    </ser:static-account>
</ser:service-account>


================ end of FTPUser.ServiceAccount====================

Step 2 : Create a build.{env}.properties file as follows

Note {env} could be the any target environment(dev/qa/prod)

=============== build.dev.properties=================


ServiceAccount.FTPUsername.value=developer
ServiceAccount.FTPPassword.value=exotic


==========end of  build.dev.properties===========


Step 3 : Create the build.xml as follows

================build.xml========================


<project name="SOA_OSB_Deployment" basedir="." default="updateCredentials" xmlns:ac="antlib:net.sf.antcontrib">
 
 <taskdef resource="net/sf/antcontrib/antlib.xml">
  <classpath>
    <pathelement location="/automation/ant-contrib-1.0b3.jar"/>
  </classpath>
</taskdef>


<target name="replaceTokens">
    <!-- replace ${} tokens into <param1>.new file -->
    <echo message="Replace Tokens in ${param1}"/>
<echo message="properties file in ${param2}"/>
    <copy file="${param1}" tofile="${param1}.new" overwrite="true">
      <filterchain>
        <replaceregex pattern="\$\{" replace="{"/>
        <filterreader classname="org.apache.tools.ant.filters.ReplaceTokens">
          <param type="propertiesfile" value="${param2}"/>
          <param type="tokenchar" name="begintoken" value="{"/>
          <param type="tokenchar" name="endtoken" value="}"/>
        </filterreader>
      </filterchain>
    </copy>
  </target>

    <target name="updateCredentials" description="Replace ServiceAccounts with correct passwords">
    <tstamp prefix="Start updateCredentials"/>


     <input message="Please enter the environment to deploy:" addproperty="env"/>
     <property file="${basedir}/build.${env}.properties"/>
    <property name="sa.source.dir" value="${basedir}/source"/>
<property name="sa.target.dir" value="${basedir}/target"/>
    <delete dir="${sa.target.dir}" verbose="${OSBVerbose}" failonerror="false" includeemptydirs="true"/>
    <mkdir dir="${sa.target.dir}"/>
    <unzip src="${sa.source.dir}/sbconfig.jar" dest="${sa.target.dir}"/>

    <echo message="Parse service accounts"/>
 <for param="sa.file">
<path>
<fileset dir="${sa.target.dir}" includes="**/*.ServiceAccount"/>
      </path>
      <sequential>
        <echo message="Parse service accounts file: @{sa.file}"/>
         <antcall target="replaceTokens" inheritAll="No">
          <param name="param1" value="@{sa.file}"/>
          <param name="param2" value="build.${env}.properties"/>
        </antcall>
        <copy file="@{sa.file}.new" tofile="@{sa.file}"/>
        <delete file="@{sa.file}.new" quiet="true"/>
   
        <zip destfile="target/sbconfig.jar" basedir="${sa.target.dir}" update="false"/>
     </sequential>
    </for>
    <tstamp prefix="finished updateCredentials"/>
  </target>
  </project>

================end of build.xml===================

Step 4 : Create a base folder called automation in the file system
            Create sub folders source and target under automation folder

Step 5:  Copy the build.dev.properties and build.xml files under automation directory

Note: ensure that you have ant-contrib-1.0b3.jar in the path mentioned in the build.xml
 <pathelement location="/automation/ant-contrib-1.0b3.jar"/>

Step 6 : copy the to be updated sbconfig.jar to the source folder

Step 7 : open command prompt and go to the automation folder and run ant
 automation > run
on prompt give the target environment  dev/qa/prod.

The updated sbconfig.jar will now appear in the target folder.

To verify the results unzip the sbconfig.jar and verify the Service Account values.

Now this new sbconfig.jar can be deployed to the target environment

For task in ANT

We might require to use for task in the ANT scripts.

By default for task in not available with basic ANT installation

Using the ant-contrib jar file we can define "for" task in build script.

Copy the ant-contrib-version.jar to the ANT_HOME lib folder and in the build script add the following section

  <project name="SOA_OSB_Deployment" basedir="." default="init">
     <taskdef resource="net/sf/antcontrib/antlib.xml"/>
     <target name="init">
            <for list="1,2,3,4,5" param="letter">
<sequential>
<echo>Letter @{letter}</echo>
</sequential>
</for>
      </target>
  </project>

Note : we should use net/sf/antcontrib/antlib.xml  not  net/sf/antcontrib/antcontrib.properties

Also the ant-contrib-version.jar can be kept any location, but we need to explicitly tell ant script where to locate the file


<taskdef resource="net/sf/antcontrib/antlib.xml">
  <classpath>
    <pathelement location="/usr/share/ant/lib/ant-contrib-version.jar"/>
  </classpath>
</taskdef>