How to Avoid the OC4J Servers Got Automatically Restarted

If you notice that your production OC4J servers that house SOA/BPEL are automatically restarting every now and then, don't panic. Here are the reasons why and how to avoid it.

First let's understand what restarts the OC4J servers

Usually if there are automatic restarts it will be the OPMN that does it. OPMN stands for Oracle Process Monitoring and Notification. There is an opmn daemon process running on all cluster nodes pinging the JVM and HTTPD processes for health checking and death detection. If the response from an OC4J or HTTP server exceeds a configurable timeout (ping timeout), OPMN may consider  the server dead and decide to restart it. More explanation of the OPMN Automatic Restart can be found here.

The OPMN Ping operation is configured through timeout, retry and interval. The default Ping Timeout is 20 second.

If the OPMN is checking the health of an OC4J JVM, then there is a reverse ping operation as well.

Second, let's see how to avoid it.

Sometimes, even if an OC4J server's response to the OPMN ping operation exceeds the ping timeout, it is not because it is dead, but only that it was under heavy load, or there is something wrong with the network.

Therefore two ways to avoid the OPMN restarting the OC4J in in such circumstance.

1. You want to find out if your server is overloaded. If it is, tune the performance of the server accordingly. There are plenty of postings and documentations about performance monitoring and tuning of SOA servers running in OC4Js.

2. You can configure the ping and reverse ping to change how the OPMN detects servers death. The configuration is in the opmn.xml on each node. My friend Chintan Shah's has beautifully summarize the details in his blog http://chintanblog.blogspot.com/2010/12/opmn-ping-timeouts.html

Where and How to Set JVM Parameters in Oracle SOA 11G

Let's say you experience some performance issues of your Oracle SOA 11G servers, and after some research you know that you need to tune your JVM heap size, and turn on some JVM debugging to monitor such as Garbage Collection behaviour. After some due diligence of reading WebLogic documents and Internet postings, you found the right flags to use, and the right values to set. It typically looks something like

-Xmx4096M -Xms4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M -Xloggc:/tmp/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails

But the problem now is: WHERE and HOW to set these JVM arguments and debug flags? Should I do it through <server-start/> via Console or config.xml? Should I do it at one of those scripts under DOMAIN_HOME/bin? Exactly which script and which variables?


And to go further, what should I do if I want to set JVM differently for different servers within the same domain?

Here is what you could do in a typical Oracle SOA 11G installation.

First, choose the right way to specify JVM arguments

There are two options as the places for you to specify the JVM arguments:

1. Via the "Server Start" tab at Administration Console , which is equivalent to the <server-start/> tag in config.xml
2. Via the startXXX.sh and setXXX.sh scripts under DOMAIN_HOME/bin/ of each server

The problem of Option 1 is that the JVM parameters you put in <server-start/> via either Console or directly via config.xml will take effect only if you start the server from NodeManager. If you start the server via scripts, e.g.,  startWebLogic.sh, the parameters in <server-start/> in config.xml will be ignore.

For Option 2, if you start servers via scripts under DOMAIN_HOME/bin, specifying JVM parameters will of course work. And as long as you set the StartScriptEnabled=true and StartScriptName=startWebLogic.sh, as required by the Enterprise Deployment Guide, starting servers via NodeManager will also call the same startWebLogic.sh scripts, and hence will pick up the JVM arguments you set there. So you are covered both ways.

Therefore, I would like to recommend Option 2 to specify the JVM arguments in the scripts under DOMAIN_HOME/bin/"

Second, choose the right script and right variables

But the problem is that there are many scripts under the DOMAIN_HOME/bin. And those scripts even reference the scripts outside of the bin/ directory. Meanwhile, there are a lot of variables that are related to JVM arguments: MEM_ARGS, USER_MEM_ARGS, JAVA_OPTIONS, JAVA_PROPERTIES ... to just name a few.

Among these scripts and variables, I recommend you define the JVM arguments via the USER_MEM_ARGS and JAVA_OPTIONS variables in setSOADomainEnv.sh.

The USER_MEM_ARGS and JAVA_OPTIONS variables

When you examine the contents of startWebLogic.sh/setDomainEnv.sh/setSOADomainEnv.sh, you can conclude that you should customize the JVM memory settings (i.e. max and min heap size, PermGen size, etc) via variables USER_MEM_ARGS, and specifying any other JVM arguments (i.e. debug flags, system properties, etc) via JAVA_OPTIONS:

# from setDomainEnv.sh

# IF USER_MEM_ARGS the environment variable is set, use it to override ALL MEM_ARGS values

# from setDomainEnv.sh:

# JAVA_OPTIONS    - Java command-line options for running the server. (These

#                   will be tagged on to the end of the JAVA_VM and

#                   MEM_ARGS)

The setSOADomainEnv.sh script

In a typical SOA installation, you find that startWebLogic.sh under DOMAIN_HOME/bin sources setDomainEnv.sh, and the setDomainEnv.sh source setSOADomainEnv.sh.

Further, if you go to the end of the setSOADOmainEnv.sh, you will notice the following:

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} 
echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"

These lines print the values of JVM memory settings to the *.out file. So it makes sense to insert the settings of the JVM heap size and debug flags right above them  Otherwise, the actual settings may be different from what the setSOADomainEnv.sh print to the *.out file and would cause confusions.

Therefore, it is obvious that the proper place to set the USER_MEM_ARGS and JAVA_OPTIONS is right above the afore mentioned code snippet in the setSOADomainEnv.sh

Here is an example of how to set the JVM heap size and debug flags at setSOADomainEnv.sh:

USER_MEM_ARGS="-Xmx4096M -Xms4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"

JAVA_OPTIONS="${JAVA_OPTIONS} -Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"

Third, to make it a bit more robust.

Now, what if I want to have different settings for different servers? For examples, I may want my managed servers to run with a bigger heap size than the AdminServer, while turning on the debug flags on all of servers:

In such case you can script your JVM settings based on the server names:

if [ "${SERVER_NAME}" = "WLS_SOA1" -o "${SERVER_NAME}" = "WLS_SOA2" ] ; then
    USER_MEM_ARGS="-Xms4096M -Xmx4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"
fi

JAVA_OPTIONS="${JAVA_OPTIONS} -Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"


echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"

If each server has its own DOMAIN_HOME/bin directory, then you need to make sure the same code snippet are copied to the same setSOADomainEnv.sh script for all servers.

Forth, make the settings future proof.

So far so good with the modification of the setSOADomainEnv.sh, until you found the following comments at the top of the file:

# WARNING: This file is created by the Configuration Wizard.
# Any changes to this script may be lost when adding extensions to this configuration.

That means the changes you put in may be gone. Actually I have experienced this first hand when you upgrade the servers from the a new minor release (such as a patch set).

So, to be safe, it is better to move the settings to an external script (e.g. setCustomEnv.sh), and source this external script from setSOADomainEnv.sh:

setSOADomainEnv.sh (please note the line starts with a dot ('.'):

. ${DOMAIN_HOME}/bin/setCustomEnv.sh

echo LD_LIBRARY_PATH=${LD_LIBRARY_PATH} echo "."
echo USER_MEM_ARGS=${USER_MEM_ARGS} 
echo "."
echo "*****************************************************"
echo "** End SOA specific environment setup"
echo "*****************************************************"

setCustomEnv.sh:

#!/bin/sh

echo "Settings from setCustomEnv.sh"
if [ "${SERVER_NAME}" = "WLS_SOA1" -o "${SERVER_NAME}" = "WLS_SOA2" ] ; then
    echo "Customizing USER_MEM_ARGS for SERVER_NAME ${SERVER_NAME}"
    USER_MEM_ARGS="-Xms4096M -Xmx4096M -Xmn3072M -XX:PermSize=1024M -XX:MaxPermSize=1024M"
fi

JAVA_DEBUG_FLAGS="-Xloggc:${DOMAIN_HOME}/servers/${SERVER_NAME}/logs/verbosegc.txt -XX:+PrintGCTimeStamps -XX:+PrintGCDetails"
JAVA_OPTIONS="${JAVA_OPTIONS} ${JAVA_DEBUG_FLAGS}"
echo "Setting JAVA_OPTIONS from setCustomEnv.sh: JAVA_OPTIONS ${JAVA_OPTIONS}"
echo "End settings from setCustomEnv.sh"

In the event when the setDomainEnv.sh was overwritten by product patching or upgrades, you can still very easily put back that single line of sourcing the external script.

In the case where each server has its own copy of DOMAIN_HOME/bin directory, you can put the setCustomEnv.sh at a shared file system. So you keep only one copy of setCustomEnv.sh where all the JVM settings are centrialized. This single copy of setCustomEnv.sh can be referenced by the individual setSOADomainEnv.sh for each server.

Now, you are good to go!

Configure Adapter Threads in Oracle SOA 10G

In a separate posting, I mentioned you can configure multiple threads of inbound (polling) adapters of Oracle SOA. However, the ways to configure multiple threads vary between adapters and product versions, and the information scatter across multiple documentations. Hence it is worth to consolidate them here.

This post is for Oracle SOA 10.1.3.x. I have another blog post for configuring adapter threads in Oracle SOA 11G.


1. JMS Adapter

• For BPEL: Set 'adapter.jms.receive.threads' as activation agent properties in bpel.xml
  

<activationAgents>
<activationAgent className=“…" partnerLink="MsgQueuePL">
...     <property name="adapter.jms.receive.threads”>5</property>
</activationAgent>
</activationAgents>

• For OESB 10.1.3.3 onward: Set as endpoint property via JDeveloper (10.1.3.3) or directly at *.esbsvc file

2. AQ Adapter

This blog summarizes the different kind of settings for AQ Adapter threads very well. Let me copy and paste it below:

#### FOR ESB/BPEL 10.1.3.4.x
<property name="adapter.aq.dequeue.threads">NO_OF_THREADS</property>
#### FOR BPEL 10.1.3.3.x
<property name=activationInstances">NO_OF_THREADS<property>
#### FOR ESB 10.1.3.3.x
<endpointProperties>
   <property name="numberOfAqMessageListeners" value="NO_OF_THREADS"/>
</endpointProperties>

3. MQ Adapter

Oracle Support Tech Note How to limit number of threads for reading messages from a queue (Doc ID 1144847.1] describes the details of how to setup the threads for Inbound MQAdapter . Copying here:

There exist a parameter called "InboundThreadCount" which is valid for both 11g and also it is tested on SOA 10.1.3.5, And Confirmed to be working on 10.1.3.4.

To set the parameter, Please add the following to the Inbound MQ Adapter

<jca:operation

ActivationSpec="oracle.tip.adapter.mq.inbound.SyncReqResActivationSpecImpl"
MessageType="REQUEST"
QueueName="INBOUND_QUEUE"
Priority="AS_Q_DEF"
Persistence="AS_Q_DEF"
InboundThreadCount="1" <==== This parameter 

Expiry="NEVER"
OpaqueSchema="true" >
</jca:operation>



4. Database Adapter

It takes multiple steps to configure database adapter threads.

Step 1: Configure distributed polling. The query in the polling database adapter needs to be a distributed polling in order to avoid data duplication.

To set usesSkipLocking in SOA 10.1.3.x, you must first declare the property in ra.xml, then set the value in oc4j-ra.xml. No re-packaging or redeployment of DbAdapter.rar is needed.

Step 2. Set activationInstances as activation properties at bpel.xml to achieve multiple threads in database adapter.

Note: There is another property called NumberOfThreads. This property is NOT supported in clustered environment or when activationInstances>1 in SOA 10.1.3.x, or even at versions prior to SOA 11.1.1.3.


Step 3. Tune MaxTransactionSize and MaxRaiseSize to throttle the incoming messages along side with activationAgents/NumberOfThreads. These two properties are configured either through the DbAdapter wizard at JDeveloper, or manually directly at the WSDL file of the DbAdapter


5. File/FTP Adapter

File/FTP adapter's has a separation poller thread and processor thread (comparatively, JMS/AQ adapters always use the same thread to poll and process). There is always only one poller thread, while there could be multiple processor threads. In SOA 10.1.3.x, the processor threads are globally shared among File and FTP adapter instances, while in 11G you have an option to configure private processor thread pool per adpater *.jca file.



In SOA 10.1.3.x, the configuration file for you to set the File/FTP adapter processor threads are:
[SOA_HOME]\bpel\system\services\config\pc.properties
[SOA_HOME]\integration\esb\config\pc.properties (need to rename from pc.properties.esb)


The property name is:
oracle.tip.adapter.file.numProcessorThreads=4

If BPEL and ESB co-located at the same OC4J container, the pc.properties for BPEL takes precedence over that of ESB . In such cases, the values set in SOA_HOME\bpel\system\service\config\pc.properties will suffice

Configure Adapter Threads in Oracle SOA 11G

In in earlier posting, I mentioned you can configure multiple threads of inbound (polling) adapters of Oracle SOA. However, the ways to configure multiple threads vary between adapters and product versions, and the information scatter cross multiple documentations.

Hence it is worth to consolidate them here. Below are how to configure Adapter threads in Oracle SOA 11G.


1. JMS Adapter

• Property Name: adapter.jms.receive.threads

• Configuration File: adapter binding at composite.xml
  
  
• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABCGCEC

• Example:

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/jms/textmessageusingqueues/textmessageusingqueues/dequeue%2F#wsdl.interface(Consume_Message_ptt)"/>
<binding.jca config="dequeue_jms.jca">
<property name="adapter.jms.receive.threads" type="xs:string" many="false">10</property>
</binding.jca">
</service>

2. AQ Adapter

• Property Name: adapter.aq.dequeue.threads

• Configuration file: composite.xml

• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABDEBEE
  

• Example:

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/aq/raw/raw/dequeue/#wsdl.interface(Dequeue_ptt)"/>
<binding.jca config="dequeue_aq.jca">
<property name="adapter.aq.dequeue.threads" type="xs:string" many="false">10</property>
</binding.jca>
</service>

3. MQ Adapter

• Property Name: InboundThreadCount

• Configuration File: *.jca file

• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABDEBEE

4. Database Adapter

It takes multiple steps to configure database adapter threads.

Step 1: Configure distributed polling. The query in the polling database adapter needs to be a distributed polling in order to avoid data duplication. Please follow the two best practices in the documentation to establish the right kind of distributed polling.

Step 2. Set activationInstances as adapter binding property at composite.xml (SOA 11G) to achieve multiple threads in database adapter.

Alternatively, you can set NumberOfThreads in the jca file (SOA 11.1.1.3 onward). Technically, activationInstances and NumberOfThreads properties work differently, in such that NumberOfThreads works in the scope of per activation agent. Before SOA 11.1.1.3, NumberOfThreads is NOT supported in clustered environment or when activationInstances>1. At SOA 11.1.1.3 onward, you can use either activationInstances or NumberOfThreads properties to achieve the multi-threading effect. But if for some reason you set both, the total concurrent threads will be activationInstances x NumberOfThreads. For example, if you activationInstances=2, and NumberOfThreads=5, that means there are are 5 threads running within each activation instance.


Step 3. Tune MaxTransactionSize and MaxRaiseSize to throttle the incoming messages along side with activationAgents/NumberOfThreads. These two properties are configured either through the DbAdapter wizard at JDeveloper, or manually directly at the *.jca file.

<endpoint-activation portType="poll_ptt" operation="receive">
<activation-spec className="oracle.tip.adapter.db.DBActivationSpec">
...
<property name="PollingStrategy" value="LogicalDeletePollingStrategy"/>
<property name="MaxRaiseSize" value="5"/>
<property name="MaxTransactionSize" value="10"/>
...
</activation-spec>
</endpoint-activation>

5. File/FTP Adapter

File/FTP adapter's threading model is a bit complex. In essence, there is a separation poller thread and processor thread, except in the "Single Threaded Model" (comparatively, JMS/AQ adapters always use the same thread to poll and process). There is always only one poller thread, while there could be multiple processor threads. Please go through the documentation thoroughly so that you can choose a threading model appropriate to your application.


Step 1: Choose a threading model

• Default Threading Model
• Modified Threading Model 1: Single Threaded Model
• Modified Threading Model 2: Partitioned Threaded Model

Step 2: Configure threads depending on the threading model you choose

If you choose the Default Threading Model, you can set the thread count of global processor through the oracle.tip.adapter.file.numProcessorThreads property at the pc.properties file. This pc.properties is read from the classpath. So you could, for example, copy it to some directory under ORACLE_HOME and then reference it in the WLS classpath in setDomainEnv.sh. However, the Partition Threading Model is recommended over the Default Threading Model (see below) if you do have a need to define processor threads.

If you choose the Single Threaded Model, set the SingleThreadModel=true at the *.jca file. And as the name applies, you don't worry about any thread counts.

<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
<property../>
<property name="SingleThreadModel" value="true"/>
<property../>
</activation-spec>

If you choose the Partitioned Threaded Model, you can set the thread count for processor threads per adapter at the *.jca file:

<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
<property../>
<property name="ThreadCount" value="4"/>
<property../>
</activation-spec>

Please note the value of the ThreadCount is closely related to the kind of threading model you choose:

• If the ThreadCount property is set to 0, then the threading behavior is like that of the single threaded model.
• If the ThreadCount property is set to -1, then the global thread pool is used, as in the default threading model.
• The maximum value for the ThreadCount property is 40.

Tune Audit Trail in SOA 11G to Avoid Memory and Transaction Problems

Until 11.1.1.3, BPEL audit trails are saved to database in the same JTA transaction as the main transaction. This causes three main problems:

1. Most common: Latency of saving audit trail to database is included as part of the overall latency of the main business transaction.
2. Often seen: When the main transaction is rolled back for whatever reason, the audit trail did not get saved, because audit trails are saved in the main transaction as well. Thus no trace of what had happened can be found on the BPEL Console (or EM Console in 11G). This gives more difficulties for debugging.
   
3. Happen when you have large while loop: when a BPEL process instance has large number of activities (typically from using large while loop), the amount audit trails stored in the memory gets so large that, the BPEL service engine either encounters an OutOfMemoryException, or, thanks to the maxRequestDepth property, it commits the transaction early to flush out the audit data from memory to database to avoid OutOfMemoryException. In doing so BPEL service engine introduce extra transaction boundaries into the main transaction which sometimes cause undesirable behavior.
   

Since SOA 11.1.1.3 (i.e. SOA 11G PatchSet 2), management of audit trail memory has been greatly enhanced. Not only the size of the memory is reduced, but also when and how audit trail is stored is improved. The above three problems now have solutions.

The audit trail enhancement comes in the form of the following configuration properties

auditStorePolicy

Use this flag to choose the audit store strategy

• syncSingleWrite - would store audit data synchronously when it saves the cube instance, this is done as part of the cube instance transaction.

This is the default value. And the behavior is the same as in the 10.1.3.x version.

• syncMultipleWrite - would store audit data synchronously using a separate local transaction.
  

By "synchronously" it means the BPEL service engine is saving the audit trail in the same thread as the main transaction. However, it is doing it in a "separate local transaction".

Because they are on the same thread, latency of saving audit trail is still included into overall latency of the main transaction.

However, because they are on separate transactions, you can configure BPEL engine (using AuditFlushByteThreshold and AuditFlushEventThreshold) to flush out the audit trail from memory to database periodically regardless how long the main transaction would take. Moreover, having them on two separate transaction means the rollback of main transaction will NOT affect the audit trail transaction. That is, you will still see audit trail even if the main transaction rolls back.

• async - would store the audit data asynchronously using an in-memory queue and pool of audit threads.
  

This is almost the same as "syncMultipleWrite", except that it is done not just in a separate transaction but also in a separate thread.

The pros is the audit trail latency is NOT directly included in the latency of main transaction (but because they still share the computing resources and database, the latency is still indirectly related).

The cons is that because audit trails are being saved asynchronously, the audit trail may be out of sync from the main transaction (as the name 'async' implies).

AuditFlushByteThreshold and AuditFlushEventThreshold

When auditStorePolicy=syncMultipleWrite or auditStorePolicy=async, you use these two flags to control how often the engine should flush the audit events. These two properties do NOT apply to auditStorePolicy=syncSingleWrite.

auditFlushByteThreshold means after adding an event to the current batch, the engine would check if current batch byte size is greater than this value or not. if yes, then it would flush the current batch. The default value is 2048000 (byte), i.e. 2MB.

Similarly, auditFlushEventThreshold means this limit is reached, the engine would trigger the store call. The default value is 300 (event)

Both values need to be tuned based on the application and requirements.

AuditDetailThreshold

This is the maximum size (in bytes) an audit trail details string can be before it is stored separately from the audit trail. The default value is 50000 (byte). This is the same property as in 10.1.3.x. Please refer to the SOA Management MBean for its detail explanation.

AuditLevel

This is the same property in 10.1.3.x.

How to Configure

All of the above properties can be configured via the SOA EM Console. The path is EM -> SOA Infrastructure -> SOA Administration -> BPEL Properties -> More BPEL Configuration Properties

Change WebLogic Server Mode from Development to Production and JDK Mode from Client to Server

What does it take to change WebLogic 10.3.x server mode from development to production?

You quickly search on the Internet and quickly land to documentation such as here: http://docs.oracle.com/cd/E21764_01/apirefs.1111/e13952/taskhelp/domainconfig/ChangeRuntimeModes.html . The doc says you just need to go to the WebLogic Console to do it. That is easy, right?

But then the next link or so you may found a blog like http://blogs.oracle.com/learnwithpavan/entry/run_weblogic_in_production_mod , which says you need to set some Java parameter to make that happen.

So, which one is right?

Well, the truth of the matter is, to completely change a server from Development to Production Mode, two things need to happen:

1. From Console -> Domain -> Production mode, check the Production Mode check box. This change will go into config.xml
2. In setDomainEnv.sh, set PRODUCTION_MODE=true.

You will notice that if you only do #1, the Java will still start with "-client" option. Only when both of these settings are done, the server will start with ‘-server’ in its Java command line.


For differences between Development mode and Production mode, please refer to http://docs.oracle.com/cd/E21764_01/web.1111/e13814/wls_tuning.htm#i1111053 for details.

Retrieve BPEL payload from the database

Given an instance id, how to get the payload used to invoke the BPEL, from the dehydration database (called SOA infrastructure database in 11G)?

You have two options, each with its pros and cons:

Option 1. Extract payload from process instance data (i.e. "cube instance")

• Lookup instance handle using conversation Id or instance Id
• Call the com.oracle.bpel.client.IInstanceHandle.getField("bpelVariableName") API to get the payload. If the variable is a message type, the returned object is a Map and you can get the Element via map.get("partName"). If the variable is an element type, the returned object is an Element object.

Caveats:

• If your BPEL process change the value of the input variable, this API It will get the latest value, not the initial one.
• It may not be able to get the variables from the faulted instances. It may only work with successful instances

Option 2. Extract payload from audit trail

• Lookup instance handle using conversation Id or instance ID
• Fetch audit trail from instance handle using IInstanceHandle.getAuditXXX() API
• Use substring functions to extract initiating payload

Caveats:

• You need to do your own parsing of XML in order to extract the initial payload
• If you use auditLevel to dial down the audit trail to a certain degree, or even totally suppress it, you may not be able get the payload from audit trail