Configure Adapter Threads in Oracle SOA 10G

In a separate posting, I mentioned you can configure multiple threads of inbound (polling) adapters of Oracle SOA. However, the ways to configure multiple threads vary between adapters and product versions, and the information scatter across multiple documentations. Hence it is worth to consolidate them here.

This post is for Oracle SOA 10.1.3.x. I have another blog post for configuring adapter threads in Oracle SOA 11G.


1. JMS Adapter

• For BPEL: Set 'adapter.jms.receive.threads' as activation agent properties in bpel.xml
  

<activationAgents>
<activationAgent className=“…" partnerLink="MsgQueuePL">
...     <property name="adapter.jms.receive.threads”>5</property>
</activationAgent>
</activationAgents>

• For OESB 10.1.3.3 onward: Set as endpoint property via JDeveloper (10.1.3.3) or directly at *.esbsvc file

2. AQ Adapter

This blog summarizes the different kind of settings for AQ Adapter threads very well. Let me copy and paste it below:

#### FOR ESB/BPEL 10.1.3.4.x
<property name="adapter.aq.dequeue.threads">NO_OF_THREADS</property>
#### FOR BPEL 10.1.3.3.x
<property name=activationInstances">NO_OF_THREADS<property>
#### FOR ESB 10.1.3.3.x
<endpointProperties>
   <property name="numberOfAqMessageListeners" value="NO_OF_THREADS"/>
</endpointProperties>

3. MQ Adapter

Oracle Support Tech Note How to limit number of threads for reading messages from a queue (Doc ID 1144847.1] describes the details of how to setup the threads for Inbound MQAdapter . Copying here:

There exist a parameter called "InboundThreadCount" which is valid for both 11g and also it is tested on SOA 10.1.3.5, And Confirmed to be working on 10.1.3.4.

To set the parameter, Please add the following to the Inbound MQ Adapter

<jca:operation

ActivationSpec="oracle.tip.adapter.mq.inbound.SyncReqResActivationSpecImpl"
MessageType="REQUEST"
QueueName="INBOUND_QUEUE"
Priority="AS_Q_DEF"
Persistence="AS_Q_DEF"
InboundThreadCount="1" <==== This parameter 

Expiry="NEVER"
OpaqueSchema="true" >
</jca:operation>



4. Database Adapter

It takes multiple steps to configure database adapter threads.

Step 1: Configure distributed polling. The query in the polling database adapter needs to be a distributed polling in order to avoid data duplication.

To set usesSkipLocking in SOA 10.1.3.x, you must first declare the property in ra.xml, then set the value in oc4j-ra.xml. No re-packaging or redeployment of DbAdapter.rar is needed.

Step 2. Set activationInstances as activation properties at bpel.xml to achieve multiple threads in database adapter.

Note: There is another property called NumberOfThreads. This property is NOT supported in clustered environment or when activationInstances>1 in SOA 10.1.3.x, or even at versions prior to SOA 11.1.1.3.


Step 3. Tune MaxTransactionSize and MaxRaiseSize to throttle the incoming messages along side with activationAgents/NumberOfThreads. These two properties are configured either through the DbAdapter wizard at JDeveloper, or manually directly at the WSDL file of the DbAdapter


5. File/FTP Adapter

File/FTP adapter's has a separation poller thread and processor thread (comparatively, JMS/AQ adapters always use the same thread to poll and process). There is always only one poller thread, while there could be multiple processor threads. In SOA 10.1.3.x, the processor threads are globally shared among File and FTP adapter instances, while in 11G you have an option to configure private processor thread pool per adpater *.jca file.



In SOA 10.1.3.x, the configuration file for you to set the File/FTP adapter processor threads are:
[SOA_HOME]\bpel\system\services\config\pc.properties
[SOA_HOME]\integration\esb\config\pc.properties (need to rename from pc.properties.esb)


The property name is:
oracle.tip.adapter.file.numProcessorThreads=4

If BPEL and ESB co-located at the same OC4J container, the pc.properties for BPEL takes precedence over that of ESB . In such cases, the values set in SOA_HOME\bpel\system\service\config\pc.properties will suffice

Configure Adapter Threads in Oracle SOA 11G

In in earlier posting, I mentioned you can configure multiple threads of inbound (polling) adapters of Oracle SOA. However, the ways to configure multiple threads vary between adapters and product versions, and the information scatter cross multiple documentations.

Hence it is worth to consolidate them here. Below are how to configure Adapter threads in Oracle SOA 11G.


1. JMS Adapter

• Property Name: adapter.jms.receive.threads

• Configuration File: adapter binding at composite.xml
  
  
• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABCGCEC

• Example:

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/jms/textmessageusingqueues/textmessageusingqueues/dequeue%2F#wsdl.interface(Consume_Message_ptt)"/>
<binding.jca config="dequeue_jms.jca">
<property name="adapter.jms.receive.threads" type="xs:string" many="false">10</property>
</binding.jca">
</service>

2. AQ Adapter

• Property Name: adapter.aq.dequeue.threads

• Configuration file: composite.xml

• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABDEBEE
  

• Example:

<service name="dequeue" ui:wsdlLocation="dequeue.wsdl">
<interface.wsdl interface="http://xmlns.oracle.com/pcbpel/adapter/aq/raw/raw/dequeue/#wsdl.interface(Dequeue_ptt)"/>
<binding.jca config="dequeue_aq.jca">
<property name="adapter.aq.dequeue.threads" type="xs:string" many="false">10</property>
</binding.jca>
</service>

3. MQ Adapter

• Property Name: InboundThreadCount

• Configuration File: *.jca file

• Documentation: http://docs.oracle.com/cd/E21764_01/core.1111/e10108/adapters.htm#BABDEBEE

4. Database Adapter

It takes multiple steps to configure database adapter threads.

Step 1: Configure distributed polling. The query in the polling database adapter needs to be a distributed polling in order to avoid data duplication. Please follow the two best practices in the documentation to establish the right kind of distributed polling.

Step 2. Set activationInstances as adapter binding property at composite.xml (SOA 11G) to achieve multiple threads in database adapter.

Alternatively, you can set NumberOfThreads in the jca file (SOA 11.1.1.3 onward). Technically, activationInstances and NumberOfThreads properties work differently, in such that NumberOfThreads works in the scope of per activation agent. Before SOA 11.1.1.3, NumberOfThreads is NOT supported in clustered environment or when activationInstances>1. At SOA 11.1.1.3 onward, you can use either activationInstances or NumberOfThreads properties to achieve the multi-threading effect. But if for some reason you set both, the total concurrent threads will be activationInstances x NumberOfThreads. For example, if you activationInstances=2, and NumberOfThreads=5, that means there are are 5 threads running within each activation instance.


Step 3. Tune MaxTransactionSize and MaxRaiseSize to throttle the incoming messages along side with activationAgents/NumberOfThreads. These two properties are configured either through the DbAdapter wizard at JDeveloper, or manually directly at the *.jca file.

<endpoint-activation portType="poll_ptt" operation="receive">
<activation-spec className="oracle.tip.adapter.db.DBActivationSpec">
...
<property name="PollingStrategy" value="LogicalDeletePollingStrategy"/>
<property name="MaxRaiseSize" value="5"/>
<property name="MaxTransactionSize" value="10"/>
...
</activation-spec>
</endpoint-activation>

5. File/FTP Adapter

File/FTP adapter's threading model is a bit complex. In essence, there is a separation poller thread and processor thread, except in the "Single Threaded Model" (comparatively, JMS/AQ adapters always use the same thread to poll and process). There is always only one poller thread, while there could be multiple processor threads. Please go through the documentation thoroughly so that you can choose a threading model appropriate to your application.


Step 1: Choose a threading model

• Default Threading Model
• Modified Threading Model 1: Single Threaded Model
• Modified Threading Model 2: Partitioned Threaded Model

Step 2: Configure threads depending on the threading model you choose

If you choose the Default Threading Model, you can set the thread count of global processor through the oracle.tip.adapter.file.numProcessorThreads property at the pc.properties file. This pc.properties is read from the classpath. So you could, for example, copy it to some directory under ORACLE_HOME and then reference it in the WLS classpath in setDomainEnv.sh. However, the Partition Threading Model is recommended over the Default Threading Model (see below) if you do have a need to define processor threads.

If you choose the Single Threaded Model, set the SingleThreadModel=true at the *.jca file. And as the name applies, you don't worry about any thread counts.

<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
<property../>
<property name="SingleThreadModel" value="true"/>
<property../>
</activation-spec>

If you choose the Partitioned Threaded Model, you can set the thread count for processor threads per adapter at the *.jca file:

<activation-spec className="oracle.tip.adapter.file.inbound.FileActivationSpec">
<property../>
<property name="ThreadCount" value="4"/>
<property../>
</activation-spec>

Please note the value of the ThreadCount is closely related to the kind of threading model you choose:

• If the ThreadCount property is set to 0, then the threading behavior is like that of the single threaded model.
• If the ThreadCount property is set to -1, then the global thread pool is used, as in the default threading model.
• The maximum value for the ThreadCount property is 40.

Tune Audit Trail in SOA 11G to Avoid Memory and Transaction Problems

Until 11.1.1.3, BPEL audit trails are saved to database in the same JTA transaction as the main transaction. This causes three main problems:

1. Most common: Latency of saving audit trail to database is included as part of the overall latency of the main business transaction.
2. Often seen: When the main transaction is rolled back for whatever reason, the audit trail did not get saved, because audit trails are saved in the main transaction as well. Thus no trace of what had happened can be found on the BPEL Console (or EM Console in 11G). This gives more difficulties for debugging.
   
3. Happen when you have large while loop: when a BPEL process instance has large number of activities (typically from using large while loop), the amount audit trails stored in the memory gets so large that, the BPEL service engine either encounters an OutOfMemoryException, or, thanks to the maxRequestDepth property, it commits the transaction early to flush out the audit data from memory to database to avoid OutOfMemoryException. In doing so BPEL service engine introduce extra transaction boundaries into the main transaction which sometimes cause undesirable behavior.
   

Since SOA 11.1.1.3 (i.e. SOA 11G PatchSet 2), management of audit trail memory has been greatly enhanced. Not only the size of the memory is reduced, but also when and how audit trail is stored is improved. The above three problems now have solutions.

The audit trail enhancement comes in the form of the following configuration properties

auditStorePolicy

Use this flag to choose the audit store strategy

• syncSingleWrite - would store audit data synchronously when it saves the cube instance, this is done as part of the cube instance transaction.

This is the default value. And the behavior is the same as in the 10.1.3.x version.

• syncMultipleWrite - would store audit data synchronously using a separate local transaction.
  

By "synchronously" it means the BPEL service engine is saving the audit trail in the same thread as the main transaction. However, it is doing it in a "separate local transaction".

Because they are on the same thread, latency of saving audit trail is still included into overall latency of the main transaction.

However, because they are on separate transactions, you can configure BPEL engine (using AuditFlushByteThreshold and AuditFlushEventThreshold) to flush out the audit trail from memory to database periodically regardless how long the main transaction would take. Moreover, having them on two separate transaction means the rollback of main transaction will NOT affect the audit trail transaction. That is, you will still see audit trail even if the main transaction rolls back.

• async - would store the audit data asynchronously using an in-memory queue and pool of audit threads.
  

This is almost the same as "syncMultipleWrite", except that it is done not just in a separate transaction but also in a separate thread.

The pros is the audit trail latency is NOT directly included in the latency of main transaction (but because they still share the computing resources and database, the latency is still indirectly related).

The cons is that because audit trails are being saved asynchronously, the audit trail may be out of sync from the main transaction (as the name 'async' implies).

AuditFlushByteThreshold and AuditFlushEventThreshold

When auditStorePolicy=syncMultipleWrite or auditStorePolicy=async, you use these two flags to control how often the engine should flush the audit events. These two properties do NOT apply to auditStorePolicy=syncSingleWrite.

auditFlushByteThreshold means after adding an event to the current batch, the engine would check if current batch byte size is greater than this value or not. if yes, then it would flush the current batch. The default value is 2048000 (byte), i.e. 2MB.

Similarly, auditFlushEventThreshold means this limit is reached, the engine would trigger the store call. The default value is 300 (event)

Both values need to be tuned based on the application and requirements.

AuditDetailThreshold

This is the maximum size (in bytes) an audit trail details string can be before it is stored separately from the audit trail. The default value is 50000 (byte). This is the same property as in 10.1.3.x. Please refer to the SOA Management MBean for its detail explanation.

AuditLevel

This is the same property in 10.1.3.x.

How to Configure

All of the above properties can be configured via the SOA EM Console. The path is EM -> SOA Infrastructure -> SOA Administration -> BPEL Properties -> More BPEL Configuration Properties