Enabling Apache Axis2 Clustering

By Kamal Wickramanayake, 21 May, 2008

I tried enabling clustering support in both standard and war distributions of Apache Axis2 version 1.4. It was not as simple as said. I could see possibly a bug related with coding in war distribution. And in both distributions, I could see improper axis2.xml files and several mistakes in packaging jars. Here's how you go about enabling clustering support:

Environment:

Operating System: Fedora 7 (Linux)
Java: jdk 1.6.0_03
Axis2: 1.4 (axis2-1.4-bin.zip and axis2-1.4-war.zip)
Tomcat: 6.0.14

Axis2 1.4 Standard Distribution:

The clustering related section of the axis2.xml file has couple of mistakes. Here's what I found with the distribution (You need to uncomment the section):

<cluster class="org.apache.axis2.cluster.tribes.TribesClusterManager">
        <parameter name="param1">value1</parameter>
        <parameter name="domain">apache.axis2.domain</parameter>
        <parameter name="synchronizeAll">true</parameter>
        <parameter name="maxRetries">10</parameter>
        <configurationManager class="org.apache.axis2.cluster.configuration.TribesConfigurationManager">
            <listener class="org.apache.axis2.cluster.configuration.DefaultConfigurationManagerListener"/>
        </configurationManager>
        <contextManager class="org.apache.axis2.cluster.context.TribesContextManager">
            <listener class="org.apache.axis2.cluster.context.DefaultContextManagerListener"/>
        </contextManager>
</cluster>

Here's the correct version:

<cluster class="org.apache.axis2.clustering.tribes.TribesClusterManager">
        <parameter name="param1">value1</parameter>
        <parameter name="domain">apache.axis2.domain</parameter>
        <parameter name="synchronizeAll">true</parameter>
        <parameter name="maxRetries">10</parameter>
        <configurationManager class="org.apache.axis2.clustering.configuration.DefaultConfigurationManager">
            <listener class="org.apache.axis2.clustering.configuration.DefaultConfigurationManagerListener"/>
        </configurationManager>
        <contextManager class="org.apache.axis2.clustering.context.DefaultContextManager">
            <listener class="org.apache.axis2.clustering.context.DefaultContextManagerListener"/>
        </contextManager>
</cluster>

Changes:

All the package names are changed from org.apache.axis2.cluster.* to org.apache.axis2.clustering.*
There were no classes called TribesConfigurationManager and TribesContextManager with Axis2 distribution. I replaced them with DefaultConfigurationManager and DefaultContextManager. "Clustering Guide" found in Axis2 documentation has sample axis2.xml file content and correctly uses "DefaultContextManager" though.
Furthermore, "Clustering Guide" sample axis2.xml file content has a "<replication>" section including parameters names that should not be replicated. I didn't try with them. But hopefully, they should be present in the file.

The standard distribution is missing two jar files that are required for clustering support. I could find the two files in Apache Tomcat (I used version 6.0.14). The lib directory of Tomcat has "catalina-tribes.jar" file and bin directory contains "tomcat-juli.jar" file. Copy both of them to the lib directory of Axis2.

After these, I copied Axis2 directory (AXIS2_HOME) (including everything inside) to two machines and started Axis2 (in both machines). Clustering worked!

Axis2 1.4 War Distribution:

To correct the problems, you need to unzip the war distribution to a temporary directory. You will see axis2 directory being created. As I remember, if you need this to work in JBoss, change the directory name from axis2 to axis2.war (Yep, let that be the directory name).

axis2.xml has the same problems mentioned above with the standard distribution. You need to correct them. The file is located in axis2/WEB-INF/conf directory.

axis2/WEB-INF/lib directory has servlet-api-2.3.jar file which is not required since Tomcat will provide that. You can keep it. However, Tomcat log file (log/catalina.out) will complain about it.

You need to download Axis2 standard distribution to get a key jar file missing in Axis2 war distribution. Copy the axis2-clustering-1.4.jar found in lib directory of standard distribution to axis2/WEB-INF/lib directory.

You should now be able to copy this axis2 directory (including everything inside) to apache-tomcat/webapps directory. If it's JBoss, you will copy axis2.war directory (not the file!) to jboss/server/default/deploy directory if you use the JBoss default configuration.

Start Tomcat and clustering will work! I deployed a cluster sensitive service that replicates data and tested. Here's the POJO service I used:

package mypkg;

import org.apache.axis2.context.ConfigurationContext;
import org.apache.axis2.context.MessageContext;

public class StatefulClusterTester {

    public void setName(String name) {
        ConfigurationContext configurationContext =
            MessageContext.getCurrentMessageContext().getConfigurationContext();
        configurationContext.setProperty("name", name);
    }

    public String getName() {
        ConfigurationContext configurationContext =
            MessageContext.getCurrentMessageContext().getConfigurationContext();
        return (String) configurationContext.getProperty("name");
    }
}

I used wsdl2java to generate stub code and invoked setName on one machine and getName on the other machine.

<service name="StatefulClusterTester">

<messageReceivers>
    <messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-only"
        class="org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver"/>

    <messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-out"
        class="org.apache.axis2.rpc.receivers.RPCMessageReceiver" />
</messageReceivers>

<parameter name="ServiceClass">mypkg.StatefulClusterTester</parameter>

</service>

I could use ServiceContext data replicated as well. In this case, I had to set scope of the service as "application" in the services.xml file:

package mypkg;

import org.apache.axis2.context.MessageContext;
import org.apache.axis2.context.ServiceContext;

public class StatefulClusterTester {

    public void setName(String name) {
        ServiceContext serviceContext =
            MessageContext.getCurrentMessageContext().getServiceContext();
        serviceContext.setProperty("name", name);
    }

    public String getName() {
        ServiceContext serviceContext =
            MessageContext.getCurrentMessageContext().getServiceContext();
        return (String) serviceContext.getProperty("name");
    }
}

<service name="StatefulClusterTester" scope="application">

<messageReceivers>
    <messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-only"
        class="org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver"/>

    <messageReceiver mep="http://www.w3.org/2004/08/wsdl/in-out"
        class="org.apache.axis2.rpc.receivers.RPCMessageReceiver" />
</messageReceivers>

<parameter name="ServiceClass">mypkg.StatefulClusterTester</parameter>

</service>

Problems With URL Configuration:

Axis2 can be configured to use an axis2.xml file and a repository (containing services and other modules) accessible through urls. You need to copy the axis2.xml file and the entire repository directory found in standard distribution or the WEB-INF/services and WEB-INF/modules directories found in war distribution to a directory that can be accessed through a web server. Here's what I created in a web shared directory in my Linux machine:

axis2repo/
|-- axis2.xml
|-- modules
|   |-- addressing-1.4.mar
|   |-- axis2-scripting-1.4.mar
|   |-- mex-1.4.mar
|   |-- modules.list
|   |-- ping-1.4.mar
|   `-- soapmonitor-1.4.mar
`-- services
    |-- services.list
    `-- version-1.4.aar

Then I wrote the following class, compiled it and run with all the jar files in Axis2/lib directory added to the class path.

import java.net.MalformedURLException;
import java.net.URL;

import org.apache.axis2.AxisFault;
import org.apache.axis2.context.ConfigurationContext;
import org.apache.axis2.context.ConfigurationContextFactory;
import org.apache.axis2.transport.http.SimpleHTTPServer;

public class Axis2URLBasedStarter {

public static void main(String[] args) throws AxisFault, MalformedURLException {
    ConfigurationContext configctx = ConfigurationContextFactory.createConfigurationContextFromURIs(
        new URL("http://192.168.0.10/axis2repo/axis2.xml"),
        new URL("http://192.168.0.10/axis2repo/"));

    SimpleHTTPServer smt = new SimpleHTTPServer(configctx, 8080);
    smt.start();

    }
}

192.168.0.10 is the IP address of the machine where I placed the above repository directory hierarchy including the axis2.xml file.

Pretty good! It worked even with clustering enabled.

However, the War distribution seems to have problems. Of the war distribution, WEB-INF/web.xml file has some sections that can be used to configure Axis2 to use a URL based configuration file and a repository.

The web.xml file declares a Servlet called AxisServlet. Within the declaration, several init parameters are present which are commented by default. Note that to correctly set parameters, for each <param-name> and <param-value> pair, you need to use <init-param> tag (which is not suggested in web.xml file that comes with Axis2). Here's how you may ago about using a remote configuration file and a remote repository:

<init-param>
    <param-name>axis2.xml.url</param-name>
    <param-value>http://192.168.0.10/axis2repo/axis2.xml</param-value>
</init-param>
<init-param>
    <param-name>axis2.repository.url</param-name>
    <param-value>http://192.168.0.10/axis2repo/</param-value>
</init-param>

However, setting the axis2.xml.url did not work. It generated the following error in my machine and failed to start Axis2 instance:

[INFO] org.apache.axiom.om.OMException: com.ctc.wstx.exc.WstxEOFException: Unexpected EOF in prolog
at [row,col {unknown-source}]: [1,0]

This may be a bug in coding. I didn't try to find why this was generated or correct the problem. I commented out this init-parameter setting and kept axis2.repository.url. It worked. However, the last "/" (of http://192.168.0.10/axis2repo/) was important. If it was missing, again I could see an error.

No Clue!

I still couldn't figure out the purpose of ConfigurationManager. May be it's not working in the way it should be. I couldn't find any article or guide about using it except for this. Assuming that at least the ConfigurationManager would notify the cluster nodes of a newly added service, I tried the following:

1. Got clustering to work with the remote repository
2. Started two Axis2 instances (in Tomcat)
3. Placed a new service in the repository, added the .aar file name to services.list file
4. Stopped one Axis2 instance and restarted the same (Say instance 1)
5. Tested the availability of the new service. It was present in instance 1, but not in instance 2.

What could be the purpose of this ConfigurationManager?

Java

Web Services