2014年10月17日星期五

JBoss Fuse/A-MQ - Easy way to achieve fault tolerant messaging Part 1

Following some of the questions I got from the people during the enablement in APAC, one of the question I get is how to do achieve fault tolerant, I believe there are already many works done out there that talks very technical and search deeply in this area.

But as always, I am going to show few simple steps to achieving fault tolerant messaging with A-MQ.

The simplest way to make your messaging system fault tolerant is to make your broker into Master/Slave group. Therefore when one broker goes down a backup server will takeover.


So we can now ensure our broker will always be up and running, Fuse does a beautiful job of detecting and switching.

There are 2 ways to spin up an the broker, one with the beautiful management GUI with Fuse, or the old hardcore command line.

Please make sure you have installed Fuse and boot up the Fabric before you begin.
Logon to JBoss Fuse management Console, under Runtime -> MQ-> +Broker






Add group, broker name and choose MasterSlave kind, then click on create broker, after that, you broker diagram should appear in the list, click on the red triangle in the middle.








Create the container by giving a name to it.






And you will see 2 containers with one actively listens to the client and one standby container.


Now, because the way we spin up our message broker, an port will automatically assigned to each connector to the port, to avoid conflict, it is going to assign ports that is not in use. But for the clients to connect to the broker, you probably want your program to detect the ports of the broker automatically. With A-MQ it has a convenient and easy way of detecting with Zoo Keeper registry. I will be showing it through a camel route.

The configuration is really easy, instead of setting your messaging address to a specific address and port like below:

  <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
      <property name="brokerURL" value="tcp://localhost:61616"/>
      <property name="userName" value="admin"/>
      <property name="password" value="admin"/>
  </bean>

or using the failover that must point out the address before hand (static), where the client must know where every single one of


  <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
      <property name="brokerURL" value="failover:(tcp://localhost:61616,tcp://localhost:61617)"/>
      <property name="userName" value="admin"/>
      <property name="password" value="admin"/>
  </bean>

But, we can always use the fabric discovery protocol to find out all the broker registry. the only configuration that will need is to add the discovery configurations to the broker URL.

  <bean id="jmsConnectionFactory" class="org.apache.activemq.ActiveMQConnectionFactory">
      <property name="brokerURL" value="discovery:(fabric://blogdemo)"/>
      <property name="userName" value="admin"/>
      <property name="password" value="admin"/>
  </bean>

As for library dependency, there are a few artifacts you will need to add to your project other then messaging libraries, are fabric related artifacts like, fabric-groups, fabric-zookeeper and the osgi ones.  So to do that, make sure you add the following dependency into your pom.xml

    
    
      org.apache.activemq
      activemq-camel
      5.9.0.redhat-610379
    
    
      org.apache.activemq
      activemq-pool
      5.9.0.redhat-610379
    
    
      org.apache.activemq
      activemq-osgi
      5.9.0.redhat-610379
    
    
    
    
	    org.jboss.amq
	    mq-fabric
	    6.1.0.redhat-379
	
	
	    io.fabric8
	    fabric-groups
	    1.0.0.redhat-379
	
	
	    io.fabric8
	    fabric-zookeeper
	    1.0.0.redhat-379
	
	
	
	
	    org.osgi
	    org.osgi.core
	    5.0.0
	
	
	    org.osgi
	    org.osgi.compendium
	    5.0.0
	

And another important configuration, because we are going to take advantage of the Zoo Keeper registry, so we must tell the application where to lookup the registry and add the zoo keeper password to the pom.xml making sure we were authorized to lookup into the registry, not just some random application happens to float around.

  
    <!-...... -->
    localhost:2181            
    admin           
  

And the rest is just the same for a normal messaging application.

In my demo, I had 2 route,

1st route will send a message to the broker every 5 mins,









2nd route read and then log the message










When starting the demo, you can see the messages are starting to send into the active broker,



Once we turn off the container, the standby container will takeover,


and the client application will automatically switch to the new active broker.










Next part of the topic, I will do a live video, and publish the code on github.


2014年9月30日星期二

JBoss Fuse - How to setup cluster of servers in Fabric

Staying in Beijing for a technical enablement, one question I have been ask a lot is how to setup a Fabric in a cluster of servers with JBoss Fuse?

I am going to setup 2 Fuse on 2 different servers, and then join them together to form a fabric cluster.


First we need to create a fabric, so unzip the JBoss Fuse zip file in Server1, setup password by editing etc/user.properties.

And please do the same thing to server2. If you are testing, and tries to setup on the same machine, please remember to go to "org.apache.karaf.management.cfg" under etc/ and change the RMI related port.



Now, getting back to setting the cluster of fabric, start up JBoss Fuse by going into bin/ and execute fuse on server one. After it starts up, create a fabric by entering the following command:

fabric:create --wait-for-provisioning

This spins up a fabric on a container call root, now, start up JBoss Fuse on server2 by going to bin/ and execute fuse on server two. And instead of creating a fabric, we are going to join, by entering following command, as fabric:join [options] zookeeperUrl [containerName] .

fabric:join --zookeeper-password admin 192.168.0.1:2181 root1

To find out what the zookeeper URL, go to your fuse command line console in Server1 and type
config:edit io.fabric8.zookeeper
config:proplist

this will give you your zookeeper details

                       JBossFuse:karaf@root> config:proplist
            service.pid = io.fabric8.zookeeper
            zookeeper.password = ZKENC=YWRtaW4=
            zookeeper.url = 192.168.0.1:2181
            fabric.zookeeper.pid = io.fabric8.zookeeper

And now we have successfully created a fabric on 2 servers. If you type in container-list in the command line, you should be able to see we now have 2 working server.

JBossFuse:karaf@root> container-list 
[id]                           [version] [connected] [profiles]                                         [provision status]
root*                          1.0       true        fabric, fabric-ensemble-0000-1, jboss-fuse-full    success
root1                          1.0       true        fabric                              

And if you login to Fuse management console, under Runtime-> Containers you will see the 2 root containers on both servers


To add a new container, click on +Create on top right hand corner of the page, select the parent container so it will run on the server base on the parent container. Then choose the profile you need to install on the container.

And we are done!




2014年9月21日星期日

JBoss Fuse on xPaaS - IoT Mash-up Demo

There are 3 common protocols when it comes to messaging on Internet of Things,

MQTT - is OASIS standard, adopted by various vendor and companies, it is an extremely light weight transport, and is excellent for low computing power device, and platforms. And it only supports publish and subscribe mechanism. 

STOMP - is simple and lightweight (not as light as MQTT), it supports transport over websockets, and it is basically sending text through wires, very much like telnet.

AMQP - is a vendor neutral protocol over the wire, it supports rich sets of communication patterns, and it is accessible through a wide variety of APIs, such as JMS, JCA, WCF, Qpid, etc

All these are very easy to implement using Apache Camel. So I came up with a demo, showcasing these 3 common protocols. 

The basic architecture of demo is basically like this: 


Where I have created different connectors, like MQTT, AMQP, STOMP, WebSocket and Openwire for a broker on top of OpenShift, or it's similar if you want to create this on a Local JBoss Fuse. 
To do that, go to the boker.xml setting (depending on your environment, if you create a new broker profile on Fuse, make sure you make a copy of broker.xml from mq.base profile to your broker profile.), add transportConnector with different protocol name and port. 

<transportConnector name="amqp" publishedAddressPolicy="#addressPolicy" uri="amqp://${OPENSHIFT_FUSE_IP}:5672"/>       

<transportConnector name="mqtt" publishedAddressPolicy="#addressPolicy" uri="mqtt://${OPENSHIFT_FUSE_IP}:1883"/>      

<transportConnector name="stomp" publishedAddressPolicy="#addressPolicy" uri="stomp://${OPENSHIFT_FUSE_IP}:31613"/>      

<transportConnector name="websocket" publishedAddressPolicy="#addressPolicy" uri="ws://${OPENSHIFT_FUSE_IP}:31614"/>

Then install 3 Camel Routes, all of them are using built-in Components in JBoss Fuse to poll in the message from broker. With the built-in component, it will save lots of code implementation time.

AMQP Component: 
       amqp:queue:myAMQPDemo

MQTT Component: 
       mqtt:bar?subscribeTopicName

STOMP Component: 

        stomp:topic:chat.general

As you can see from the camel route, after we retrieve messages, we will process it and put them all into a big queue, which will be picked up later by web application and displayed. 

Because we are using openshift as our broker, we need to export the internal port by using the port-forward command provided by the openshift client tool.

rhc port-forward containerName

And lastly, we are going to startup a local web application, so we can have a nice web interface let you play with different protocol and display the result in a web console. And within the web application, we also use Qpid API and MQTT API to send messages into the queue or topics. 

You can find the demo code here:
https://github.com/weimeilin79/iot-mashup

Here is the video demo: 



2014年8月9日星期六

Red Hat JBoss - HP IT Summit in Taiwan, Openshift and Devops

Last Friday, I had a JBoss talk in HP taiwan IT summit, normally, developers doesn't really go to the forums like this, HP are know for their Hardware and infrastructure solutions, but I was there to break the boundary between developers and infrastructure gentlemen/ladies :)

It was a very short 40 minutes session, and it was the last session of the day, to make sure everyone is still awake and concentrating, I also did a little demo too.

As a developer, I wanted to discuss how Openshift has really boost the DevOps, and make our life much easier from a developer point of view, and also allowing system administrators to be relieved from their boring day to day routine job.

Automation place a big part of the DevOps world, OpenShift let developers create and setup a runtime environment on their own, so no more email or phone calls between 2 parties, developers can spin up an application server in few seconds without bothering his/her colleague. Inside Openshift, there are various cartridges you can choose from, the cartridges are how OpenShift installs Languages & Middleware, so there will be an JBoss Enterprise Application Server, or JBoss Fuse for developers to choose their middleware platform, and there are also cartridges of MongoDB or Postgresql and more, if developer needed a database. The instance developers create can be deploy both on premise or cloud, it is completely transparent to developers. They will not need to worry about anything can can then concentrated on their work. With JBoss developer studio it seamlessly combines with openshift, so developers can quickly update their code through the git to the repository inside their openshift nodes.

To make sure we make the most of DEVOPS, we should seriously consider breaking our code/project into smaller chunks, (Believe me, It is so common in Taiwan that I bumped into a WAR or EAP even a JAR file with 1-200 MB, even seen one with more then 700MB). Modulize it, It will improve the development cycle

1. Shorten the developement cycle with less code to maintain and test
2. Less complex code, so it's easier to maintain.
3. Modules can be reused. Even Test code are easier to create because we have limit what each module can do.
4. And be replace or remove with less impact, because it's less sticky in code.

To modulize it and make it more suitable to run in cloud, it is important that we keep it stateless. Or keep the state in your data. Openshift is also a great xPaaS, meaning it provides a great platfrom that integrates everything together with the help from JBoss Fuse technology. With the combination of JBoss Fuse and Openshift, it allows administrator to dynamically create as many instance as the hardware can handle. Give more flexibility to them.

I hope I crack a little hole in the wall between devlopers and system adminstrator with Openshift. :)
Thanks! And here are the slides on that day.

Red Hat JBoss - HP IT 高峰論壇 , Openshift 與 Devops

上週五亂入HP 的論壇去講JBoss, 怎麼JBoss 的開發者會去HP 這種超級硬邦邦的場子。沒錯JBoss 也可以去平台論壇喔!

今天講的主題是 DEVOPS 因為只有短短的40分鐘其實並沒辦法講得很深入。而且小姐我喜歡玩DEMO 多過於在台上一直念一直念催大家入睡.

因為我是開發者,當然要從IT 開發者的角度去好好的討論一下。到底OpenShift 在DevOps 的世界裡面有什麼功能跟好處呢。

首先在DEVOPS 世界中,有個很重要的Keyword 就是 Automation, 透過Openshfit 可以讓開發的程式人員自己啓動自己想要的環境。 在Openshift 中,有很多的Catridge 可以提供給大家自由選擇,例如說,Java 的開發者可以透過自己本地環境的開發工具,直接建立想要的執行環境。而且這個環境可以同時安裝在公司內部的系統或是外部的雲端上。 現場立馬建立環境,並且部署到Openshift 上。

為了配合成功的DEVOPS, 開發者在寫程式的時候,不能像以前一樣,有事沒事就寫一個1-200MB 的大包程式,開發者應該要朝著分散的程式區塊來寫。所以就可以加快上線的速度,而分散的程式的好處就是,

1. 開發週期變短
2. 需要維護的程式碼較不複雜 (天兵把簡單的事寫複雜則不在此限)
3. 測試模組較好維護
4. 可以隨時置換抽離

可是相對地,分散的區塊與區塊之間,因為拆得開,所以盡量就不要是Stateful 的了。
因為要部署的區塊多了,Openshift 可以讓你可以決定要起幾個環境。然後以一個整體的角度去管理了。對於開發者現在的問題,反倒是,我要如何將這些區塊完整地聯結呢?Openshift 與 JBoss Fuse 結合後,開發者可以透過設定讓模組與膜組之間的聯結更動態。所以在抽患之間又可以更隨意。透過 這兩個結合而成的xPaaS 平台,開發者也可以隨自己喜好建制需要的instance了!

OpenShift 可以消彌開發跟維運單位之間那條界線,讓開發的人可以建立自己的環境,讓維運的人可以整體的來看架構,而非一台一台的維護。寫得好累,用講的果然比較快,要睡了..

當天的Slide 放在這裡,請查收囉~

2014年7月22日星期二

Red Hat JBoss Fuse - Getting Started, Home Loan Demo Part 5

This is the last one of my getting started demo for JBoss FUSE 6.1. For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

This series of videos are focused on playing with the integration framework in JBoss Fuse, which base on the open source project Apache Camel. In this video I am going deploy the integration projects onto the OSGi container and manage it with Fuse Fabric.

The story behind the home loan demo is we have a system, that takes in XML files from different vendor's home loan application into a folder, and we will need to process the data give house an appraised value and then provide the information back to the vendors. 

First demo was to take in and process the information, and put it into a message broker,  and the second demo,  was about processing the customer data, by reading the xml file and store it into a Postgres Database. The third demo, was slightly complex, by adding value to the house by the number of surrounding schools to the house. we referenced Google App Engine (Google Geocoding API to get the co-ordinate from Address, then use Google Places API to get number of surrounding schools) and then updated the new appraised value into house table in Postgres Database.

And from last demo, we have expose an restful web service endpoint for Vendors to query the new house value and data via customer's Nation ID. 

In this demo, the project is now ready to be deployed into a Testing environment. By using the fabric8 plugin in JBoss Fuse, we are going to start up JBoss Fuse, and create a broker and container in our Fuse management console. Deploy our project onto the OSGi container in JBoss Fuse 6.1. 



In this demo, you will learn, 

Deploy your Camel Project via Fabric8 plugin 
This plugin makes it easy to create or update a fabric profile from your maven project.
When you deploy your project to a fabric profile with this plugin the following takes place:

  • uploads any artifacts into the fabric's maven repository
  • lazily creates the fabric profile or version you specify
  • adds/updates the maven project artifact into the profile configuration
  • adds any additional parent profile, bundles or features to the profile.

Create a new container in JBoss Fuse
The Fabric in JBoss Fuse makes it really easy to provision, automate, configure, and manage from a central location

There is one thing I forgot to mention in the Video,
For Fabric8 plugin to deploy the profile into your JBoss Fuse, need to edit your ~/.m2/settings.xml file to add the fabric server's user and password so that the maven plugin can login to the fabric..

e.g. add this to the <servers> element:

<server>
  <id>fabric8.upload.repo</id>
  <username>admin</username>
  <password>admin</password>
</server> 



Thanks for following this series of getting started. :)


2014年7月21日星期一

Red Hat JBoss Fuse - Getting Started, Home Loan Demo Part 4

This is the 4th part of my getting started demo for JBoss FUSE 6.1. For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There will be a series of videos coming after this one, we will focus on playing with the integration framework in JBoss Fuse, which base on the open source project Apache Camel. And in the last video of the series I will deploy the integration projects onto the OSGi container and manage it with Fuse Fabric.

The story behind the home loan demo is we have a system, that takes in XML files from different vendor's home loan application, they will place their customer input into 2 different XML files, one with it's customer data, the other with their housing details. Both files will be place into same folder either by FTP or Batch generated overnight.

First demo was to take in and process the information, first we need to separate the 2 kinds of files, because they are handled differently. And appraised the value of the house before sending it to a messaging broker.

Second demo,  was about processing the customer data, by reading the xml file and store it into a Postgres Database.

Third demo, we were added value to the house by the number of surrounding schools to the house. we referenced Google App Engine (Google Geocoding API to get the co-ordinate from Address, then use Google Places API to get number of surrounding schools) and then updated the new appraised value into   house table in Postgres Database.

In this demo, the vendors are asking for an service, that provides all the data with the updated appraised house value back to them. And they are requesting if the information can be provided through a Restful Web service. It's a piece of cake for JBoss Fuse developer to expose an service via Web Service, we simply setup and configure the endpoints. 

In this demo,  you will learn

CXFRS Endpoint

The cxfrs: component provides integration with Apache CXF for connecting to JAX-RS services hosted in CXF.

SQL Endpoint 
The sql: component allows you to work with databases using JDBC queries.



You can also find the source project in my github.
https://github.com/weimeilin79/homeloan-part3