Thursday, April 9, 2015

JBoss Fuse - Error Handling workshop

This time I want to talk about the error handling in Camel, when there is calculation, there will be exceptions. I know 99% of the time, it doesn't go the exception, but we don't want this 1% to screw up our entire application. So it is rather important to handle these exceptions carefully.

In Camel, there are three main ways you can handle exceptions,

Error Handler, any exceptions in the message will trigger this handler, so normally I'll use this for a more general base error handling.

On Exception captures the specific exceptions and can then apply different ways to redeliver messages. I often use this with Error Handler, so the on Exception handles the exception that I want to deal with specifically, and let the Error handler handles the rest of other possible errors.

Try and Catch works exactly like what is does in Java, you can use this to handle errors in any particular sections in your route.



The three error handling methods can be define in different scopes. For both Error handler and On Exception, you can define them in both Camel Context scope or Route scope.But there is a little difference, if you want to configure Error Handler to the Camel Context, you have to place your configuration outside of the camel context tag, For On Exception in the camel context scope, you place it inside the camel context tag.

All of three exception handling can be define in the route scope, for Error handler, you place the configuration outside the route tag, and then reference it in the route. For both On Exception and Try n Catch all you have to do is simply add them to your route.

So how about the routes that did not reference anything, well, it's quite simple, the route will inherit the error handler of the camel context.

So there are 5 different types of Error Handler, but here I am only going over the most commonly used ones.

First up is the DefaultErrorHandler, as it's name suggested, this handler is camel's default handler, basically what is does, is to capture the exception, log and redeliver the messages if configured, when everything fails, it'll return the message to caller, which is the very beginning of your route.
The other very commonly used on is the DeadLetterChannel handler, the major difference between this handler and the default one is that this handler moves the message to another location instead of returning it to the caller. The location can be any thing from a messaging queue, database or simple a file folder.

For on exceptions, camel will automatically finds the closest exception it can find and then apply it's redelivery policy to handle the error.
So what are the redeliver policies? Redeliver policies are sets of rules you can configure to control your redelivery methods, such as what is the maximum time you can re-send you messages, what is the gap time between each attempts? Set special patterns to each delivery also sets the log level for different situations.

Try and Catch are exactly like java, where you are going to surround the process that you want to capture the errors with a doTry tag, and each doCatch can catch one or more exception and handles them individually. Note there are several definition you can use inside catch to help you, such as onWhen, onRedeliver and retryWhile.

So here are my slides,



And this time in our lab. we are going to create a brand new project. In this project, we are receiving xml files from partner, and before we actually process the data, we are going to validate it with a xml schema. If the validation failed, I want to place the file into a garbage folder. If it's a valid xml file, I am going to send it to another route to start processing.

In side my other route, I have a process, that is going to randomly throw Exception, it has 30% chance of throwing IOException, 60% of throwing FileNotFoundExceptions and 10% chance of passing the process. For each different exception, we will handle it differently, and if all redelivery failed, we are going to send the xml to a data file for later investigation.


Here are the lab instructions.

Monday, April 6, 2015

JBoss Fuse - Tips, Offline Repository in Fuse

JBoss Fuse relies very heavy on maven, as it uses maven to control the dependencies for projects and jars it is using. Most of the repository settings can be found in this configuration file:

Under $FUSE_INSTALL_FOLDER/etc/ org.ops4j.pax.url.mvn.cfg

Inside this file, take a look at 2 configurations,

  • org.ops4j.pax.url.mvn.defaultRepositories
  • org.ops4j.pax.url.mvn.repositories
Or if you are using fabric, the same settings are available under io.fabric8.agent.properties in your  default profile.

The org.ops4j.pax.url.mvn.defaultRepositories, sets the location of the local repository for your Fuse to look up.  The default location are under the system folder.

And of course org.ops4j.pax.url.mvn.repositories are the external repository settings for your fuse to scan. Here are the list of default repos.   

  • https://repo.fusesource.com/nexus/content/groups/public/
  • http://repository.jboss.org/nexus/content/repositories/fs-public/
But there are often time where you don't have limited access to the world wide web, maybe it's just bad internet connection or security issues.

Inside Red Hat Fuse documentation, they showed you how to create a custom repository, where they showed you how to properly use the "" plugin to generate the necessary repo. I highly recommend you read through the documentations if you have time.

However a lazy person like me, I always tried to find some easy way out of configuring and setups. Just want to warn you, this may not be the proper way. But it works for me :)

First thing first, you got to have maven installed in your local machine, and that's about it! And I am assuming you have setup to connect to the Fuse and other needed repo too.

Find the setting.xml file, with

  • Mac or Unix, 
    • ~/.m2
  • Windows 
    • D:\Documents and Settings\{your-username}\.m2
  • Linux
    • /home/{your-username}/.m2/


And under the settings.xml, find the tag name:

<settings>
  <!--.....-->
  <localRepository>/path/to/local/repo</localRepository>
  <!--.....-->
</settings>

if you have a brand new maven installed, then you may have to create the xml file and add the tag by yourself.

OK, now,create a NEW temporary working folder and place the path in  "localRepository" tag. I prefer to put it in the same folder as the local system folder as later on I need to compare it using my script.

<settings>
  <!--.....-->
  <localRepository>//Desktop/jboss-fuse-6.1.1.redhat-412/tmprepo</localRepository>
  <!--.....-->
</settings>

Next, go back to your IDE, run a simple mvn clean install, so what this does, it actually starts download all the needed dependencies into the folder, by all, I mean it. So it is going to take a very long time to download.


Once you got everything downloaded, I want to quickly show you something, check your local repo in the system folder with your new temporary working folder. you will find there are lots of duplications of files. That is because we tried to download all the dependency to local, and we don't know which one has already existed, that's why it downloads them all.

Now we have the entire dependency repo, you can either point to this local repo directly in your org.ops4j.pax.url.mvn.defaultRepositories setting, or you can clean it up a bit using my script.

#create directories in the new repo that are not exist in the system folder
diff -r tmprepo/ system/ | grep "Only in tmprepo" | xargs -n2 | grep tmprepo | tr ":" "/"| tr -d " \t\r" |xargs -I{} bash -c 'if [ -d {} ]; then mkdir -p newrepo/{} ; fi '

#create directories in the new repo that are not exist in the system folder
diff -r tmprepo/ system/ | grep "Only in tmprepo" | xargs -n2 | grep tmprepo | tr ":" "/"| tr -d " \t\r" |xargs -I{} bash -c 'if [ ! -d {} ]; then mkdir -p newrepo/$(dirname {}) ; fi '

#copy all files over to the new repo that are not exist in the system folder
diff -r tmprepo/ system/ | grep "Only in tmprepo" | xargs -n2 | grep tmprepo | tr ":" "/"| tr -d " \t\r" | xargs -I{} cp -rf {} newrepo/{}

This script checks for the repo that are not in system, can copy them out to another folder called "newrepo". So I have actually clean up the files and jars that are existed in the system folder.
(Tested on Mac and Linux, please contribute the Window version, sorry, I am not a windows person :p )

So far so good, now we are ready to point the org.ops4j.pax.url.mvn.defaultRepositories to our new repo
   file:${karaf.home}/newrepo/tmprepo/@snapshots 

And remove all proxy maven look up by adding "#" in-front all of them.


That's all. For me, this is much easier then to config a bunch so set ups. But again, this is just a tip, the recommended way is to use the one in the documentations!


Thursday, April 2, 2015

JBoss Fuse - Fuse workshop 101 - Part Six

This is the last part of Fuse workshop 101, we have gone through the development part, what about in real production environment? This workshop is all about the operation and management side of JBoss Fuse. The technology we use for this purpose is called Fabric8. Fabric8 mainly takes care of  provision, automate, configure, and manage from a central location.
Since in real production environment, we will need to manage tens and hundreds of Karaf containers, we need a way to coordinate between containers and to be able to discover all the available service on them, this is where zoo keeper comes in. To discover all the service, we have a place to store all the runtime information, it's call runtime registry. And all the configuration for containers are grouped into something called profile, where you can apply the profile to sets of containers. Hawtio is the GUI for you to visualize the entire environment. 

So to provide this fabric service, we will need a server to actually run the fabric, the server running the fabric is called "Fabric Server", it provide the registry service, maintains a replicable database of information about the state of the fabric. And in order to make sure our server does not became the single point of failure, we will create multiple Fabric server to achieve high availability. It's called ensemble. 
So within every Fabric server, the runtime state of the fabric is stored, in side the run time registry you will find all the relevant information about each container. 
As well as the runtime status, Fabric also stores the configuration data in a git repository, each fabric server has the complete copy of this data. And because we are using git, all the configurations are versioned. And we combined a set of configuration into a profile, and then apply these profiles into the container we want it to run in.

Inside each Profiles, you will have static configuration files like, json, xml, properties and the links or URL of where to get your code(bundle), it also has information on what features and bundles you want to include in this profile.
Just to clarify, the program package you as the developer develops are called bundles. (It's actually a jar with some META-DATA of your endpoint). A feature is a set of bundles and also features that you combine together.
So, last thing I want to talk about before we move on to how to deploy, is the maven proxy embedded inside fabric. The proxy actually connects to 2 different set up maven repository, one is the local repo, the other is the remote one locate outside your machine. The container uses it to locate the actual program describe in the profile and download it to the container to run.
To make it absolutely clear, the way we normally deploy is

  1. Write your code and then package it into bundle
  2. Create a profile in the fabric, add the necessary feature and bundle you need for the profile, including your program
  3. In fabric, create a new container (or use the existing one)
  4. Add the profile you created to the container.
Or you can simply deploy your program using the fabric8:deploy maven plugin. It will automatically installed the profile and settings according to the configuration that is set in the pom.xml. 

There are several way to operate in fabric, using the command line using the fabric client, or through the GUI interface Hawtio. Below is a picture of what each tab takes you to in the fabric. 

The Slides are here:

This time for the lab, we are going to just deploy our camel route into the fabric. And see it running. 

And the lab instruction:



Workshop part 1 - Camel basic and components
Workshop part 2 - Data Transformation 
Workshop part 3 - Enterprise Integration Pattern
Workshop part 4 - Bean Registry
Workshop part 5 - OSGi Container

Tuesday, March 31, 2015

JBoss Fuse - House Keeping, where to download and what version of JBoss Fuse?

First of all, until today most of my post are done on JBoss Fuse 6.1.x and the version of tool that I am working on is JBoss Developer Studio 7 

Where to download JBoss Fuse?

Go to  http://www.jboss.org/products/fuse/overview/ and click on download



Until today, 2015/3/31 I am still using 6.1.x for my examples and demos. 

Where to download JBoss Developer Studio?

Go to  http://www.jboss.org/products/devstudio/download/, scroll down to download the tested version of JBDS 7.1.1 GA 

Until today, 2015/3/31 I am 7.1.1 GA version for my demos and example, I tend to use the supported and GA version for demo, to avoid any problem. 

What version of JDK?

I use JDK 1.7_xx 

What versions of Camel, CXF and A-MQ are you using?

Reference Red Hat document : 

So here is a guide telling you what versions we are actually using with Fuse. 
Basically, depend on when I did the demo, 
For demo I did with JBoss Fuse 6.1.0 (most of the case)
  • Camel  2.12.0.redhat-610379
  • CXF     2.7.0.redhat-610379
  • AMQ    5.9.0.redhat-610379 
  • Fabric8 1.0.0.redhat-379

and demo I did with JBoss Fuse 6.1.1 I use these version of camel
  • Camel  2.12.0.redhat-611412
  • CXF     2.7.0.redhat-611412
  • AMQ    5.9.0.redhat-611412
  • Fabric8 1.0.0.redhat-412
They are basically very similar.

Where are the repository?

Since JBoss Fuse rely very heavily on maven, make sure you are connect to the right repository. So the repository I use is this one. (Note, I don't normally use the tech preview or snapshot version for demos.)

       <repository>
      <id>release.fusesource.org</id>
      <name>FuseSource Release Repository</name>
      <url>http://repo.fusesource.com/nexus/content/repositories/releases</url>
      <snapshots>
        <enabled>false</enabled>
      </snapshots>
      <releases>
        <enabled>true</enabled>
      </releases>
    </repository>

If there is anything you would like to know, please leave a message.

Monday, March 30, 2015

JBoss Fuse - Fuse workshop 101 - Part Five

So far we have focused on the routing capability of Fuse, which is the wire and plug that integrate different systems together.And for now let's just put Camel aside for a second and take a look at the container our little Fuse project can run on. And some of the tools that might help you while developing.

With JBoss Fuse, you have a number of containers.

  • Standalone Camel process
  • Apache Karaf container
  • JBoss Enterprise Application Server (WildFly)

We have been running it as a single Camel process. In this workshop I want to take a look at the the OSGi container inside JBoss Fuse, which comes from the community project Apache Karaf.

OSGI break the application into multiple modules so it is easier to manage the dependency between each modules, and this nature allows it to become very lightweight without the need to load unused modules. You can make dependency base on different version. Also OSGi helps you set boundaries between modules, so you can hide packages that you don'r want to export.


Within this lightweight container, there will be something already installed for you, for example, the log module actually includes various kind of logging libraries, so you application can use without problem. Deployer that scan for new applications, and provisioning of your application via maven, files, or features... etc. It also has exposed JMX for administration and monitoring. A-MQ is bult-in in the base container, but you have a choice of removing it if needed later.


Speaking of deployment, the unit of deployment is called "Bundle", but it is actually a "jar" file. There are several way you can deploy it onto the container, either place your bundle inside the "deploy" folder(JBoss Fuse installation folder) or you can install it by using a install command and specify the maven detail to load it into the container.

    osgi:install -s mvn:org.apache.camel/camel-core/1.5.0

And after successful installation, your bundle will become a service in side this container, and this service will be visible to other service inside, and uses them if needed.

The console for the container is really powerful, here you can see the list of thing you can see via the console.


Can I use the container in my local environment for development? Yes, through the use of JBoss Developer Studio, it is very easy to debug through it's tooling. Go the the Fuse integration perspective, it will automatically detect JBoss Fuse, and here you could simple run your code on the by container by dragging the project to the bundle folder in JBoss Fuse. And see the execution result.



This is the slide for workshop 5.


Our 5th Camel Ride is all about the Karaf container, after this lab, you should be able to run your camel application on top of it!



Here are the lab instructions.



Here are the other parts of the workshops!

Workshop part 1 - Camel basic and components
Workshop part 2 - Data Transformation 
Workshop part 3 - Enterprise Integration Pattern
Workshop part 4 - Bean Registry

Wednesday, March 25, 2015

JBoss Fuse - Fuse workshop 101 - Part Four

This is probably the easiest part of the entire Camel 101 workshop.  It's very similar with Bean injection in Spring, so with the IoC in mind, in Camel we do the same thing.

First we create our POJO, In side each POJO has either your customize and the business logic, which are the most valuable part in the entire system, or just any helper code you would like to use while developing. The reason for this is to keep things easy to maintain and to have a much more light weight framework.

As soon as we got the POJO or Java Bean, we are ready to register it into the "Registry". In Camel there are few registry available.

  • Simple
  • JNDI
  • ApplicationContext
  • OSGi 

Basically, they are just different place to store your Registry, you can probably guess it by it's name. We use SimpleRegistry mainly in Testing, when you want to use Maps to store your Registry. And obviously JNDI is the default in Camel project. And if you have Spring or OSGi you can always register it there! With JBoss Fuse Project, there is no need to worry about this. We use mostly ApplicationContextRegistry. Then we can "reference" this bean from the route within each Camel Context.

The diagram below shows you how to do it.


It's as simple as that!
About the parameter you pass into the method of POJO, Camel is going to automatically map everything from the body for you, if you have not specifically specified in your reference. Camel actually use runtime look up, so don't expect any error message of misconfigure or misspell during compilation time.

The slide for the workshop are here:


In this exercise, we continue on building the stock trading system, putting real business code that calculate how much money of this person that needs to deduct from this stock trade. As well as that, to speed up the process for VIPs, the vendor decide to introduce Messaging Queue to the game. We now need to deliver the trading into via messaging.


 
Lab instructions are here:



Here are the other parts of the workshops!
Workshop part 1 - Camel basic and components
Workshop part 2 - Data Transformation 
Workshop part 3 - Enterprise Integration Pattern

Monday, March 23, 2015

JBoss Fuse - Fuse workshop 101 - Part Three

It's really difficult to design or architect an Integration project, especially within an Enterprise, there are so many factor to take into considerations.
  • System characteristic
  • Data mapping
  • Interface of systems
  • Risk management
  • Security 
  • And even office politics comes into play, I have to say this is the most time consuming and can quickly become nightmares be not deal carefully. 

This quickly became a headache for architects, and people constantly ask for best practice, I have to say, there are no single best practice, because it is not an ONE SIZE FIT ALL answers, integrating system is all about problem solving. 

So how do we solve an complex integration problem? YUP! you are right! break it down to smaller problem and then tackle them one by one! And fortunately this book "Enterprise Integration Patterns" by Gregor Hohpe has most of the solutions. This book "provides a consistent vocabulary and visual notation to describe large-scale integration solutions across many implementation technologies". 

To put it in my words, it's a collections of patterns, each pattern is to solve a design problem, so by implementing the patterns to the pieces of problem you have already broken down and put the patterns together will give you an idea of where to start from. Of course for every pattern it has it's pros and cons and many things to consider. That why using Camel as the tool that supports EIP will allow you to modify or refine your design with a few easy modification steps! 
For this workshop, we talk about Enterprise Integration Patterns, I will skip all the parts that talks about individual pattern, seriously, it's going to take too long, and there are so many resource on the internet that talks about them. I would recommend you visit the official EIP site to learn what each and every pattern does, and maybe read the book if you have time! It's really helpful. 

Let's look at this Ordering system. Shops from different location sending in order data, it needs to go into both inventory system to check for product availability and accounting system for finance check up. Then the inventory sends the product ready for shipment. At the same time it sends a confirmation email to the customer along with some advertisement and coupons. 


OK, so here is the problem, how do I make sure inventory and accounting system gets the message as soon as possible? what if there is something wrong with customer's payment? How do I make sure the shipments are shipped efficiently? And what about the notification system, how do I deliver it? 

Here are my thoughts on the solution, 
To make sure both inventory and accounting system receives the message at the same time, we will use the "publish and subscribe" pattern, so a copy of the message are send to all receivers. so the inventory and start the inventory check and the shipping process immediately and accounting system can start process it's finance and billing process.


If the payment ever goes wrong, a message will be send to a "invalid message channel", that will be pickup by the inventory system and cancel the shipment.  

The shipment message should be sorted according to different delivery priority and area to delivery and "aggregate" into a bigger chuck so it's easier for delivery truck to divide it's duty, and the content send to delivery system should be "filtered" so it only contains necessary data. As for notification, we also want to make sure we send it to the notification system at the same time, but send an cancelation letter or a successful letter in our "Recipient List". And as well as that, we will also "enrich" the content with coupon and the advertisement that might interest the customer. 


OK, what happen if the online shopping team comes in, and they would also like to do the same thing, but their ordering data comes in with a totally different format, so first, we will certainly reuse what we have ready done. (There is a fine line between reusing code and rewriting it all over, but here we assume the process behind is exactly the same). So by "transforming" the data the same as shops, then publish the content to the existing channel will certainly saves you lots of time!  


Slides for the workshop:

As for lab, continue from last one, this time we are going to separate the processing route for VIP and normal stock purchase. Also we want to introduce another vendor that trade with our system but with orders in one large file, we need to spilt the content before reusing our existing route! 


Lab Instructions:


Here are the other parts of the workshops!
Workshop part 1 - Camel basic and components
Workshop part 2 - Data Transformation