Monday, August 31, 2015

Red Hat JBoss Fuse - Getting Started Home Loan Demo version 6.2 Part7

Finally the last part of my getting started demo for JBoss FUSE 6.2, an update from my old demo project.  For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There is a series of blogs/videos, you can find the previous post here:
First part of demo was to take in and process the information, first we separates the 2 kinds of files, because they are handled differently. And send it to separate messaging broker.
Second demo was about processing the customer data, by reading the xml file and store it into a existing database table.
Third part starts to process housing data by getting number of schools surrounding the house address via SAAS. Forth part does the actual housing value appraisal and store it into database.
Fifth part demo, was the last part of developing this application, which as to add a Restful Web service, providing all the housing appraisal result combining the customer details. 

That was the end of the development, last blog, we took a look at running a local standalone camel route, we have also tested by running the latest JBoss Fuse 6.2 development debugging tools in JBoss Studio, 

At last, the project is now ready to be deployed into a running environment. By using the fabric8 plugin in JBoss Fuse, we are going to startup JBoss Fuse, and deploy our project onto the OSGi container. Since we are breaking the application into microservice and deploy them onto separate distributed running instance(container), we probably need a power tool to ease the work and effort we need to put in when provisioning and maintaining the application. Fuse Fabric make the perfect tooling for it. It provides a visual tooling for you to see what is actually going on, and underneath the GUI, it allows service to be registry and then discover by other, does load balancing and provisioning options.


Video:

In this demo, you will learn, 

Deploy your Camel Project via Fabric8 plugin 
This plugin makes it easy to create or update a fabric profile from your maven project.
When you deploy your project to a fabric profile with this plugin the following takes place:
  • uploads any artifacts into the fabric's maven repository
  • lazily creates the fabric profile or version you specify
  • adds/updates the maven project artifact into the profile configuration
  • adds any additional parent profile, bundles or features to the profile.
Create a new container in JBoss Fuse
The Fabric in JBoss Fuse makes it really easy to provision, automate, configure, and manage from a central location

To run it, you can find the source project in my github.
https://github.com/jbossdemocentral/jboss-fuse-homeloan

Tuesday, August 25, 2015

Red Hat JBoss Fuse - Getting Started Home Loan Demo version 6.2 Part 6

This is the 6th part of my getting started demo for JBoss FUSE 6.2, an update from my old demo project.  For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There is a series of blogs/videos, you can find the previous post here:


First part of demo was to take in and process the information, first we separates the 2 kinds of files, because they are handled differently. And send it to separate messaging broker.
Second demo was about processing the customer data, by reading the xml file and store it into a existing database table.
Third part starts to process housing data by getting number of schools surrounding the house address via SAAS. Forth part does the actual housing value appraisal and store it into database.
Fifth part demo, was the last part of developing this application, which as to add a Restful Web service, providing all the housing appraisal result combining the customer details. 

It's time we start running this application and see how it works, first we are going to start up JBoss Fuse, as our example needs to have a running message broker to deliver data between each distributed microservices. And new in JBoss Fuse 6.2, it has introduce this very helpful and nice ability in the development tool to allow user to do debugging while developing this route. While running the debugging mode, we can configure which node we would like to place the breakpoint, so when the message reach the breakpoint the route will pause to allow user to see how the application is doing,  you will be cable to see the running thread, value of the Camel header, body, this is particular useful, because we will have better idea of how the message has been changed by each processor. There is an option to set condition breakpoint which it will only pause then the criteria it meet. 

Please see the video for more details about the debugging tools.


To run it, you can find the source project in my github.
https://github.com/jbossdemocentral/jboss-fuse-homeloan

Tuesday, August 18, 2015

Red Hat JBoss Fuse - Getting Started Home Loan Demo version 6.2 Part 5

This is the 5th part of my getting started demo for JBoss FUSE 6.2, an update from my old demo project.  For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There is a series of blogs/videos, you can find the previous post here:

The story behind the home loan demo is to prepare housing appraisals base on the content given real estate vendor. The appraisal needs to talk to local database, external google map API and also provide information through a web Restful API. We have divided this project into 4 independent integration applications, deploy in a distrusted environment, each handle single functions, and using message broker as the asynchronous medium to pass the message and data around.

First part of demo was to take in and process the information, first we separates the 2 kinds of files, because they are handled differently. And send it to separate messaging broker.
Second demo was about processing the customer data, by reading the xml file and store it into a existing database table.
Third part starts to process housing data by getting number of schools surrounding the house address via SAAS. Forth part does the actual housing value appraisal and store it into database.

Fifth demo, the vendors are asking for an service that provides all the data with the updated appraised house value back to them. And they are requesting if the information can be provided through a Restful Web service. It's a piece of cake for JBoss Fuse developer to expose an service via Web Service, we simply setup and configure the endpoints. Previously we need to state the interface with Java interfaces or classes, but with the latest new Rest DSL in Camel, we can simply set the endpoint through simple XML setting.


You will find detail instruction in the video.


In this demo,  you will learn:

Rest DSL to expose Restful Endpoint services, map the Restful service using Camel-Servlet. 
Allow end users to define REST services using a REST style, it builds Rest endpoints in Camel routes. But the actual REST transport is supported by components that has native REST integration

<rest path="/summaryservice">
    <get uri="/nationalID/{clientId}">
      <to uri="direct:customerSummary"/>
    </get>
</rest>

<restConfiguration component="servlet" contextPath="/homeloan" host="localhost" port="9999" bindingMode="json" >
  <dataFormatProperty key="prettyPrint" value="true"/>

</restConfiguration>

SQL Endpoint 
The sql: component allows you to work with databases using JDBC queries.

You can also find the source project in my github.
https://github.com/jbossdemocentral/jboss-fuse-homeloan


Friday, August 14, 2015

Red Hat JBoss Fuse - When size and time does matter

It is very so often in the integration space we need to deal with large amount of data. When designing the integration solution, we really need to stop and take a good look at how to deal with these data.
You may find yourself have to handle large data in the following situations,

  • Incoming data 
  • Processing Data
  • Providing output 

From my experience, when having large amount incoming data, for me it means the data comes in with very high frequency, as well as high volume of messages, the risk of having too much content flooding our application is high, my approach is restricting the data coming into the application, so it runs the maximum capacity but at the same time avoid jamming the system. In this case I will

  • Try to use Polling Consumer if possible, there are many components in JBoss Fuse support polling mechanism. Such as File, FTP, JPA, Quartz2.. etc, it supports configuring how frequently the polling should be. 

  • If no polling consumer available or the data are just coming in too fast, try EIP, the Throttler Pattern, not only it can set the frequency of polling it will also allow you to control how many messages goes in per poll. 

When applying the above pattern, just make sure the place you receiving the input is large enough to temporary hold the incoming data, try to extend the capacity of the service either by have more service to share the load or simply allocate more hardware resources. 



Another situation when it comes to large amount of data is actually handing the large message, because of the way JBoss Fuse process the messages, it loads the message into memories, when the messages gets too big, it will soon run into out of memory problem. So, when dealing with one large chunk of data my approach will be

  • Splitting your content, when dealing with large files, it is always best to split it into smaller chunks if possible, for many reasons, avoid large memory consumptions at once, by splitting, we can even process these smaller chunks in parallel, instead of processing one part after the other. 

  • Enable streaming, when this is turned on, instead of holding the entire message in memory it sends big streams to file, you can even configure the StreamCachingStrategy to customized the size, location, buffer size, memory limits, etc. 

  • Filter the content, it is often the case, with large data, not all part of the file is needed for further processing, at the same time the original message is needed later, I would then use the Claim Check pattern to first filter the data send, and then retrieve the original data when needed.

Last but not least, providing large amount of data to broad audiences, clients. We try do least data sending as possible, the most sensible way then is to place a buffer in-between the client and the output procurer.

  • Publish and subscribe(Topic) in messaging, this is probably the first scenario that comes into my mind, but it guarantee the subscriber to get the message as for the producer , it only have to write it once. 

  • Caching medium, when the messages needs to be repetitively read by client, then placing the content into a caching medium such as database or even faster ones like memory caches, is  better then messaging, as messages will be gone as soon as all the recipients receive the data.


As you can see, there are many possible way to handle large data in JBoss Fuse, because of it's flexibility, you have the freedom of choosing the perfect strategy for different situation, there are many more options and combinations we can do, what is your approach when dealing with large data? I am curious of all the genius way people solve their problem, let me know!

Tuesday, August 11, 2015

Red Hat JBoss Fuse - Getting Started Home Loan Demo version 6.2 Part 3

This is the 3rd part of my getting started demo for JBoss FUSE 6.2, an update from my old demo project.  For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There is a series of blogs/videos, you can find the previous post here:

  • Part one - File connector and simple EIP
  • Part two - Persisting Data to Database

The story behind the home loan demo is to prepare housing appraisals base on the content given real estate vendor. The appraisal needs to talk to local database, external google map API and also provide information through a web Restful API. We have divided this project into 4 independent integration applications, deploy in a distrusted environment, each handle single functions, and using message broker as the asynchronous medium to pass the message and data around.

First part of demo was to take in and process the information, first we separates the 2 kinds of files, because they are handled differently. And send it to separate messaging broker.
Second demo was about processing the customer data, by reading the xml file and store it into a existing database table.

This part of the demo start really appraising the value of the house, base on the address and information given by the vendor, one of the important factor is the number of schools surrounding the house, to get this information, we are going to integrate with google app engine, by getting the geolocation of the address and then query the surrounding schools base on the latitude and longitude.



Video,

In this demo,  you will learn

Endpoint reading from activemq messaging broker
The ActiveMQ component allows messages to be sent to a JMS Queue or Topic; or messages to be consumed from a JMS Queue or Topic using Apache ActiveMQ.

  <bean id="activemq" class="org.apache.activemq.camel.component.ActiveMQComponent">
      <property name="brokerURL" value="tcp://localhost:61616"/>
      <property name="userName" value="admin"/>
      <property name="password" value="admin"/>

  </bean>

Properties Place holder
Placing data into properties, to set base on different configurations.
  <propertyPlaceholder location="classpath:sql.properties,googleapp.properties" id="properties"/>

RecipientList Endpoint with dynamic Restful URL 
Which allows you to route messages to a number of dynamically specified recipients.



Http4 Endpoint
The http4: component provides HTTP based endpoints for calling external HTTP resources

That's all for this demo!

Friday, August 7, 2015

Red Hat JBoss Fuse - Managing microservice architecture in JBoss Fuse

Moving away from a monolithic application towards microservice architectures, that means the application goes from a centralized big bulk of package deploy on single instance, to small modularized chunk deploy on multiple distributed environment. As mircoservice brings you the advantages of easier to maintain the code, and allowing more developer to work simultaneously, but it also comes with additional problems like a more complex environment, and become difficult to manage.

It will be very irritating to check every single instances, so it's important to have a centralized view of the entire environment, and have that summarized. In JBoss Fuse, we uses something called Insight, which consolidate view on both the log and Camel events, so you have better idea of what is going on in your system in a broader view. By default JBoss Fuse collects data into Easticsearch. It is a is a powerful search engine, which handles distributed datasource efficiently with Restful interface open for query. By taking advantage of the fast search ability, we can store and load the log and camel metrics content. Have data is not enough, without a proper display view, it is also just a bunch of text, and numbers. In JBoss Fuse, we uses Kibana is the search dashboard for Elasticsearch, which analyze the data, display the query result. 


Enabling the Insight is pretty easy, First of all, we need to create a few container to act as the distributed datastore for the instance, please locate a large amount of memories for these instance, as the algorithm requires loading indexes in memory for calculations. I am creating more then one datasource instance because I would like to have HA clusters for these data. 


And add the Insight console (Kibana) profile to the root container, also add the log and camel metrics profile too to also collect data from root. 


Once the profile is installed, you will find the Insight view on the top lefthand corner, click on it, 


Here are the dashboard consolidated view on log and camel metrics of the entire fabric, for the managements to see. 



And under Elasticsearch tab, it shows how the data are distributed

To run an example, I am going to create 2 running instances (containers), deploying a basic camel example writing logs in every 5 seconds. To start collecting data, please also add the insight-log and insight-camel to the container. 




Once all container started, 

Go back to the Insight view and you will see under Logs tab, all the logs in the fabric are collected. 

And the camel events are picked up as well. 


By having this centralized view, we can see the distributed instance in one single view, making it easy to manage just like what we used to do in monolithic application.

Tuesday, August 4, 2015

Red Hat JBoss Fuse - Getting Started Home Loan Demo version 6.2 Part 4

This is the 4th part of my getting started demo for JBoss FUSE 6.2, an update from my old demo project.  For people that are getting started to know Fuse, and wants to get their hands dirty and develop a JBoss Fuse project.

There is a series of blogs/videos, you can find the previous post here:

The story behind the home loan demo is to prepare housing appraisals base on the content given real estate vendor. The appraisal needs to talk to local database, external google map API and also provide information through a web Restful API. We have divided this project into 4 independent integration applications, deploy in a distrusted environment, each handle single functions, and using message broker as the asynchronous medium to pass the message and data around.

First part of demo was to take in and process the information, first we separates the 2 kinds of files, because they are handled differently. And send it to separate messaging broker.
Second demo was about processing the customer data, by reading the xml file and store it into a existing database table.
Third part starts to process housing data by getting number of schools surrounding the house address via SAAS.

Since we have the surrounding schools, we can now start to appraise the value of the house by the information provided, such as number of rooms, number of bathrooms, and even the size of the property. This business rule is built in a Java POJO, by calling the POJO with the data, it then calculate the actual value and store it into Database.


Here is the video,


In this demo,  you will learn

SQL Endpoint 
The sql component allows you to work with databases using JDBC queries. Also setting datasource for the sql component

<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close">
  <property name="driverClassName" value="org.h2.Driver"/>
  <property name="url" value="jdbc:h2:file:~/h2/homeloan;AUTO_SERVER=TRUE"/>
  <property name="username" value="sa"/>
  <property name="password" value=""/>
</bean>
 
<!-- configure the Camel SQL component to use the JDBC data source -->
<bean id="sql" class="org.apache.camel.component.sql.SqlComponent">
  <property name="dataSource" ref="dataSource"/>

</bean>

Properties Place holder
Placing data into properties, to set base on different configurations.
  <propertyPlaceholder location="classpath:sql.properties,googleapp.properties" id="properties"/>

Reference Java POJO bean in camel context
<bean id="Appraiser" class="org.blogdemo.homeloan.processor.AppraisalProcessor" />

Data Mapping tools with JAXB transformation
Translate one data format(XML) into another(Pojo) using JAXB




You can also find the source project in my github.
https://github.com/jbossdemocentral/jboss-fuse-homeloan