Message queuing

Message queuing is an often forgotten technique to provide asynchronous communication between systems or services where the server and client do not need to interact with the message at the same time. MSMQ is Microsoft’s message queue implementation.

Some of the advantages of message queuing are:

  • Load leveling Most systems do not have an even load. If you design your system for peak load, you are wasting resources. If you design your system for average load, you will not be able to handle peak load. When using message queuing the calls get queued up and the service can process them in its own pace.
  • Asynchronous communication The server and the client do not need to interact with the message at the same time. The client does not need to wait for the server to process the message.
  • Reliability A client may need to work against a service even when it is disconnected. By utilizing MSMQ the client can queue up messages to a local queue and then send them to the service when it is connected.

A MSMQ queue can be configured to be either a transactional queue or a non-transactional queue. Messages sent to a transactional queue are transferred exactly once.

There are typically three transactions involved in a message transfer (if the client and the server are not on the same machine).

 

 

Client transaction The client transaction takes place when the client pushes a message to the client queue. If the client commits the transaction, the message is pushed to the queue. If the client aborts the transaction the message is rejected from the queue.

Delivery transaction MSMQ is responsible for delivering the message from the client queue to the service queue. If the network is down, or the service machine has crashed, message removal from the client queue is rolled back. MSMQ will eventually deliver the message when the service is up.

Destination transaction The service tries to remove the message from the service queue. If the transaction aborts, message removal is rolled back. If the transaction is committed, the message is removed from the queue.

SOA’izing the message queue Working directly with MSMQ is hard. The developer must handle transaction commits and roll backs explicitly. Retry functionality must be implemented on the client and on the service. The messages must be hand-crafted and throttling behavior must be implemented manually. In the SOA world (Service Oriented Architecture) we are used to not think in terms of messages on a queue, but transport neutral business operations.

WCF to the rescue WCF can act as an abstraction layer on top of MSMQ to shield you from many of the pains of using MSMQ directly. In the SOA world we are used to communicate via contracts, often exposed as WSDL documents. The service contract tells the client how it is supposed to communicate with the service, how transactions are implemented, which security settings should be used, throttling behavior etc. The data contract tells the client what data needs to be sent. A proxy is generated on the client side based on the contracts. The proxy does all the heavy lifting of converting WCF-messages to MSMQ messages, enlisting in transactions etc. There is not a direct mapping of WCF messages to MSMQ messages. The client posts messages to a queue, not a service which means that all calls are asynchronous and disconnected.

 

Another important aspect is throttling. If there are 15 messages on the queue the service may not want to have 15 concurrent instances and worker threads. The throttling behavior in WCF controls how many instances and sessions can be handled at once. Maxed-out messages stay in the queue.

This has been a brief introduction to what the benefits of message queuing is and how WCF makes life easier when it comes to message queuing. For a more technical reference on leveraging MSMQ in WCF I can really recommend Programming WCF Services by Juval Löwy.

Posted in: •Integration  | Tagged: •Message Queuing  •MSMQ  •WCF 


Talking the right language

When should you talk about TOGAF or any other architectural framework with your business representatives? The answer is NEVER!

Have you ever had the feeling at a party or any social gathering that people starts to talk about their jobs and what they doing? I have, and it is so strange; because when the nurse talks about their job everyone is excited and interested to listen, same goes for the fireman, carpenter and taxi driver. Everyone can relate to what these people do and have a nice discussion which all can relate to.

But when it comes to you and your story is IT they all stop talking and you never get that second question. People just go back to the nurse, carpenter, doctor and continue to talk about just how important and interesting their job is…

What I just mentioned happened to me a few weeks back when I joined a fantastic initiative called Rosa Landet (http://rosalandet.se). This was a group of ~10 people that started from the upper north of Sweden and bicycled down to Skåne in the very south, a ride for about 2500 km all this to collect money for the breast cancer foundation in Sweden. I joined the group for one stage between Falun and Uppsala. During the trip which lasted about 9 hours there were a lot of interesting discussions going on and most of the people had jobs like fireman, doctors, salesmen or electricians, anyway jobs that people understand what they actually do; Except me! So when we were biking and talked I was very keen on learning more about how a typical working day looks for a fireman, of course there were loads of questions and the women whom I asked answered enthusiastic so we had a really nice conversation. After about an hour we started to talk about me and of course after a while she asked what my occupation is. I answered IT and that was pretty much the end of the discussion…

Today most people love what IT can do for them but they really don’t care about how the apps and information really got there. This is a mystery for us IT-people who just can’t stop ourselves from thinking about how things are integrated and what framework is used to get the pieces in place. But for normal people this should just happen by magic. Sad for us IT-people, but true. It isn’t that strange though, look how much you care about how your car gets fixed, you don’t; isn’t it so. So why should you load details into your customer?

As an example when you leave the car at the garage. You just don’t care what tools your mechanics are using you just care about that your car is returned in better condition than it was when you left it there. If you feel that the relation with the price you paid for fixing your car and the condition the car is in now, then you probably will give the garage your trust next time your car breaks down again (because it will for sure).

This is exactly what you should do next time you meet your business colleagues! Take the opportunity to understand the business challenge and do not expose your internal challenges. You would probably prior to the meeting have safeguarded that your IT is ready to meet the new challenge and now it is time to show that your efforts are well worth the money spent because you are ready to implement what your business requires This is exactly what your car mechanic does prior to a car brand releases a new model. He prepares himself for the new stuff coming.

Like with the car engine that needs oil; TOGAF is your architectural engines lubricant which gets things going smoothly and without too much friction. Like lubrication this should not leak outside the engine because when it does then there is an indication that something is wrong. If you as a driver gets a warning signal it puts you in a state of insecure. This was probably not your intention with exposing your architectural tools but it is a natural reaction if someone don´t understand your language. Probably your business representative will try to get a second opinion from someone that seems to know what he is doing and then you are suddenly not the captain of the IT decision anymore.

For me as an architect it is always good to compare myself with other jobs that people just don’t care how it works, think about the people fixing your car. You just want it to work! And as an advice from a friend; Do not ever compare yourself with the fireman because people do not see you as the savior from the fire, they see you as the representative that make things work.

My personal lesson learned from this story is that next time I am at a party and someone asks me what I am working with, I will tell a lie. My answer will be that I am a fireman but I am soon out of business because I am doing such a good proactive job so I don’t have any fires to chase any longer. Hopefully I will get a second question and then I will find something interest to talk about. Bicycling maybe :-)

 iBiz Solutions Integration Framework gives you a helping direction in your dialogue with the business and also it directs you on how to build your integration landscape.

Posted in: •Integration  | Tagged: •Integration Framework  •Rosa Landet  •TOGAF 


A shout out from the rookie

My name is Robin Ericsson and I have been at iBiz Solutions for about two months. I am currently “in training” to become a part of the iBiz Solutions integration team.

During my first time at iBiz Solutions my focus has been to learn to work with and understand system integration with an emphasis on Microsoft BizTalk Server. I have had a great help from internal courses, course material and exercises that we have in the company. By doing the exercises and reading the material I have gotten a basic understanding and knowledge in the subjects.

As I am pretty much fresh out of school, I have been wondering about why there are not more courses at universities that focus on system integration or widely used system integration solutions such as Microsoft BizTalk Server or Tibco. I did some searches at antagning.se (a portal for university studies in Sweden) for “system integration”, “systemintegration”, “Microsoft BizTalk Server” and “Tibco”. What I could find was there are some courses that take up system integration as small part of a course but there is only one course that focuses totally on system integration and that is being held at Karlstad University.

This is why I want to do a shout out to all of the universities in Sweden that have courses and/or programs in IT to consider adding a course in System Integration, as we all know it is one of the most important aspects in IT today and I think that students would have great use of knowledge on the subject.

Posted in: •Integration  | Tagged: •BizTalk  •Education  •System Integration  •Tibco 


BizTalk is not dead!

On October 3rd 2012 I held an informational event on BUGS in the premises of Informator in Stockholm about running BizTalk 2010 R2 (or whatever the name in the end will turn out to be)in Windows Azure. Great news for us professional system integrators who have had over 10 years of pleasure working with BizTalk. The fact that Microsoft allows us to seamlessly move our integration engine from on-premise to the cloud (Microsoft BizTalk 2010 R2 comes with license mobility) will increase availability as most local IT-departments simply do not have the economics, power, resources or the organization to match those of Microsoft. Microsoft will have better SLAs than most other. BizTalk will now run as an infrastructural component directly in the cloud (IaaS). Microsoft is also providing the new EAI and EDI functionality which is based on long established BizTalk patterns. The latter provides the platform as a service (PaaS).

I find it especially nice that Microsoft allows us to setup our own private networks directly in the cloud and to top that they also provides us with the infrastructural configurations to easily connect the on-premise network and the cloud network through a static VPN tunnel. This means that we can move our BizTalks to a more reliable hardware platform and pay as you go. These machines in the cloud are of course easily accessible and manageable from everywhere. BizTalk machines in the cloud can use this VPN-bridge connect to LOB-systems on premise. So there is no reason to wait any longer, you can have your development and test environments up in only a few hours. Production environments should of course wait until final release, which probably lies within 6 months from now.

Microsoft has not yet announced what BizTalk 2010 R2 running in the cloud will cost but indications are that the established path with no costs up front and pay as you go will be the model.

If you found this article interesting and want to read more, please download my full presentation from BUGS

 

https://windows.azure.com/

http://msdn.microsoft.com/en-us/library/windowsazure/hh689864.aspx

http://video.ch9.ms/teched/2012/eu/AZR207.pptx  

http://video.ch9.ms/teched/2012/na/AZR211.pptx

http://www.informator.se

http://www.ibiz-solutions.se

http://biztalkusergroup.se/

Posted in: •Integration  | Tagged: •ABizTalk  •BUGS  •IaaS  •Paas 


And the techniques, they are changing…

Are you having a hard time keeping up with the pace on the introduction of the cloud, users that want to go mobile, project sites that have stopped using your traditional databases and moved out to the cloud? Too bad for you, but if it is of any comfort, you are not alone; and there is a way to get out of there.

If you ought to summarize the last 10 years in IT you would probably, like myself, have somewhere on your top 5 list the word change. Changes are of course something that is not IT unique. We see it in every business where lifecycles of products is getting shorter which makes the time we have to develop products shorter. We seem to have a never ending demand of experiencing new emotions driven by mobile phones or any kind of new devices and accessories. Change is a good thing and without change life would of course be boring and probably most of us wouldn’t have a job either. With Change constantly happening like in IT and you never really have time to finish what you are currently up to before a new thing is on the arise. Maybe you start to dream about that the evolution of IT should stop for at least a couple of years so that we get the time we so well deserve to clean up what we have today and start all over again. With all the wisdom we have collected during the “wild years”. Unfortunately this could only happen in our dreams…

But maybe there is another way to get out of the feeling of chasing techniques and never really reach the finish line until the next version is knocking at the door. Instead of setting focus on the devices or whatever the new hot thing is, you should change focus and aim for a goal that you can reach with a reasonable amount of investment.

Think about it; How many different types of E-commerce solutions, PC’s, mobile devices have been around for the last decade? I have lost count and a few years ago my webhosting company sold me application hosting and now they are suddenly in the cloud; I don’t know how that happened there! But think again and remind yourself what have happened to the information? An order, customer or a product look pretty much the same as it did 10 years ago from an Information perspective. You would probably say that not very much have happened there, a few more attributes on the product, maybe geo position is added to your customer if you are a company in time and working with location analytics. But it isn’t that dramatic change in Information compared to what my mobile phone can do today and what it did 10 years ago.

This is really good news for all of us because if we start to think that information matters and information is moving in a pace so that we can actually follow the changes and maybe even be a bit proactive. Then we can suddenly feel a little bit better and start to think that there is at least one area that I am in control of. So now we are in control! Think a little bit further and create an imaginary world where you also have connected the enterprise business entities into well-defined and communicated interfaces and services. Then yet again the world looks better because in that imaginary world you have built a foundation where you can meet the evolution in devices and technique by looking at interfaces and information flows rather than fixing thing based on project demands and your services interfaces will most probably live much longer than the device that is currently consuming it. If we are honest to ourselves, not so much have happened the last 15 years when we are looking at integration. Of course the possibilities increased dramatically with the introduction of XML at the end of the 90’s and with the use of integration middleware like Tibco, BizTalk and others. But those who did focus on Information and integration in the beginning of 2000 have been well prepared to support the technical evolution during this time. Because what happened from an integration perspective is that we went from XML-RPC to SOAP and for the last years we see REST-services and JSON-structured data being very popular especially on the web for mobile devices. But if you are on a modern integration platform it doesn’t really care if the information is presented in JSON or Xml or any other formats for that sake. It is still the same old information.  

Figure 1 Information is the most stable occurrence you have. By working with information close to integration and business driven interfaces you will be better prepared to meet changes

So what do we do then? In a world in a rapid change you need to have a plan! We know that we cannot stop the world from evolving so you will not get very far by trying to stop Google, Apple and Salesforce from providing us with new exciting devices and services. We need to put focus on something that is under our control. We know that we at least can affect our own organization even if it takes a little longer time in some organizations due to size or other factors but it is for sure possible. By finding the right ambassadors and by starting to implement the new thinking of information and integration and how it can make your IT productive and give great value to your organization you will see that you will sleep better at night. Find the low hanging fruit in your company to gain trust and qualify the plan we just worked out. By finding the low hanging fruit that delivers value you will be lifted by success stories rather than pontifications and enforcing standards that cannot be linked by the business to business opportunities. Just make sure your first low hanging fruit is achievable…

By working information and integration closely together you will be better prepared when your colleagues wants to go mobile or move a module out to the cloud because you will now work with identifying what need of information the piece of functionality that you are moving requires and map the information need to your Enterprise services.

iBiz Solutions Integration Framework provides a methodology and framework that gives you a jump start to be more prepared for a world in a rapid change. Integration Framework addresses both architecture and tools on how your information integrates into your IT-landscape. It also gives support for strategic planning for coming opportunities with city planning that makes it easier to manage the information flows in your organization. Important aspects as the governance of SOA-services and how integration patterns links to information in your organization are also covered by Integration Framework.

So let’s start to meet the techniques that are a changing with a stable foundation built on Information and integration and a plan to put them together with the help of Integration Framework.

Posted in: •Integration  | Tagged: •Enterprise Architecture  •Integration Framework 


iBiz Solutions, one of very few Super Companies in Sweden 2012!

PAR, has in association with Veckans Affärer just announced the Super Companies of 2012, and this year, only 317 of ALL Swedish companies with a turnover of 10 million SEK (48000st) handled the fierce requirements of becoming a Super Company. Only 25 companies in the IT and Telecom business passed the eye of the needle, and iBiz Solutions is one of these!

Super Companies - PAR in cooperation with Veckans Affärer

Super Companies is a partnership between PAR and Veckans Affärer, where PAR analysts have developed the model for identifying the Swedish Super Companies. The selection and ranking of becoming a Super Company is based on the Swedish companies’ economic performance over the past four years. The model takes into account and consider the individual companies:

• Growth
• Earnings
• Return
• Efficiency
• Capital Structure
• Financing

I’m very happy and very proud that we received the award as Super Company 2012 says Allan Hallén, CEO at iBiz Solutions. I think it shows ourselves, our customers and the market in big that we are doing a lot correct now, have done so for a long time and we have had a clear focus. We have a clear and long-term goal towards becoming No. 1 in the Nordic countries in the integration area, and is already one of the very best. We have good cooperation with many large, well-known and exciting clients in the Nordic region, where it undoubtedly shows that we are well qualified and are at the forefront in the field. Of course there are many determining factors, and this must surely include our highly competent and skilled staff that contributes greatly to customers to create what many aspire to - to establish a manageable integration. Both we and our customers gets great value and effect of our concepts and tools that we over time have developed and refined - we make a difference, continues Allan.

I know from previous years that the VA’s assessment criteria are rock hard, and the eye of the needle is extremely small to meet the requirements, so it is especially exciting today. Our ambition is to be a Super Company for employees, customers and partners today and in the future.

Posted in: •About iBiz Solutions  | Tagged: •2012  •Super Company 


iBiz Solutions is sponsoring Karlstad Championships in programming organized by the Department of Computer Science at Karlstad University October 6.

“For us it is very important to sponsor this type of creative, fun and challenging student competitions. The tasks has historically been quite challenging and it’s very nice that the students in Karlstad get the chance to try this locally in Karlstad. We wish the contestants good luck,” says Caj Rollny, Professional Services Manager at iBiz Solutions.

The contest, which is a part of the Nordic Championships in programming, takes place in the house “Vanern” Saturday October 6 2012.

A team of up to three people will solve 8-12 programming problems in five hours. Each team has access to one of the university’s computers to do the programming on. The tasks is presented to the student when the race starts and they should be resolved by team members without the help of outsiders. Each program is then tested by a judge who approves or return the task. After the competition a list of results is compiled, and the team that solved the most tasks correctly wins.

The purpose is to encourage interest in programming by giving participants a chance to compete with contestants from Karlstad and other universities in the country, for example Lulea, KTH, Chalmers and o, which all run the contest the same day and time. In addition to the honor, the top three teams get prizes sponsored by iBiz Solutions.

Read more about the contest: http://www.cs.kau.se/ncpc12/

Posted in: •About iBiz Solutions  | Tagged: •iBiz Solutions  •Karlstad  •Nordic Championships in programming  •sponsor 


MSMQ between BizTalk and Legacy Systems

In this post I will discuss a problem encountered a while ago when I was responsible for developing a Microsoft .NET Class Library (DLL) that was going to be used by .NET applications handling messages between BizTalk and old Legacy systems. The .NET applications retrieves information from and store information into the Legacy system and communicates with BizTalk via MSMQ queues. Picture 1 shows the conceptual solution where messages are sent between BizTalk and an old Legacy system.

Picture 1. Messages sent between BizTalk and Legacy system via MSMQ queues.

At first it was straight forward programming using standard .NET Message Queuing classes from_ System.Messaging_ namespace. The code block below  demonstrates the classes used.

using System.Messaging;
…
MessageQueue queue = new MessageQueue();
Message message = new Message(“This is a message!”);
queue.Send(message, ”Message Label”);
…

Problem encountered When it was time for Integration Tests a problem was encountered. The BizTalk MSMQ Adapters were not able to interpret the messages sent by  .NET applications and vice versa.

When sending or receiving messages using the MessageQueue class something called a Message Formatter is used. A Message Formatter is a class implementing the IMessageFormatter interface and it determines how the messages are serialized/deserialized into the body of the message when they are sent/received. The System.Messaging namespace contains the following Message Formatters:

1. ActiveXMessageFormatter
2. BinaryMessageFormatter
3. XmlMessageFormatter

If no Message Formatter is explicitly used when calling Send/Receive on MessageQueue the  XmlMessageFormatter will be chosen by default.

Why is this a problem?
The XmlMessageFormatter wraps information around the actual data being sent and expects certain information wrapped around received data. The ActiveXMessageFormatter and BinaryMessageFormatter serialize the data into binary representation.

Since there is no way to configure BizTalk MSMQ Adapter to use a Message Formatter it is not possible to configure the BizTalk MSMQ Adapter to be compatible with the .NET Applications. The problem had to be solved in the .NET Class Library since it was desired to solve it in one place and it should work for both directions. A possible solution might have been to solve it with a Custom Pipeline Component. However, since there was a clean and elegant solution to the problem that could be implemented in the .NET Class Library we naturally chose that solution.

Solution The solution to the problem was to write a Custom Message Formatter which serializes/deserializes the data to/from a byte stream with UTF-8 encoding witout adding any additional information to the message body.

For more information on MSDN about creating Custom Message Formatter, see this article http://support.microsoft.com/kb/310683

Posted in: •Integration  | Tagged: •BizTalk  •Custom Message Formatter  •Integration  •Legacy System  •MSMQ 


iBiz Solutions gives presentations on BUGS

BizTalk User Group Sweden (BUGS) is a community whose goal is to increase the understanding and knowledge of Microsoft products and solutions from the Connected Systems Division, mainly around the brand and the platform BizTalk. The target audience is individuals with a technical background (or interest) from consulting organizations interested of systems integration on the Microsoft platform.

On October 3, BUGS have a meeting in Stockholm and two of iBiz Solutions experienced architects in the area, Richard Hallgren and Michael Olsson will give two presentations.

Richard will talk about effective system documentation in an integration project where he will discuss and illustrate the various levels of documentation that we use, what you should document manually and what to automate. Richard will also outline the different platforms and tools that are available to manage system documentation, its advantages and disadvantages as well as some general principles for successful documentation of a project, and look at BizTalk Documentor and BizTalk Web Documentor.

Michael will then hold a presentation about BizTalk IaaS, Paas - Hybrid-based integration solutions using BizTalk locally and / or in the cloud, where the focus will be on future opportunities to run BizTalk 2010 R2 and Windows Azure together.

When: October 3 at 18:00 Where: In the premises of Informator at Karlagatan 108, 115 26 Stockholm

More information and registration for the event can be found at: http://biztalkusergroup.se/

http://swebug20121003.eventbrite.com/?ref=ecal#

Posted in: •About iBiz Solutions  •Uncategorized  | Tagged: •BizTalk  •BUGS  •IaaS  •Paas  •system documentation  •Windows Azure 


Integration “done right” makes advanced business intelligence simple!

The last couple of years the BI (Business Intelligence) world has gone through a paradigm shift. OLAP (On Line Analytic Process) based solutions that previously been hugely successful within the enterprises have been more and more replaced by in-memory alternatives. 

OLAP solutions once revolutionized BI by storing pre-aggregated results and allowing users to make advanced analytics over huge data sets in the blink of a second. The problem is however that as the data needs to be pre-aggregated KPIs (Key Performance Indicators) has to be defined in forehand and the model that creates the cube is more or less fixed which in the end of course restricts the possibilities for the end user. OLAP solutions have traditionally also been complicated to implement as they need its own specialists to design and feed the cube with data – a competence that differs a lot from the more commonly known relational data skills. 

The performance of relational data products has however vastly improved over the years, 64-bit computing has become a commodity and fast computer memory (RAM) has become cheaper and cheaper. This has opened the field for in-memory BI solutions like Spotfire, Qlikview, Tableau, and PowerPivot to name a few. An in-memory solution will not use a cube directly but rather read the whole relational data set into memory. This approach combined with modern and attractive user interfaces made the solutions more efficient, faster and much more flexible than the traditional OLAP alternatives.

Both BI approaches however has one thing in common; without correct and current data the solution is more or less worthless.

Sometimes “correct and current data” is easier said than done. As best of breed system architectures get more popular and organizations start taking advantage of Software as a Service (SaaS) solutions, important data gets spread over a vast number of different systems. This makes the collection of data for the BI solution both complicated and fragile.

The “integration done right” in the title of this post refers to an integration that is based on a comprehensive integration strategy, that uses shared conical internal schemas, is loosely coupled and well documented. If this is the case creating a central data warehouse as a base for the BI solution of relevant data in a proper format is a simple task.

Receiving and feeding the data warehouse with data directly via the integration is also often superior to actually trying to have the BI solution reach into the different systems and read data. The integration option is superior as the alternative complicates the whole architecture, creates a fragile solution, puts extra load on online data within the actual system – but also uses extra time before the data can be analyzed. In a world where two seconds can be a business advantage as current data as possible is critical and waiting on for example a scheduled batch-extract to run before we can receive data from a system isn’t acceptable!

To summarize: an integration investment and strategy within an organization needs to be implemented correctly and from the beginning strive for flexibility and honor things like loose couplings, documentation, canonical formats etc. With a BI implementation on the other hand is equality important to consider integration as a vital part of the project and not end up with a complicated, fragile and non-manageable, non-optimal solution.

Posted in: •Integration  | Tagged: •Architecture  •Business Intelligence (BI)  •Integration