Calling all integration experts!

Remember the old Universal Translator as modeled here by the late Mr. Spock? One of the first (or perhaps future?) examples of integration solutions, and certainly one of the most fondly remembere! But at its heart, it is also an almost perfect representation of the integration challenges today. Many years ago, there was EAI (Enterprise Application Integration) which was all about integrating homegrown applications with purchased package applications and/or alien applications brought in from Mergers and Acquisitions activity. The challenge was to find a way to make these applications from different planets communicate with one another to increase return on assets and provide a complete view of enterprise activity. EAI tools appeared from vendors such as TIBCO, SeeBeyond, IBM, Vitria, Progress Software, Software AG and webMethods to mention just a few.

Then there came the SOA initiative. By building computer systems with applications in the form of reusable chunks of business functionality (called services) the integration challenge could be met by enabling different applications to share common services.

Now the eternal wheel is turning once again, with the integration challenge clothed in yet another disguise. This time it is all about integrating systems with completely different usage a resource characteristics such as mobile devices, IoT components and traditional servers, but also applications of completely new types such as mobile apps and cloud-based SaaS solutions. In an echo of the past, lines of business are increasingly going out and buying cloud-based services to solve their immediate business needs, or paying a third-party developer to create the App they want, only to then turn to IT to get them to integrate the new solutions with the corporate systems of record.

Once again the vendors will respond to these user needs, probably extending and redeveloping their existing integration solutions or maybe adding new pieces where required. But as you look for potential partners to help you with this next wave of integration challenges, it is worth keeping in mind possibly the most important fact of all; a fact that has been evident throughout the decades of integration challenges to date. Every single time the integration challenge has surged to the top of the priority list, the key differentiator contributing to eventual success is not the smarts built into the tools and software / appliances on offer. Rather it is all about the advice and guidance you can get from people with extensive experience in integration challenges. Whether from vendors or service providers, these skills are absolutely essential. When it comes down to it, the technical challenges of integration are just the tip of the iceberg; all the real challenges are how you plan what you are going to do and how you work across disciplines and departments to ensure the solution is right for your company. You don’t have the time to learn this – find a partner who has spent years steeped in integration and listen to what they have to say!

Oracle BPM improving

I sat in on the latest Oracle webinar yesterday to hear about its latest developments with its Oracle BPM offering. have to say I was pleasantly surprised. I have been a little harsh in the past about Oracle BPM, but it seems Oracle is finally getting its BPM act together. Process Composer (the Oracle ‘business user’ environment) now offers Oracle BPM Web Forms, an intuitive and easy to use tool to allow user forms to quickly be assembled. The business analyst or architect can assemble whatever user form makes the most sense for each step of the workflow, using a palette of handy selections for such elements as phone numbers, addresses, text input boxes etc.. The mechanism for adding a rule into a process flow is also pretty simple, although of course it assumes a developer has already set up the relevant options for rule specification. Oracle has even started to add Process Accelerators to provide process templates for a small selection of business needs, for example employee onboarding.

I did get one surprise though – this may not be new, but it certainly was to me. Apparently, as well as offering the ability to run Oracle BPM on Oracle’s WebLogic Suite, Oracle also supports IBM WebSphere as the application server layer ūüėģ

New Lustratus Research Report – A Competitive Review of SOA Appliances

Just a short note to say that I’ve uploaded a new report to our web store at

The report, entitled A Competitive Review of SOA Appliances focuses on Intel’s SOA Expressway, IBM’s DataPower range and Layer7’s SecureSpan SOA Appliance. In the report I compare and contrast the technical and strategic approaches each vendor takes to addressing the task of creating, managing, accelerating and securing a service oriented architecture using appliances.

The report can be found here.

Steve Craggs

SOA / ESB confusion

confusionI recently commented on a query in another blog about ESBs, and what they are in relation to SOA.

Since this is a subject that I continue to get asked every now and then, even though I have blogged on it a number of times before, I thought I would reproduce the response I gave.

It starts right at the ESB beginning, and concludes with a few old Lustratus paper references that I believe are still relevant in introducing the ESB and SOA concepts.

When the Enterprise Service Bus concept started off life in the mid 90s it was as an extension of a messaging pipe ‚Äď that is, message based communications. Prior to the ESB, messages were sent with tools such as IBM‚Äôs MQSeries (now WebSphereMQ), Progress Software‚Äôs Sonic MQ and a range of others, particularly JMS (Java Messaging Service) implementations. Users quickly realized that more was needed than just the ability to send a message from A to B ‚Äď value add capabilities were needed such as data transformation, message enrichment and dynamic routing of messages based on content or other circumstances. This resulted in the emergence of ‚Äėmessage brokers‚Äô ‚Äď pieces of code that acted as a ‚Äėhub‚Äô, where any required actions could be taken on in-flight messages. This is where the ‚Äėhub and spoke‚Äô concept that was the basis of the EAI market came from. Messages from A to B would go via the hub so that the required intelligence could be applied, with the A and B endpoints requiring little intelligence at all.

However, two things happened that caused the ESB concepts to emerge around 1996-97. Standards activity in the integration marketplace increased and took root, and users wanted to find ways to lower the entry price for integration ‚Äď having to buy a hub was very expensive, particularly when connections were few in the early stages of integration development. There were also fairly groundless concerns about availability with the hub and spoke model due to the perceived single point of failure. As a result, the ESB emerged.

The ESB did two important things ‚Äď it leveraged major standards, in particular the web services standards offering a standard way to invoke the messaging capabilities of the bus, and it adopted a ‚Äėbus‚Äô rather than ‚Äėhub and spoke‚Äô architecture which resulted in a much lower deployment cost, at least in the early stages of integration development. The bus concept involves placing more intelligence at each node, so that messages can flow A to B without going through the hub. In flight message processing happens at the individual nodes rather than at the hub.

So, an ESB is actually just a ‚Äėsmart‚Äô communications pipe, providing not just a way to transfer the messages from A to B but also the in-flight capabilities (often called mediation services) required. In addition, this is all available under a layering of standards. This is why typically ESBs are used with web services invocations, and often utilize JMS servers for the actual transfer mechanism.

SOA is something much bigger. It is a way of architecting your IT programs around a service-oriented concept. The absolute key to this is that an SOA service relates to a BUSINESS piece of functionality as opposed to some programming activity. So, you do not have an SOA service to read a record from a file, but rather an SOA service to ‚ÄėGet Customer Details‚Äô which internally will end up reading customer information from files and so on. The secondary characteristic is that this service must be able to be invoked from anywhere, with no requirement to know where the target service will actually run. Therefore, it is clear to see that SOA requires some sort of communications capability, and while this does not have to be an ESB, the ESB fits the role very well particularly with its affinity to standards such as web services.

There are a number of free white papers at that discuss this topic in more detail, particularly ‚ÄėSOAs and ESBs‚Äô, ‚ÄėWhat is an SOA Service‚Äô and ‚ÄėThe Year of ESB-ability‚Äô.


Software AG sitting pretty?

Software AG seems to be defying predictions and surprising the market at every turn.

Once seen as a sleepy European software house based largely around legacy system technologies, it has taken major strides to transform itself into a major global software industry player. Its acquisition of webMethods a few years ago surprised the market, with many analysts unconvinced that it could make a go of the move into integration / SOA middleware, but it has done a fair job of building some momentum by tying the webMethods portfolio up with its own CentraSite governance technology, providing service-oriented architecture (SOA) with integrated governance.

Then it once again shocked the market¬†by snatching IDS Scheer, the well-known supplier¬†of modelling tools, from under SAP’s nose. Given that the IDS Scheer technology is used by most of the major SOA suppliers across the world for modelling, and in particular is a key part of the SAP portfolio, this would appear to give Software AG lots of¬†cross-sell opportunities across the two customer bases and throughout the SAP world.

Now it has announced its 2Q09 results, and they make pretty good reading ont he surface. A 9% increase in product revenues is particularly noteworthy give that so many companies are struggling to show any year-on-year growth in product sales. However, before getting too carried away it is worth delving a little deeper into the numbers. The product revenue numbers include maintenance as well as license sales. Licensesales actually fell, as with most other companies. Maintenance revenues jumped by 20% Рdoes this mean that the company has built a much larger maintenance base, or is it actually a reflection of a more aggressive pricing policy? Then there is the split between the legacy business (ETS) and the SOA/BPM business(webMethods). License revenues in this segment were down 15% Рnot very encouraging since this is the strategic business unit. Also, it is noticeable that maintenance revenue in each segment increased by about 20%, suggesting that this rise does indeed reflect a price hike.

However, taking all this into consideration, Software AG is still looking to have moved forward substantially from a few years ago, and assuming the IDS Scheer acquisition goes through OK there should be lots of opportunities for the company. Of course, a cynic might point out that by adding IDS Scheer to the webMethods portfolio, the company has made itself a highly attractive acquisition target to someone Рperhaps SAP?!


SOA success, and what causes it

I was recently pointed to an article in Mainframe Executive magazine written by David Linthicum on the subject of “Mainframe SOA: When SOA Works/When SOA fails”.

I think the friend who suggested I read it was making mischief, knowing my views on the subject of SOA and guessing (correctly) that this article would wind me up.

In summary, the article says that SOA is a large and complex change to your core architecture and working practices and procedures, and that the success or failure is dictated by questions such as executive buy-in/resourcing/funding/skills, and not technology selection.

The truth about success with SOA is that it has little to do with the technology you want to drag into the enterprise to make SOA work, and more to do with the commitment to the architectural changes that need to occur

I have two problems with the opinions stated in this article. The first is to do with changing attitudes to SOA, and the second with the technology comments.

Let me first state that I am well aware that if a company wants to adopt an enterprise-wide SOA strategy designed to take maximum long-term benefit from this new way of leveraging IT investments, then this requires all ofthe areas brought up in the article to be addressed Рskills, management buy-in, political will, funding and a strategic vision coupled with a tactical roadmap. I have no beef with any of this.

But I would contend that the world has changed from two years ago. The financial constraints all companies are experiencing have more or less forced the long-term strategic play onto the back burner for many. Some analysts actually like to claim that SOA is dead, a statement designed to be controversial enough to gain attention but to some extent grounded in the fact that a lot of companies are pulling back from the popular SOA-based business transformation strategies of the past. In fact, SOA is absolutely not dead, but it has changed. Companies are using SOA principles to implement more tactical projects designed to deliver immediate benefits, with the vague thought of one day pulling these projects together under a wider strategic, enterprise-wide SOA banner.

So, as an example, today a company might look at a particular business service¬†such as ‘Create Customer’, or ‘Generate Invoice’, and decide to replace the 27 versions¬†of the service that exist¬†in its silos today with a single shared service. The company might decide to use SOA principles¬†and tools to achieve this, but the planning horizon is definitely on the short term – deliver¬†a new level of functionality that will benefit all users, and help to reduce ongoing cost of ownership. While it would have¬†been valid a¬†few years ago to counsel this company to deliver this as part of an overarching shift to an SOA-oriented style of operations, today most companies will say that¬†although this sounds sensible, current circumstances dictate that focus must remain on the near term.

The other issue I have with this article is the suggestion that¬†SOA success is¬†little¬†to do with the technology choice. Given that the topic here was not just¬†SOA but mainframe SOA, I take particular exception to this. There are a wide range of SOA tools¬†available, but¬†in the mainframe arena¬†the quality and coverage of the tools vary widely. For example, although many¬†SOA tools claim mainframe¬†support, this may in actuality simply be anMQ adapter ‘for getting at the mainframe’. Anyone taking this route¬†is more than likely to fail with¬†SOA, regardless of how well it has taken on the¬†non-technical issues of SOA.¬†Even for those SOA tools¬†with specific mainframe support, some of these offer¬†environments alien to mainframe developers,¬†thereby causing¬†considerable problems in terms of skills utilization. It is critical that whatever technology IS chosen,¬†itcan be used by CICS or IMS-knowledgable folk as well as just disributed specialists.¬†Then there is the question of¬†how intuitive¬†the tools are. Retraining¬†costs can destroy an SOA project before it even gets¬†going.

For anyone interested,¬†there is a free Lustratus report on selecting¬†mainframe SOA tools available¬†from the Lustratus store. However, I can assure companies that, particularly¬†for mainframe SOA, technology selection absolutely IS a key factor¬†for success, and that while all the other transformational¬†aspects of SOA are indeed¬†key to longer term, enterprise-wide SOA there are still benefits to be gained with a more short-term¬†view that is more appropriate in today’s¬†economic climate.


Pragmatism is the theme for 2009

I have just returned from a couple of weeks around and about, culminating in presenting at the Integration Consortium’s Global Integration Summit (GIS), where I presented the Lustratus ‘BPM Sweet Spots’ paper.

One message seemed to come out loud and clear from the conference – pragmatism is the watchword for 2009.

There were two other analyst presentations apart from the Lustratus one, and I was surprised to see that both presenters pitched a message along the lines of ‘you will never succeed with SOA/Integration/BPM unless you get all the strategic planning and modelling for your enterprise done first’, combined with a suggestion that the presenter was just the resource to ask for help! This contrasted sharply with my own presentation of choosing tactical targets for BPM rather than going for a strategic, enterprise-wide, fully modelled approach.

I was wondering if I had read the mood wrong in the marketplace, but then the eight or so user case studies all proved to be tactical strikes for specific business benefits rather than the more extensive strategic approach more common a year or so ago. It was nice to be vindicated – it looks like 2009 really IS the year of pragmatism and short-term practical considerations.


The forgotten SOA benefit – Visibility

There has been a lot of chatter recently about measuring SOA ROI – take a look at Loraine Lawson’s recent blog for instance.

or Gartner’s¬†results of a UK-based survey of SOA adopters. However, one¬†of the benefits that I think a lot of people miss, or do not attribute enough¬†importance to at least, is Visibility.

Basically, the visibility story goes that with SOA, since you break up operational components into discrete business services, then it becomes easy to monitor entry and exit to these services and hence business operations flow. This gives a clear picture in business terms of execution and performance Рnot just what is happening, or how many times, but HOW business is being carried out.

Gartner did touch upon visibility,

Improved Efficiency in Business Processes Execution – Isolating the business logic from the functional application work enables a clearer view of what a process is, and the rules to which it adheres. This can be measured by lower process administrative costs, higher visibility on existing/running business processes, and reduced number of manual, paper-based steps; better service-level effectiveness; quicker implementation of process iterative or of variants of the same process for different contexts.

However, the Gartner focus was only on visibility as it relates to execution efficiency.¬†In fact, SOA-based visibility offers another benefit which, in today’s tough times particularly,¬†can be a real big hitter for executive management. It enables management to see how process are being executed – in other words, it provides the ideal tool to monitor compliance against a growing raft of regulatory requirements across just about every industry. In order to demonstrate that your systems comply, it is necessary to be able to see what they are doing and how they are doing it. This is what SOA delivers.

So how does improved compliance management fit into the ROI picture? True, it is very hard to attach a dollar amount to compliance – however it certainly matters. With the amount of public and political scrutiny of corporations today, it is absolutely imperative that executives can show they are faithfully adhering to regulations and guidelines. Failure to do so will not only risk severe penalties, but¬†also probably lose them their jobs! Now THAT’s a compelling business case….


Does Microsoft ESB Guidance have a future?

As one might have expected, Microsoft tried to ignore the Enterprise Service Bus (ESB) movement for a long time, but eventually it had to do something to answer demands of its customer base looking for SOA support.

Its response was Microsoft ESB Guidance, a package of

architectural guidance, patterns, practices, and a set of BizTalk Server and .NET components to simplify the development of an Enterprise Service Bus (ESB) on the Microsoft platform

Let’s be honest. This is a typical Microsoft ‘fudge’. Microsoft ESB Guidance is not a Supported Product, but is instead a set of guidelines and one or two components. It is a Microsoft Patterns and Practices offering – in other words, you are on your own. This may be fine if you are a Microsoft development shop, but far more worrying if you are real business user with extensive Microsoft presence. It has a lot of the disadvantages of Open Source, but you still have to pay for Bizt Talk etc..

So what does the future hold? Will trying to bring the Microsoft server world into the SOA domain always be a matter of risk and going it alone? Will Microsoft productize Microsoft ESB Guidance? Are there any alternatives other than just consigning the Microsoft platform to run in isolation on the fringes of the enterprise?

Fortunately, the Microsoft model may actually be working here. I do not believe Microsoft will ever productize ESB Guidance Рafter all, they have had two years and are still maintaining there are no plans to do this. However, what this position does do is it encourages opportunists to jump in and develop products based around the Microsoft technology and guidance materials. An example is Neuron-ESB, from Microsoft specialists Neudesic.

So, while¬†Lustratus strongly cautions users about¬†the¬†effort, cost¬†and risk of using Microsoft’s own ESB Guidance package,¬†the idea of utilizing a Microsoft-based¬†supported ESB product from a specialist vendor is much¬†more attractive. Of course, whether these new Microsoft-based ESBs are any good is yet to be seen….


Why do so many SOA adopters moan about low reuse levels?

I was reading a recent post from Joe McKendrick the other day on measuring SOA success…

…and it reminded me of a related issue – that of measuring services reuse. SOA adopters often moan to me that despite having implemented SOA and deployed many services, reuse rates are down at the 1-1.2 level – in other words, virtually no reuse. They seem to want to pick a fight with me because as an advocate of SOA I have often pointed to reuse as one of the more measurable benefits. After all, achieving a high level of reuse is a clear indicator to business executives that efficiency is increasing, since the implication is less development is required to do new things.

I am starting to get pretty short now inthese conversations.¬†I wish, wish, wish that people would heed¬†my previous advice¬†– don’t think of SOA delivering reusable services, think of it¬†as a great tool for SHARED services. Obviously reuse will come¬†through services being shared –¬†so what point am I trying to make? The problem is people are choosing to build ‘reusable services’ with SOA and assuming that others will start reusing them. It is the old ‘Build and they will come’ philosophy. This rarely works – it is worse than a scatter-gun approach. If users instead think about what services would be good candidates for being shared first, and then develop these as SOA services, reuse levels will definitely improve.

So, when getting started with SOA, don’t encourage everyone to start building code into services and hope that reuse will come as if by magic. Start off by deciding on the logical services to build that will be shared – things like get customer history, or create new customer. Then go ahead and build these shared services candidates, and see reuse levels climb….hopefully making it easier to justify your SOA investments to the business.