The Promised Land of SOA

Here again is another post about S.O.A (Service-Oriented Architecture), but from a different perspective then my previous ones.

I have just realised a major error that I have been making when I have been talking to folks about SOA, I have not appeared excited about SOA and as a result I think I might have come off as negative or disapproving of the approach. I only had this cognition yesterday when I was asked advice on a couple of SOA issues.

I think what has happened is that I have been banging on for so many years now about elevating good Object Oriented design principals beyond a single application to level where it could apply to a collection of applications I have gotten sick of my own voice on the issue. Since 1998 when I started to playing with CORBA and model driven architecture I realized that the future was in items like components, application as services and distributed application where the boundary lines where greyed.

So what I plan to do in this post is describe to you what it is like to live in the promised land of SOA, what the SOA landscape actually looks like and most importantly how to get to this promised land. Just to cover the above three areas one could fill a book or two so I am going to distill thing down.

The promise of SOA is simply vastly increased agility to respond to organisational demands without paying more. This means that IT can develop new application quicker and integrate existing applications in new ways to meet changing business demands. It would be wildly irresponsible for me to say SOA is a silver bullet but it is very close.

Just as a side point there has been an attempt to define a maturity model for SOA which appear to based on the CMMI from Carnegie-Mellon Software Engineering Institute. I offer a slightly different take on this model, this is slightly more result orientated.

• Level 1 - Web enabled Applications (CRM - ERP - Productivity Applications)
  
  • Internal & External Users
  • Securer but course Access Control
    
  • Connected & Discounted Operations
  • Unified User Experienced
  • Reliability & Scalability
  • Platform Centric
    
• Level 2 - Composite, Mash-up & Portals
  • Various internal & external data-sources
  • Internal & External Users
  • Personalisation & Roles Base Security
  • Platform Agnostic
• Level 3 - Business Process Driven
  • Systems and Application are described in terms of services they offer
  • Low level of interdependency between services
  • Services can be easily brought together to automate business processes
  • High levels of standardisation across all systems yet allowing for localization
  • Zero function duplication in systems
    

We know SOA works and we know it can drive down operating and development cost by about 20%, so what is the catch? If one exist it is that it is not easy, by my last count there where at least 50 standards group work on various aspects of SOA and the majority of these groups are not working together, anyhow more on this issues when we talk about how to get to the promised land.

This lead to what a SOA implementation looks like, instead of quoting the reference model here (which by the way is worth reading) I am going to share an image from Sun which I think is the best diagram I have seen that explains SOA from a before and after perspective. Unfortunately it lacks the rich details and references to technologies so I shall elaborate further on these.

The benefit of the above diagram is that it shows a before and view of an architecture and via this contrast we can see how different things are. For example within the SOA above when a new application needs to created the designers would be asking themselves how do they use the existing component's/services to stitch together a new application. They are composing a application rather then retro fitting or running pipes everywhere and affect data states in non confined or defined ways.

Within the level labeled as Data Repository in the before/after diagram we would expect to see our core corporate applications. Certain place our data warehouses and marts here, the interface at this level does not necessary have to direct access to the database, API's are equally acceptable this level. The other thing that is commonly found here are integration buses and wrapper type systems that expose the underlying infrastructure.

My general rule of thumb for a demarcation line at this level is that the interfaces exposed here should not reflect how the business works but how the infrastructure works, put in other words an engineer should able to map in his or her mind how a particular database or message bus participants in the interface. At the same time a business unit manager should not need to understand what is happening at this level and these interfaces should not map to any particular business function.

"Business Services" are designed around and built upon the interfaces that are exposed at the Repository level. The requirements for a business service should be mapped to a self contained function that ideally is performed across the organisation. In the before/after diagram 'Check Customer Status' is used as a example. There would by a number of different situations in the course of business that would required that status of a customer to be check and depending upon the outcome of that check certain actions would be allowed or not allowed.

It is the job of these business services to deliver clearly defined actions, other example of business services not included in the diagram would include, Ship Product, Validate Credit Card, Request Inventory, Search (Free Form & Structured) etc. Coupled with these business type services, there are other supporting services that help the architecture work, these are items like, Data Services which house dedupers, parers, verifiers etc, Security Services which cover such things as single-sign-on and authentication of users and applications.

All of these services are contained within registry that catalogs and describes each of the services. It is import to point out here even tho the registry is in essence a directory it sits insides a structure that provides a number of base function to the services, like version control, monitoring, common user interface templates, data dictionary etc, basically these are common services which can be consumed by services or are used to operate and monitor the services.

Now that we have a tangible collection of services an analyst can work with a business unit manager to string together the 'business services' into a business process, these processes sit within the composite application level. An example of a process could be insurance claim, this shown below.

There a also a number of emerging technologies in the process construction area and they warrant a posting by themselves, but for now let just look at the key concept in this process area with is orchestration and choreography. We over complicate things sometimes, the basic difference between the two is that one's macro and the other micro. Orchestration is said to micro, focusing on one process and controls the messages that send and received by the process to services, in this area languages like BPEL are starting to develop and likely to become the future if it is not already the standard.

The macro or choreography side of things is not that clean, firstly choreography deals with the conversations that services have amongst themselves and how this conversation is defined, it protocol is not as clear as that of a application talking to a service (Micro - Orchestration). I personally am punting on WS-Choreography but due to huge vendor interest in this area there is competition and proprietary approaches like BPMN, BPSS, BPEL4WS, WSFL and XLANG.

Our user facing applications sit at the top of architecture levering services via processes, here is one of the key pay offs of SOA, these applications are composites they don't need to reinvent the wheel and there is uniform that comes as part of the structure, rather then being dependant on guidelines and good practice.

Which way to the promised land, I am sure everyone can see the benefit of SOA, but the question is how do we achieve it, how do you get to this place, where applications can put together quickly and easily that results in flexible structure?

Ones needs a map or at lease a general outline which will help avoid shallow waters and sea monsters, in reality the technical challenges, will fade into insignificance compared to the organisational, commercial and business issues. Therefore these need to faced upfront and addressed, the map below attempts to pull forward all the sea monsters and rough patches yet spaces them out so that you are not dealing with too many twisters at once.

To achieve a service orientated architecture and service oriented applications is journey that does not have an end point where all the benefits are deliver, there is no pot at the end of the rainbow, rather benefits come with each step that the organisation takes. I can hear the relief that previous statements brings, I think everyone is happy with instead gratification.

This is underpinned and is only made possible by an system architectural constitution, or what some call a governance. The Architectural Constitution is a manifestation of the commitment by key organisational members to a service orientated information technology approach. The constitution is a published document that sets out the best practices for the organisation, description of services and their suggested reuses and standards and process that can be leveraged to achieve reuse.

Hanging off this constitution are the standard practices of requirements gathering, problem solving and prototyping.

The Code Base is an Asset

Having recently completed a maintenance engagement, yes architects do get involved with maintenance, hence I am going to use this posting to rave on and on about code as an asset.

The concept of custom developed software as an Asset totally escapes most organisations and as result of poor asset management, they end upstanding under the shower ripping up hundred dollar bills to keep their software applications running.

In my example, you have an organisation that has developed an online application to facilitate the exchange of information between thousands of commercial users. The nature of the exchange means that there are financial impacts for the users and the application host, so there is a degree of motivation and funding to keep the application operating.
Here is the amazing part; they invested large amounts of money in building the application but did not give one thought to how they would keep the application running and evolve it to meet changing requirements.

The strongest evidence of this was present in the code, you could see how things began, there where unit test for all the early functionality, the naming convention was clear and the design of the application appear to be thought out, and the underlying theme was MVC. Then you could see the first round of changes to the application, which was a walk on the wild side, the second around trying to keep with the original approach and the final around, was a perfect demonstration of over engineering. The whole concept of less is more had escaped some of the folk that worked on this application.

So why does this matter, if the application is doing its job and working why are you banging on about all this internal stuff. Simple it effects productively and productivity equals dollars, in other words you end up paying over the mark not by a little but by a ton, are you interested yet?

Now it is easy for me to sit here and say folks should be more professional, but that is unrealistic, often the developers who work on this stuff have ten or least years experience in the industry they are not aware of how terms of engagement can affect their output let alone know how best manage these situations.

The root cause for the above problems which results in wasted dollars on maintenance is ignorance, ignorance of the promise of OO and how to harvest the benefits, ignorance that code is an asset and it can be and should be managed like any other asset. Let us get practical with an example, Application A has taken 3,500 man-hours to build at a cost of about $612K dollars, there is annual hosting & operating cost of $25K (excluding software maintenance and changes cost). This plus the depreciated cost of the development results in $230,000 expense on the P/L annually, this is the start of the hard questions.

Is the system meeting its financial and operational objectives and at what level will it start not to meet those objectives, if maintenance cost rise (putting aside changes for the moment as they normally come with their own set of productive gains or new revenue opportunities).

This is starting point for a software maintenance programme; we need to be able to say if we send more than X this year on maintenance of y over the next z years (z being the effective remaining life of the application) then the viability of the application is under question.

From here, a maintenance strategy can be constructed, which is based upon a code audit, which identify areas of risk, these can then be proactively refactored to mitigate the risk arising from them.

This is a good time to share you with you the number one question I am often asked about maintenance as separate issue away from change. If the application is working well now, why would it break and if there is no reason for it to break then why carry out maintenance?
It is actually a very good question, if the application is running well now and has been for a while, then the risk of it stoping or breaking tomorrow morning is low. That is not the type of risk maintenance looks to mitigate. Maintenance focuses on parts of the application that can break easily when change occurs, for example, let say that backend database needs to be upgraded or a new functionality needs to added, the application should be in such a state that those changes should not break the application.

Another practical example let us say a new feature needs to be added to the application and the original estimates where 20 hours work for the implementation of this new feature. In implementing this feature using best engineering practices, the security mechanism of the application is broken and to change the mechanism so it can work with the new feature will require 80 hours. All of sudden a 20 hour improvement has turn into a 100 hour job which could render the new feature uneconomical to deliver, after all 100 hours is about 3% increase in the total hours invested in the application.

In a refactoring situation, parts of the security mechanism can be changed at a leisurely pace without needing run resources in parallel to meet the deadlines. It still might result in 80 hours work but it cost dramatically less due to resourcing.

I hope this posting can help you see how quickly maintenance can get out of control and why you need, a plan and ideally locked in contracts with vendors prevent these situations from arising. If you know the risk, you can transfer it to a vendor.

Web 2.0 + SOA = Web Services

It is easy to be caught up in the hype and the latest terms, certainly, when your cabbie ask you about Web2.0 you know that marketing engine has been working in overdrive. Well that is actually, what happened to me today.

Whilst catching a cab today, my friendly cabbie asked me what I did, I replied with the standard line 'I work with computers' , Before I knew it he was telling about his son the wiz and asked me about Web2.0 and how I thought I would change the 'net'.

Therefore, this posting is for you George...

Web2.0, SOA, Web Services are all so interconnected I cannot do the subject justice by covering just one area. The whole push behind web2.0 is to allow people to work to collaborate and discovery new synergies. Web2.0 are software applications which are more like work environments that are inclusive and aims to bring the best out of your colleagues. As to what functionality these include, this is only limited by our own creative thoughts but these application will seek to leverage collective wisdom of people who where normally separated by distances, time zones and social economic barriers.

Social Bookmarking at del.icio.us is a great example of web2.0, its value and utility increases as more people participate; clearly, wiki is another great example that follows this rule.

However, here is the interesting thing the 'Technologies' behind the above two example are not new the same applies to many things we associate with Web2.0. I can a few voices off in the distance they are saying "What about AJAX"? Yes, the term is new, yet the enabling technology XMLHttpRequest has been around since 1999 if I recall correctly.

Yes, standards have been refined and new tools have made it easier to implement functionality, but as for enabling technology, most of it has been around for at least five years. Therefore, it would be fair to say that Web2.0 is the application of web/internet technologies in new and creative ways. This in itself speak mountain for the human creative spirit, I like to think of it as Web1.0 is characterized by humans duplicating what they had in the physical world online and Web2.0 is characterized by human discovering new way of doing things without the shackles of physical work practices, traditions and limitations.

So what does this all have to do with SOA (Service Orientated Architecture)? Well like Web2.0, SOA is about doing things differently, without the restrains of old world thinking and old world habits. Now there are some, which believe that this new world thinking is a fad and the cite pass things such as Modular Programming of the 70's, Event-Oriented Deign of the 80's and Component Approaches of the 90's. For anyone who has lived each of the above they would not say it is fad rather evolution.

Clearly, I was not coding in 70, but I have maintained allot of code from that, era and anyone can see that Modular Programming has influenced the single responsibility concept that all us OO Architects bang on about today. The same applies to the Event-Orientated approach that was so hot in the 80's, which has greatly influenced messaging today. The same applies for Components in 90's which today is still realized in many of our current designs. All of these approaches have stood upon the shoulders of their parents and the same applies to SOA.

OK we know that SOA is way of design a style if you will and we know where is has come from, it heritage but what is it?

From my perspective as an Architect, Service Orientated Architecture is way of putting together a system or application that uses existing systems/services and newly built components systems/services in such a manner that quickly, and cost effectively meets an organisations objectives whilst cutting across ownership boundaries.

Let us get practical; in this example, the brief is to deliver an email and SMS marketing application. After studying and decomposing the requirements, we can draw some big boxes on a white board. The first box is something to send and receive emails and another box to do the same for SMS's, another to compose the messages and one to maintain a list of folks who would get marketing message then on top of all that we need another box to report outcomes from these activities.

Reality check... if the above was the requirements we could probably find a few services that provided it all today and we just hook-up to their API to move data back and forth and write some report stuff.

However, let us for this example assume such a service does not exist and will not likely to exist in the future.

In the old days, we would have just built a 'sendmail' server and hooked up the serial post of the same server to a mobile phone to drive the SMS stuff. (Clearly not thinking about scalability issues here) Then we would have written heap and heaps of code and then probably a GUI and an import and export function so we could move data to and from the customer database.

Being that this is the enlighten age we would use a third party SMS provider or if one did not exist we would seek out a partner to own and operate a SMS service after we built it. Being the enlighten age we will build the SMS box/gateway in a fashion that was independent of the rest of the application, so it could be used by other application and other parties in the future.

In doing this, we would be concerned about coupling at the service level as well as the implementation level, we would ensure that way of communicating with this SMS box/gateway and getting to do stuff like sending SMS's and dump back replies where clearly define and that the inter-workings where clearly abstracted away.

Our objective here is to build a standalone SMS gateway that is capable of evolving to become the best SMS gateway around (will it be evolved is another matter which is commercial in nature).

We follow the same approach for our boxes, firstly looking for existing services, which are built to standards and using these if possible, then we consider building it with others and lastly build it in house (Clearly using another order if your core business was the provision of services).

In the ideal world, there would be services out there on the net that one could 'discover' and integrate together.

The above is the SOA style or way of building an application in addition to my perspective and there are generally accepted principal of SOA and reference implementations I strong recommend that every architect study and implement on their own.

The design principals are nothing new and you are probably using day to day as an architect, they are:

Abstraction

Autonomy

Compos-ability

Coupling

Discoverability

Encapsulation

Optimization

Reuse

Service/Interface Contracts

The only one in the above list I probably need to elaborate on is Discoverability, and certainly, this is an advance design feature. It should be noted that within the standards from OASIS, which I have strongly suggest, that everyone read, discoverability is necessary feature. In the commercial setting, unless you are building for a commercial provider it will probably not make the final cut on the specifications.

Discoverability is similar to reflection, it allows for the finding and querying of a service to get it measurements so to say. In a SOA sense where one is, building web services there is a couple discovery protocols you need to understand, being, UDDI and WS-Discovery. They both are different in that UDDI is more like a phone book (directory or service broker) and WS-Discovery is multicast 'who is out there' type protocol.

Also not to confuse but there is one more protocol to get you head around and that is WSDL, which is really simple it is basic a XML format that describes the web service , stuff like end points on the Net, address and ports and list of available command and any special data types.

You may have noticed the introduction of the term of web services in the last few paragraphs, well our SOA approach has lead us to build these boxes, which are, known as web services. Therefore, we can say that one of the artefacts of SOA is web services.

Within this realm or at this level also is a set of standards and specifics that can be used to find our out way out of the dark. These are from W3C and are again easy to get your head around, these standards position web services as client/server that exchanges XML messages that follow SOAP. There are alternative ways of doing this, but this is probably the simplest way, and there are plenty or reference implementations.

Therefore, as we can see web2.0 or peoples' creativity is driving the demand for new applications and the new way of building them is SOA when combined results in web services, that simple.

Leveraging Google

With the buzz about Web2.0, Mash up and SOA, I thought this would be an opportune time for a quick post on how an enterprise can leverage Google so that can go where no man has been before.

When we think Google we think search and rightly so they have had a major impact on that landscape, but search is not the only thing we should think. As of this moment there is probably some half dozen other major Google online applications which is not within the primary search space.

These applications which I am not including with the primary search space are Google Documents, Calendars, Base Data, AJAX, Checkout, Email and Blogger. If you include the rest like searching, mapping, ad words/sense, etc the list just sky rockets. What is more important is that Google have developed stable and well documented API's for these services.

Google clearly want folks to build applications that use their API, because these become additional channels, channels which are very sticky simple because once an organisation invest in learning and implementing an API they happen hang around for very long time. That said and understanding that one could spend a whole posting on the commercial issues, let move along to what we can do with this stuff.

Via the Google API's all of the Google Services have become 'Web Services' therefore a custom smart client or web application can be written to replace the existing heads. This in itself is not very interesting, given that the Google interface is so functional. The next step is integrating single services, like the Google Check service, which in short is a merchant facility for online transactions.

The immediate benefits are clear here, reduced development time and you get access to a bullet proof facility, but go a step further and use Google Spreadsheets Data API and you can easily hook-up a whole reporting and management system which produces Graphs on sale numbers and operational targets.

Another Cool combination would be to use the Gdata API to store the products online that you would like to sell. So all of a sudden you don't have to maintain your own database, Google is doing for you and somehow I think they can do a good job of it.

List of possible uses are almost unlimited, you could use blogger to data amongst a group of analysis, data which can read by a human eye as well as being machine readable. Anyhow as you can see the list could continue forever. As a thought exercise image a desktop smart client application that relied on web services for all it' s major components, how things would change not only in what we could do but also how quickly we could do it.

Software Architecture within a Commercial Context

Inspired by a recent post by Paul Reedman I was driven to make this response, before diving in let me go on record and say that having been a readers of Paul’s Blog for while now I usually find myself saying ditto to all his post.

But this one and in particular the responses which it generated has fired me into action.

The line "In the software industry it's all about writing code and getting a product out the door. Good design and architecture has disappeared." Sends the message, to me at least, that somehow we have sold off good design for the sake of money or commercial pressures.

I can understand those comments but I also need to say that is not way it has to be, in fact it is only that way in some organisations because us as a group of Software Architects have not been innovative enough to prevent this from happening.

My view point is this, Software is developed within a commercial context and the commercial model of supply, demand and free market forces needs to drive everything we do. Recent history has shown us this is a force that can’t be resisted.
So as Architects we need to embrace this reality, much like many of us have embraced change as a fact of every project we all need to embrace commercial constrains of time and budget and design around it.

Now that does not equal compromise rather it should cause us to innovate. Let me share some experiences here in short form, I will save the details for a book, or for the next time we have a drink together.

• Just Like Any Other Feature - Get the time and budget constrains into the project requirements and map the relationship between other constrains and these commercial ones.

• More bang for the buck – Everything needs to used more than once, write development documents that can be rehashed for user documentation, use API test suits as a basis for API Client libraries, practice OO reuse all the time. (Email me if you want more of these ideas I have hundreds)

• Less is more – Be strategic why spec two functions that have 70% overlap, give it one use case and extend it, same applies at implementation stage.

• Use a pattern – Patterns are distilled experience which is free and only have a positive impact on the project, so understand them and use them all the time.

• Parlay It Up – Stop looking at projects in isolation, ask yourself what is one thing we can do that will have less than a 1% impact but give us some value that we can use in the next project. (Poor man’s framework and code base)

• Don’t remake the wheel. For the pride, if there is a third party component, server, application, API, whatever that can help you deliver; trade some budget to save some time. Clearly do this with care; it can work out if you do your homework.

• Show me the Money – Releasing early and releasing often not only keeps a client happy but can work to simply your project. Something magical happen you make the client live and breathe an early version of the application. They start to discover how they can optimize the functionality and this can highlight the useless stuff that can be cut.

Now that I made all the motherhood statements let me hit with some hard realities. You cannot change the laws of physics; total project budget divided by time constraint divided by the number of plan iterations will always limit what one can achieve.
The defining factor of a software architect and where the art comes into it is how much you can achieve with what you have to work with.