Well it has been a big Christmas and it has also been too long since I have posted, the last six months have been very hectic, I have kick off a couple of large software development projects and between designing, fighting specs and training developers I have not had much time for anything.
I have decided that these post are going to be written more for my benefit, some would call this selfish but in my case this blog is my dumping ground of thoughts and ideas, the objective is to get this stuff out of head so I can move on.
An open journal of sorts...
Friday 28 December 2007
Monday 16 July 2007
The Promised Land of SOA
Here again is another post about S.O.A (Service-Oriented Architecture), but from a different perspective then my previous ones.
I have just realised a major error that I have been making when I have been talking to folks about SOA, I have not appeared excited about SOA and as a result I think I might have come off as negative or disapproving of the approach. I only had this cognition yesterday when I was asked advice on a couple of SOA issues.
I think what has happened is that I have been banging on for so many years now about elevating good Object Oriented design principals beyond a single application to level where it could apply to a collection of applications I have gotten sick of my own voice on the issue. Since 1998 when I started to playing with CORBA and model driven architecture I realized that the future was in items like components, application as services and distributed application where the boundary lines where greyed.
So what I plan to do in this post is describe to you what it is like to live in the promised land of SOA, what the SOA landscape actually looks like and most importantly how to get to this promised land. Just to cover the above three areas one could fill a book or two so I am going to distill thing down.
The promise of SOA is simply vastly increased agility to respond to organisational demands without paying more. This means that IT can develop new application quicker and integrate existing applications in new ways to meet changing business demands. It would be wildly irresponsible for me to say SOA is a silver bullet but it is very close.
Just as a side point there has been an attempt to define a maturity model for SOA which appear to based on the CMMI from Carnegie-Mellon Software Engineering Institute. I offer a slightly different take on this model, this is slightly more result orientated.
We know SOA works and we know it can drive down operating and development cost by about 20%, so what is the catch? If one exist it is that it is not easy, by my last count there where at least 50 standards group work on various aspects of SOA and the majority of these groups are not working together, anyhow more on this issues when we talk about how to get to the promised land.
This lead to what a SOA implementation looks like, instead of quoting the reference model here (which by the way is worth reading) I am going to share an image from Sun which I think is the best diagram I have seen that explains SOA from a before and after perspective. Unfortunately it lacks the rich details and references to technologies so I shall elaborate further on these.
There a also a number of emerging technologies in the process construction area and they warrant a posting by themselves, but for now let just look at the key concept in this process area with is orchestration and choreography. We over complicate things sometimes, the basic difference between the two is that one's macro and the other micro. Orchestration is said to micro, focusing on one process and controls the messages that send and received by the process to services, in this area languages like BPEL are starting to develop and likely to become the future if it is not already the standard.
The macro or choreography side of things is not that clean, firstly choreography deals with the conversations that services have amongst themselves and how this conversation is defined, it protocol is not as clear as that of a application talking to a service (Micro - Orchestration). I personally am punting on WS-Choreography but due to huge vendor interest in this area there is competition and proprietary approaches like BPMN, BPSS, BPEL4WS, WSFL and XLANG.
Our user facing applications sit at the top of architecture levering services via processes, here is one of the key pay offs of SOA, these applications are composites they don't need to reinvent the wheel and there is uniform that comes as part of the structure, rather then being dependant on guidelines and good practice.
Which way to the promised land, I am sure everyone can see the benefit of SOA, but the question is how do we achieve it, how do you get to this place, where applications can put together quickly and easily that results in flexible structure?
Ones needs a map or at lease a general outline which will help avoid shallow waters and sea monsters, in reality the technical challenges, will fade into insignificance compared to the organisational, commercial and business issues. Therefore these need to faced upfront and addressed, the map below attempts to pull forward all the sea monsters and rough patches yet spaces them out so that you are not dealing with too many twisters at once.
To achieve a service orientated architecture and service oriented applications is journey that does not have an end point where all the benefits are deliver, there is no pot at the end of the rainbow, rather benefits come with each step that the organisation takes. I can hear the relief that previous statements brings, I think everyone is happy with instead gratification.
This is underpinned and is only made possible by an system architectural constitution, or what some call a governance. The Architectural Constitution is a manifestation of the commitment by key organisational members to a service orientated information technology approach. The constitution is a published document that sets out the best practices for the organisation, description of services and their suggested reuses and standards and process that can be leveraged to achieve reuse.
I have just realised a major error that I have been making when I have been talking to folks about SOA, I have not appeared excited about SOA and as a result I think I might have come off as negative or disapproving of the approach. I only had this cognition yesterday when I was asked advice on a couple of SOA issues.
I think what has happened is that I have been banging on for so many years now about elevating good Object Oriented design principals beyond a single application to level where it could apply to a collection of applications I have gotten sick of my own voice on the issue. Since 1998 when I started to playing with CORBA and model driven architecture I realized that the future was in items like components, application as services and distributed application where the boundary lines where greyed.
So what I plan to do in this post is describe to you what it is like to live in the promised land of SOA, what the SOA landscape actually looks like and most importantly how to get to this promised land. Just to cover the above three areas one could fill a book or two so I am going to distill thing down.
The promise of SOA is simply vastly increased agility to respond to organisational demands without paying more. This means that IT can develop new application quicker and integrate existing applications in new ways to meet changing business demands. It would be wildly irresponsible for me to say SOA is a silver bullet but it is very close.
Just as a side point there has been an attempt to define a maturity model for SOA which appear to based on the CMMI from Carnegie-Mellon Software Engineering Institute. I offer a slightly different take on this model, this is slightly more result orientated.
- Level 1 - Web enabled Applications (CRM - ERP - Productivity Applications)
- Internal & External Users
- Securer but course Access Control
- Connected & Discounted Operations
- Unified User Experienced
- Reliability & Scalability
- Platform Centric
- Level 2 - Composite, Mash-up & Portals
- Various internal & external data-sources
- Internal & External Users
- Personalisation & Roles Base Security
- Platform Agnostic
- Level 3 - Business Process Driven
- Systems and Application are described in terms of services they offer
- Low level of interdependency between services
- Services can be easily brought together to automate business processes
- High levels of standardisation across all systems yet allowing for localization
- Zero function duplication in systems
We know SOA works and we know it can drive down operating and development cost by about 20%, so what is the catch? If one exist it is that it is not easy, by my last count there where at least 50 standards group work on various aspects of SOA and the majority of these groups are not working together, anyhow more on this issues when we talk about how to get to the promised land.
This lead to what a SOA implementation looks like, instead of quoting the reference model here (which by the way is worth reading) I am going to share an image from Sun which I think is the best diagram I have seen that explains SOA from a before and after perspective. Unfortunately it lacks the rich details and references to technologies so I shall elaborate further on these.
The benefit of the above diagram is that it shows a before and view of an architecture and via this contrast we can see how different things are. For example within the SOA above when a new application needs to created the designers would be asking themselves how do they use the existing component's/services to stitch together a new application. They are composing a application rather then retro fitting or running pipes everywhere and affect data states in non confined or defined ways.
Within the level labeled as Data Repository in the before/after diagram we would expect to see our core corporate applications. Certain place our data warehouses and marts here, the interface at this level does not necessary have to direct access to the database, API's are equally acceptable this level. The other thing that is commonly found here are integration buses and wrapper type systems that expose the underlying infrastructure.
My general rule of thumb for a demarcation line at this level is that the interfaces exposed here should not reflect how the business works but how the infrastructure works, put in other words an engineer should able to map in his or her mind how a particular database or message bus participants in the interface. At the same time a business unit manager should not need to understand what is happening at this level and these interfaces should not map to any particular business function.
"Business Services" are designed around and built upon the interfaces that are exposed at the Repository level. The requirements for a business service should be mapped to a self contained function that ideally is performed across the organisation. In the before/after diagram 'Check Customer Status' is used as a example. There would by a number of different situations in the course of business that would required that status of a customer to be check and depending upon the outcome of that check certain actions would be allowed or not allowed.
It is the job of these business services to deliver clearly defined actions, other example of business services not included in the diagram would include, Ship Product, Validate Credit Card, Request Inventory, Search (Free Form & Structured) etc. Coupled with these business type services, there are other supporting services that help the architecture work, these are items like, Data Services which house dedupers, parers, verifiers etc, Security Services which cover such things as single-sign-on and authentication of users and applications.
All of these services are contained within registry that catalogs and describes each of the services. It is import to point out here even tho the registry is in essence a directory it sits insides a structure that provides a number of base function to the services, like version control, monitoring, common user interface templates, data dictionary etc, basically these are common services which can be consumed by services or are used to operate and monitor the services.
Now that we have a tangible collection of services an analyst can work with a business unit manager to string together the 'business services' into a business process, these processes sit within the composite application level. An example of a process could be insurance claim, this shown below.
Within the level labeled as Data Repository in the before/after diagram we would expect to see our core corporate applications. Certain place our data warehouses and marts here, the interface at this level does not necessary have to direct access to the database, API's are equally acceptable this level. The other thing that is commonly found here are integration buses and wrapper type systems that expose the underlying infrastructure.
My general rule of thumb for a demarcation line at this level is that the interfaces exposed here should not reflect how the business works but how the infrastructure works, put in other words an engineer should able to map in his or her mind how a particular database or message bus participants in the interface. At the same time a business unit manager should not need to understand what is happening at this level and these interfaces should not map to any particular business function.
"Business Services" are designed around and built upon the interfaces that are exposed at the Repository level. The requirements for a business service should be mapped to a self contained function that ideally is performed across the organisation. In the before/after diagram 'Check Customer Status' is used as a example. There would by a number of different situations in the course of business that would required that status of a customer to be check and depending upon the outcome of that check certain actions would be allowed or not allowed.
It is the job of these business services to deliver clearly defined actions, other example of business services not included in the diagram would include, Ship Product, Validate Credit Card, Request Inventory, Search (Free Form & Structured) etc. Coupled with these business type services, there are other supporting services that help the architecture work, these are items like, Data Services which house dedupers, parers, verifiers etc, Security Services which cover such things as single-sign-on and authentication of users and applications.
All of these services are contained within registry that catalogs and describes each of the services. It is import to point out here even tho the registry is in essence a directory it sits insides a structure that provides a number of base function to the services, like version control, monitoring, common user interface templates, data dictionary etc, basically these are common services which can be consumed by services or are used to operate and monitor the services.
Now that we have a tangible collection of services an analyst can work with a business unit manager to string together the 'business services' into a business process, these processes sit within the composite application level. An example of a process could be insurance claim, this shown below.
There a also a number of emerging technologies in the process construction area and they warrant a posting by themselves, but for now let just look at the key concept in this process area with is orchestration and choreography. We over complicate things sometimes, the basic difference between the two is that one's macro and the other micro. Orchestration is said to micro, focusing on one process and controls the messages that send and received by the process to services, in this area languages like BPEL are starting to develop and likely to become the future if it is not already the standard.
The macro or choreography side of things is not that clean, firstly choreography deals with the conversations that services have amongst themselves and how this conversation is defined, it protocol is not as clear as that of a application talking to a service (Micro - Orchestration). I personally am punting on WS-Choreography but due to huge vendor interest in this area there is competition and proprietary approaches like BPMN, BPSS, BPEL4WS, WSFL and XLANG.
Our user facing applications sit at the top of architecture levering services via processes, here is one of the key pay offs of SOA, these applications are composites they don't need to reinvent the wheel and there is uniform that comes as part of the structure, rather then being dependant on guidelines and good practice.
Which way to the promised land, I am sure everyone can see the benefit of SOA, but the question is how do we achieve it, how do you get to this place, where applications can put together quickly and easily that results in flexible structure?
Ones needs a map or at lease a general outline which will help avoid shallow waters and sea monsters, in reality the technical challenges, will fade into insignificance compared to the organisational, commercial and business issues. Therefore these need to faced upfront and addressed, the map below attempts to pull forward all the sea monsters and rough patches yet spaces them out so that you are not dealing with too many twisters at once.
To achieve a service orientated architecture and service oriented applications is journey that does not have an end point where all the benefits are deliver, there is no pot at the end of the rainbow, rather benefits come with each step that the organisation takes. I can hear the relief that previous statements brings, I think everyone is happy with instead gratification.
This is underpinned and is only made possible by an system architectural constitution, or what some call a governance. The Architectural Constitution is a manifestation of the commitment by key organisational members to a service orientated information technology approach. The constitution is a published document that sets out the best practices for the organisation, description of services and their suggested reuses and standards and process that can be leveraged to achieve reuse.
Hanging off this constitution are the standard practices of requirements gathering, problem solving and prototyping.
Saturday 7 July 2007
Developer Manager Divide
Something inside of me was stirred recently when I came across a blog by Rob Walling, clearly Rob is an experienced and insightful developer which is passionate about his work, further he has experience the many sides of our industry.
One particular post 'An Open Letter to Software Managers of the World' really got me thinking. I have hinted at the conflict issues before in my posting that can arise between managers and developers, I feel that I have some experience in this having been on both sides of the fence.
What got me in particular about the above post was that Rob lays out a dozen or so responsibilities that managers and developers need to accept. This is great starting point, but it still has the under tones of us and them, which is the reason the list sits sightly uneasy in my mind. This all in all is not a bad thing because it has acted as a catalyst of thought and the reason for this posting.
After much consideration I have decided not to approach this issue as a developer, manager or architect, rather as a human being with a vested interested in the industry and in software engineering. So I hope to share from my perspective as that person whom has enjoyed a good living from this industry and wants to continue to do so. Firstly you would be a fool not to accept that there has been historical conflict between developers and managers.
Clearly the solution of eliminating one group is just not viable, in the words of A.Cockburn development is a game, a game that needs a group of people with various skills to work together to make things work. We need participants to map out a path ahead, others to manage clients, others cut code and ride the small details, others to deploy and integrate, others to manage and take the damage when we slip up. The reality is when the participants do act together as a team the game can be profitable and rewarding to all.
Therefore this is the first premise that needs to acknowledged, that this game take various participants with diverse skills working in concert to win. The alternative is that one can play with ones self and maintain an illusion that you are taking part in the big game, which clearly you are not.
Acceptance of the first premise leads to the question as to how do we optimize the roles and interactions between the team members?
This is where I think a common set of rules or principals might work better then a list for this type of person and another for other type. So here is my attempt to list my three key rules, explain why they are important and how it relates to the list in the open letter.
1. Understand what the other team members do.
Understand their constraints and how they do their job and what they produce when they have completed their job. For example a developer looking at a sales or business development person, needs to understand that he is out their fighting for a contract that will put food on the table of the developer, that he needs to deal with all different personality types which are receiving alternative offers. He is reject hundred of times and totally shutdown most of the time, this is clearly a hard job, he is at risk of losing his job if the contracts are not delivered and further is at risk of burning is reputation if the software is not delivered.
From the perspective of the business development guys he needs to understand that the developer has to pay the price emotionally and physically for the constraints established within the contract. It is the developer which is not going to go home at night to see their family it is the developer which is not going have time to spend they pay packets because of the unreal expectations that have been set. This applies to all the roles with in team,
If understanding and balance is not achieved between all team members, ultimately it is the stakeholder who pays the price. It is only when we don't attempt to balance profit against the human factors does the scales default back to profit, but that is not say that a profit will ultimately be delivered.
2. Take Personal Responsibility to empower other team members.
Clearly if everyone is operating at optimum levels the team overall have a better change of winning the game. I can already hear some people say but what power do I have, I am just the newbie tester. Well we all have power in a development team, and we need to use that power to ensure that our team mates can do their job better. For example, as tester I can empower a developer by really working with him to help him improve his code, I can do this not only by doing my job well but also taking responsible to communicate openly and directly and respecting the other person at all times.
As a Business Development person I can talk to the developers, architects etc before the terms of engagement are in stone and ask them how would they like to see the engagement carried out. As a developer I can work with architects, project managers and business people to come up with innovative solutions that the business development team can take to clients. As someone who has budget and resourcing responsibility over developers I can ensure that they have the best tools so they can perform at their optimum.
3. Communication and Ethics
Despite the best efforts and good intentions of all team members, there will be conflicts, stuff-ups and less then ideal situations. It is how we overcome these event that determines if we are winners or losses in the game over the long term. Communicate everything without fear of judgement or retribution, if you hear something you consider stupid it does not tell you anything about the other person who said it rather it tell you that you have failed to empower that person and bring them along with you.
Communicate in an open manner not to position others or cover your own short comings, this is the only way to build trust. If there are short coming never lie about them, highlight them, find a solution, fix it and move on.
That is it! Probably sounds very idealistic but it is applicable. Let show you using the list in the open letter.
Managers List
It is not unrealistic, it can all be achieved..... Yes there is a missing piece from the above but I have told you plenty for you too put your heads together to work it out. I will even give you a clue.
There is one extra thing you need to do on the commercial side to make it all work.
Basically if you can answer the question you have team that could win in the development game, if you cannot, it is time to start understanding that you might need others on your team.
Good Luck
One particular post 'An Open Letter to Software Managers of the World' really got me thinking. I have hinted at the conflict issues before in my posting that can arise between managers and developers, I feel that I have some experience in this having been on both sides of the fence.
What got me in particular about the above post was that Rob lays out a dozen or so responsibilities that managers and developers need to accept. This is great starting point, but it still has the under tones of us and them, which is the reason the list sits sightly uneasy in my mind. This all in all is not a bad thing because it has acted as a catalyst of thought and the reason for this posting.
After much consideration I have decided not to approach this issue as a developer, manager or architect, rather as a human being with a vested interested in the industry and in software engineering. So I hope to share from my perspective as that person whom has enjoyed a good living from this industry and wants to continue to do so. Firstly you would be a fool not to accept that there has been historical conflict between developers and managers.
Clearly the solution of eliminating one group is just not viable, in the words of A.Cockburn development is a game, a game that needs a group of people with various skills to work together to make things work. We need participants to map out a path ahead, others to manage clients, others cut code and ride the small details, others to deploy and integrate, others to manage and take the damage when we slip up. The reality is when the participants do act together as a team the game can be profitable and rewarding to all.
Therefore this is the first premise that needs to acknowledged, that this game take various participants with diverse skills working in concert to win. The alternative is that one can play with ones self and maintain an illusion that you are taking part in the big game, which clearly you are not.
Acceptance of the first premise leads to the question as to how do we optimize the roles and interactions between the team members?
This is where I think a common set of rules or principals might work better then a list for this type of person and another for other type. So here is my attempt to list my three key rules, explain why they are important and how it relates to the list in the open letter.
1. Understand what the other team members do.
Understand their constraints and how they do their job and what they produce when they have completed their job. For example a developer looking at a sales or business development person, needs to understand that he is out their fighting for a contract that will put food on the table of the developer, that he needs to deal with all different personality types which are receiving alternative offers. He is reject hundred of times and totally shutdown most of the time, this is clearly a hard job, he is at risk of losing his job if the contracts are not delivered and further is at risk of burning is reputation if the software is not delivered.
From the perspective of the business development guys he needs to understand that the developer has to pay the price emotionally and physically for the constraints established within the contract. It is the developer which is not going to go home at night to see their family it is the developer which is not going have time to spend they pay packets because of the unreal expectations that have been set. This applies to all the roles with in team,
If understanding and balance is not achieved between all team members, ultimately it is the stakeholder who pays the price. It is only when we don't attempt to balance profit against the human factors does the scales default back to profit, but that is not say that a profit will ultimately be delivered.
2. Take Personal Responsibility to empower other team members.
Clearly if everyone is operating at optimum levels the team overall have a better change of winning the game. I can already hear some people say but what power do I have, I am just the newbie tester. Well we all have power in a development team, and we need to use that power to ensure that our team mates can do their job better. For example, as tester I can empower a developer by really working with him to help him improve his code, I can do this not only by doing my job well but also taking responsible to communicate openly and directly and respecting the other person at all times.
As a Business Development person I can talk to the developers, architects etc before the terms of engagement are in stone and ask them how would they like to see the engagement carried out. As a developer I can work with architects, project managers and business people to come up with innovative solutions that the business development team can take to clients. As someone who has budget and resourcing responsibility over developers I can ensure that they have the best tools so they can perform at their optimum.
3. Communication and Ethics
Despite the best efforts and good intentions of all team members, there will be conflicts, stuff-ups and less then ideal situations. It is how we overcome these event that determines if we are winners or losses in the game over the long term. Communicate everything without fear of judgement or retribution, if you hear something you consider stupid it does not tell you anything about the other person who said it rather it tell you that you have failed to empower that person and bring them along with you.
Communicate in an open manner not to position others or cover your own short comings, this is the only way to build trust. If there are short coming never lie about them, highlight them, find a solution, fix it and move on.
That is it! Probably sounds very idealistic but it is applicable. Let show you using the list in the open letter.
Developers List
We will do what it takes to get the job done without being asked, including working extra hours (as long as it does not violate clause 1 in the section below).
We will not complain when we are assigned boring tasks, bad problems, or have to maintain someone else's code (as long as it does not violate clauses 4 or 5 in the section below).
We will bring issues to your attention constructively and with proposed solutions.
We will seek to understand a decision before questioning it.
We will build the best software we are able to.
We will be available when you really need us.
We will fully document our code and designs.
A professional would not do anything else
We will happily coach and mentor new developers.
I know I was a newbie too many moons ago.
We will tell our friends how cool it is to work at our company.
I will always to tell the truth
An issue of understanding, no one should burn themselves and extra hours should be so unusual one should have trouble remembering them. (You can see the agile guy in me coming out), Communication leads to understanding of work loads and they are what they are, you can not change the laws of physics.
We will not complain when we are assigned boring tasks, bad problems, or have to maintain someone else's code (as long as it does not violate clauses 4 or 5 in the section below).
Task 'niceness' is in the eye of the beholder, I personally think maintenance work is one of the most challenging jobs one can do. Anyhow can write a connecting pool-er but it takes skill to retro fit a new database into a old one without breaking it. The key point here is understanding and communication, one just should not commit to delivering something unless you can get someone to signed up to do it. Respect means influencing and not forcing.
We will bring issues to your attention constructively and with proposed solutions.
Does not need discussion, it is about communication and ethics.
We will seek to understand a decision before questioning it.
To question is good, but understanding and empowerment would cause you are apart of the decision if it effects you.
We will build the best software we are able to.
Does not need to be stated, no person wants to do a bad job, we all need to feel proud of our work.
We will be loyal to the company and our team. Loyalty comes as part of trust, and the only way to build that is communication and ethics.
We will be passionate about the software we build. Needs not be stated.
We will be available when you really need us.
I will alway be available to empower a team mate
We will fully document our code and designs.
A professional would not do anything else
We will happily coach and mentor new developers.
I know I was a newbie too many moons ago.
We will tell our friends how cool it is to work at our company.
I will always to tell the truth
Managers List
You understand that "crunch time" is an unexpected part of software development. Unless we have substantial equity in the company, crunch time will not exceed 3 weeks during any 6 month period.
You will give us powerful, best-of-breed PCs, huge hard drives, large monitors, and the latest development software.
You will listen and take action when we constructively bring a problem to your attention.
You will ensure that at least 80% of our time is spent on good problems.
It is in my interest to get involved with everyone and sign up for things that I think I can do well. If I have accepted a task and it turns out to be a bad situation I need to take responsibility to complete it and ensure that i does not happen again by empowering those around me with the knowledge and understanding as to avoid the situation.
If you plan to call us when software breaks, we will be given time to refactor and stabilize it as needed.
You will not ask us to serve as technical guides for highly paid contractors only to be held responsible when their code single-handedly brings our operations to a grinding halt.
If marketing is allowed to set our deadlines based on their knowledge of software projects, we will be allowed to set their budget and/or revenue expectations based on our knowledge of marketing.
You will not ask us to compromise a solid, stable, and maintainable design in order to meet an unrealistic deadline.
You will communicate expectations to to the stakeholders. You will ensure that before we begin building an application, all stakeholders spend ample time reviewing and understanding the specification.
You will ensure that as new requirements arise we will be given the corresponding amount of additional development time.
You will pay attention to your people more than your bottom line.
You will make our company a cool company to work at so we're not lying to our friends.
If I have created crunch time I have failed to understand what others do and have not communicated adequately to explain work loads etc. Just because someone has substantial equity does not mean I can drop the ball.
You will give us powerful, best-of-breed PCs, huge hard drives, large monitors, and the latest development software.
I understand that tools does not make it the programmer but they sure as hell improve productivity and I want to empower developers to be their best, for that matter I want everyone to be the best they can.
You will listen and take action when we constructively bring a problem to your attention.
Does not need discussion, it is about communication and ethics.
You will ensure that at least 80% of our time is spent on good problems.
It is in my interest to get involved with everyone and sign up for things that I think I can do well. If I have accepted a task and it turns out to be a bad situation I need to take responsibility to complete it and ensure that i does not happen again by empowering those around me with the knowledge and understanding as to avoid the situation.
If you plan to call us when software breaks, we will be given time to refactor and stabilize it as needed.
I will clearly set the expectations when I am asked to sign up for a task.
You will not ask us to serve as technical guides for highly paid contractors only to be held responsible when their code single-handedly brings our operations to a grinding halt.
So Obvious it does not need an answer.
If marketing is allowed to set our deadlines based on their knowledge of software projects, we will be allowed to set their budget and/or revenue expectations based on our knowledge of marketing.
What can I say understanding and mutual respect.
You will not ask us to compromise a solid, stable, and maintainable design in order to meet an unrealistic deadline.
What can I say understanding and mutual respect.
You will communicate expectations to to the stakeholders. You will ensure that before we begin building an application, all stakeholders spend ample time reviewing and understanding the specification.
What can I say understanding and mutual respect.
You will ensure that as new requirements arise we will be given the corresponding amount of additional development time.
What can I say understanding and mutual respect.
You will pay attention to your people more than your bottom line.
What can I say understanding and mutual respect.
You will make our company a cool company to work at so we're not lying to our friends.
Never Lie.
It is not unrealistic, it can all be achieved..... Yes there is a missing piece from the above but I have told you plenty for you too put your heads together to work it out. I will even give you a clue.
There is one extra thing you need to do on the commercial side to make it all work.
Basically if you can answer the question you have team that could win in the development game, if you cannot, it is time to start understanding that you might need others on your team.
Good Luck
Wednesday 4 July 2007
Not all architects are the same
The Art and Craft of Great Software Architecture and Development: How to spot the dreaded non-coding architect
Came across the above whilst surfing and just had to add my two cents worth.
I came across this post by Frank Kelly; clearly, his life has been made hell by what he calls the dreaded non-coding architect.
Therefore, I had to respond as the non-coding architect. I am going to follow the format put down by Frank because I think it is the best way to highlight some of the differences. Frank uses the annotation of NCA for non-coding architect and CA for coding architect, I am adding one myself, EA for experienced architect.
Scenario: Need to map an object data model to the database
NCA: According to the J2EE spec we should be using Entity Beans.
CA: Entity beans are terrible - let's write our own DAOs or use Hibernate/iBatis/Toplink. Why do we need an App Server for this? Let's try to stay with J2SE.
The essential issue: J2EE Entity / Session beans were too slow and cumbersome for too long - you also need an app server (which cost mucho $$$$ if you go IBM/BEA). Why not use lighter frameworks such as Spring which will work pretty much anywhere? The NCA wants to adhere to the "spec" from Sun. The CA has been burned before and knows better.
The EA approach would be to consider the environment, including time constrains, available skills, the overall architectural strategy (what the organisation has done before that has worked and fitted with the way the businesses needs to operate). In addition, one would consider their own experiences for what they are, and arrive a preferred option.
All things being equal, I personally would go with simple and easy and write our own data access objects. Once I got buy-in to this approach or during the process, I would prototype of some interfaces and mock objects to demo how I thought we needed to implement.
Then down at pub on Friday night I am going to try to convert everyone to DB4O and after the five drink talk about evil things like how to bring RDBMS to their knees.
Scenario: We need only one instance of an Class per JVM
NCA: Hey design patterns are cool - let's use a Singleton!
CA: How do I code thread-safe singletons in Java? How do I unit-test a Singleton? Let's try to avoid this if we can?
The essential issue: Design patterns are great - but sometimes the pattern is more trouble than it's worth. It's hard to code singletons right, especially for thread-safety. Also they're hard to unit test because singletons are stateful classes whereas unit tests should be as stateless as possible
The EA approach again would be to consider what we are doing from a big picture point of view, asking questions like are we going to maintain the code, is that application mission critical or a little tool that is going to have a 1 year half life.
Assuming that we are going to live with this for years and that I have not shot myself in the foot with deadlines (getting better at this these days). I am going to go singleton, knowing that it will take extra investment now which I hope to earn back in later maintenance cycles, even if I am no longer at the organisation during those maintenance cycles.
But that is not where I am going to leave this issue, clearly folk are worried about the implementation of the singleton so I am ensure the project manager reallocates time to the task and I am going to get as hands on as I can, to help out.
Scenario: Users say our app is too slow
NCA: We need to create and perform a performance test suite across all our supported operating systems, and supported database platforms. We need to buy (insert your favorite expensive and unwieldy toolset here) and test all user workflows.
CA: I know where the bottleneck is because I live with the code daily - we keep loading the same static data over and over again. We'll do some analysis but at least this will get us most of the way.
The essential issue: These days you don't have the time! The first few performance improvements in a new application are usually easy to spot (if you work with the code daily). Some analysis though to verify your hypothesis is required though. Do those initial improvements first and then you can perform the full-blown analysis.
The EA approach is instincts plus benchmark. I don’t know about you guys but the times I have heard this comment I get a feeling in my stomach like I have had a bad curry, this is followed quickly by a flash back to the Tuesday afternoon when I made the call not parameterize and not to thread a certain package.
However, experience has taught me to stop and measure, understand what the user thinks is slow and define it. You will be amazed at the number of times there is a total mismatch between your gut and that the user is really talking about. Work out if this code issue or a UI issue.
Most importantly profile stuff, but not everything that executes, you know the stuff that could be problems, times when you have to reconstitute objects, opening connections, when you have been casting madly. The point is profile, refractor and optimize the really fat things areas where you can get a measurable result.
Scenario: Our CIO says we need to use SOA / Web Services now!
NCA: We should use the WS-* specifications
CA: You know the WS-* specs are still in development and somehow apps with Web Services and SOA are proceeding along fine - let's be careful about which standards we adopt and have a plan in place in case they change. Hey what's this REST thing? That's pretty simple!
The essential issue: Simplify, simplify, simplify! WS-* specs are just getting too heavyweight and are getting bogged down by vendors. Don't get locked in analysis paralysis or worse adopting so many WS-* standards your dev team needs 6 months of training before writing a line of code.
The EA would had his or her CIO medicated early be right inside their head, so something like web services would not come out of the blue. He/she the architect would have already played around with the technology and would be aware of what others are doing out there.
He/she would understand that Web Services are about strategic integration of applications by the bar is still very low out there. Most folks that are going to consume your services are capable of doing stateless soap messages. Therefore WS-* are probably something we need to know able but probably a project for next year. Gold rule kiss, Keep it simple stupid.
REST is ok, but I would look initially to what we already know or have, if a .net house, then WCF (Yuck – personal opinion only) if Eclipse a work in the part. Not make which tool, get the contract done upfront, step out of the day-to-day and think from the perspective of the person who needs to consume the service.
Again, if you as the Architect have done a good job on the design, where the web service is going to plug into to your application is obvious, and you should have not cut across issues that things like security can bring up.
Scenario: We need logging in our application
NCA: We should use Commons Logger as our default logger
CA: Log4j gives you so much more useful functionality that gets abstracted away by Commons Logger - let's use that instead. Log4j's appenders are just soooo cool!
The essential issue: You log file is a very important artifact and you'll want to do different things with different parts of it (e.g. send emails on errors, log some stuff to the database, send something to a message queue etc.). Log4j lets you do that - but in most cases you need to use the XML configuration. Commons wrapper let's you use Log4J underneath but only with the log4j.properties files - you lose access to many of the cool appenders with pattern matching capabilities.
The EA would say yes logging is key critical, and in fact, there should be a standardised approach over all applications.
Because I am, a bit twisted on this issue I would be pushing to build a logger that is flexible and fits with the way the organisation is going to do things. If not possible Log4j is cool, easy to use, but for the benefits a homemade system can deliver, I will be going to CTO to look for some dollars for a little internal project.
Scenario: Our J2EE vendor provides a useful non-standard set of functions - should we use them or write our own?
NCA: Let's make sure to write all our EJB code without hooking in to specific WebSphere/WebLogic etc. extensions. Let's stay vendor neutral. So let's write it ourselves!
CA: It would take us longer to write our own code to do this - so why not use the extensions and just ensure we wrap them in some facade / abstraction that doesn't leak anything container-specific.
The essential issue: Save time! Anyway, what are the odds you will switch J2EE app server vendors?
The EA would say vendor lock in can be a problem but let us get real; it is an issue of degree. This is the type of issue where the architect really can come into play; he/she need to balance the upside with against lock in issue.
Often it is matter of selecting which of the extensions you are going to implement and how you are going to implement those extensions. Again, it is key to look at the environment, are you building an application for commercial resale, or is this an in house application (this can affect how likely the server need to be swapped out)
Bottom line, if you have great relationship with the vendor and their product is hot and it is any in-house app that you understand the life cycle off, then I would say leverage leverage leverage (notice the use of the architect word ‘leverage’) and use those puppies to make you life easier.
I would call myself a non-coding architect, more focused on the big pictures then the correct use of semaphores. Luckily, I have come from a strong coding background and can understand some problems a coder can face day to day.
However, I believe that if you call yourself a coder, you need to work towards being the best coder that you can be; you need to take a work document from an architect and bang it out in record time in the correct form, with test cases and adequate documentation. I am saying that if you code do that professionally.
As an architect I write code to test things, prototype against new API’s, demonstrate new methods for doing things, etc. This is in fact a some part of my job, a lot of my time is spent working out what to do and how to do it, and getting others ‘manager’ on the same page.
I believe that development without an architect is just as dangerous as installing a bad architect in a team. I can accept that there is some bad architects out there as there are some bad developers, what sets us apart from the non-professional amateur hacks is how we deal with them.
What is a Software Architect
There seem to be more and more confusion over the role of a Software Architect, it seems the definition can change day to day from person to person. So what I am going attempt to do in this posting and some followup posting is define what I believe the role is from my own experience and research.
To kick things off I have to say that 70% of the 'Architects' I have met should not have the title, they are doing things in which they have no interest or skill, a better title for some of these folks would be 'fellow software engineer', or just 'Red Hot Developer'.
My first assertion, experienced and skilled developer DOES NOT EQUAL Architect.
The Three Architectural Practices
You can not infer from this that an Architect is not a 'skilled' developer, just that good technical skill alone make not the good Architect. To bring some concrete definition to the role I have identified the three key practice areas that person must be skilled in to deliver ready benefits to an organisation as an Architect. Those areas are Technical, Strategic and Organisational Awareness.
I hope that this posting is has cast some light of what a software architect is, the skill links in each practice area will allow you to drill deeper into that it take to have the right stuff.
To kick things off I have to say that 70% of the 'Architects' I have met should not have the title, they are doing things in which they have no interest or skill, a better title for some of these folks would be 'fellow software engineer', or just 'Red Hot Developer'.
My first assertion, experienced and skilled developer DOES NOT EQUAL Architect.
The Three Architectural Practices
You can not infer from this that an Architect is not a 'skilled' developer, just that good technical skill alone make not the good Architect. To bring some concrete definition to the role I have identified the three key practice areas that person must be skilled in to deliver ready benefits to an organisation as an Architect. Those areas are Technical, Strategic and Organisational Awareness.
Technical Practice
Technically one needs to manage and overcome the frog of war to intimately understand the impacts of organisational decisions on the technology choices which are going to be made down the line. The best way to understand this is compare the task to what a developer would face, as a developer one would require a clear definition of what is needed and then one would go about implementing a solution to meet the requirements.
That clarity is what needs to be created by the Architect with consideration for opposing objectives & priorities, existing technology choices and commercial limitations. The Architect go's about removing the ambiguity by building models, preparing documents and presentation for stakeholders to bring them to consensus. Once achieved the Architect can translate that consensus into cohesive and lucid design that can implemented.
Clearly to achieve the above technical deliverable the architect needs a broad and varied set of skills, which is acquired via years of experience.
Technically one needs to manage and overcome the frog of war to intimately understand the impacts of organisational decisions on the technology choices which are going to be made down the line. The best way to understand this is compare the task to what a developer would face, as a developer one would require a clear definition of what is needed and then one would go about implementing a solution to meet the requirements.
That clarity is what needs to be created by the Architect with consideration for opposing objectives & priorities, existing technology choices and commercial limitations. The Architect go's about removing the ambiguity by building models, preparing documents and presentation for stakeholders to bring them to consensus. Once achieved the Architect can translate that consensus into cohesive and lucid design that can implemented.
Clearly to achieve the above technical deliverable the architect needs a broad and varied set of skills, which is acquired via years of experience.
Strategic Practice
This is the most difficult area of practice, it is a quadratic equation with concepts and many shades of gray, where in the technical practice one is distilling and refining information, here one is providing feed back and input that will ultimately shape the organisations business decisions and in turn it's technology choices. The feedback component is the provision of digestible information as to the organisations technology capabilities and the interpretation of how those capabilities impact product strategies and objectives.
The input component requires creativity and visions in discovering ways to leverage the existing capabilities and new technology to deliver strategic advantages and improved value to the business. Combining this two elements a technology strategic is conceived that acts as a background influencing factor in development of all designs carried out by the Architect.
Many of the skills required in the strategic practice are innate to the architect, they certainly can not be learnt, yet they can be improved upon and strength with research and hard work.
This is the most difficult area of practice, it is a quadratic equation with concepts and many shades of gray, where in the technical practice one is distilling and refining information, here one is providing feed back and input that will ultimately shape the organisations business decisions and in turn it's technology choices. The feedback component is the provision of digestible information as to the organisations technology capabilities and the interpretation of how those capabilities impact product strategies and objectives.
The input component requires creativity and visions in discovering ways to leverage the existing capabilities and new technology to deliver strategic advantages and improved value to the business. Combining this two elements a technology strategic is conceived that acts as a background influencing factor in development of all designs carried out by the Architect.
Many of the skills required in the strategic practice are innate to the architect, they certainly can not be learnt, yet they can be improved upon and strength with research and hard work.
Organisational Awareness Practice
This is the practice area that brings many unstuck, the reality is that an organisation is made up of it's people, each person having their own view point and their own set of immediate priorities to satisfy. Because technology cuts across an organisation, often support is needed by a number of stakeholders across the business to see successful delivery of a project or architecture.
The art of understanding the business and the key players is key to getting buy in and maintaining that buy in over a period of time.
The nature of the role of Architect is that he has influence but no power, so he needs to work to keep his influence and to bring people along willingly. Like the strategic skills these are innate to the architect and very difficult to teach or learn.
This is the practice area that brings many unstuck, the reality is that an organisation is made up of it's people, each person having their own view point and their own set of immediate priorities to satisfy. Because technology cuts across an organisation, often support is needed by a number of stakeholders across the business to see successful delivery of a project or architecture.
The art of understanding the business and the key players is key to getting buy in and maintaining that buy in over a period of time.
The nature of the role of Architect is that he has influence but no power, so he needs to work to keep his influence and to bring people along willingly. Like the strategic skills these are innate to the architect and very difficult to teach or learn.
I hope that this posting is has cast some light of what a software architect is, the skill links in each practice area will allow you to drill deeper into that it take to have the right stuff.
Tuesday 3 July 2007
The Online Office
I think we all knew it was coming and there have certain have been previous attempts to deliver the corner stone applications of word processing and spreadsheets online but they have all just been warm-ups compared Google's efforts. For a while now one could access and use Google Docs & Spreadsheets and in the usual Google fashion they have evolved the offering quietly and thoughtfully.
Then in the last year they have packaged the offering into Google Apps, which is in short an online office productivity suite with hosted email, single sign-on and swag of other online goodies. They have backed this up with a pricing model which appears simple and workable, basically a per user annual fee. Now this all very nice and in someways impressive especially considering the quality of the Google word processor and spreadsheet, but what got my attention is some of the other technologies Google are playing with at the same time.
Namely Google Gears, for those that don't know, the vision here is offline access to a web application, yes a local copy that can be synchronised back to the web application. Added to this what I consider one of the best if not the best API in the industry, it does not take intellectual giant to see where all this is going. Granted that Gears can not really be easily used for Docs and Spreadsheet because you need more a file system rather then a DB achieve this type of stuff but the seed has been planted.
For reasons I have stated above coupled by the fact there is just so much offered now by the Google API/s I think that anyone that is in the process of designing a productivity and/or collaboration application needs to consider how can they leverage the services that Google is offering.
I can hear all the objections now! Let me answer some of those questions:
Objection One.
One can not use or leverage the google services in a commercial application because you can't build a organisation off open source free stuff and service out there on the net, you need solid contractual terms that you can rest upon.
Well you actually do have this, each of the google services does have terms of use, which does facilitate their services being used in a commercial manner.
Objection Two.
We need some kind of SLA or assurance that they will provide these services and will be there tomorrow.
At the time of writing the market cap is 162Billion and more importantly they have a net profit margin of 27% which is based upon business model that they offers these service within, one can take a degree of comfort that they might outlast any application that relies upon them.
Objection Three.
They don't offer anything that we can't build ourselves.
At the risk of using a buzz word 'SOA', why reinvent the wheel?
Anyhow you get the idea, my core point is that Google is becoming more then a search engine and their services can be leveraged not only by web sites but by commercial corporate applications.
Monday 18 June 2007
The Code Base is an Asset
Having recently completed a maintenance engagement, yes architects do get involved with maintenance, hence I am going to use this posting to rave on and on about code as an asset.
The concept of custom developed software as an Asset totally escapes most organisations and as result of poor asset management, they end upstanding under the shower ripping up hundred dollar bills to keep their software applications running.
In my example, you have an organisation that has developed an online application to facilitate the exchange of information between thousands of commercial users. The nature of the exchange means that there are financial impacts for the users and the application host, so there is a degree of motivation and funding to keep the application operating.
Here is the amazing part; they invested large amounts of money in building the application but did not give one thought to how they would keep the application running and evolve it to meet changing requirements.
The strongest evidence of this was present in the code, you could see how things began, there where unit test for all the early functionality, the naming convention was clear and the design of the application appear to be thought out, and the underlying theme was MVC. Then you could see the first round of changes to the application, which was a walk on the wild side, the second around trying to keep with the original approach and the final around, was a perfect demonstration of over engineering. The whole concept of less is more had escaped some of the folk that worked on this application.
So why does this matter, if the application is doing its job and working why are you banging on about all this internal stuff. Simple it effects productively and productivity equals dollars, in other words you end up paying over the mark not by a little but by a ton, are you interested yet?
Now it is easy for me to sit here and say folks should be more professional, but that is unrealistic, often the developers who work on this stuff have ten or least years experience in the industry they are not aware of how terms of engagement can affect their output let alone know how best manage these situations.
The root cause for the above problems which results in wasted dollars on maintenance is ignorance, ignorance of the promise of OO and how to harvest the benefits, ignorance that code is an asset and it can be and should be managed like any other asset. Let us get practical with an example, Application A has taken 3,500 man-hours to build at a cost of about $612K dollars, there is annual hosting & operating cost of $25K (excluding software maintenance and changes cost). This plus the depreciated cost of the development results in $230,000 expense on the P/L annually, this is the start of the hard questions.
Is the system meeting its financial and operational objectives and at what level will it start not to meet those objectives, if maintenance cost rise (putting aside changes for the moment as they normally come with their own set of productive gains or new revenue opportunities).
This is starting point for a software maintenance programme; we need to be able to say if we send more than X this year on maintenance of y over the next z years (z being the effective remaining life of the application) then the viability of the application is under question.
From here, a maintenance strategy can be constructed, which is based upon a code audit, which identify areas of risk, these can then be proactively refactored to mitigate the risk arising from them.
This is a good time to share you with you the number one question I am often asked about maintenance as separate issue away from change. If the application is working well now, why would it break and if there is no reason for it to break then why carry out maintenance?
It is actually a very good question, if the application is running well now and has been for a while, then the risk of it stoping or breaking tomorrow morning is low. That is not the type of risk maintenance looks to mitigate. Maintenance focuses on parts of the application that can break easily when change occurs, for example, let say that backend database needs to be upgraded or a new functionality needs to added, the application should be in such a state that those changes should not break the application.
Another practical example let us say a new feature needs to be added to the application and the original estimates where 20 hours work for the implementation of this new feature. In implementing this feature using best engineering practices, the security mechanism of the application is broken and to change the mechanism so it can work with the new feature will require 80 hours. All of sudden a 20 hour improvement has turn into a 100 hour job which could render the new feature uneconomical to deliver, after all 100 hours is about 3% increase in the total hours invested in the application.
In a refactoring situation, parts of the security mechanism can be changed at a leisurely pace without needing run resources in parallel to meet the deadlines. It still might result in 80 hours work but it cost dramatically less due to resourcing.
I hope this posting can help you see how quickly maintenance can get out of control and why you need, a plan and ideally locked in contracts with vendors prevent these situations from arising. If you know the risk, you can transfer it to a vendor.
The concept of custom developed software as an Asset totally escapes most organisations and as result of poor asset management, they end upstanding under the shower ripping up hundred dollar bills to keep their software applications running.
In my example, you have an organisation that has developed an online application to facilitate the exchange of information between thousands of commercial users. The nature of the exchange means that there are financial impacts for the users and the application host, so there is a degree of motivation and funding to keep the application operating.
Here is the amazing part; they invested large amounts of money in building the application but did not give one thought to how they would keep the application running and evolve it to meet changing requirements.
The strongest evidence of this was present in the code, you could see how things began, there where unit test for all the early functionality, the naming convention was clear and the design of the application appear to be thought out, and the underlying theme was MVC. Then you could see the first round of changes to the application, which was a walk on the wild side, the second around trying to keep with the original approach and the final around, was a perfect demonstration of over engineering. The whole concept of less is more had escaped some of the folk that worked on this application.
So why does this matter, if the application is doing its job and working why are you banging on about all this internal stuff. Simple it effects productively and productivity equals dollars, in other words you end up paying over the mark not by a little but by a ton, are you interested yet?
Now it is easy for me to sit here and say folks should be more professional, but that is unrealistic, often the developers who work on this stuff have ten or least years experience in the industry they are not aware of how terms of engagement can affect their output let alone know how best manage these situations.
The root cause for the above problems which results in wasted dollars on maintenance is ignorance, ignorance of the promise of OO and how to harvest the benefits, ignorance that code is an asset and it can be and should be managed like any other asset. Let us get practical with an example, Application A has taken 3,500 man-hours to build at a cost of about $612K dollars, there is annual hosting & operating cost of $25K (excluding software maintenance and changes cost). This plus the depreciated cost of the development results in $230,000 expense on the P/L annually, this is the start of the hard questions.
Is the system meeting its financial and operational objectives and at what level will it start not to meet those objectives, if maintenance cost rise (putting aside changes for the moment as they normally come with their own set of productive gains or new revenue opportunities).
This is starting point for a software maintenance programme; we need to be able to say if we send more than X this year on maintenance of y over the next z years (z being the effective remaining life of the application) then the viability of the application is under question.
From here, a maintenance strategy can be constructed, which is based upon a code audit, which identify areas of risk, these can then be proactively refactored to mitigate the risk arising from them.
This is a good time to share you with you the number one question I am often asked about maintenance as separate issue away from change. If the application is working well now, why would it break and if there is no reason for it to break then why carry out maintenance?
It is actually a very good question, if the application is running well now and has been for a while, then the risk of it stoping or breaking tomorrow morning is low. That is not the type of risk maintenance looks to mitigate. Maintenance focuses on parts of the application that can break easily when change occurs, for example, let say that backend database needs to be upgraded or a new functionality needs to added, the application should be in such a state that those changes should not break the application.
Another practical example let us say a new feature needs to be added to the application and the original estimates where 20 hours work for the implementation of this new feature. In implementing this feature using best engineering practices, the security mechanism of the application is broken and to change the mechanism so it can work with the new feature will require 80 hours. All of sudden a 20 hour improvement has turn into a 100 hour job which could render the new feature uneconomical to deliver, after all 100 hours is about 3% increase in the total hours invested in the application.
In a refactoring situation, parts of the security mechanism can be changed at a leisurely pace without needing run resources in parallel to meet the deadlines. It still might result in 80 hours work but it cost dramatically less due to resourcing.
I hope this posting can help you see how quickly maintenance can get out of control and why you need, a plan and ideally locked in contracts with vendors prevent these situations from arising. If you know the risk, you can transfer it to a vendor.
Subscribe to:
Posts (Atom)
