Revamped Services Fabric Blog

I decided I'd spend thisn weekend tarting-up this blog and making use of the new blogger template gadets etc.I've also added more meaningful labels to make filtering on a specific topic easier (e.g. click VPEC-T below to see all VPEC-T related posts)

Other news: Richard Veryard created a Lenscraft wiki that promises to be a interesting place for developing a number of themes my Twitterati pals and I've been discussing for a while.

Photo Credit ShoZu on Flickr

The Great Granularity Debate

Events of the past week have led me back to the "Great Granularity Debate" that goes hand-in-glove with Service Orientation. I was discussing this with some colleagues last night - I described the problem I was dealing with as a 'nano-Lego' problem. This problem seems to come about when technically-focused architects define a 'SOA' without binding it to business drivers and objectives - this results in a plethora of  fine-grained 'architecture-for-architecture-sake-services-for-god's-sake technical services that look suspiciously like re-usable 'OO' objects (they didn't get reused either did they?).

In this particular case, the business would like to move away from their old monoliths to more granular architecture that would allow for more efficient change. They don't seem to be bothered about reuse and put performance much higher on the list. They also recognise that they're not experienced in doing things a 'Service Oriented' way and can see some of the problems in funding cross-project service development. 

All this tells me that the most appropriate SOA for these guys would be a coarse-grained and business focused. Finer grained services might be developed later as their maturity in things service oriented develops.

So my message to the techies - put the tweezers away and find some heavy lifting-gear to put those chunky business services in place first.

Why Service Orientation should start with Systems and not (always) end in systems

In publishing this I'm throwing caution to the wind (and ignoring, in part, the good advice of a respected friend!). Chris Bird, suggested that I should avoid talking about Systems Thinking, Service-oriented Architecture and other such consultantese. But I've decide I will talk about Systems Thinking and SOA as they relate to a broader world-view of services (given the focus of this blog).

To start the discussion, however, I'd like to quote Chris: “ SOA starts in the wrong place. The tools are tools and not grand strategies. I don't look at the screwdriver in my hand and say, "Cool, now what projects can I undertake?" I think about what needs fixing in my house and what tools I need (aside - the checkbook is my favorite tool)”.

Mindful of Chris's words and a client's SOA-related organisational challenges in mind, I thought it was about time I added my two-pennies-worth to the SOA-doesn't-work debate. My observation is that many SOA projects start in the wrong place - that being in the technology weeds i.e. Conversations around ESBs, WS*, Registries, EJB, XML et al. I believe the first step towards 'service orienting' a business is taken by applying Systems Thinking (that is Systems in an ecosystem sense) rather than thinking about Services per se and certainly before technology view of SOA). Most importantly, the notion that a business is comprised of multiple, interacting, 'Systems' of people, processes and technologies (agents of the system) that cannot be viewed in isolation one from another but accepts each system/sub-system works within a unique set of values.

Taking a fresh Systems Thinking (capital 'S' Systems from now on) perspective helps to breakdown the more traditional organisational and process bounded views of the business and that the complexity of the behaviour of the business is best tackled by examining the interactions between 'Systems' that often span traditional boundaries. This then helps layout the organisation as a set of 'System' behaviours that can, for example, be examined as core or context to the business operation/strategy/well-being. I've found that with this approach, its possible to evolve certain 'Systems' into 'Services' by defining the Consumers, the Policies/Contracts that apply, the Events that trigger action and the Content being exchanged (physical and/or informational). This 'Big Services' (Systems)view allows the business to see how to chunk-up aspects of the operation in new ways that helps with business problems such as: simplifying post M&A situations, executing major transitions, outsourcing. And, from an IT perspective, where/how to apply SOA, COTS packages or SaaS for that matter.

I've been accused of being too idealistic when I say Service Orientation starts in the boardroom not the IT department. I agree, it's often hard, if not impossible, to get SO on the business agenda but if your SOA is being 'sold' as a way to achieve 'business relevant' efficiencies and associated cost reductions through service reuse, then board members must be the sponsors. This becomes most obvious when organisations realise they must change the way they fund software development and/or procurement projects to realise the desired sharing and reuse.

So how can 'Systems Thinking' be introduced to the CxOs without appearing to be too academic?
The rule is to avoid talking about Systems Thinking and, for that matter, SOA. Instead, the discussion is focused on delivering business value around a topic that is front-of-mind for the board: Compliance, Profitability, Green-agenda or Strategy execution might be such topics. Then the trick is to use simple 'Systems Thinking' techniques and tools (akin to S.W.O.T. or Forcefield and Mindmaps or PowerPoint) to start to describe the emergent services.

As Chris points out, it's important to be looking at the cross-cutting concerns of the enterprise (and the extended enterprise - but extended from a business sense, not an IT sense). I believe that getting the CxOs to buy-in to SOA via Syetms Thinking - (without necessarily trying to teach them the Theory!) - is the way to make SOA work, and for that matter, improve many aspects of their business operations and strategy execution with or without SOA or with or without IT systems.

To bring to a close, I'd like to paraphrase Chris again:

“We must focus on what the enterprise wants to achieve. There are many ways of getting "it done". Goldblatt in "The Goal" makes some useful analogies. The goal in manufacturing isn't about keeping the machines busy, it is about increasing value - converting raw materials into products at the optimal rate for making profits (even if you have to underutalize resources). In the SOA world, the corporate goal is not to maximize the use of IT tools, (to suborn everything to the technical services oriented architecture), but to look for services that deal with the Events that the business has to deal with. I think we have to get the very loose coupling done first before thinking about SOA in companies. Until the businesses think in terms of hand-offs instead of commands, they won't get any of the benefits possible anyway”.

I believe subtly applied Systems Thinking will help us put controls where controls are needed, don't control what doesn't matter and help us answer the question; “How do you focus on what is important, and at the same time not miss the critical details?”

The Cynefin framework and 5D lens are both tools that can help introduce Systems Thinking to broader audiences.

As always, I welcome your comments.

Services Fabric and Data Management

Reflecting on the previous posts referring to events, content and semantics it's apparent to me that data management is ever more important in a world of services. Just because a database is buried deep behind a service interface doesn't mean we can ignore the basic principles of data management.

As we know semantics are critical at the service interface or the definition of an event, but how do we know that data passed through an interface is handled correctly? Given that we're moving to a service world in many cases by service enabling legacy applications we need to establish a framework for managing data across the service enabled landscape. We can not rely on a sea of services that are going to perform every validation for us - much of this will still be left embedded in applications. After all how would our services perform if every valdiation required a service invocation?

The framework needs to deal with:

Synchronisation (including translation) of content - ensuring that reference data or master data (not my favourite terms - see below) are up to date and distributed to where it's required when it's required. As above real time look up of content will not be performant in many situations. Therefore data needs to be 'cached' as part of the service implementation.

Synchronisation (including translation) of definition - as with any language, establishing a single dialect for the business and implementing it in its information systems is probably not a reality, especially when many system components are sourced from outside the enterprise and/or need to interoperate with customer and supplier services. Is Oracle's semantic model the same as SAP's?

Data (and service) ownership - as those that have tried will know, until this fundamental principle of data management is established many of the advertised benefits of service orientation will remain elusive. Unless ownership of data and associated rules are established multiple definitions and implementations will remain along with multiple service implementations performing similar functions.


As mentioned above these are the basic principles or needs of enterprise data management. However, apart from a few execution tools aimed at areas such as master data manaagement and canonicals I see very little discussion on this aspect of the services fabric. Of course this may be limited to whether SOA is treated as an IT or Business issue. The development and management of business ontologies has to be a cornerstone for establishing true SOA.

In this vein, what is the collective for the 'things' the business cares about (e.g. orders, customers, accounts)? Objects? Entities? Types?

(Note: Reference data or master data are not my favourite terms as these terms appear to lump in to one bucket what are probably a number of critical subject areas that define the enterprise. Indeed, MDM appears to be an IT persons response to the problem when of course managing information about the basic assets of the organisation is a business problem, requiring ownership and process.)

One Registry to Bind Them All

I’ve been wrestling with what seems to be a common problem – finding the right language to describe, arrange and catalogue services/sub-services.

I need to describe everything my client organisation offers its customers in a common easy-to-consume ‘Service’ language, regardless of how the service is delivered – some services are technology-based (e.g.. SaaS or Data Centre related) and some are competency-based (e.g. software lifecycle expertise or SME advisory). I want to be able to describe all the services offered in a consistent, non-technical, language that is readily understood by both service consumers and service governors alike. And just to make life more interesting, services are described at varying levels of granularity which requires a degree of consolidation of some of the finer-grained services into ‘Service-Bundles’ – if only for ease-of-consumption. When I look at the prevailing services registry technology, it seems the current SOA-based registry offerings are too limited in their scope. Interestingly, although my client is building to an SOA, the usefulness WSDL/UDDI-focused registry is unclear today.

What is needed is a more expansive set of tools that deal with full service lifecycle management. This includes the classification, marketing and consumption of all the services involved in business transactions beyond the narrower world of SOA. So it seems there is a need to support a broader landscape of all the services both provisioned and consumed by a given business domain. Features might include:

• Service Taxonomy (relationships between services – across levels of granularity)
• Delivery Medium
• Lifecycle Management
• Proto-Services (candidate services not yet exposed)
• Version Management
• Service Withdrawal
• Event Definitions (real-world or technology-based triggers and exceptions)
• Content Definitions (any human-based or technology-based information exchanges)
• Policies and Policy Management (any rule, law or binding contract that constrains the service)
• Value and Trust calibration (inter-dependency and degree of desired or necessary binding).

I’m sure there are many other dimensions that might be included, but you get the general idea. It seems that even within the die-hard SOA camp, the once prevailing UDDI/Service-Broker-view of the world is giving way to a slightly more traditional perspective on how we provide and consume services (i.e. how business was transacted way before anyone uttered S-O-A). SOA thinking, however, is forcing a much needed degree of rigour on the definition, management and operation of services. Might a combination of these two perspectives (traditional services and SOA) drive out some interesting thoughts around what it really takes to operate in a marketplace of interacting services and therefore what a comprehensive Services Registry might look like?

Capability vs Service: What's the difference? Does it matter?

So what exactly is a Service? If we are seeking a common definition of what SOA is (and much is made of the lack of this on the web), then we could at least start by defining the 'S' (in SOA).

There is some emerging consensus in some SOA communities that 'services are a way to gain access to, or make use, of a capability'.

The uninitiated might propose that this sounds rather cryptic, but it is actually important because defining the difference between a service and a capability is one of the key concepts to get to grips with, in order to design an SOA that actually is workable.

To illustrate this we need to look at a business example, as making IT architectures more business-meaningful what SOA is about. (I've written before on my Enterprise IT blog about how the implicit goal of SOA is to create IT architectures that are structured explicitly like the business they support, rather than being structured explicitly in IT construction terms I.e. systems I build, data I store, software I deploy, hardware I install, etc). So let's look at a business example.

Imagine I run a practice in a consulting business. I offer services externally to my clients (defining supply-chain strategies, say). I also offer services internally to other parts of my consulting business (maybe sales-support to the sales team, solution audits to the delivery management team, etc etc). Against all of those services I need to know what my value proposition is to those who consume my services. I need to carefully define the quality of service (or service levels) that I can offer. And I need to know how I’m going to generate 'value' for me (directly as revenue, or indirectly) and what my 'costs' are going to be for those services (again directly or indirectly).

Of course I will require some capabilities that I don't have, or don't want to carry myself, which I can obtain through consuming services from others (maybe HR and Finance services from the common back-office for example).

But the services that I offer to others, and the ones that I consume from others, cover a fraction of the capabilities that I or they need to have in order to operate. For example, I need to have a management capability for a start, but it’s not some thing I'm ever going to be to offer as a service to anyone, externally or internally. But it's still a capability I need to operate. In particular I need it in order to deploy my resources optimally, to grow my practice's capabilities appropriately, and generally run efficiently.

So I define the services that I offer, and I define the capabilities I need to support them. They are not the same thing, and it is important to differentiate. The services that I offer define my value proposition to my customers, partners, or other consumers. The capabilities I have define my operating model, internal mechanisms, ways-of-working, and even technologies. Of course, of the capabilities that I need, I may have them in-house or I might source from somewhere else via a service that they provide to me (at a service level that I find acceptable for my needs).

The key point is here that if you treat everything as services (as some SOA dogma would either have you do), then there’s a strong argument to say that you’re devaluing the concept of service to the point where it becomes meaningless.

If you were to do so, you'd be treating your capabilities as services, which they are not. The two have very different characteristics, and need to be defined in different ways. Furthermore you'd be massively increasing the number of services you need to manage, which is just asking for trouble as services are complicated things, with overheads that you need to manage around the service levels, policies, contracted interoperability etc. I would advocate that you only want as many as it is valuable to offer. The rest are better served as capabilities.

So if you're having trouble defining your services with sufficient clarity and rigour, take a minute to check whether that's actually because the service you're looking at is not meaningful to the outside world. Maybe you're actually over-reducing the service being offered to the level of the process it uses internally - maybe it would be better treated as a capability?

I should mention that Steve Jones has also recently put down some of his thoughts on a similar subject in his blog. He has some interesting thoughts on how you draw the boundaries.

Technorati Tags: SOA Service Oriented Enterprise IT Enterprise Architecture EA

Enterprise Architecture v Services Federation Architecture

This post is based on a number of recent discussions (many of you will recognise some of the thoughts). It posits - A Service Federation approach would provide a better way of understanding, aligning and delivering business and IT services than that of traditional Enterprise Architecture.

‘Services Federation Architecture’ (SFA) is a simplified, more realistic and risk reducing approach to building IS systems. That is, realistic in the sense that the model provides a better representation of how the real-world actually operates.


Enterprise Architects usually approach their work top down– starting with business vision and strategy and working down to the implemented system components and the activities of the people that use them. This way of working has had a degree of success within the four walls of the organisation (in truth, however, rarely applied enterprise-wide). But more importantly, it breaks down across highly federated business scenarios like supply chains. Here multiple business domains interact around a common purpose and at the same time, operate within the context of their own business mission and behavioural traits. This world is less precision-engineered and more negotiated as events unfold – but within a simple model of well defined but relatively simple “touch-points”.

The need for a simplified approach services (from definition to delivery) is obvious in the highly federated world of international trading partners. For example, the joining up both human-based and technology-based services across an international supply chain involving different originations across multiple countries and many languages. Anything other than a simple model of the interfaces between interacting services would simply fail to be meaningful. Each of the members of this supply focused Value System ( “system” in the General Systems Theory abstract sense) operate both within the context of their own independent Value System and within that of the overarching supply chain.

Each constituent organisation is potentially both a consumer and provider of services (within the context of a particular business activity or task). However, despite the apparent complexity, these interacting services are naturally understood by the individuals involved and as is the notion of service governance through policies, contracts and SLAs between parties. The need to document the descriptions of the services provided and the associate information passed between each service (either verbal, paper-based or electronic) is also intuitive. There is a clear understanding of responsibility across the interacting domains and little desire to ‘peek into’ others processes as long as service levels are met or exceeded.

Contrast this with artefacts prescribed by many Enterprise Architecture frameworks. These frameworks focus on engineering rigour and traceability from the coarse-grained business strategies down to very fine-grained technology widgets. The architecture and engineering process wraps itself within its own language and with it, introduces multiple layers of translation between the originating business concept and the final operational outcome. This has the effect of dis-integrating IT away from the business and introduces delay and increases the risk of misinterpretation.

So what? - I believe we might be better off describing, creating and managing services using a “Service Provision in a Supply Chain” pattern rather than the “Construction Blueprint” or even, the slightly more applicable, “Town Planning” pattern. By doing this, perhaps, we can find a way to simplify how we conceive and deliver IT-enabled capabilities and at the same time, reconnect IT to the business.

There are many aspects to this discussion and as always, the interesting stuff is probably to be found somewhere between engineering rigour and operational pragmatics. I would like to explore this thinking here over the coming weeks. I also believe this is fundamental to the whole Services Fabric theme. While similar discussions have taken place in the past within the context of Object Orientation for example, I believe now is the right time to explore a more holistic, abstracted, macro approach to both business (e.g. trading-economies and global ecological needs) and IT (e.g. Web 2.0, Semantic Web, CEP.. et al). I feel this might drive out new insights and principles that can be applied at a macro level (e.g. pan-nation trade) right down to services delivered within an enterprise or by an individual - in all cases; without regard for the service delivery mechanism (e.g. IT-based or human-based).

Other related areas for discussion that come to mind:

• Fractal Model (i.e. a model that is recursively constructed or self-similar, that is, like a shape that appears similar at all scales of magnification)
• Value Networks and Domains of Control
• Service Granularity: How to get in right – the Holy Grail!
• A better representation of the real world and real-world-aware systems
• Social networking and trading models
• Multiple concurrent centres of gravity
• The importance of understanding Events in a federated model.
• Business and IT communication
• The relative importance of re-use and of what?

Demise of the Application?

Does SOA at last move us beyond the 'traditional' notion of an application and elevate the space between 'applications' to that of a first class citizen in the world of information systems? Today, users and IT practioners alike reinforce the notion of traditional application stovepipes in the way that they (applications) get discussed as servicing particular user communities. This affects many aspects of how we do IT, from how things are run to what is paid for.

SOA and event management operate in the world between 'applications' and as such require us to re-evaluate the boundaries of our functional and data blocks. Maybe EAI was the first IT trend to bring attention to the space between applications, but the Application was still central. SOA by definition places emphasis on Interface. We need to manage interfaces and the infrastructure that enables them, not just on a per project or application basis, but enterprisewide.

This is where applications move beyond the database in to the world of communications; not at only at the network transport level but at the semantic level. Indeed, this a paradigm that may be more readily understood by telecoms people, who talk in terms of protocols and interfaces, whereas application people talk in terms of databases, business logic and user interfaces. A user interface being no more than an adapter (a service wrapper?) for a particular form of external system, all be it a rather demanding and unpredictable one.

This emphasis on interfaces and boundaries between systems aligns well with the enterprise architects view of the world but naturally creates a tension with the application owners and developers. Ultimately for SOA to be successful we have to unlearn some of these traditional IT practices and look to other disciplines to help us.

The State-Machine in the Sky

The jokes about solving all information problems with the utopian concept of a single 'database in the sky' that knows everything are well known. But is the new utopia that some describe of centralised top-down process management just as bad, just as impractical? Service-Oriented Architecture (SOA) is often presented as wedded to this concept of centralised top-down process management, but it is not. SOA may require composition (creating services composed of other services) and choreography (sequenced interaction of service providers and consumers), but it doesn't necessarily 'depend' on centralised state-machine-type orchestration. I sometimes worry that people are chasing another conceptual utopia which is not fit-for-purpose to be applied across-the-board in the way that some seem to suggest it should be, and that's one of the ways that business-cases don't stack, and expectations are missed.

Of course this is not to say that BPM isn’t great, I'm certainly not saying that BPM doesn't have its applications, and Process Management is an important discipline. I just feel more and more that it shouldn't be touted across-the-board utopian end-state as it sometimes is. I have a strong gut-feeling that people's desire for visibility and coordination across businesses can often be served through ways other than enforcing top-down control (and by association, top-down constraints).

I had a conversation around a similar topic with Alec Sharp at a data management conference a few years ago and even though of course he used different classes of process to explain his framework, I've always found it very interesting. He described three classes of 'processes' (and I'm using my words here because I can't remember his exactly):-

1.Formalisable processes like certain well-known back-office processes (e.g. Purchase-to-Pay) or certain supply-chain processes.

2.Dynamic sub-processes like customer-interaction where actually you really needed to focus on how to respond in certain scenarios, because you either couldn't predict or formalise the end2end process, or it's not useful to do so.

3.Totally unpredictable processes where actually you just needed to focus on getting information together in front of a sufficiently skilled person who would decide what to do from it.

If I use Alec's categories I might speculate that the first class is most suitable for top-down process management (although in many industries, off-the-shelf application-centric flavours of process management may be more practical than pure BPM).

I might speculate that for the second class you really need to focus more on Event Management, and semi-formalised responses to business scenarios underpinned by information availability, rather than Process management, but there are some obvious similarities.

I might speculate that for the third class you need to focus on information management rather than anything else. And that doesn't mean data aggregation and databases. It's far more likely to mean providing means for federated groups to share their own information repositories, be able to search for it and find it in ways they can understand and aggregate. There is of course a lot of Web 2.0 implications in here, as this is more like searching, linking, tagging and subscribing, than it is like old data management solutions of consolidating databases, creating static interfaces etc.

One last kicker to this point is that conventional wisdom might split the three categories into different types of 'role', maybe something like 'procedural', 'management' and 'strategy'. But this runs the risk of underplaying the pervasiveness of the second and third classes of requirement. If you position them as just for a top few top-management, you miss the point that actually most Enterprise roles these days involve all three types and architectures for these can be just as important as, sometimes more than the one's focused on centralised process-management.

Technorati Tags: BPM Process Management SOA Service-Oriented EA Enterprise Architecture