Role of Systematic Reuse in Enterprise Integration

January 7, 2012

Systematic reuse in the context of enterprise integration provides several benefits:

  • allows the team to evolve consistent set of data interfaces across integration efforts – if multiple systems need to integrate with yours, can you define a generic data model for your domain-specific concepts? This could be file layout(s), service interface schemas, or platform specific APIs
  • enables common interfaces for domain specific services and business processes
  • integration points often need cross-cutting concerns such as logging, transaction management, exception handling, etc. that can be reused
  • reusable utilities can help with support and testing tools – for instance,  reply failed messages, drain message queues/topics, generate high volume test data, etc.

Question is – are your teams taking advantage of integration points? or is each integration point implementing identical or similar functionality in an inconsistent fashion?


Software Reuse Quick Tip #29

January 1, 2012

Tip #29 – Automate Documentation on Reusable Assets

Happy new year 2012 :-)

What is one key reason developers have a difficult time finding and evaluating existing assets? Lack of robust documentation including what the reusable asset isn’t meant to do. Though it is a critical success factor, maintaining documentation manually is a time consuming task and is the first item that gets left out when the development team is racing to meet a deadline. It will be useful to generate documentation on service clients or library client code snippets alongside the provider code. Automate documentation as much as possible – this will come in handy when fixing bugs, integrating new consumers, as well as integrating documentation within IDEs. Here are a few examples of doing this:

  • Maven javadoc plugin for example can generate javadoc style HTML documentation for various java and web modules
  • Maven site deploy can be used to publish generated artifacts to a remote host
  • XSL stylesheets can be used to generate HTML documentation from XML schemas (XSDs) – this can be handy when exposing reusable services (e.g. using XS3P)

Enabling Self-Service for Easier Service Integrations

May 31, 2011

There are a number of benefits with exposing services and enabling application to application integration through them. However, when the number of services increase, the support and integration effort involved goes up as well – moving to a self-service model for service integrations makes it simpler and faster for clients, cheaper and more reliable for the service provider. Here are a few tips to make this happen:

  • Ensure all service interfaces are URL accessible – e.g. WSDL, associated XSD schemas etc. are hosted in a consistent structure that tools like WSDL2Java or wsdl.exe can point to and generate client side code
  • Templatize the WSDL and XSD to capture client-specific information such as application identifier and name (e.g. if your service interface mandates these values, you can default them to use the appropriate client-specific value).
  • Capture authentication and authorization requirements and similar to above point, generate code to set appropriate message headers (e.g. SOAP Header or JMS Header as appropriate) – this will be common to most clients except the exact credentials will vary
  • Keep documentation up to date as part of the build and deployment process – e.g. if you are deploying services to a client facing environment (e.g. integration testing), deploy documentation as well as part of the exercise. For instance, generate javadoc style XML schema documentation and maven site-deploy target can be leveraged to upload the generated artifacts to a web application.
  • Capture and maintain service metadata – this metadata should not only cover functional aspects but non-functional attributes as well. For instance, if your service cannot honor a response time requirement, it is critical to know that during the integration effort and not after the client has migrated their code to a production environment. This could translate to a service enhancement or an alternate integration approach altogether

This is by no means an exhaustive list but meant to give a window into the opportunities and benefits that self-service can provide.


Software Reuse Quick Tip #28

May 2, 2011

Distribute Configuration For Various Testing Needs

If a reusable component requires runtime configuration (properties, XML files etc.) alongside the binaries, make it easier for the asset’s consumer to integrate these artifacts in their tests. For instance, using maven assembly the configuration files can be packaged as a jar file and added in the test scope. This will make the configuration available in the classpath when executing unit tests in the module. The configuration can be packaged per environment and using the dependency classifier the appropriate artifact can be used. This approach works for integration and performance testing as well – for instance, performance tests might use a different set of values for the configuration artifacts since they execute in a dedicated environment.


9 Quick Tips to Reducing Technical Debt

April 13, 2011

Wrote earlier about the importance of refactoring and continuous alignment within the context of systematic reuse effectiveness. Reducing technical debt is an integral aspect of refactoring. This post provides tips for reducing technical debt in your software assets:

  1. Minimize redundant definitions of key domain objects (i.e. competing, conflicting definitions of the same domain object across related applications)
  2. Minimize similar solutions for slightly varying business processes and instead create common process flows
  3. Loosen tightly coupled integration/transport logic with business logic
  4. Provide consistent strategies for implementing cross cutting concerns
  5. Replace tactical implementation for a problem that has a better open-source alternative
  6. Eliminate redundant approaches for managing configuration information
  7. Harmonize multiple, incompatible interfaces and make them more domain relevant
  8. Minimize excessive coupling with a particular technology/implementation stack
  9. last but not the least – create a comprehensive suite of automated tests

Are there similar themes in your development efforts? What steps are you taking to ensure technical debt is addressed?


Guest Post: Succeeding With Reuse In The Enterprise

March 18, 2011

Who doesn’t like the idea of reuse? It’d be like saying you don’t like mom, apple pie and puppies (or kittens if you’re a cat person). We can all say that we like the concept – but putting a successful reuse program into action has been a challenge.

To help increase the chances of success in running a reuse effort, we can embrace two ideas – having a target and leveraging the use of patterns.

Let’s start with the idea of having a target. In thinking about adopting reuse within an organization, a quote that I like is: “If you don’t know where you are going, any road will get you there.”[1] To help guide us in figuring out where we want our reuse program to take us, we can refer to a maturity model. A maturity model provides a summarized view\framework that incorporates industry experience, research and guidance. It’s a first step in reuse – reuse the experiences of others in setting up your reuse effort.

A maturity model can help us figure out where we are: What are our strengths? What are our weaknesses? There are a number of different models out there – but the important thing is to find a model that works for our situation. An example, the Reuse Maturity Framework[2] is shown on the right.

Reuse Maturity Model

We could quibble about a few of the aspects and characteristics – but it certainly gives us a great starting point in figuring out where we are today and where we want our reuse program to go. Each of the columns represents a stage in the maturity of the reuse effort. Moving from left to right shows an increase in reuse maturity. Each row maps to a characteristic\aspect of the reuse effort – highlighting that such an effort has multiple facets. Some of the key things that we can glean from this framework include:

  • Recognition that culture is a major component and we need to get past just incentivizing and make reuse a part of the organizational culture
  • Planning for reuse is part of the strategic effort within an organization
  • Assets that are acquired\created need to be managed and classified. If assets cannot be found easily and quickly – mapping to the needs of the moment – then they provide no value.
  • We need to consider the capabilities of the tools that we use and how they support reuse.
  • And related to tooling – not only do we need to capture metrics – but we strive to find ways to automate the collection of data and analysis of results.
  • Recognition that in most organizations it is the breadth of reuse that drives the need for greater structure and tooling, so that the cost of reuse is minimized while the value\return is maximized.

Moving on from the maturity model, we can now look at the use of patterns. As you may recall, patterns provide a proven best practice solution to a known, recurring problem. Patterns provide a concrete and tangible way for people to experience reuse. Which all sounds great – but the success of using patterns is limited when we take an ad hoc, inconsistent approach to using them and fail to measure and consider the impact that they deliver.

So we need to go beyond the simple statement of “use patterns” and look for an approach that provides more support. More specifically, we should follow Patterns-Based Engineering (PBE) – a systematic, disciplined and quantifiable approach to using both written and automated patterns in delivering software solutions. With PBE we look at how to improve how we identify, produce, manage and consume patterns. Recognizing that not only will we work with patterns provided by others – but we will also look at identifying and creating our own patterns. PBE help us to get started with reuse AND it helps us to reach higher levels on the maturity framework.

Those that have adopted PBE, have found it a great way to get started with reuse. It has helped them improve the quality of their solutions, deliver solutions more quickly, and better leverage skilled resources. To find out more about PBE, including a detailed case study, a development practice and a set of PBE patterns and guidelines – I encourage you to visit patternsbasedengineering.net.

About the author

Lee Ackerman, a Distinguished IT Specialist at IBM, has extensive experience with patterns, model-driven development, XML, data, and SOA. He has spent the past few years helping IBM build its Patterns-Based Engineering approach and offerings and has written and presented extensively on these topics.

You can follow Lee on Twitter or via his personal blog. Lee can be contacted via lee@patternsbasedengineering.net.


[1] Lewis Carroll

[2] Image is sourced from: DeCarlo, J., et al. 2008. Strategic Reuse with Asset-Based Development. IBM Redbook. www.redbooks.ibm.com/abstracts/sg247529.html which is based on work done by Koltun and Hudson as discussed in Lim, W. 1998. Managing Software Reuse: A Comprehensive Guide to Strategically Reengineering the Organization for Reusable Components. Upper Saddle River, NJ:Prentice Hall.


Benefits of Decoupled Service Contracts

March 13, 2011

The Decoupled Contract pattern separates the service contract or interface from the physical implementation. The service interface is independent of implementation yet aligned with other services in the domain or enterprise service inventory.This is a follow up to the podcast episode on building contract-first services

Distributed development technologies such as .NET and Java allow easy creation of service contracts from existing or newly created components. Although this is relatively easy to accomplish the service contract tends to get expressed using the underlying technology platform – one of the many anti-patterns to avoid. This inhibits interoperability and increases integration effort for service consumers. Additionally, service consumers get forced to a specific implementation thereby considerably increasing technology coupling. Finally, this is problematic for the provider as well. If there is a need to upgrade/replace/change the service implementation, there is high probability that existing consumers will need to change their code.

The Decoupled Contract pattern solves above problems by decoupling the service contract from the implementation. Instead of auto-generating service contract specifications via a WSDL document, this pattern advocates the creation of contracts without taking implementation environment/technology into consideration. This contract will be free of physical data models instantiated in a backend data store and proprietary tokens or attributes specific to a technology platform. Consequently, the service consumers are bound to the decoupled contract and not a particular implementation.

The service and the consumers will still be coupled with any limitations associated with a service implementation (for instance, an implementation may not be able to realize all features of the contract or guarantee policy requirements etc.). It is also possible for a particular implementation technology to impose deficiencies across the service inventory in its entirety. But even given these impacts, the contract-first approach that this pattern facilitates is significant and foundational to service-orientation.

This pattern is beneficial to several design techniques that use Web Services or benefit from a decoupled service contract. Contract Centralization and Service Refactoring patterns are greatly enhanced by this pattern. Service Façade is often applied alongside this pattern as well to realize Concurrent Contracts.


Agile Software Reuse Design Practices Primer

March 5, 2011

Pursuing systematic reuse the agile way? This primer will cover a variety of design practices to help your development teams. It covers:

  • Building reusable assets from existing applications
  • Designing new reusable components and services
  • Design patterns, product line practices, and more!

Rationale for Compensating Service Transactions

March 5, 2011

Compensating Service Transaction pattern helps consistently handle composition runtime exceptions while eliminating the need for locking resources.

Service compositions could generate various runtime exceptions as part of fulfilling service functionality. For example, imagine a service composition that invokes a decision service to validate request data, proceeds to update a customer entity via an entity service, and then sends a message on a queue for an external process to consume. Consider the three steps as part of a single unit of work – a service transaction that execute in sequence.

If runtime exceptions associated with the composition are unhandled there is an increased likelihood of compromising data and business integrity. Similarly, if all the steps are executed as an atomic transaction each service invocation will tie up backend resources (e.g. using various kinds of locks) hurting performance and scalability.

The compensating service transaction pattern introduces additional compensation steps in order to handle runtime exceptions without locking system resources. These additional steps can be included as part of the composition logic or made available as separate undo service capabilities. Continuing the earlier example, a call to undo the customer update (essentially resetting the data back to the earlier state) can be made to place the underlying data in a valid state.

The Compensating Service Transaction pattern provides several benefits:

  • Eliminates the need for executing distributed service invocations within a transactional context.
  • Provides a mechanism for services/resources that don’t support atomic transactions to participate in service compositions.
  • Reduces load on underlying systems and databases by only invoking compensation routines when failure occurs. Instead of locking resources in case failures happen, the idea is to handle runtime exceptions when they actually occur.
  • Allows reuse of compensation steps across service compositions. E.g. two business services that update a data entity can reuse the undo Update capability.

Compensating transactions tend to be more open-ended when compared to atomic transactions and their actual effectiveness varies. The extent to which the service composition can apply this pattern is directly dependent on the undo capabilities provided by various service capabilities being invoked. Consequently, this pattern cannot be leveraged if compensating transactions don’t exist. Likewise, it is necessary to ensure that the compensating steps themselves execute successfully. Error alerts/notifications may need to be sent in case compensations fail and manual interventions are needed.


Abstract Utility Functions for Effective Service Reuse

February 23, 2011

The Utility abstraction pattern enables non-business centric logic to be separated, reused, and governed independent of other services.

Utility Abstraction pattern encapsulates cross-cutting functionality across service capabilities into a separate component. For instance, cross-cutting functionality such as logging, notification, and auditing are required by several service capabilities. Dedicated services based on the utility service model guided by enterprise architecture specifications.

This pattern provides several benefits: 

  • Reduces and potentially eliminates cross-cutting logic from individual services. This keeps the individual services domain-specific and lightweight.
  • Reduces and potentially eliminates redundant cross-cutting logic that might be implemented across the service inventory. This will reduce development and testing costs while minimizing duplication.
  • This pattern also enables the reuse of utility capabilities across both business processes and service capabilities.
  • In addition to being the central component for a cross-cutting function, this abstraction will facilitate changes to implementation. For example, logging may use a file store and later switch to a database-driven solution. Likewise, with a utility abstraction component, it is simpler to migrate to an alternate provider – replace an in-house implementation with a cheaper cloud provider.

Separating the utility component also has benefits from a non-functional standpoint. The utility function can be executed asynchronously (to save response time for a service) or additional instances can be supported to enable concurrent processing.

This pattern also makes it easier to perform additional functions surrounding the core cross-cutting function. Taking logging as an example again, if archiving/backup policies change there will be one piece of logic to update rather than touch individual services. It is important to note however that this pattern can increase the size, complexity, and performance demands of service compositions.

There are a variety of strategies to realize this pattern – via service mediation layer in a service bus based architecture or using a lightweight proxy that intercepts service methods.


Follow

Get every new post delivered to your Inbox.