November 21, 2009
Problem Statement
A consuming application/business process wants to integrate with your reusable asset. However, the consumer doesn’t want to take your standard message – wanting a specific format that will make it easier for them. Why? Could be because of several reasons: lack of time/money/skills or technical limitations (e.g. their system can handle only delimited files and cannot parse XML).
Suggested Approach
The immediate, tactical (and often tempting) solution would be to just format the data per the consumer’s format and integrate in a point-to-point fashion. Don’t settle for this option too quickly! Prefer to publish a standard publication in that is in line with your long-term direction (e.g. offer reusable message publications in XML format indicating changes to critical data entities or state changes in business processes). Create a subscriber that tranforms the standard message to the consumer-specific format.
Rationale
If point to point integrations go ungoverned, you will end up with a rat’s nest of tightly coupled integrations that ultimately hurt business agility.Your long-term intent is to have multiple consumers (web applications, backend systems, business processes, etc.) consume reusable message publications.If several business processes need the same core data or notifications about state changes, why would you want to integrate with them separately? Publications will not only reduce integration effort, they place less strain on your system resources – publish once and subscribe several times. No need to query databases or invoke external systems every time there is a new integration.
Note: The additional transformation logic will require extra logging/error handling but over the long haul is a better option than going for a point-to-point integration approach.
This will enable the asset provider, to integrate new applications faster (no need to custom develop messages and integration code for every new consumer) and reduce the cost of maintenance. Reducing application to application coupling is a key motivation for pursuing this approach as well.
Like this post? Subscribe to RSS feed or get blog updates via email.
40.099443
-74.932568
Leave a Comment » | 
Design, Reuse, SOA | Tagged: integration, loose coupling, messaging, point to point, publications, reuse recipes, software reuse | 
Permalink
Posted by vijaynarayanan
November 20, 2009
I introduced the data services product line in earlier posts here and here. As a follow up I wanted to set the context for a common architecture for this product line. The idea is to have all products in a services product line need to share the same underlying architecture. The common architecture enables the development of a set of common software components and also facilitates variation management, product configuration, and consistent behavior across all the services.
The architecture typically consists of both domain agnostic technical assets and domain specific assets. The domain agnostic assts includes logging, error handling, auditing, metrics, and legacy service invocation modules. The domain specific assets includes data assets, relationships, domain events, and business process workflow components that get triggered based on rules. The common architecture also provides components to integrate with the workflow engine, initialize business processes, as well as provide approval, routing, and escalation steps for any kind of enterprise data entity.
The architecture needs to be reviewed to ensure that it meets the product line’s non-functional requirements such as availability, scalability, and runtime performance. For instance, supporting high availability can be tricky when data repositories undergo scheduled maintenance on a periodic basis. Scalability is also a key quality attribute that needs to be addressed by the architecture where the need for connection pools, thread pools, and messaging session pools are extensively utilized. Additionally, the architecture can provide caching capabilities for most accessed data records as well as integrate with a lightweight rules engine for fast and efficient rule execution.
Data visibility is another key capability that the architecture needs to address – it can do so by providing an extension point in the processing lifecycle to integrate visibility logic (e.g. filtering attributes, masking data, or utilizing a third party authentication service to validate the identity of the caller etc.). Ability to assign priorities to jobs in the processing queue based on SLA needs is another feature that could be provided by the common architecture.
Finally, the architecture needs to support several utility features: manage connectivity to each data source, allocate dedicated processor threads based on message volume, and monitor the health of each integration point outside the data service logic. The common product line architecture can potentially address both domain agnostic & domain specific needs.
In a follow up post, I will provide more detail on each of these capabilities and how they align with an agile software reuse approach.
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | 
Planning, Reuse, SOA | Tagged: architecture, common, product line, software reuse | 
Permalink
Posted by vijaynarayanan
November 19, 2009
One of the often-cited concerns with leveraging reusable software is design complexity. This is indeed a legitimate concern and as designers, we ought to ensure that it is managed appropriately. In this post, I want to provide some strategies for tackling integration complexity:
- Build Iteratively: this is undoubtedly an effective way to avoid over-engineered assets. Building assets iteratively means realizing functionality in small bites, over multiple releases even. Instead of trying to implement a perfect reusable asset, prefer building in increments. This has several benefits: reduced risk, increased relevance for your applications, early feedback on whether the asset has captured domain relationships appropriately, and opportunities to remove code or refactor behavior on a continuous basis.
- Capture natural variations in the domain: reusable assets that don’t reflect the natural variations in the problem domain run into lots of issues. If you keep scratching your head trying to infer what the asset is trying to accomplish – examine the consumer-facing interfaces and ask yourself, does the interface reflect domain variations or is it providing needless variations or worse, ignoring must-have ones?
- Prefer convention over configuration: This is one of the foundational principles behind why frameworks such as Ruby on Rails are so popular. You can use this idea and simplify assets in many ways! For example, instead of forcing configuration for files, maybe a standard location would suffice. This idea can be leveraged with scripts that setup developer environments, automated regression tests, and reading/saving program output etc. There might also be cases where input data is used to determine class instantiation or stylesheet selection. Again, if you come up with a simple convention, many lines of configuration can be eliminated.
- Loosely Couple Capabilities: Loosely coupled capabilities are easier to change and integrate. By creating reusable assets in a loosely coupled manner, you will also make it beneficial for consumers. Loose coupling provides another important benefit – making it easy to isolate assets and test them as individual components. If you are building service capabilities, explore the use of asynchronous message publications for real-time notifications to data/status updates.
- Strive for consistent naming and behavior: consistent naming reduces learning curve for developers as well as makes it easy for the asset provider to debug, integrate, and test reusable assets. Consistent behavior should go beyond simple method calls – you can extend this to services and business processes as well.
- Make Assumptions Explicit: A lot of design complexity can arise due to incorrect assumptions – for instance, there may be operating assumptions about security, data integration, tracking, and error handling. There are a lot of design assumptions that get made as a natural part of the development process (e.g. every customer will always have a address, or that every customer needs to get authenticated prior to instantiating a business process). Make sure these assumptions are put in the open and for everyone to validate. It often turns out that an asset doesn’t have to implement a feature or that it may be implement an existing feature incorrectly.
- Provide consumer-friendly interfaces: Start designing from the consumer’s standpoint and strive for simple yet functionally rich interfaces. This has several benefits: you won’t expose needless internal complexity associated with the asset to the consumer (i.e. achieve right level of encapsulation) and also make it simple for consumers to integrate with the asset. If you have 10 options for a reusable asset but most customers use 2 frequently, why not set the other parameters with sensible defaults? Consumer friendly interfaces also ensure that you build assets that have tangible business value.
- Avoid abstractions for the sake of technical elegance: not every abstraction is meaningful, especially with respect to your problem domain. I wrote earlier about the domain-specific nature of variations and why one set of abstractions isn’t always appropriate for your problem. Experiment and iterate to get the right abstractions – they will help establish a shared language within the team and reduce needless complexity because of overly generic interfaces.
- Minimize compile-time and runtime dependencies: Reducing the number of moving parts – both in terms of compile time libraries and runtime libraries, services, and systems will make it easier to manage design complexity. Always, ask yourself – is this dependency absolutely essential? Does it introduce a single point of failure in the overall architecture? Is there a way to cache frequently accessed data or return that isn’t 100% up to date?
- Provide Mock Interfaces: When possible provide mock data/objects that can help consumers integrate and test assets quickly. This is related to the earlier point about minimizing dependencies but is also useful for customers to get a flavor for the kind of data or business rules that get executed as part of the asset’s functionality. Mocking also helps with another key benefit: asset co-creation. If you are developing in parallel with a consumer, mocking is a great way to agree on interfaces and develop in parallel.
What is your view on these strategies? Can you share some of the ideas/approaches that you have pursued to tackle integration complexity?
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | 
Design, Reuse, agile | Tagged: agile, convention, Design, domain variations, integration, iterative, software reuse | 
Permalink
Posted by vijaynarayanan
November 18, 2009
List of Return Codes
I wrote earlier about the idea of using consistent error codes for reusable assets. As a follow up, here is a document with a list of reusable return codes that can be used when building service capabilities as part of SOA initiatives. They are categorized into:
- request processing
- data processing
- dependency access
- transactions
This isn’t an exhaustive list by any means but can get you started in terms of achieving consistency across services and projects.The service consumer can use the return code to understand the service provider’s response. This consistent, uniform categorization of return codes can help you reuse error handlers (e.g. handle a particular error in the same way regardless of which service capability raised it). It will also help with production support and troubleshooting – less learning curve for support staff and developers to categorize errors at runtime.
Note: The return code derived based on errors need not necessarily be the return code sent back to the service consumer. It is entirely possible that you return a friendly error to the consumer and a detailed error to production support.
Are there additional ones to include in this list?
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | Design, Quality, Reuse | Tagged: exceptions, return codes, Service-oriented architecture, SOA, software reuse | Permalink
Posted by vijaynarayanan
November 17, 2009
When you develop reusable components quality is very important. The higher a reusable asset’s usage, the higher the need for robustness. Unlike monolithic code, defects with reusable assets can rapidly impact several business processes and applications. All the more reason for the criticality of automated testing. In addition to testing, code analysis tools can detect defects, warn regarding unsafe/potentially buggy code, and also recommend various source code style/formatting improvements. All of these contribute to higher quality. I have used findbugs, pmd, and checkstyle in this space and they are very useful to analyze source code. You can also include these in your continuous integration suite.
Many readers have requested that I post sample code and scripts – so here is a sample apache ant build script that you can use with your code. Just be sure to modify the properties file to point to appropriate folders. The script will produce html reports for pmd and checkstyle as well as a find bugs output file (for viewing it with the findbugs client).
Code Analysis Script
Pre-requisites prior to running this script:
Set JAVA_HOME to your JDK 1.5 or above folder
Install apache ant 1.6+, findbugs, pmd, and checkstyle in your environment
Make sure the findbugs, pmd, and checkstyle jar files are in ant’s CLASSPATH.
Let me know if you have issues with using this script. Enjoy!
Like this post? Subscribe to RSS feed or get blog updates via email.
2 Comments | Quality, Reuse | Tagged: checkstyle, code analysis, findbugs, pmd, Quality, software reuse | Permalink
Posted by vijaynarayanan
November 16, 2009
The template method design pattern defines a generic algorithm and specify hooks for plugging in extension points and variations. How does this fit in with systematic reuse? It supports two levels of reuse: reuse the overall template algorithm over and over. Basically, an entire set of steps that gets repeated as-is or reuse specific steps across one or more template algorithms.
Template method can be used in a variety of scenarios. For example, the following steps could be performed for handling requests:
- validate input request using security token
- standardize data – remove whitespaces, unify upper/lower case, correct misspelling
- pre-processing e.g. transform request prior to processing
- processing step – e.g. executing specific business logic
- post processing e.g. transform response message in a specific format, encrypt sensitive data
Key question here: what are the variations in the above steps?
- authenticating requests – using a LDAP provider vs. using X509 certificates
- standardizing data – use custom implementation or open source tools to fix request data and perform pattern matching
- transformation: might vary by format (XML, plain text, JSON etc.)
- processing: might vary by use case
The security module itself could be used by other apps requiring authentication. the transformation module could be used for a batch process and processing logic could be reusable for more than one project as well. As always, it is key to keep the template algorithm simple, cohesive, and loosely coupled via well designed application programming interfaces.
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | Design, Patterns, Reuse | Tagged: design pattern, software reuse, template method | Permalink
Posted by vijaynarayanan
November 15, 2009
Here area set of ideas, candidate assets if you will, of reusable software capabilities for your business processes. Please don’t take these as capabilities you have to build from scratch. Instead, view them as part of your overall BPM software infrastructure. Most of these capabilities could be provided by a single vendor or using an abstraction layer, you can realize these using multiple ones. You can also prioritize this list when evaluating vendor offerings.
- Rule driven Exception Handler: Integrate exception handling with business rules engine. The rules could determine how to resolve, mitigate, and handle errors. It will also specify routing instructions for errors requiring human intervention.
- Internationalization Support: Status messages driven by a resource bundle as opposed to a single locale. Additional locale-specific resource bundles can be added as required for your business needs.
- Seamless Cross Channel Movement: Enable a business process to start in one sales channel and get completed in another. Idea is to support business processes that start, pause, and get reactivated via an alternate channel. For example, a user can start ordering a book online and request a customer service rep to finish the process. The idea is to have the user’s in-progress process instance data get placed in appropriate work queues in a new business process or in a new state on the original one .
- Business Process Completion API: Ability to determine % complete for any business process based on the state of the process instance . This API can provide the ability to get current status, estimate completion time (e.g. based on historical data), and what steps are remaining for the process instance to finish.
- Business Activity Hold & Retry API: Facilitate pause, resume, and delegation of any business activity based on business rules. This interface would put processing on hold as well for one or more process instances.
- Authentication: Enabling integration with a LDAP store, or providing digital certificate based security for interfaces that instantiate the business process are examples here.
- Entitlements API: Enable fine grained authorizations for business activities and business processes. Role based entitlements for events – i.e. if a particular user role cannot initiate a business process or execute a particular activity, the software infrastructure should prevent those actions.
- Content Management integration: Integrate with content management system to serve targeted content based on state of business process or to augment data in a process instance.
- Event Integration API: Capture or derive events and handle them using one or more business processes. This could also impact existing process instances that are in-flight.
- Indexed Search Integration API: Ability to execute full-text search as well as categorized search using an indexed search engine against completed and in-progress business process instances. E.g. search all process instances from a particular division or with a particular customer code, or category etc.
- Process Dashboarding: Provide key business process metrics – report real time status of availability, performance, usage statistics, escalations, etc. Also provide ability to override/adjust in-flight process instances based on business scenarios.
- Business Process Preferences API: Manage a set of process-level preferences. E.g. default logging level for all tasks could be set to high or all notifications get delivered to additional recipients, or data fixes will get audited a different way etc.
- Document Integration: Attach documents using information in a process instance and attach/route content as part of the business process orchestration.
- SLA Adherence API: Manage service level agreement specifications associated with end to end business process as well as key business activities. Ability to define/handle, and proactively prevent SLA violations.
- KPI Definition and Monitoring API: Monitor business processes to capture key process indicators based on business process. E.g. number of accounts opened today, number of accounts opened after multiple validation errors etc.
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | BPM, Reuse | Tagged: BPM, business process, capabilities, integration, process, software reuse | Permalink
Posted by vijaynarayanan
November 13, 2009
#4 Separate Message Construction from Message Delivery
There are a variety of scenarios where you will build a message and deliver it to a destination. The message itself could be free-form text, delimited/fixed-length text, XML, or some other format. The destination itself could be a system, service, application, a server, or a human user or group of users (in the case of email). If you decouple message construction from its destination, that will drive in reuse of both of these assets. For example, you can take a message and send it via email or send to a message queue. In the same vein, you can potentially use the same code that sends email to send a newsletter message and a reminder message.
To illustrate this idea, take a look at this code fragment:
MimeMessage message = new MimeMessage(session);
message.setFrom(new InternetAddress(from));
message.addRecipient(Message.RecipientType.TO,
new InternetAddress("someone@xyz.com"));
message.setSubject("Hello there...");
FileReader fr = new FileReader("some-template.html");
StringBuffer sb = new StringBuffer();
BufferedReader reader = new BufferedReader(fr);
String line = reader.readLine();
while (line != null) {
sb.append(line);
line = reader.readLine();
}
reader.close();
message.setText(sb.toString());
// Send message
Transport tr = session.getTransport("smtp");
tr.connect("host-name", "user", "some-password");
message.saveChanges();
tr.sendMessage(message, message.getAllRecipients());
tr.close();
There is a file that is read from the file system and a data buffer is being constructed. The data is then used to send an email. However, this class isn’t very reusable because of the tight coupling between message construction and delivery logic.
Now examine, a newer version that has modularized this code – Publisher sends the message using an EmailConfiguration and TemplateBuilder constructs the message using an instance of Template.
Publisher publisher = new Publisher();
EmailConfiguration eConfig = new EmailConfiguration();
publisher.setConfiguration(eConfig);
TemplateBuilder builder = new TemplateBuilder();
Template template = builder.build();
publisher.publishMessage(template);
Now, Publisher, EmailConfiguration, Template, and TemplateBuilder are all candidate assets that are available for other classes to reuse. If you have other types of configurations, they can be included as well and bound polymorphically. This concept can be further extended based on the quantity and quality of variations. For instance, the message construction can be realized using a common interface with multiple implementations. The message itself might have standard and personalized content.
You may move to a more modular version iteratively. Which version do you think facilitates reuse, faster integration, and higher productivity? This new design will make it easier to test message construction independently from publisher.
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | Reuse, agile, refactoring | Tagged: agile, decoupling, interfaces, iterative, refactoring, software reuse | Permalink
Posted by vijaynarayanan
November 13, 2009
There are two critical risks with reusable components – they are needless complexity and domain irrelevance. Needless complexity results in more learning curve for developers to consume the asset and increased development and maintenance costs for the provider. Domain irrelevance results in assets that are of little business purpose and inhibits the ability to reuse capabilities across applications within a domain. Adding business value to a reusable asset should be your #1 priority above everything else. I will focus here on broad categorizations to illustrate this success factor:
Increasing Revenue: reusable assets should help reach new markets, create new products and services. Reusable assets need to be domain-specific for the most part and strive to map to business capabilities. Most importantly, strive to focus on what provides competitive advantage for your firm. Invest in reusable assets that no one else in the market can build because they lack the expertise or resources. Finally, reusable assets should help you drive time to market down. If it takes longer to integrate and test reusable assets than to develop from scratch, guess what your developers and tech leads are going to prefer?
Decreasing Costs: Reusable assets should help by saving you duplicate work. Likewise, reuse can help reduce maintenance costs – but only if the assets are relevant. If your products are for a domain that has no reusable assets, you won’t be able to save costs. Also, when you reuse, there is potential to consolidate vendor solutions, data feeds, and proprietary implementations as well. Do factor in less licensing and ongoing upgrade costs. Note: don’t underestimate the cost of integration – at least during the initial stages of your reuse efforts. There is additional training and integration cost that has to be accounted for, before cost savings can materialize.
Increasing Agility: You can also add value by focusing on increasing agility. This is critical to avoid big design up front (BDUF) and big investments. Instead of trying to build perfect assets, try to build assets that meet your needs. If reusable assets can help you evolve capabilities that can benefit multiple applications, business processes, and services that will aid organizational agility as well – e.g. you can provide vendor agnostic abstractions, provide simple interfaces to complex orchestrations, automate repetitive manual steps etc.
Increasing Productivity: Reusable assets can drive increases in productivity for both business users and developers. For business users, reuse can facilitate access to same or similar capability across applications and processes. For developers, reuse can reduce learning curve (when they are using something repeatedly) as well as manual steps involved in integrating certain assets together (assuming you have not only thought about assets in isolation but when how they work together as well).
Reusable assets have to provide value so there is incentive to create, maintain, and consume it over and over again. This is easier said than done but it will make a big difference to your business objectives.
Like this post? Subscribe to RSS feed or get blog updates via email.
Leave a Comment » | Reuse, agile | Tagged: agility, business value, decreasing costs, increasing revenue, software reuse, success factor | Permalink
Posted by vijaynarayanan
