Build near real-time capabilities even if your consumers don’t want them. Your consumer may not want a real-time interface or maybe unable to integrate with one. It is tempting to just go for a batch based solution because, that is what your consumer is asking for. For now at least. However, if you build one off batch file extracts or directly expose your legacy system to make one consumer happy, you will encounter the ill effects of tightly coupled systems.
There are several approaches to achieve long-term reuse goals and address the immediate customer need:
- Publish a standard message that you will want to treat as a reuse candidate going forward. A subscriber can drain, accumulate messages, transform them to a customer-specific format, and and append it to a file. A scheduled job or process can transfer this file to your customer.
- Create a reusable service capability that provides the data in the target format that you want to maintain/evolve going forward. You can create a batch process that shares the same interface that the real-time service uses. Note: volume is a critical factor here though – you don’t want to make several atomic calls when it is more efficient to fetch data in bulk. You could have a configurable parameter for fetching multiple records at a time – the real-time service can use a much smaller number when compared to the batch process.
- For large data volumes, consider populating a read-only data store using database replication. File extracts can then be driven off this new database. This has the advantage of reducing load on your operational data stores at the same time facilitating additional consumers who might prefer a SQL interface or a file extract based solution. Downsides: additional moving parts and increased cost for a new data store.
When you build real-time capabilities, adding consumers doesn’t involve too much effort. Are there additional approaches to pursue?