Wednesday, July 3, 2013

Efficient integration development?

I have been involved in many integration development projects during my career. Some of them have been successful and efficient, some haven't. What is the magic behind efficient integration developent?

Preparation phase

Open conversation about the goals and the means is crucial. Every involved party must have uniform understanding about goals and functionalities that can be used to achive them. When everyone have clear picture about the challenge, it is much more easier to engage and try to put best effort to achieve it. Motivate project with positive awards.

Patterns and reusability: whenever possible try to identify known solution patterns by which desired fuctionality can be achieved. You should not reinvent a wheel every time, except you have room for innovation in our schedule or known bottleneck in the pattern.

Realistic schedule and resource planning: if you have a complex systems, processes or information objects, do not underestimate time schedules. I have witnessed many cases where decent processes / exchanged information were not known after specification phase but many testing-revising cycles were needed in SIT / UAT environments and estimations and schedule were freezed based on the original specs. This lead to feeling of hurry, extended hours, weekend work etc. which decreased the efficiency and quality of the outputs. One tip from the funding point of view: do not argue too much about time estimations but argue on price if you need to spare or meet some limits.

However set multiple deadlines: it is not a secret that nothing motivates better than deadline.

Specification phase

Spent time to think all the possible scenarios. Rethink. And think again. Produce test cases from and for the systems if possible. Simplify. Consider which party is responsible for the needed functionality and could it be handled at that party instead of the middleware. Innovate and procude material also for the exception cases.

Specify (or reuse) functionality for easy to view and ensure content of the flowing messages before and after conversions etc.  The easier way to check these message contents the more valuable  it will be at the SIT & UAT with complex business processes and structures involved.


Implementation phase

Step by step development and testing: implement one functionality / mapping rule  at the time and module test that it works as expected with the realistic testing data received from the specification phase. Do not try to implement and test all at once, because you will not be (or at least I'm not) attentive enough to check huge amount of the things in one shot.

If possible try to avoid personification, let development and testing parties to spread knowledge and work inside their teams. This can pay you back in one day when the only developer / tester get a flu and is not available for awhile.


System integration test phase

Arrange testing workshop(s)  where all the parties are in place and technical or logical issues can be debugged and fixed immediately. Run and rerun all the system acceptance test cases through in these workshops with increased logging and debugging levels to collect material and experience that these testing cases are working. Try to play with cases and cause exceptions.

Fix dataflows one by one. Do not run multiple dataflow testing and debugging streams with the same resources  at parallel. Your focus will be disturbed and your resources need time to refocus to different subjects and are not as efficent as they could be. Also arrange enough rooms for testing teams so that other subjects do not distract their focus.

Do not move to UAT too early

There are always extra work when debugging and fixing things on two environment parallel, especially you get extra work from evaluating what is set differently in the systems because they are not working the same etc. It is not unusual that some fixes or settings are forgotten to order to or copy to next testing environment. Another thing that might be issue from efficiency point of view is that in some cases deployments to UAT enviroment can be done only in certain time windows and by decent processes.

Friday, June 28, 2013

Question matrix for IT -projects

For each IT -project success, it is essential that every important questions have been asked and answered before or during the planning. I promptly collected evolving question matrix to help me to build questions and check if they have already been covered. Hopefully you can benefit from it too ;.) Any feedback is appreciated.

What How When Who Where Why
Business process
Change management
Project management
Partner management
Vendor management / selection
Deployments / release management
Application Management

For example: What benefits do we get? How do we get those benefits? Who will benefit from these new functionalities? When do we expect these benefits to become reality? Where will these new functionalities be taken in place? Why should we market these new functionalities internally? . . .

Monday, June 24, 2013

Homemade application from middleware / ESB layer point of view

While I was laying in the hammock during the midsummer weekend  I let my mind wander free. Suddenly my free mind picked up the path that led me to think about past integration projects and what I can learn from them.

I have been involved more than a decade in developing integrations from and to the complex applications. I strongly agree the idea of making integrations with vendor independent standard ways. Best approach, because of strong standardisation, so far have been to use Web Services (WS) as much as possible to consume services, fetch from or upload data to applications.

At the beginning we had no previous experience about designing WS's. We did services what we thought --at that time- to be suitable, but now we have learned that certain things could be done differently.

First of all GUI, business logic and WS part of the application should be packed into their own (J2EE) applications and persistences. This allows better possibilities to update user experience and business logic part while WS part remains the same and can be up and running while the service break for the others. Especially if you are developing homemade application with agile, tight scheduled intervals it could be usefull not to stop WS part (connectivity services) or not to use energy to build complex retry logic to middleware / ESB implementations to support product updates. And if the WS part is done like I propose below, the real business logic and data models could be updated too without touching the integrations or information objects used to interact with service consumers / clients. Separated GUI, business logic and WS parts can be also scaled individually based on the realised workloads.

Second lesson learned is that application specific and the other perhaps more common services the project produces should not be mixed in the same application and persistences. They should be held strictly separated in their own. Reasoning can be the same as above.

The level of the abstraction and granulation of the WS:s were deeply discussed in the past, but now we have learned that in some projects we chose to publish  too elementary services. For example in one project one needed to use several services / calls to upload data to the application. It was fine when there were a few connected systems and users. When volumes increased there were too much xml -parsing and serialization and logic processing for the one J2EE application that provided also web interface for the users at the same time. There were severe problems with application server resources (memory and cpu) and the response times for the WS:s calls and it caused unnecessary errors (time outs or internal server error responses) and management work to the middleware / ESB layer too.

Near time instead of the real time!

From the middleware / ESB layer's point of view the interface should be fast and reliable also with great volumes. So with the applications that applies complex computing (like route calculation) to the data it receives or produces one should consider to separate the real time application data from the data WS -layer (connectivity services) inserts or updates.

My suggestion is that one consider to use  the following model --idea copied from the SAP- for the complex applications.
  1. Received data will just be put to the persistence to wait for the complex processing and uploading to the application. Middleware / ESB layer gets the acknowledgement back that data were received by the WS part (connectivity service)  and   the status of the processing can be tracked with id xxxx.
  2. Just the syntax of the received data should be validated against the schema while interacting with the middleware  / ESB layer, not the content.
  3. Own services / operations should be provided for tracking the status of the complex data import / update to the application.
  4. Perhaps there could be usefull to implement a GUI from which business users can manually edit and reprocess failed / incomplete data from the WS part (connectivity service) to the application.
  5. Perhaps there should also be caches for the data produced or published with complex business logic. Cached preprocessed data could then be published with simple and light logic with great throughput.

In the picture above I try to express ideal architecture that helps to make maintainable, scalable and reliable application from middleware / ESB layer point of view.

Wednesday, March 13, 2013

Lesson learned about WS -interface design

I read a document IEC 61968-100 / Application integration at electric utilities –
System interfaces for distribution management –
Part 100: Implementation Profiles. Great advices how to implement WS-interfaces, on your application infrastructure, that comply with the standard. Some notes and thoughts below.

Start from Information Objects   

First thing for designing new WS-interfaces is to consider information objects that the new functionalities pertains to. If possible try to find standard  that defines objects and their naming conventions (noun) for you. There is a good chance that the same standard defines actors, roles and interaction patterns (verb) in that specific domain. If so, follow them as straight as possible.

Draw interaction and sequence diagrams

Draw interaction and sequence diagrams based on the use cases the interfaces are meant to be used for.

Define Service Semantics

Based on the diagrams, services are identified for the entire message flow. The naming of the services are based on their service pattern (based on the role in entire flow). Each service will have the same operation as indicated in the diagrams as messages, but have different service name.

Conventions for version controling

Use targetNamespace and version attributes to define the version of the used schema or wsdl.


<xs:schema targetNamespace="http://ORGANISATION/DOMAIN/STANDARD/RELEASE DATE OR YEAR/VERB + NOUN" version="1.0">

Add detailed version information with annotations

<xs:schema targetNamespace="http://ORGANISATION/DOMAIN/STANDARD/RELEASE DATE OR YEAR/VERB + NOUN" version="1.1">
       Major version 1.0 created in 2013/03/12
       Minor version 1.1 created in 2013/03/13 - one optional element added for the object xxx

Don't change targetNamespace in the minor backward compatible changes to avoid unnecessary WS-client rebuild. Data types are typically defined in the external schema files and imported in wsdl instead of being embedded for better version control.

Conventions for the wsdl 

Service and operation naming patterns can be used to implement strictly typed web services.

Wrapped Document style is normally used. 

  • document style means an XML document is included in a soap message. In the normally case, it is directly placed in the <soap:body>
  • if wsdl:operation name is the same as the input element name wsdl is considered as wrapped document style wsdl
  • The input message has a single part
  • The part is an element
  • The element has the same name as the operation
  • The element's complex type has no attributes

Operation name follows the Verb + Noun naming convention. The plural form of the information object name is used as a noun to avoid collisions within the xsd. For example CreateEndDeviceEvents

Service name -  <Service pattern name>+<Noun>. For example ReceiveEndDeviceEvents

Service patterns can be defined from used standard or the use cases the new interfaces are meant to be used for.

Role of the actor (service provider) in the message flow defines which service name pattern to use

Some common service naming conventions

To provide (send) information (information object) for public (enterprise) consumption. To be invoked by the system of record for the business object and only when the state of the business object has changed. This is normally used in conjunction with the verbs created, changed, closed, canceled and deleted. If ESB integration pattern that loosely couples the service provider and the consumer is used this naming pattern is applied with the interface the ESB provides for the consumers.

To consume (receive) information (business object) from an external source. This is used in conjunction with the verbs created, changed, closed, canceled and deleted. If ESB integration pattern that loosely couples the service provider and the consumer is used this naming pattern is applied with the interface the ultimate service provider provides.

To request another party to perform a specific service. This is used in conjunction with the verbs get, create, change, close, cancel and delete. If ESB integration pattern that loosely couples the service provider and the consumer is used this naming pattern is applied with the interface the ESB provides for the consumers.

To run a service provided to the public, which may include a state change request or a query request. This is used in conjunction with the verbs create, change, close, cancel and delete. If ESB integration pattern that loosely couples the service provider and the consumer is used this naming pattern is applied with the interface the ultimate service provider provides.

To reply with the result of the execution of a service (by the Execute service). This is used in conjunction with the verbs created, changed, closed, canceled and deleted.

To provide (show) information (business object) for public (enterprise) consumption, when the state of the business object is not changed, by the system of record or other system that has a copy of the same business object.

To request specific data of a business object to be provided.

Create template xsd:s and wsdl:s 

Create (or use those ones that comes with the used standard) template xsd:s and wsdl:s to support commonly used interaction patterns in your organisation. These templates will help you to follow selected best practices and reduce time consumed to produce new interface definitions.