Process Automation is always in my radar since I had a lot of opportunities to implement lots of workflow solutions when I worked in enterprise environment. After moving to new place, I thought workflow might be something in the past. Surprisingly, it keeps coming, everywhere at the core of complex software solutions I went through. That tells me something about the important of this technology.
Although workflow software is everywhere, it comes in different forms with different technology stacks - BPM/BPMS in enterprise, Workflow in a small to medium company and Process Orchestration in microservice context. They are fundamentally the same.
Recently, I read a book “Practical Process Automation” which covers every aspects of process automation or workflow in every contexts mentioned above.
In this post, I’d like to summarize key points that I learned or inspired by this book.
What is Process Automation
Process means steps we follow to achieve something. Even a simple task like brewing a cup of coffee has multiple steps. In order to achieve something faster and more reliable, automation is key especially in a software dominated world. To automate a process, we need tools such as BPMS, Workflow or Custom-built software.
Why Process Automation
Every organization has one or more business processes and it tends to be complex. If you can make it fast and reliable, that means more value and revenue will be created to the organization. It also means cost reduction or having more resources to do more valuable tasks.
A good Process Automation tool
As mentioned, you need a tool to automate a process. A good tool needs to be easy to understand and easy to create or update process model. It also needs to be mature enough to support real world process modeling which has a lot of exceptional or edge cases. It needs to support simple tasks like event/timer trigger, sub-process and looping. I found that tools that support BPMN notation modeling make it easier to check these points.
Process Automation in distributed system (microservice)
Microservice architecture is the most popular architecture for modern technology organizations these days. The reason behind this is the need of modular software for organizations to survive in a more and more complex world. One problem everyone faces when use this architecture is that more communication between services and teams are needed as opposed to monolithic architecture.
We can use process automation tool in conjunction with event-driven communication style to tackle this problem. My preferred choice of service-to-service communication is by publishing an event from the source service, then a process consumes that event, determines next action and send a command to another service.
Process Automation in microservice architecture
With this approach, consumer service does not need to know the source service, thus they are loosely coupled. We can also see what is happening using process visualization. This also make it easy to track an issue for a particular transaction.
Choreography or Orchestration
There are 2 styles in the realm of event-driven communication, Choreography and Orchestration.
Choreography means event produced from the source consumed directly by consumer service.
Orchestration means event consumed by a coordinator such as process automation server which control the overall process between services. The mechanism described above is an example of this process orchestration.
In practice, Choreography style should be used in a simple coordination between one or more services. On the other hand, Orchestration style should be used in a more complex communication between services. One notable benefit of using orchestrator is that there is a place where you can see the overall process progression. Also, this makes the process more transparent, traceable and loosely coupled.
Traditional vs Modern tools
Traditionally, process automation tools used in enterprise is a single centralized tool, one huge instance for everyone and guarded by a centralized team. This proved to be ineffective because of bottle neck created by a centralized team. A more modern tool tends to use decentralized approach, it can run in a distributed fashion and works in modern environment.
Adopting Process Automation
To adopt this new capability, firstly, you need to select a tool, cloud-based or open source based. Example for Cloud-based are AWS, Google and Azure. The problem with cloud-based is that they all use their own proprietary solutions which are incompatible. Example for open source based are Netflix Conductor, JBPM and Zeebe. They are also incompatible. Personally, Zeebe is an interesting choice since it’s distributed by nature and focus on performance but still in early development.
To introduce the new tool into your organization, one suggestion is to use it with new process. You need to design a way to integrate your service with the tool as parts of your architecture like event-driven. When you successfully acquired this new capability, try applying it to a more complex process to test and adapt your new ability. Then distributing this knowledge to other team by forming a starter team (maybe using Enabling team topology) and building Community Of Practice (COP) as a way to sustain this capability in your organization.
Saga vs Process Orchestration
You might have heard the word “Saga” in the event-driven community. If not, it is a pattern originally describes a way to do atomic transaction in long running process across multiple services (as opposed to short-live transaction we are familiar in database technology). When there is a problem in one step, all other steps need to be rollback or compensated to maintain consistency between services.
The main different between Saga pattern and Process Orchestration is that in Process Orchestration, not all services need to be rollback. The process flow needs to decide what to do next in case of failure then proceeds. In short, Process Orchestration is a more flexible way to control the process.
BPMN in practice
As mentioned in “A good process automation”, BPMN meets all of the expectation. This might be opinionated but it’s based on my experience with this technology. At first, I thought it was too complex to describe a process with is notations but I learned shortly after that these notations was necessary in order to describe real world process. The fact that it’s still in used despite being old is kind of proof. One very nice thing I should talk about is that BPMN is designed for visualization - you can see every detail of your process just by looking at it both at the modeling and run time.
Conclusion
For me, Process Automation is the future, especially, in the age of AI, things have been automating more and more. Process Automation capability is vital to an organization to be a leader in the fast moving world.