Workflow Engines

Workflows

Most of us are familiar with workflow diagrams.  We likely have even created a number of them ourselves to graphically convey the flow of information of a particular aspect of  business. These drawn workflow diagrams, drawn by hand or digitally, by themselves, don't do anything. The most one could do is to distribute one to everyone, who would then print it out and use it as a reference, but not much is done beyond that.

A workflow engine takes a coded workflow diagram and orchestrates that workflow through an actual project.  It controls the stream of data and information that is created, transformed or transferred at any process to other dependent processes.  It defines the various trackable stages in a project and monitors the project is it goes through each stage. Additionally, it can interact with external services to import or export data.

Workflows are often not linear.  There may be parallel components as well as parts that need to be redone which necessitates an ability to push back to earlier stages.   The workflow may branch out into separate streams with a decision conditional process that executes the logic to determine which streams to take.  Alternatively, the workflow engine may redirect back to redo certain parts of the workflow. When executing a project through a workflow, a workflow engine ensures that all of the processes are completed in the order and logic of the defined workflow.

Project Breakdown

Most organizations break down their work into groups of projects or jobs.  A group of people are then assigned to the project and work on it together, producing content and communicating as needed to complete the project with as high a quality as possible in the least amount of time.

Within a project, data and files are constantly being created, distributed, usually through email and then stored somewhere, everywhere, and possibly nowhere.  Usually this task is left up to the individuals to figure out how everything is stored. This is usually done in haste and needless to say, difficult for others to decipher. In any given workplace, data is transferred from individual to individual or individual to machine or machine to machine in a rather haphazard chaotic manner use a variaty of tools to communicate with each other such as email or instant messaging apps.

Where are the inefficiencies?

While all the work directly related to the project takes place within a process and most often done by competent individuals, it is often the connection between processes that create significant overhead.  This is especially true when communication needs to occur between departments. Each department often acts an isolated silo where one department must formally or informally request work from another external department.  For example, you may need a member of the legal team to look at a document. These types of requests often requires a department to use their internal resources to gather the necessary materials, files and data together in an email and then communicate this to the receiving party.

Because of the time taken between each request, a project coordinator then has to manage all these moving parts as simultaneously as possible, trying to keep all the information up to date.  The complexity increases as the project size increases as well as the number of projects. Problems occur. Errors cause incorrect work to be done. Time is wasted preparing and managing all of the communication and requests.  Jobs sit in waiting for extended periods of time. So even though everyone on the project is working on the project as efficiently as possible, the overhead of the connection between the processes causes significant delays. This ultimate results in late projects or lower quality or incorrect work done.

Workflow Management

In these situations, projects can quickly spin out of control, particularly when the jobs are a little more complex than just a straightforward checklist or if there are many of them going on at the same time. Sometimes it goes well ... others decidedly not, however, in the end, the overall performance suffers.

In order to solve this, projects require some structure or their outcome will not be trackable or predictable.   Without tracking the progress of a project, analysis on the performance of the project cannot be done and management is often left blind as to what is actually happening

 A workflow engine ensures people are notified when things need to be done, it communicates what exactly needs to be done and provides all of the necessary information upfront to do that task.   While working on a task, an individual or an automated process will create or transform data in the form of new files, changed files, comments, feedback and statuses. The workflow engine will record this information.  And when a task is completed it will look at the workflow definition for the next dependent processes and then deliver the necessary data to those proceeding processes. This will continue through all the processes in the workflow until the project is completed.

A workflow engine with asset management is able to present the necessary information at each stage of the workflow.  Because the workflow engine is aware of the context of where in the job a particular process occurs, it is able to provide a customized experience for each user.  It removes the overhead that occurs between the work processes. Each user receives the correct information and files needed for their task without having someone to manually prepare it.  They get this packaged information when the job is ready for them with all the required materials to complete the task. Since the communication of the details of the task is automatically generated as soon as the previously dependent processes are completed, the overhead of gathering this information is eliminated.

Automated actions introduce more efficiencies.  This is done with specific nodes that are triggered to execute at the appropriate point in the workflow.  Since the workflow engine is aware of all of the assets and data associated with the job, a process can automatically trigger actions to occur.  This includes custom scripts or interacting with external services.

Once a workflow engine is incorporated a number of distinct benefits will automatically be attained.

Project Tracking

As the workflow engine pushes data and information along the workflow, it records the status and timing of all events.  This gives project managers an accurate overview of the actual real time status of the project.

Critical to the smooth execution of complex jobs is the ability for the project manager to manually interrupt or restart any aspect of the workflow.  No matter how well a workflow is defined, it is always possible that it does not capture every exceptional case. This handling of exceptions allows for certain aspects of the workflow to be reset and restart when it is required.

Digital Asset Management

The TACTIC workflow engine has built-in Digital Asset Management.  It is able to track all of the project files and organizes them automatically with process in the workflow of the job.  This self-organizing of the assets ensures that you know exactly why, when and for what purpose a file was added to the system.  This is unlike a shared folder, Sharepoint or an independent DAM system, in that the organization of these files is left up to the individual users.  This inevitably creates a mess over time. Much time is spent organizing these files using tags or keywords.

Feedback and Review

When people do their work, they will inevitably need to have feedback or have their work reviewed.  The workflow engine is able to take feedback data and possible annotations and deliver everything to the person who has worked on the task.  This feedback mechanism can be informal or occur in a more formal setting where a bunch of submissions are reviewed together.

Reporting

Because all actions are recorded, it becomes possible to create and analyze timing and report on them.  This will expose problem areas that contribute to either poor quality or late delivery on a project. Real jobs have many moving parts and they quickly become overwhelming to manage, especially if there are a lot of jobs occurring simultaneously.

Reporting is an essential aspect of a workflow engine.  Since the workflow engine records all activity and timing of a job, reports can be generated to analyze bottlenecks where either processes or communication between the processes are causing delays in the completion of the project.

Streams

    The workflow engine can use decision criteria to either send back to redo work or choose one or more streams to progress.  Different jobs may be different parts that need to be executed depending on a variety of factors. For example, a particular job may require input from the legal department.  Conditional process nodes can determine which streams to take based on any condition within a workflow.

Security

Another important  aspect of a workflow engine is that is separates the users who are working on the content to the action that publishes work to external services such as Salesforce, Sharepoint or social media sites such as Facebook or Twitter.  The authentication credentials of communication with these external services can be encapsulated within an automated process so that the people working on the project are not directly publishing the content. This reduces the number of people who have direct access to critical services in the organization.

Compliance

A workflow engine ensures that the job will go through certain predefined paths and reach well defined milestones.  While it is doing so, it will track all the stage and have the ability to report on them. This ensures that any job using a workflow engine will meet any regulatory compliance standards specified for the job and, importantly, demostrate that this is the case with a simple report.

A workflow engine is a missing component of most organizations.  The lack of definition of the work process results in a chaotic work process that cannot be tracked, is time consuming to report on and leaves no room for automation.  Coupled with an Asset Management system, communication between task can be effectively packaged and communicated, eliminating the overhead between processes. Workflow is now starting to be recognized as an essential business component, but understanding what a workflow engine is how to use it properly is critical to the success of its deployment.