Architecture Kata "Dropbox Workflows"

Collaboration in the office is often ad hoc. This is what makes high productivity in the office so challenging. However, there are always opportunities to systematize collaboration. Workflows are one way of doing this (or protocols, routines).

Workflows define which steps are to be carried out in order to react to an event.

Examples of events: an invoice arrives and must be checked before payment, a vacation request is submitted and must be approved, an order is placed and must be processed.

Messages or documents in the form of files are accepted as events.

Workflows define how these events are always handled in the same way. They are coded practice. Workflows make it easier to train new employees, as the activities to be carried out are no longer just in someone's head, but are explicitly recorded. With a workflow application, the necessary work steps are then remembered by the relevant employees themselves.

Audit" scenario

An invoice arrives and is scanned into a special directory. This event triggers the workflow "Audit" to.

The workflow software is running for the relevant employees - accounting, purchasing, managing director. As soon as the invoice arrives in the inbound directory of the workflow, an order is generated for it, which reads "joint Check invoice".

First, the accounting department is notified in order to formally check the invoice. Are all the details required for tax purposes included? If so, the order is forwarded to the purchaser if it is below a threshold amount. (e.g. 5000€). Otherwise it flows immediately to the purchasing manager.
Buyers or purchasing managers approve or reject the invoice or feel unexpectedly overwhelmed.
In the event of excessive demands, they escalate the review and the managing director has the final say. He rejects or approves.
The order flows back to the accounting department, which either settles the invoice or handles it differently.

Collaboration between roles is automated by the workflow application. Nobody has to keep an eye on the whole thing. Everyone just looks at their workflow client: orders for processing appear in it when they arrive at a work step for which a role is responsible.

This makes the processing of orders reliable. Even if an employee does not complete a work step, the workflow application will not miss it. Each order is assigned a deadline for each work step. If this is not met, it is automatically escalated.

The world of workflows

Workflows include:

Start eventA workflow begins when an event occurs. In this case, this is the receipt of a file in a directory.
OrderThe file of the start event is packed into a job. This order is processed according to the work steps linked in the workflow.
Work step: Each work step (or also station) stands for a task that is to be completed with the order. Work steps are marked with a Result completed. Which work step follows the current one depends on the Result from.

The elements of a workflow in action:

Each employee only observes "his" work steps, the purchaser "Buyer.invoice check", the bookkeeping "Accounting.invoice validate" and "Accounting.invoice complain" and "Accounting.invoice pay" etc.

The employee responsible can enter order data in each work step (e.g. the invoice amount). At the very least, however, a work result is assigned, on the basis of which the order flows on to one or other of the next work steps (e.g. "approved" or "<5000€").

Orders carry a logbook of the stations through which they flow. Processors can also store notes in this logbook.

Work steps are mainly carried out manually (Boxes with rounded corners in the image above). The main purpose of a workflow application is to organize this reliably.

However, it is also possible to perform a work step using a program. (Workflow-Bot) to be processed (Box with sharp corners in the picture above). Communication with a Bot takes place via the command line. It has read access to the workflow definition and read and write access to the job in question and associated documents.

Workflow elements

Each workflow definition contains...

Title (e.g. "Audit")
Work step*
- Title (e.g. "Accounting.invoice validate")
- Possible result* (e.g. "approved", "<5000€")
- Property to be recorded* (e.g. "Invoice number")
- max. processing time (e.g. "2 hours" or "3 days")
- [WIP Limit]
- [Bot] (Command line to call up a program, e.g. "mono rechnung_verteilen.exe")
Source* (e.g. "Incoming invoices")
Sink* (e.g. "Paid Invoices")
Relationship*
- (Source | Work step result) (e.g. "Incoming invoices", "Buyer.invoice check.approved")
- (work step | sink) (e.g. "Paid Invoices", "Accounting.invoice pay")

Work steps, sources, sinks and results could also be assigned IDs via which they are related.

Order elements

Each job file contains...

Order number
Incoming order time stamp
Recorded property*
- Name (e.g. "Invoice number")
- Value (e.g. "20170106-1")
Appendix* (relative Path to a file in the job directory, e.g. "rechnung123.docx")
History
- Work step*
  - Work step title
  - Arrival time stamp
  - Work start time stamp
  - End of work stamp
  - Result (e.g. "approved")
  - Identifier for the processor (e.g. "dwalin@acme.com" or bot name)
  - Note*
    - Text
    - Timestamp

Dropbox as a foundation

The file system should be used to store workflows, orders and documents. If Dropbox directories are used, it is even possible to work together across the boundaries of a company network.

The file system or Dropbox as the foundation instead of a database technology such as RDBMS or NoSql should make it easy to deploy the workflow application. No special infrastructure is required. The application only needs to be installed on participating employee computers and requires minimal configuration. This is intended to lower the inhibition threshold for using workflows to systematize work. This is particularly important for small companies without an IT department. A fileshare or dropbox (resp. another cloud storage, even Git would be possible) can be found everywhere.

A desktop application also seems easier to use at first. "sell". If workflows are only to be processed in the local network, there is no need to set up a web server. And for workflows that go beyond the local network, there is no need to operate a server in the cloud; Dropbox or similar is sufficient.

Alternative: The workflow application is delivered as a server that not only hosts the workflow engine, but also offers a REST API to which HTML/JS or Elm clients can connect. (The HTML/JS/CSS/image files are also supplied by the server).

Folder structure

All workflows under one root.
The job directory and the job file are created automatically when a file arrives in a source.
The order list flows through the work steps. (The Order file could be structured as Evensource).
Each work step consists of three directories for its orders
- /todo: Orders awaiting processing. With manual processing, the employee can see what still needs to be done.
- /inprogress: Orders that are being processed. With manual processing, the employee explicitly accepts an order.
- /done: Orders whose processing was completed with a result in the work step.

Paths in the json files are always relative to the directory of the file.

Jobs in /done are no longer displayed in manual work steps. They are moved by the workflow engine to the next work step in its /todo according to the workflow definition and the current work step result.

Workflow application

Workflow client

The workflow client is used by employees to execute work steps. They use the client to monitor selected stations and are informed about incoming orders (/todo). You then select orders for processing (/inprogress), record data, write notes and conclude the work step with a result.

Focus on work steps of a workflow
- What orders are pending?
- What is in progress?
- Data acquisition during processing

Employees log in to the client with their e-mail address.

Workflow engine

The workflow engine establishes the flow of orders.

It monitors the sources and creates new jobs for incoming files.
It monitors sources and /done directories of the work steps and moves jobs to the next /todo directory or a sink according to the result.
It monitors /todo and /inprogress directories to determine whether jobs have exceeded their deadline, so that it can be processed with the result "timeout" to push on.
It executes bots during automatic work steps. It transfers the current workflow, the current work step and the current order to the bot. The bot produces results in the job directory and returns with an exit code. If the exit code is not equal to 0, the job step is completed with the result "exception" completed; ditto if the bot crashes or takes longer than 60 seconds.

The workflow engine can initially be included in the client; otherwise it runs as a service on a single computer.

If the workflow engine is running in the client, it can be called up immediately for each completed work step. In addition, a client would only have to run for the sources in order to compile jobs when new files arrive.

Attention: Job directories must not be edited manually! In particular, all files in the job file must be noted. Otherwise, it will not be possible to determine whether all files have already arrived during Dropbox synchronization on the recipient side.

Workflow Designer

Workflow definitions are json files. These can initially be coded by hand. However, end users need a workflow designer, visually or not to ensure error-free workflow definitions.

Workflow bots

Bots are programs that can be called from the command line. They only need to be accessible to the engine. The platform on which they are developed is irrelevant. The contract only covers the command line and the exit code (and Standard output for log messages).

Example call:

Kata Dropbox Workflow - Clean Code Developer Akademie 3

The standard output and the exit code of a bot could be included in the order as a note.