After an automation goes live, someone will ask whether it worked. If nothing was measured before, nobody can answer.
This article describes measures that a firm can collect on its own. It gives no targets, because a target from another firm says little about yours.
The problem
Without a starting point, a change feels successful or not depending on who you ask. A busy week can hide a gain, and a quiet week can hide a problem.
What is known
Vendors report large gains for their own software, and our article on vendor research explains why those figures need care. We found no independent study that measures the effect of automation on U.S. law firms, so we give no target and no expected improvement.
The only reliable comparison is the firm's own before and after.
What to do
- Write the question first: what should be different? For example, replies sent sooner, fewer retyped details or fewer missed dates
- Choose two or three measures that answer that question
- Count before the change, for a few weeks, with the method you will use after
- Count after the change in the same way and for the same length of time
- Write down what else changed in both periods, such as a holiday, a new hire or a busy month
- Ask the people who use the system what got easier and what got harder
Measures a firm can collect itself
- Time from a new inquiry to the first reply
- Inquiries with no owner after a day
- Details typed more than once for a matter, counted by hand on a sample
- Missed or late deadlines and reminders
- Requests for help after launch, and how long each took
- How long a routine task takes, timed on a sample of tasks
An illustrative example
Illustrative example, not a client case. A firm wants quicker first replies. For a few weeks it writes down the date and time of each inquiry and of the first reply. After the change it does the same.
It compares the typical wait, not the best case, and notes that one of the weeks had a holiday.
What to look for when you implement it
- Do not count only what the system makes easy to count
- Do not compare a quiet month with a busy one
- Keep a note of the errors the automation made and who caught them
- Review on a fixed date, for example after the first month and after the third
- Decide in advance what result would lead the firm to change or stop the automation