By: Dan Janowak, Consultant, Epic Practice
The following is a fictional scenario.
A slight preamble
In the real world, things aren’t always nice and orderly, and events sometime fly by in seemingly random sequences. The first paragraph is a mad rush that might accompany a particular fictional Epic system scenario that could happen just after go live.
And so it begins (the story that is)
Your phone rings and there is an emergency change control meeting being called. Three days ago, a change was made to the system definitions, configuration, and SUP (customers’ daily copy of production). It had been overwritten and the former system definitions aren’t available. Someone had already checked back to an older environment and nothing seemed to have changed to cause the problem. It was suggested that it was possible the change was made and undone, but no one can identify what had changed in the system definitions. From the Data Courier reports, it was seen that the entire record was sent instead of just the one or two items that were most recently intended to be moved to production. But in fact, the person who had made that most recent change had a JXPORT of their change before and after they made the change.
A narration
The time just after go live can be quite the whirlwind, especially for those who take stress and anxiety seriously, or have no choice. In the story, a change in live functionality that was undesired had been detected, but not fast enough. And based on how most Epic customers are set up, the ability to recover changes made the day before weren’t useful because the change in live functionality had been detected too slowly. Yes, the fictional people should have been more vigilant or spent more time monitoring the system, but they were probably all very busy taking care of other issues that had arisen after go live.
Fortunately the fictional organization had a strong change control team with the primary rule being ‘Don’t Break Prod.’ Except, there wasn’t a required policy that certain very important records and pieces of information were always recorded – usually by an Epic technology called JXPORT to record the ‘before’ and ‘after’ a change to enable putting production back to its former state of being. The good news is that the strong change control team had a document that was required reading for each team member that gave advice on how to better manage their changes and increase the ability to mitigate or reverse changes that later turned out to have unintended effects.
The story continues…
After the meeting and more investigation of the Data Courier table, it turned out that a different change had occurred a couple days ago but hadn't been noticed as having the undesired effect. The change from three days ago had also been recorded with JXPORTs before and after the change. Those changes were triaged, updated, and a new set of changes were tested and moved to production.
The Closing
What happened in the story above is that two changes had been made and the users who made those changes had gone above and beyond change control requirements, but they used change control suggested methods to record their changes. Days later, it was determined that the most recent change was blamed, but vindicated for causing an undesired shift in live functionality and the earlier change was then identified, validated, tested, and updated to produce the desired shift in live functionality. No real harm, no real foul. These things happen, and the more tools we have available to prevent or mitigate them, the better.
Without the record of the changes before and after each set of changes, the investigation into what broke production, or rather what caused an undesirable shift in live functionality could have taken weeks to figure out and restore instead of a single afternoon. The results could have been the wrong person being disciplined or fired.
In general, the faster things can be restored, figured out, or made to work as expected, the more confidence and good will that gets generated. We also become better partners to each other, our stakeholders, and our respective organizations. Reputations can be significantly impacted.
No matter how strong the change control team, a good, strong document on how to prevent and mitigate undesired changes to live functionality can save the day.