A CASE FOR LONGITUDINAL CLINICAL TESTING OF ORDER SETS
By Marilyn J. Cushing, RNC, MSN, MBA
Most clients are familiar with writing test cases for and performing unit, functional, and integrated
testing as part of pre-activation requirements. These cases validate the source code and compliance with
client-established configuration standards (unit test); validate that all elements per the reference
documents are available in the application and display and function as specified (functional test); and
that data/information is received as expected within the application and/or passed appropriately to
downstream systems as well as verifying that all pieces of the application work as designed (integrated
test).
Within the order set design/configuration team for Eclipsys’ Sunrise Clinical Manager (SCM) at
one client site, these tests were defined more specifically. Unit testing was designed to identify
inconsistencies in screen, dictionary, and/or table and file components within the order sets. This included
validation of content in configuration tools as well as in the SCM application itself, and was performed by
configuration specialists. Functional testing was designed to test each feature and function of the SCM
application in relation to order sets to verify that configuration met site-specific and physician author
requirements and was completed by various project team members or students, some of whom had no previous
clinical background. Integrated testing validated that all orders from the 18 selected order sets passed
through SCM and ancillary systems as expected and was performed by a designated test team. Within the
testing process and specific test cases as designed, however, this consultant identified that none of the
approaches could reliably predict whether the end user would have the required information to carry out the
orders as intended by the physician over a period of time. Therefore, an additional testing step –
identified as longitudinal clinical testing – was designed and implemented.
The goal of clinical testing was to measure overall usability and functionality of the order set in SCM
from the end user perspective, and to identify issues or problems needing correction prior to activation.
In order to validate the longitudinal concept and provide consistency in results, all clinical
testing/retesting was performed by this consultant. The approach taken was straightforward: each complicated
or high-risk order set (of the 600+ to be implemented) was reviewed as though the tester was a nurse at the
bedside for the length of time covered by the orders in the set (up to 35 days for some chemo sets). An
appropriate number of orders from an identified set was placed on a specific test patient and reviewed
daily until all orders had been completed or 35 days had passed. Orders generated or completed that day were
compared to the original order template expectations; labs were collected, tasks and flowsheets completed
and medications charted. Various order management functions were also carried out, such as review of
documentation, reinstatement of tasks, and verification of order status changes, among others. Specifically,
the tester answered the following questions: Was the information needed to safely provide patient care as
outlined in the order template present? Was each order complete, did it make sense as written, and could it
be documented against if appropriate to do so? Did each order become active/generate occurrences/complete
when expected? Did it appear correctly on the eMAR (if appropriate) and in the correct medication category?
In addition, the order set was reviewed for how well it followed the site’s style guide.
Some of the issues uncovered through clinical testing were significant; while others were of lesser
importance, all offered opportunities for improvement in the product presented to the end user at acceptance
testing and activation, complied with the physician’s intent, decreased potential errors, improved
patient safety, and provided additional SCM application education to the order set team.
Some of the issues identified through clinical testing included:
- Chemotherapy Protocols – numerous issues due to the complexity of these protocols in which
certain activities occur on specific days based on the protocol start date, such as:
- Scheduled medication task AND provisional (trough) task generated for the same administration time;
trough remained after task documented against – identified as tied to a daily order with a
specific priority, ordered to begin >48h in the future (resolved with new code)
- Certain medications appeared on eMAR as yellow trough (give anytime within this time period) instead
of a yellow cell (give at this specific time) (resolved by adjusting configuration)
- Certain medications did not appear on the correct start date on the eMAR (resolved with configuration
changes)
- Certain medication dosages did not adjust as expected using the “adjust dose”
functionality (resolved with configuration mapping)
- Certain medications automatically discontinued themselves prior to the expected stop date (resolved
when medication item “stop after _ days” field selected by pharmacy was identified and
changed in configuration of the order)
- Some labs mapped to routine, start T+1 scheduled themselves for today (resolved with configuration
changes)
- Other Order sets, such as:
- Drug calculations were not always correct as expected based on advisory text. For example,
gentamicin was to be calculated on kg/ideal body weight per advisory note, but the drug calculation form
only selected kg. In this instance the patient might be given an overdose of the drug (resolved with
configuration change)
- Medication orders with system calculated start and stop dates in tapering dose sets found to have
overlapping dosages on eMAR due to misunderstanding of T (today) ending at 2359 and T+1 starting at
0000 tomorrow (resolved with education and configuration changes)
- Advisory information needed by all caregivers only appeared in Advisory note, so only seen by
ordering clinician (resolved by utilizing Additional Information field)
- Tasks appeared as unscheduled on eMAR (resolved when new frequency was added to frequency
translation table in configuration)
Due to the complexity of patient care and resultant increase in the number of order sets in use at a site,
the usual functional testing must be expanded to include analysis of the order set longitudinally. Sets must
be evaluated for usability from the perspective of the end user responsible for carrying out the
physician’s orders, as well as the functionality of the set itself. In this way clients will be
assured that their order sets will function as intended and designed, and will promote safe, appropriate,
and timely patient intervention.