Introduction
Six sigma is a widely used quality management system, introduced to the industry in the 1980s for the first time by global companies such as Motorola and General Electric. This innovative system has resulted in tremendous improvement in customer satisfaction and overall profitability [1], and has become widespread in healthcare fields from hospitals to reference laboratories [2].
Originally, sigma was a statistical value indicating the standard deviation (SD), but in the six sigma strategy, the sigma value is defined as ‘(total allowable error [TEa]-bias)/SD.’ The six sigma strategy as applied to laboratory quality control (QC) aims to reduce errors by controlling the ratio of the allowable control limit (TEa-bias) to the SD of a test. The ratio should be at least greater than 3, and ideally should be 6 [3]. As the control limit, the TEa of the product is predefined, and the bias and SD of the process should decrease for the sigma value to reach a higher target.
The point-of-care (POC) glucose meter is one of the most widely distributed and important POC devices for critical patient care. However, as POC devices are not primarily controlled and maintained in a central laboratory under a strict QC protocol by experienced laboratory staff, but are instead maintained by end users, there exist considerable chances of potential inaccuracy for the measured results [4]. To resolve this problem, it is important to assess the QC status of POC devices objectively with alternative approaches. In this situation, a sigma-based QC approach can be a good choice where conventional QC strategy fails to provide appropriate QC management.
Therefore, in this study we tried to apply six sigma metrics to the QC of POC glucose meters in a university hospital, assess the QC status, and finally achieve improvement in quality.
Methods
A total of 30 POC glucose meters (Accu-Check Inform II; Roche Diagnostics, Mannheim, Germany) installed at 26 departments in Ewha Womans University Mokdong Hospital were monitored. The departments in this study included seven internal medicine wards, 12 surgery wards, five intensive care units (ICUs), a hemodialysis unit, and a post anesthesia care unit and were labelled from A to Z (Table 1). The QC for each POC glucose meter was performed using Accu-Check Performa Controls (Roche Diagnostics) every 24 hours.
Monthly QC data from each glucose meter at two different levels (low control: 45 mg/dL and high control: 307 mg/dL) were collected from January 2013 to March 2014. Based on the raw QC data, the target value of the QC materials, monthly mean, bias (%), and SD were initially calculated every month. The bias was calculated as the measured control mean—the target QC value (% bias; bias/target value×100). The coefficient of variation (CV) was calculated as SD/mean×100. The TEa of glucose values were set to 6 mg/dL and 9.2%, at the low and high levels, respectively [5], and the mean monthly sigma metrics for 16 months at low and high concentration were calculated as (TEa-bias)/SD and (% TEa-% bias)/CV, respectively. The minimum sigma metric that was generally acceptable for routine POC measurement was set to 3, and the POC glucose meters with poor QC (<3 sigma) were identified based on relative QC data from the POC devices [6].
During the 15-month study period, the correlation between the % bias and the CV was analysed to identify any potential relationship between accuracy and precision. Data analysis was performed with Microsoft Excel software (Microsoft, Redmond, WA, USA).
On the wards with POC glucose meters with poor sigma values, we investigated the potential causes of the poor QC results by contacting the ward practitioners. At the same time, the personnel in those wards were educated with detailed instructions on the operation and QC methods for the POC devices through a newly produced educational video and documents.
After the educational intervention, the sigma metric-based QC status was prospectively tracked over the following 3 months, and the QC performance of each glucose meter was evaluated again in the same manner as described above.
Results
The mean sigma values of each glucose meter are shown in the Table 1. The mean sigma metrics of all the devices during the study period were 3.29 and 3.71 at the low and high levels, respectively. There was no statistical difference in the sigma metrics between the two levels (P=0.0718).
At the low and high controls, 36.6% (11 out of 30) and 10% (3 out of 30) of the glucose meters had a sigma value <3, respectively. All the 5 ICUs (ward Q, T, U, W, and X) were included in that category (Table 1).
The CV and % bias had a weak positive correlation (correlation coefficient, 0.45; P<0.0001), indicating that poor accuracy roughly tends to be combined with poor precision (Fig. 1).
The most frequent cause of low sigma values was the use of expired control materials, and others included prolonged air exposure of the test strip, lack of user training, and errors in device maintenance. After comprehensive instruction on the operation and maintenance of the devices with a PowerPoint slide and video resources in the wards that showed poor QC performance with sigma value less than 3, follow-up monitoring for the next 3 months showed that 23.3% (7 out of 30 at the low control level) and 6.6% (2 out of 30 at the high control level) of the glucose meters had sigma values below 3 (Fig. 2), indicating improved QC results compared with the results before the intervention.
Discussion
The main concept of the six sigma strategy is to reduce the variation and errors strictly in processes and products, thereby making the system more robust and reliable [2]. This concept is basically in line with laboratory quality improvement strategies aiming to achieve higher accuracy and precision of laboratory tests. In the clinical laboratory, the quality required by an analytical testing process must be defined. Tolerance limits in the laboratory are best expressed as a TEa [2]. Sigma metrics used for laboratory QC are expressed as (TEa-bias)/SD, where the numerator and denominator are related to the accuracy and precision, respectively.
In conventional laboratory QC systems, the accuracy and precision are measured separately (i.e., accuracy is indicated by the bias and precision by the SD or CV). Therefore, in this manner it is impossible to evaluate the overall QC state, because there is no quantitative standard to determine which is better between a system with high accuracy and low precision or a system with low accuracy and high precision. The major advantage of sigma metrics is the ability to assess the quality status with a single numerical value, representing accuracy and precision combined [2].
One of the main problems of QC for POC devices such as POC glucose meters is technical difficulty in the applicability of conventional QC strategies, such as the Levey-Jennings chart or the Westgard multi rule-based algorithm, due to a lack of delicate statistical support in the devices [7]. The lack of a solid QC program could possibly lead to inaccurate test reporting, which can be potentially serious inpatient treatment and management.
In this study, our primary goal was to evaluate the general QC status of different POC devices that are widely distributed in the whole hospital wards. We then sought to identify and resolve any QC problems. Even if all the POC glucose meters evaluated in this study were the same instrument model, it was hard to compare the QC status of each device in a traditional manner, because the number of compared devices was as many as 30, and there was no standard criteria to quantitatively evaluate the relative performances of accuracy and precision among the different devices. In this respect, the sigma metric was an ideal index to implement this QC process, as it can not only reflect both accuracy and precision in a single numerical value, but it also enables multiple comparisons among different devices without any complicated statistical processing.
The QC performance of POC devices in this study reflected by the mean sigma value was generally acceptable in both the low and high control, with mean sigma values of 3.29 and 3.71, respectively. Conventionally, a sigma value under 3 is regarded as ‘unacceptable’, requiring additional process improvement, and a sigma value of 6 is regarded as ‘world-class performance’ [2]. In this study, 36.6% (11 out of 30) and 10% (3 out of 30) of the POC devices at low and high control, respectively, were found to be unacceptable. The reason for the higher percentage of unacceptable QC found in the low control compared with the high control is the relatively greater effects of the low control measurement on the CV. This phenomenon corresponds to the common experiences found in routine clinical chemistry analysis, with relatively higher CVs found at low concentrations compared with those found at higher concentrations [8].
To minimise this problem, we applied a different standard of TEa, absolute value (6 mg/dL) for low control and relative percentage (9.2%) for high control, based on the TEa specification [5], because the absolute value-based approach for the low control could significantly decrease the abnormal distortion of the actual bias and precision. Therefore, the sigma value for low control was calculated using the SD, and that for the high control was calculated using the CV.
An interesting finding of the relative QC performances among the devices was that the majority of ICUs had unacceptable QC status. Compared with the general wards, the average QC performance of the ICUs was generally poor (average sigma of all wards, 3.49; average sigma of ICUs, 2.89). The POC glucose meters are mainly used by nurses. Nursing workload has been reported to be associated with patient safety outcomes, such as medication errors, hospital-acquired pneumonia, hospital-acquired urinary tract infection, patient falls, length of stay, and patient’s satisfaction [9,10]. Poor QC of POC glucose meters in the ICUs can be partly explained by the higher nursing workload in the ICUs than in the general wards. Another finding from the bias versus the CV correlation analysis was a weak positive correlation between the bias and the CV, or in other words, between the accuracy and the precision.
Combining these two findings, it can be hypothesised that bad QC results might be related to a busy and unstable working environment with rapid turnover of personnel, and poor accuracy tends to accompany poor precision, even if each parameter is mutually independent. These findings indicate that the main cause for QC problems might be insufficient education or lack of familiarity of the laboratory personnel with the device maintenance on these wards.
As shown by the result, the sigma-based approach could successfully identify critical quality problems, including expired control materials caused by a lack of understanding of QC material by working personnel, prolonged air exposure of sample strips, lack of user training, and errors in device maintenance. In addition, after appropriate intervention, follow-up monitoring showed significant improvement. After the intervention, a total of 23.3% (7 out of 30) and 6.6% (2 out of 30) of the glucose meters had sigma values below 3 for the low and high control, respectively, compared with 36.6% (11 out of 30) and 10% (3 out of 30) unacceptable results at low and high control, respectively, before the intervention. The mean sigma value for the ICU units also increased from 2.89 to 3.19 (data not shown).
The six sigma-based rule can also be applied to improve conventional laboratory QC procedures. Based on the calculated sigma level, the optimal Westgard multi rule has been suggested with a stricter rule applied to lower sigma levels, and vice versa [11]. This approach has the advantage of ruling out unnecessary additional QC tests without significantly sacrificing the false rejection rate, which is usually repeated in cases of conventional QC rejection. In addition, the sigma-based rule has the potential to be used as a valuable tool for interlaboratory proficiency testing or accuracy-based surveys, where standardisation issues among different devices or reagents are a matter of conflict.
However, the lack of TEa targets for many analytes and the sometimes-inconsistent TEa targets from different sources are important variables for the interpretation and application of sigma metrics in a routine clinical laboratory [12]. Therefore, further study is required before a consensus can be reached on the standardisation and verification of TEa for laboratory test items in the future.
In conclusion, QC management with sigma metrics could successfully reflect accuracy and precision at the same time in an objective and quantitative manner, expressed as a standardised index. The CV and % bias tended to be weakly correlated, indicating that devices with poor QC generally had problems with both accuracy and precision.
The sigma-based approach could also be used for quantitative follow-up monitoring and assessment for practical improvement of QC. Sigma-based QC strategies can be utilised as a useful alternative QC modality in a wide variety of clinical laboratory situations.
