Influence of different types of real-time feedback on hand washing quality assessed with neural networks/simulated neural networks

Background: Thousands of people die every day around the world from infections acquired in a hospital. Hands are the main pathways of germ transmission during healthcare. Hand hygiene monitoring can be performed using various methods. One of the latest techniques that can combine all is a neural network-based hand hygiene monitoring system. Methods/Design: Each participant performed 3 hand-washing trials, each time receiving different type of feedback. The order in which each participant of the study used the developed applications was strictly defined, thus each hand-washing study session started with performing hand washing using application A, B and C accordingly. All captured videos of hand-wash episodes were saved and later analysed with neural networks. In the end, both evaluation results were compared and evaluated. Results show that when the participants use Application Type A, they perform hand washing much faster, as well as in comparison of Application Type A versus application type C. However, the longest time spent for the hand washing was detected while using the application type B. Conclusion: Study shows that structured guidance provided during the real time hand washing could be associated with better overall performance. The Application C has confirmed its effectiveness. Proving its advantage among other applications, the Application C can be integrated into the clinical environment

The most common reasons for insufficient hand hygiene are lack of time, workload, disinfectant unavailability, lack of knowledge or hand skin irritation [19]. Studies have convincingly shown a connection between increased compliance with hand hygiene recommendations and reduced levels of healthcare-associated infections [24].
A multimodal approach to hand hygiene is recommended to improve the quality of hand hygiene [8,9,10]. This includes education, availability of hand sanitizer, reminders in the workplace, regular audits, and feedback on compliance with hand hygiene guidelines [7]. However, it should be noted that the interventions conducted to date have not always resulted in sustainable results [11,12]. Consequently, monitoring and improving hand hygiene compliance remains a challenge in healthcare settings.
Monitoring is used to follow up hand hygiene quality among healthcare workers, which is considered the standard in most hospitals. The purpose of monitoring is not only to detect errors but also to improve hand hygiene habits among medical staff [22]. Hand hygiene monitoring can be performed using various methods, such as direct observation, video filming, calculation of hand disinfectant consumption, etc. The gold standard for now is direct observation of healthcare workers performing hand hygiene [22,23,24]. Although there are many advantages of this approach, several disadvantages have been highlighted in recent publications -it is time-consuming, important details could be missed by observer and is subject to the Hawthorne Effect, in which the observed person changes their behaviour when they find out that they are being observed [22,23]. Another question is whether we are familiar with other monitoring technologies.
One of the most recent technologies providing an opportunity to improve direct observation would be a neural network-based hand hygiene monitoring system. With neural networks-based method, it would be possible to recognize hand movements and register whether and what movements are performed during the washing procedure. Supplementing digital neural network-based video monitoring with audio and/or visual feedback signals can provide continuous objective monitoring and formatting feedback. More data that are comprehensive could be collected comparing with those observed by a single person [7].
Technical solution for network-based video monitoring should comply with three main objectives: to capture visual information (live video) of hand-washing episodes; to recognize hand-washing movements done during each episode; to provide users with feedback of hand hygiene quality regarding the exact hand-wash episode. Additionally, the solution should be able to store data of all performed and analysed hand hygiene episodes for further analysis. The best way to combine all features mentioned above is to use smartphone. Modern smartphones are equipped with high-resolution cameras for video and image capturing, and provide different kinds of audio visual (AV) feedback.
We assume that feedback type may affect human behaviour a lot, so it is important to identify the best feedback type to be implemented in the solution.
The aim of this study was to estimate an association of different types of real-time AV feedback from evaluation system of hand hygiene quality with its compliance to WHO guidelines, as well as with hand hygiene quality assessed with UV light.

Material and methods
To estimate the principles of how neural network-based hand washing movement recognition and quality evaluation tool should interact with users, three different operating smartphone applications were developed.
Working principles of all applications were developed similarly -application captures user's hand washing episodes using a smartphone with integrated camera, further evaluation is performed by neural network, integrated in the app, which makes decisions based on evaluating each single captured frame of hand-washing process. Neural network calculates a probability of each captured frame to correspond to WHO recommended movements or other unclassified hand-washing movements, which the Neural network was trained to recognize.
Differences between these three smartphone applications were in the way of interacting with users and providing real-time feedback to the user of performed hand-washing quality.
Application type A is called -"Not guided" hand washing Application type B is called -"Semi-guided" hand washing Application type C is called -"Fully-guided" hand washing In "Not guided" hand wash application, users are able to wash their hands freely, using any sequence of hand washing movements as well as performing washing for as long as they want/consider necessary. The only assistance users can use while washing hands is a poster with hand-washing guidelines defined by WHO displayed on the smartphone screen.
In "Semi-guided" hand wash application the users are assisted by the application in helping to recognize the movements performed in real time, however not defying, or restricting the duration or sequence of the moves. The application shows information on which hand hygiene movements are performed in situ and displays visual information if exact movement is performed more than 7 seconds.
In application "Fully-guided" hand wash, users are strictly guided step by step, following certain sequences and durations of hand washing movements.

Study design
The study was performed at Medical Education Technology Centre, Riga Stradins University (Riga, Latvia). The study involved 72 participants. All participants were healthcare specialists (employees of the medical university, physicians, medical students etc.) and were familiar with methods of performing proper hand hygiene. Prior to the study, all participants gave their informed consent.
Each participant performed 3 hand washing trials, each time receiving different type of feedback -different hand-washing quality evaluation application.
The order in which each participant used the developed applications was strictly defined, thus each hand washing study session started with performing hand washing using application A, B and C accordingly. Before each hand-washing session participants were introduced with functionality and feedback provided by each developed application. Then participants treated hands with UV active gel covering the surface of the palm, dorsum of the hand as well as other hard-to-reach regions between fingers, fingertips etc. Pictures of the treated hands placed under the UV lamp uncovering all dirty regions of the hand were captured. After each washing session, new pictures with already washed hands under UV light exposure were taken.
These pictures were taken to evaluate the amount of UV gel left on the hands after each hand-washing episode and to determine if there were differences in hand-washing performance while using different applications -receiving different feedback. As previously mentioned, the order of developed applications used in each study stage was defined and strictly maintained in order to avoid any type of interference. Moreover, participants kept closed eyes while hands were scanned with UV lamp, thus preventing participants from being influenced by acknowledging previous results.

Hand washing quality in situ evaluation
As mentioned above, the studies, described in literature, show that use of neural networkbased technologies allows distinguishing among various WHO recommended hand-washing movements. Nevertheless, the neural network-based hand hygiene movements recognition software, used within the study and presented in this article, at the time of the study was in the construction process, hence it did not give the expected optimal accuracy. However, the purpose of this study was to compare the reaction of the involved participants and quality of the hand wash while using different types of applications. Therefore, two methods of analysis of captured hand wash videos were applied, where the first and the later one involved the use of a simulated neural network. During the study, the neural network was substituted with a human operator, an expert in hand hygiene. The research operator evaluated real time hand washing of all episodes carried out by research participants. This evaluation was executed on a computer located in an adjacent room by analysing the live-stream, which employs flask web server PC application to annotate respective videos. The annotation was carried out as follows: when the operator recognized a defined WHO hand-wash movement, it was marked accordingly. As well the operator evaluated quality of each recognized movement: the operator marked movement in state "OK" if the movement was performed fully correct, but in state "!" -if there were some inaccuracies of performing the movement. Depending on the type of application, the participant received certain feedback on the screen of the smartphone. All captured videos of hand-wash episodes were saved and later analysed with neural networks. In the end, both evaluation results were compared and evaluated.

Statistical analyses
1. Analyses of the operator annotations -average hand-washing duration for each application (total and for each movement).
Applications were compared using predefined time-based criteria -Average duration of total hand-washing procedure (sec) -Average duration of each hand-washing movements (sec) -Average duration of each hand-washing move in "ok" state (sec) -Average duration of each hand-washing move in "!" state (sec) -Average duration of "other movement" (sec).
2. Evaluation of hand washing quality based on washing time: to achieve 100% quality it was required to complete all 6 hand-washing movements, where a complete movement is a movement in "ok" state for not less than 7 sec. According to WHO recommendations based on suggestions from infectologists, it is recommended that during hand wash all 6 movements are performed in not less than (NTL) 7 sec each.
Evaluation method of hand-washing quality based on image data captured on UV light exposure.
Delta value, i.e., difference between the percentage change in hand cleanliness before and after washing, determined from measurements performed employing an UV lamp.
Evaluation process was performed as follows: to analyse the hand-washing colour proportion to image captured under UV light an image processing engine PicsArt was used. By handling Lasso crop function tool hands were separated for the backgrounds of the pictures and analysed separately in order to avoid malfunction of colour analysis. Later github open source code was modified to suit our needs and get the colour distribution histogram in the image [30]. The script works as follows -from 2 images (palm and dorsum of the hands) a collage is created -one picture, in which hands from both sides are visible without a background. Then, grade analysis and determination of the percentage of each colour (shade) is performed. The total number of clusters is defined in the code (150). This number was determined experimentally because at a smaller number of clusters, the colour spectrum is not distributed precisely enough and since the proportion of green colour after washing is mostly very small, it can only be distinguished. Further, the script generates a histogram image, where each column of the histogram has its own width (along the x-axis, which is proportional to the percentage of the given colour in the collage of hand images. Respectively, the whole width of the image is the percentage of all bars or 100% (100% corresponds to 600 pixels). Percentage of all colours -then the bar will occupy half of the histogram image (300 pixels). The obtained histograms were analyzed with an easel 3D program. Columns corresponding to the green colour and its tones (UV gel) were indicated using the select function. The program shows the total width in pixels of the selected columns. The pixels are proportional to the proportion of colour, so using the direct proportionality formula, the proportion of green colour was determined, which also means the proportion of dirt in the image.
3. Evaluation of neural network ability to recognize hand-washing movements compared to operator annotations.
4. Analyses of neural network recognition -average hand wash duration for each application (total, for each movement), evaluation of hand-washing quality based on annotations.

Results
Analyses of operator annotations -average hand-washing duration for each application (total and for each movement).

Compliance to WHO recommendations
Our finding shows that compliance to WHO recommendations depends on feedback type (A, B or C) provided. "Not guided" (type A) feedback was associated with the lowest average compliance to WHO recommendations (16.4%). However, the use of type B or type C feedback demonstrated significantly higher compliance, 75.9% and 95.6% respectively (Fig. 1). Data showed that there was a noticeable difference in average time spent for the handwashing process while using different feedback application types. Much faster hand-washing process is completed when participants are performing "Not guided" hand washing -Application Type A However, the longest time spent for the hand washing was detected while participants were using an application type B (see results in Fig. 2). Figure 3 shows the average performance of hand-washing movements and the time allocated for them during the whole hand hygiene process according to different feedback application types. As seen from the figure, when the participants perform hand washing while getting feedback from application (type A) then almost half of the total hand-washing time (22 seconds) other non-defined hand-washing movements were present. Total time allocated for defined hand-washing movements is noticeably shorter compared to the results collected using application type B or type C. Moreover, there is a noticeable longer performance of the sixth hand-washing movement compared to the time allocated to other defined hand-washing movements.  While evaluating hand-washing performance in the life stream video, the operator annotated each defined hand-washing movements labelling it with "ok", or"!". Where the label "OK" was used, it implies that the hand-washing movement was done correctly, while the label "!" implies that the movement was not done perfectly and requires additional adjustments. Moreover, the option of no labelling was present, in the cases when none of the defined hand-washing movements was present. Figure 4 gathers average time recorded for each labelled state for each movement and each application type.   . The average duration of hand-washing movements using different types of feedback applications (states "OK" and "!"showed).

Hand wash quality
To understand whether there is a real difference in hand-washing quality performance while using different feedback application types, additional analysis of pictures captured under UV light exposure was performed. Results showed that less sufficient performance was registered when application type A was used as a feedback tool. In this case, 7 SHS Web of Conferences 131, 02008 (2022) https://doi.org/10.1051/shsconf/202213102008 Int. Conf. SOCIETY. HEALTH. WELFARE. 2021 the average amount of hand surface qualitatively washed was defined as 97.26% of the total area of the hands. Thus, hand-washing performance improves while applications type B and type C are used, where the average amount of hand surface qualitatively washed showed to reach 98.40% and 99.00%, respectively (see Fig. 5).  Figure 6 shows the distribution of not qualitatively washed hand surfaces depending on the different feedback application types, presenting the percentage of non-qualitatively washed surfaces from the entire performed hand wash episodes. For example, using application type A, in 32% of all hand-washing episodes performed, UV gel was present on the top surface of the fingers.

Neural network
After the feedback experiments were completed, the hand washing episodes captured in the experiments were classified with a neural network. Then they were used to evaluate the match between the annotations created by the human operator and the neural network. The neural network was trained on a dataset that included both real-life hand-washing episodes from a hospital [29] and some of the experiments. More specifically, 20% of the experiments were classified as test data for the neural network and held out from the training phase. Subsequently, these test videos were used to test the accuracy of the neural network and its match with the human operator. The architecture of the neural network was the same as when [29], specifically, transfer learning on top of MobileNetV2 was used.
The network operated in a frame-by-frame fashion, classifying each frame individually. As some of the hand-washing movements, in particular, movement 1 and movement 3, are impossible to differentiate from seeing just a single frame, we applied time window averaging to the neural network's output in order to increase the accuracy of the classification. A simple majority voting was used within the window, and the duration of each non-overlapping window set to 1 second. Figure 7 shows the confusion matrix of the frame-by-frame classification, and Fig. 8 shows the confusion matrix when the time-window majority voting is applied. The matrices are normalized so that each row sums to 1.0, and the number in the diagonals correspond to the precision metric of each movement. The average F1 score is 0.65 in the first case and 0.75 in the second case, indicating a reasonably good match between the human operator and the neural network. If we are interested in just a binary quality measure -that is, whether all six washing movements were performed in the washing episode -then the F1 score of correct classification is 0.71. The recall metric of the classification is 1.0, but precision is lower: the human operator classified some of the videos as incorrectly performed, but the neural network did not. Finally, the washing duration was classified with 12.5% error, with the mean human-observed washing duration being 37.44 seconds per episode, and the neural network making 4.68 second average error in this metric. Once again, these initial results are promising, but leave a big room for improvement in the future.

Discussion
The main purpose of this study was to check association between real-time feedback given to person during hand-washing procedure and a level of compliance to the WHO recommendations on hand washing. Several types of feedback were tested providing a different level of assistance during hand washing procedure. The level of compliance was measured as a function of time while participants hand movements were recognized as correct. We found that full guidance feedback (Type C) was associated with much higher compliance to WHO guidelines, as was expected before. As the same time no guidance feedback was associated with lowest level of compliance mainly due to shorter time spent and numerous undefined hand movement.
World Health Organization guidelines recommend performing a 6-step hand washing method and the use of an alcohol-based disinfectant for better hand hygiene [4]. It is well structured procedure to wash all surface of the entire hand. We use special gel applied on hands before washing and UV light to detect unwashed areas. The finding of this study demonstrates that full guidance feedback is associated with the best hand washing results. Application Type A, the average amount of hand surface qualitatively washed was defined as 97.26% of total area of the hands. This result can be compared to Application Type B and Application Type C, which rated 98.40% and 99.00%, respectively. If the person can wash hands freely and use other motions for washing, as in Application Type A, we can expect much more unwashed hand surfaces. This is also reflected in our research as we found the dirtiest areas were nails and external surfaces of fingers. This finding is of major importance as clearly demonstrates benefits of guided approach even among healthcare professionals quite familiar with hand washing procedure.
To test hypothesis on non-inferiority of neural network-based assessment comparing with human assessor we perform evaluation of hand washing sessions previously assessed by human observer. During the study, neural network was trained with real data from hospitals. Our research has shown that this Neural Network model works in general. We found that: Qualitative research results depend on how well Neural Network recognizes handwashing motions. In our research, a smartphone camera fixed hand-washing episodes, after which the Neural Network performed an evaluation. The success of Neural Network is dependent on a high probability of accurate analysis for each captured frame, in correspondence with WHO-recommended motions or other unclassified hand-washing motions, which Neural Network was trained to recognize. However, if we analyse the Neural Network in more depth with and without a time-window algorithm, we receive average scores of 0.75 and 0.65, respectively.

Conclusion
Our study demonstrates that structured real time guidance provided during the hand washing is associated with better compliance to WHO recommendation and better overall quality of hand washing. Real-time full guidance feedback should be preferred pattern for integration in the multimodal monitoring solutions. Neural network-based assessment seems to be promising approach and integrated with real-time full guidance feedback have a great potential to improve quality of hand washing procedures.