“Hyperwiper”: Dancing windshield wipers Synchronizing Windshield Wipers with Sound Entertainment System Inside Vehicles

. Instead of adding to the driving cacophony, actively orchestrated windshield wipers can enhance musical audition, reinforcing a beat by augmenting the rhythm, increasing the signal:noise ratio by aligning the cross-modal rhythmic beats and masking the noise, providing “visual music,” the dance of the wipers. We recast the windshield wipers of an automobile with advanced multimedia technology, allowing wipers to dance to music.


Introduction
The basic motivation for this project is that most normal songs have a consistent (or perhaps slowly changing) rhythm that can be used as a "click track" for dancers or percussionists, including robotic windshield wipers. By hiding the sound of wipers in the rhythm (a kind of masking), cabin signal:noise ratio can be improved. Many car drivers and passengers enjoy listening to music while driving [1], and support of cross-modal, synaesthetic stimulus by coordinating auditory and visual rhythm [2] can enhance that experience, a kind of visual music.
This project is a resurrection of an earlier project 1 that featured miniaturized model car wipers, first shown publicly eleven years ago [3,4] and subsequently extended into a full-scale proof-of-concept five years ago [5]. The idea isn't totally original. Popular culture, especially television and cinema, is replete with instances of robotic or sentient vehicles, including "My Mother the Car" [6], "[Herbie] The Love Bug" [7], "Wonderbug" from The Krofft Supershow, 2 Gadgetmobile from the "Inspector Gadget" series [8,9], KITT from "Knight Rider," 3 John Carpenter's "Christine" 4 , Transformers, 5 and Pixar's "Cars" francise. 6 More recently, an inventor developed such a product [10], originally using Arduino before switching to Raspberry Pi to get more powerful processing. That project directly controlled speed of motor, which is illegal in some places, including Japan [11].
Our goal, then, was to improve upon these past efforts, crafting an "after-market" embedded system with modest power demands, integrable with vehicular wiper systems, and inherently compliant with any safety regulations. There are many subtleties, but basic operation is a phase-locked loop (pll), a well-known engineering technique featuring adaptive feedback for entraining pulse trains, aligning a delayable output pulse stream (to modulate phase) with an input pulse stream, the so-called "reference." An audio source such as music player or smartphone music app is connected to a beat detector deployed as an embedded mechatronic interface. The output of this module is a realtime pulse stream or "train," which is dynamically compared to the position of windshield wipers, as detected by a windshield-mounted magnet-based proximity sensor. Other research projects have used beat extraction to time-warp video contents [12]; we use such analysis to animate wiper motors. Entrainment software is used to synchronize the activation of the wipers with the rhythm of the audio source. The following sections detail the dataflow and deployment of our device.

Beat Detection
Transient detection systems have been developed elsewhere (See [13], for examples), but these systems usually work in computer-based architectures. Although some computers have become cheaper and smaller than those originally used for the development of the aforementioned systems, their energy demands discouraged their deployment in embedded systems where battery life is an important consideration. For instance, Raspberry Pi (a popular inexpensive micro-computer) runs at about 270 mA with 5 V power, whereas Teensy v. 5 and Arduino Uno (a similar priced micro-controller) draw about 120 mA at the same voltage (i.e., a 2-fold reduction in power requirements). Some micro-controller-based systems have become powerful enough to run complex DSP procedures in real-time, such as Fast Fourier Transformation on CDquality audio streams. For this research, we took the latter approach (i.e., a micro-controller-based system) that can be easily integrated with other projects based on the Arduino platform (programming language and hardware), as shown in Fig. 1.
In our solution, a digitized audio signal is sent to a module that computes spectrum-based Onset Detection Functions (ODFs). In this module, the monophonic signal is first Fourier-transformed, and comparison within frequency bins of consecutive frames is performed to determine the likelihood of a transient to be present. The energy per band, is computed in every k frequency bin of an n frame by accumulating the Fourier-transformed frame weighted with a Hann window w(m) on each m th sample.
To obtain a robust system that tolerates noisy and missing parts of a signal, we use a Linear Programming Component (LPC) to model predictability of the signal, sending an output even when the supposed current transient is missing or the ODF has failed to identify it. Finally, the predicted output is sent to the software and hardware that controls the motor of the windshield wiper.
Regarding the LPC, we compute the actual output of the systemx(n) by a weighted combination of past and present values (effectively, a Finite Impulse Response in an FIR filter):x where p is the order of the LP model, and a k are the prediction coefficients. These coefficients are computed using Burg's algorithm [14].

Current Status of Development
Development status regarding installment of a dancingwiper-mode module, replacement of the park/run switch of the original wiper motor with magnetic switch, development of hardware to detect windshield wiper position, development of a controlling function for dancing-wipermode and development of hardware to control the windshield wiper have been previously reported [15]. Also reported was development status regarding software to use a windshield wiper position detector, to detect a beat from audio signal, to predict next beat for sending a signal to the windshield wiper motor, to control a windshield wiper motor, to link the separate software parts together and test of dancing wipers system (software, hardware) both parked and driving. This design was subsequently refined and re-presented [5].
In 2015, we were discussing the software on two platforms: one is a Teensy (Arduino-compatible microprocessor unit, MPU), and another one is a Raspberry Pi (RPI). However, we decided to develop the software/hardware only on the Teensy platform because the RPI disadvantage of longer startup time is still a significant issue for this development. Also the reliability on having the OS (RPI) on an SD Card would be much less then and Teensy based system. [5]. Fig. 2 shows the timing diagram for a single cycle. On our custom-made printed circuit board, shown in Fig. 3, an input pin is designated to read the status of the magnetic switch, deployed as a sensor of windshield wiper position detector. We developed software to calculate the position of windshield wipers by pulling the status from the windshield wiper position detector.

Module to detect windshield wiper position
In [5], visibility of the windshield wiper was not adjusted for each car, but this new version accommodates such variety. All cars are different regarding windshield visibility from the viewpoints of driver and passenger seats. Therefore, we have to know the exact time difference of each car regarding two events: First one is time after the windshield wiper leaves its rest position before it emerges into view of the driver and passengers. Second one is the time between windshield wiper passing the magnetic switch 1 st time after the Hyperwiper module activates the windshield wiper motor and 2 nd time the windshield wiper passes the magnetic switch (after the Hyperwiper module activates the windshield wiper motor). In order to determine these time differences, we let the Hyperwiper module start up the windshield wiper motor for one preliminary wipe to allow it to calculate the time differences.

Module to control a windshield wiper motor
The embedded software determines whether to switch the motor on or not for every beat, based on the position of the wiper and coming beats. In case the software switches the motor on, by receiving the information on the beat of the music as a data from the beat detection software, it predicts the coming beat and gives a signal to the windshield wiper motor for starting a wipe by considering the time delay of the starting of the windshield wiper to let the windshield wiper move at the correct timing of the beat. Regarding the 2015 version, when the beat arrived, the motor got switched on; that is, the software only took into account the time difference between switching on the wiper motor and the windshield wiper actually moving. With this version, the visual impact was limited for certain cars in case the resting position of the windshield wipers is below the windshield. In order to overcome this point, we refined the software to let the timing of wiper motor be adjustable, such as the motor starting before the beat is recognized by the listeners. Also, we made software to measure the time difference between windshield wiper actually moving and windshield wiper actually being visual for a driver and passengers. Accordingly, visual impact of the wiper movement is stronger and more accurate.

Developing "Hyperwiper Logic" software to integrate the software modules
The software for using a windshield wiper position detector, detecting a beat from audio signal, and predicting next beat have been developed separately. All need to be linked and integrated by Hyperwiper Logic to make the Hyperwiper module work. Fig. 4 shows the entrainment timing.
The Hyperwiper logic determines the timing for a windshield wiper to start. For safety reasons (in case no music is playing while driving through rain), if no beats are detected, the Hyperwiper Logic should start the wiper motor after a brief "timeout" default interval.
In 2015, we used a display and a joystick as an interface for setting up the parameters for timing of the signals and a basic timer for minimum wiper function in case that no music is being played. However, to simplify the hardware design, we decided to abandon the interface and parameter use. With the current version, the timing of the starting motor signal can be varied by changing a potentiometer ("pan pot" variable resistor). The basic timer is fixed for 6 s, which means 10 back-and-forth wipes per minute, if no music is being played.

Applying pulse width modulation (PWM) to control windshield wiper motor
Recently, we extended the hardware and software with pwm (pulse width modulation), so that the amount of power delivered to the windshield wiper motor can be modulated. This method allows a more nuanced expression of musical beats by softening the heretofore on/off "binary" control with "analog" speed control, speeding up and slowing down the movement of the wiper.

Windshield wiper types
Mechatronics is the interdisciplinary fusion of electronic, electrical, and mechanical engineering, including robotics and systems with electronically mediated control. The Hyperwipers project, basically a simple robot, is an instance of such a mechatronic system. A vehicle's wiper design type is selected based on various criteria, such as windshield size, curvature, and outline shape. These factors are determined mostly by the overall shape of the body of the car, especially the length and angle of the A pillar relative to the roof and top portion of the firewall which separates the passenger and engine spaces. Changes in any of these factors, among various other parts, dictate the aerodynamics and wind resistance of the front of the vehicle as well as visibility, which results in multitude of different shapes, dimensions, and arrangements of front windshield wipers, as illustrated by Fig. 6. Most common arrangements include two wipers in approximate parallel, working in tandem with various levels of overlap. Single wiper type systems are less common nowadays. Where modern systems vary, however, are the wiper anchor positions. In case of double-wiper assemblies, some have one anchor in the bottom middle of the windshield, and the other in one of the bottom corners. The latter position dictates the direction of wiping, either to the left or to the right. A second type of anchor positioning is when the wipers are anchored in opposite bottom corners of the windshield [16].
Even though the two most common tandem wiper assemblies fundamentally differ in their wiping pattern, the  underlying mechanisms is common, consisting of four link mechanisms powered by a single electric motor positioned either in the center or on one of the sides relative to the linkage. An example of this system is displayed in Fig. 7 and described in the next subsection.
The mechanical system resides below the bottom of the windshield, most commonly hidden in a dedicated compartment, either between the layers of the firewall or simply covered by a heat shield to protect it from heat dissipated into the engine bay. Another common placement of the linkage assembly is from underneath the interior dashboard, on the other side of the firewall compared to the later style. Furthermore, to decrease wind resistance of the vehicle, wipers are usually shielded by the edge of the front bonnet (hood). Depending on the dashboard design, this results in increased visibility as the wipers are not in the driver's field of view when inactive. Unfortunately this presents a problem for the deployment of the Hyperwiper controller, in particular the reed switch described in §3. As the reed switch needs to be in close proximity to the mag- net placed on one of the wiper stocks when the wiper is in its resting position, on certain vehicles this might be difficult to do, especially due to the necessity of windshield or dashboard removal in some installations, which is timeconsuming and costly. Therefore, in our design, we had to account for both styles of wiper resting positions, obstructed and unobstructed. This challenge was solved by extending the software driver to include a start-up process to accommodate the extra detection.

First deployment vehicle description
The first deployment vehicle was selected to be Toyota Crown Estate. Its design [17] consisted of several challenges which had to be addressed as they are to be expected on majority of other vehicles. One challenge was the previously described obstruction of the reed switch, as it was not possible to be placed in close proximity of the wiper resting position. Fig. 8 illustrates the aforementioned cavity between the layers of a firewall.
Inside this cavity is placed another stamped sheet of metal which serves as a tray in which the wiper linkage assembly is recessed. Fig. 9 shows why the wiper resting position is inaccessible, as the bottom of the windshield is attached to the top lip of the metal tray and the interior dashboard ends above this line.
Our deployment vehicle is equipped with a center touch screen unit from which the audio, climate, and several other vehicle functions are controlled. Furthermore, the unit has a 6-disk CD changer and tape player mounted below it, with additional modules such as TV tuner and GPS receiver, mounted inside the dashboard and trunk space respectively. GPS map data is read from a disk drive placed below the passenger airbag, above the glove box, above the passenger's footwell. Again, this arrangement proved to be quite challenging regarding adding an audio signal junction for output into the Hyperwiper module. Taking out the head unit and splicing into the complex wiring harness or re-pinning the head unit's proprietary connector was considered too complicated. Therefore, we decided that the audio signal wiring route closest to the Hyperwiper module's power delivery would be best.

Implementation
As the wiper motor is controlled through a phase-locked loop circuit as described in § 3, it was necessary to locate its fuse location. Conveniently, the wiper fuse in the de-   Concluding that the power delivery is provided from the fuse box and the closest audio signal output from the head unit is the driver-side mid-range ("squalker") speaker, we chose those two junction points. However, where exactly to splice into the wiring loom was the next decision to be made.
As seen in Fig. 10, the fuse box is accessible after pulling off a simple door sill plastic trim piece, and its wiring is fully exposed after peeling the carpet off the side.  However, after exposing the harness, it became clear that the situation is even more complex than that behind the head unit. Furthermore, for the sake of safety, it is generally good practice to splice into a wiring harness in a separate branch rather than the main bundle. Further considering that the installation should be easily reversible, we chose to use a fuse tap instead of splicing directly into the power delivery loom. This fuse tap [18], shown in Fig. 11, allowed interception of the power delivery to the wiper motor and also power for the Hyperwiper module by simply plugging it into the existing wiper motor fuse slot, as diagrammed in Fig. 12.
Inspecting the vehicle's wiring further, there are several potential junction points for the audio signal output, as suggested by the speaker layout shown in Fig. 13. However, as with the power delivery, some points are harder to access than others. The simplest approach was to remove  Figure 11. Fuse tap the door panel and splice into the speaker wires just before their termination at the driver (loudspeaker). The more difficult approach was right above the fuse box, where the door sub-harness branches off from the main loom, seemingly the point of choice. However, the bottom of the instrument cluster, above the pedals, has a multitude of other sub-harnesses, the On-Board Diagnostic (OBD II) interface, as well as mechanical parts such as the steering column, brake pedal linkage to the brake master cylinder, and throttle cable. As this vehicle is rather old, many systems are analog and mechanically connected. In more modern vehicles, such interfaces are often "drive-by-wire," mechanically disconnected. Therefore, in other situations, it might be possible to place and easily connect a Hyperwiper module somewhere in the driver's footwell.
In this current case, however, it would be difficult to crimp additional terminals in this place. Therefore it was   decided to make a small hole in the door panel and feed the signal wire through it into the door pocket, then feed the power cable through the door hinge grommet, through the signal wire hole, finally connecting it to the module. (The connectivity, then, is somewhat inelegant, as the original audio signal at line level is empowered by the amplifier to speaker level, and the Hyperwiper impedance matcher backconverts it to line level for analysis by the embedded beat detector.) The same procedure was performed with the ground cable, but instead of one end containing a fuse tap, a simple eye crimp was terminated. Then it was bolted onto the fuse box bracket, which serves as the vehicle's common ground. Fig. 14 illustrates the final implementation and locations of crucial points.

Conclusions
There are many possible extensions to this system, including distinguishing the downbeat among other rhythmic pulses to more sophisticatedly align the mechanical animation with the song, more discerning beat and rhythm detection (such as those based on machine learning), coordinating multiple sets of wipers (such as both front and back sets), and more complicated wiper "dances."