6.1. Outlook
It is planned to release the software in the iOS App Store in early 2021. Before the release, at least one more piece should be added which focuses on rhythm instruments. Personally, I do not think that this app can be successful as a product. I consider the app format itself here only as a channel to deliver the experience of the artwork. Because I think that there is not really a market for it as a “product”, I plan to release the app for free. At this point it is much more important for me reach as many people as possible than to earn a few cups of coffee’s worth through sales. I will promote the app online on the website affinetuning.com and want to post inspiring videos to popular social media sites to attract players.
In the future, I would like to work with musicians on more tracks or features. Before that, some work still has to be done: The current architecture of the software is not ready to be used by outsiders or even uses any of the formats digital composers work with. This means that right now there is no sustainable workflow. Another crucial aspect is the lack of a common language, like a notation for such interactive and dynamic compositions which includes both the movement and the music notation. It will be interesting to research if there are choreography notations which also include music which could be modified for this purpose. So far, I did not look into this because of time constraints.
Figure 25. The app icon is inspired by a wave and the contrast of warm and cold.Image: Weibezahn
6.2. Reflection
One challenge about reflecting on this work is that the experience is individual for everyone. I can only write truthfully about my own experience which is probably very different from everybody else’s, because I have lived through the whole development of the software and music. I have my certain way of interacting with the system which in turn shaped how it works. I realized this time and again when I saw other people making drastically different experiences, sometimes “struggling” to make a connection at all. This is of course a result of my decision to design a “free form” system with not many rules to follow, which also results in less guidance. One interesting consequence of this approach are the connections which players try to make retroactively between the music and the movement. This is one of aspects which was most interesting for David Rokeby. One player, who was an amateur dancer, told me that she found it rather difficult and even exhausting because she could not just move to the music, and felt that she should be alert the whole time. It basically meant more work for her. In fact, by now I am quite sure that this system is something which offers a developing experience and different levels of reward over time. But I am also aware that most people won’t have the patience, motivation or simply interest for this process. Some examples: Initially, most people stay in the same spot the whole time, standing up straight, oriented towards the device. But some musical effects rely on the player to rotate, go low, or raise their feet. Also, utilizing starting, stopping, and the speed of movement as an interaction method also takes time to discover – not because it is difficult, but requires a certain mindfulness.
From a composers perspective I can only share my own experience so far, which is probably not a very relevant insight because I had almost no prior experience in music composition. The process is also very tightly coupled to the interaction design, thus I usually considered both aspects at the same time. One thing I noticed is that movements are often “quicker” than musical structures. One example: I wanted to couple the virtual room size of a reverb effect to the size of the bounding box of the players current pose. But people do not stay in a very “large” pose for more than a few split seconds, a much shorter time than the effect of reverberation needs. I ended up to use not the bounding boxes of the motion paths instead. As a result, this feedback feels much more indirect. I expect that most players will not see a connection between movement actions that occurred more than a few seconds ago and musical structures based on them. One thing which I found exciting — and in this case I think this holds true for “real” musicians as well — was that the interplay of sound, especially when using effects like a bitcrusher distortion, was often not predicteable; and this does not even include the player’s interpretation of the concrete sonic result.
Advantages and disadvantages of the platform
The app has not been released at the time of writing, so I cannot evaluate feedback from a large number of users. However, during the testing phase, I noticed that users have certain expectations of an “app” that differ greatly from, say, an installation in an exhibition. An app is perceived as a product, and serves a clearly identifiable purpose. Both interface and feedback are expected to be clearly recognizable/operable and unambiguous. Users already know before making an input what its purpose is and what result is to be expected. If this result does not correspond to the expectation, this is seen as an error in the software. The gaming sector is less strict, but the principle also applies here on the whole. Another common user expectation ran counter to my intention and the technical necessity of using the rear-facing camera: looking at and interacting with a screen. I took it for granted after a short while not to use visual feedback, but for many testers this became a real problem when positioning the device: they almost always tried to point the screen towards the room.
I had not realized another drawback until I started looking for testers: The required devices are still comparatively expensive. The cheapest compatible new iPhone costs 467 € in Germany at the time of writing. There were many people interested in testing, but they didn’t have a suitable iPhone. I had virtually traded the exclusivity of the exhibition format for another, mainly monetary threshold. In this respect, I can only hope that the technology will become more affordable over time (which is to be expected).
Technical Constraints
One challenge which consumed a lot of time was a certain unreliability in the pose detection. Time and again I tried to find algorithmic solutions especially to detect motion onsets, but they were never reliable and consistent enough. I suspect that this is also just a consequence of me not having a computer science or signal processing background and the according experience and knowledge of suitable algorithms. But even with that I am sure that some problems would not have been solved. For now, this shortcoming is something to be accepted; I must live with the noise. I tried to react to this artisticly in the last part of my composition, when noise and distortion are an essential part of the music, thinking of the interesting sonic results as a glimmer of hope.
A feature which was prevented by the tracking quality was stereo sound which changes its balance based on the players’ rotation. The result was more confusing than anything else because over time the rotation data would drift. Another technical limitation which influenced my concept early on is that the pose detection only works for one person at a time. It would have been very interesting to include interaction between two or more players in the same physical space. Apart from that, I identified two other technical constraints. One is audio performance when it comes to realtime synthesis and effects. Like in graphics, I imagine one develops an intuition for the performance ceiling, but I was lacking the experience needed for this. Unfortunately, the penalty here is much more punitive than with graphics because it results not just in a few graphical glitches or framerate stutter, but can be very unpleasant and even harmful for the ears and the equipment. Also timing problems can add up which can stretch envelopes, which in turn leads to more audio playing at the same time, effectively destroying the composition.
Another constraint which seemed rather minor at first was the fact that the pose detection only works with the back-camera. It turned out to be a real hindrance for usability. It would be much easier for users if they could see some kind of visual feedback at least for positioning themselves in the camera image. I still think that the approach of concentrating on sonic feedback is valid, but I also want the threshold of entry as low as possible. I tried to mitigate this problem with voice feedback in the setup process but it often happened that users would not comprehend this the first time and be lost.
Process
During the process I faced a number of challenges. Each on their own were not major setbacks, but, in hindsight, I was a bit oblivious to the fact that the topic of my thesis had both quite some breadth and depth. This led to some non-ideal workflows, fragmented periods of working in different areas (i.e. jumping around quite a bit) and a convoluted schedule. Ideally, this work would have been done by a team of specialists. And as everyone else during this time, I was also affected by the COVID-19 pandemic. Early on I planned to work with a choreographer from the Theater Bremen. But just before starting the collaboration, the lockdown happened, and shortly after, he fell sick. And it did not help that my partner was stuck abroad two months because of flight restrictions. It was also planned to show a version of the work at the conference “Kunst der Erfahrung” (“Art of Experience”) at the university Witten, but this event had been cancelled. After the collaboration had not happened, I worked on my own, only reaching out to testers a few months into the process. In hindsight I think the project would have benefited from earlier, continuous testing.
I realize that parts of this last chapter read a bit bleak. After all, it was a long and exhausting process and pretty isolated at times. But I still had a lot of joyful and stimulating moments as well. There were many instances when interacting with sound through my body simply felt magical. Maybe the most peculiar and interesting thing was when I got aware of the fact that I had created this weird system for me, in the first place. I have interacted with it not only in the short term feedback loops of my performances, but also at the larger timescale of development. In a way, this is true for many kinds of artworks, but the quality of the relationship in this case was pretty remarkable. I distinctively remember one time, when lagging feedback and unreliable detection actually evoked a sensation of weight and stiffness. Even though this feeling was primarily a negative one, I found the effect pretty profound, especially because there were no visuals involved, only sound.
Writing this, I am excited again to continue with this work.
· · ·
Thank you for reading.
With any questions, comments or corrections, please reach out to contact@maweibezahn.com.