The term "fluid counterpoint" may change in the course of the investigation. For now, it serves as a working concept. The term "fluid" here suggests looseness and continuums versus rigid boundaries and discrete, separable objects and events. "Counterpoint" indicates the interlinking of independent lines of activity to establish a greater whole. This interlinking is hierarchically organised in both the vertical (instance) and horizontal (time) dimensions.
The first part of the project involved the audio recording and motion tracking of a khyal performer, Tofail Ahmed. My intent is to map the motion tracking data to abstract visualisations, with the expectations of achieving this ideal of "fluid counterpoint" between the image and the music. In other words, rather than seeking an analytical solution to the "fluid counterpoint" challenge — or using my own artistic intuition and trial and error as I have in previous works — I am wagering that the spontaneous gesturing of the musician can itself act as the intermediary between the music and visual domains.
If so, this could provide clues to the construction of real-time visualisation control systems based on body tracking, allowing coherent multidimensional control of visual systems by a "visualist/interpreter" listening to music, for example.
In character animation and motion tracking, we speak of the hierarchy of the skeleton -- the descending line of inherited motions. I am working with the hypothesis that respecting the hierarchical organisation of joint movement is a key to mapping the movement data to abstract visualisation in a way that will have rich perceptual impact.
There are reasons to be cautious here.
Yes, the musician's gestures may satisfactorily evoke for the viewer aspects of the fully embodied unfolding human meaning underneath the music. But that doesn't mean that objective motion measurement can be readily mapped to another visual organisation and still carry deeply embodied semiotics of closed versus open, pointing away and pointing in, ascent and descent, tension, release, passing trajectories, etc. Further, the objective data is almost guaranteed to not carry the essence of culturally-defined symbolic gestures, or that essence is only going to survive mapping that is customised to that purpose.
Further, there is a risk of a strange twist on the uncanny valley — where an mapping of the motion data to the abstraction results in an image that is disturbing as clearly biological motion jerks around masses in a way that has none of the characteristics that give the original gesture it meaning. It should seem abstract, but instead it seems like an all-too biological monster.
So, as a person critical of much music-to-image mapping, I may just find more reasons to remain a critic.
No comments:
Post a Comment