Bézier-Spline Control Curves in Max/MSP/Jitter

I have made my Max/MSP 6 abstractions for generating Bézier-spline curves available on the software section of my web site. The package includes a Bernstein-polynomial abstraction, a 3rd-order Bézier-spline generator, and a wrapper that expresses a constrained Bézier-spline over time in a form excellent for creating control signal nuance or for reshaping linear inputs to have ease-in and ease-out characteristics.

Towards Essential Body Relationships

Though I am intending to map the motions of Tofail Ahmed to an abstract visualisation, I want to ensure that important perceptual aspects of the body motion translate effectively. So it is helpful to analyze the performance motions. My intent is to identify high-priority relationships or parameters that I will seek to honour in the abstract visualisation. What "honouring" can and will mean in practice remains to be seen.

Gesture Types

Borrowing terms from Martin Clayton's study of gesture in Khyāl performance (2007), the performances I recorded of Mr Ahmed contained physical gestures that serve as "markers", "illustrators" and "emblems".

Marker gestures indicate a specific time point in musical structure, such as beating a pulse, identifying a downbeat, etc.

Illustrators appear analogous to the melodic flow/motion. The vast majority of the motion falls into this category. It is one vast territory that covers a lot that is very interesting — and very difficult to talk about analytically. This, perhaps, is precisely why it is so valuable!

Emblems, or symbolic gestures, are more based on cultural convention and can be translated relatively readily into a verbal equivalent. An example would be indicating approval with a hand wave.

Emblems would be the most problematic type of gesture given my intent. An emblem is at high risk of disappearing in an abstraction, given the very precise body-arrangement and audience reading it entails. Fortunately, there are very few emblems in the performance I am working with. The closest is an invocation-type emblem. This general position — palms close together, often in front of the face — is important. It creates the impression of focus and preparation, and occurs at the beginning of the first and last phrases of the performance. It also appears at the start of 11 other  phrases of the 60 total phrases. This invocation emblem is also often acting as a marker of the start of phrases. It is doing dual duty.

Invocation: Hand position at start of phrase 1

I am wagering that the element of proximity/closedness is as crucial here as the fact that this can be read as a sign of invocation/gathering.

If the invocation emblem often marks the start of a phrase, the most common marker of a phrase start involves the fingers oriented towards each other horizonally, at mid-body level. The example below is appears at the start of phrase 3:

Rest Position: Hand position at start of phrase 3

I am calling this the rest position. 27 of the 60 phrases start with the hands arranged in this general position. Five others are in closely related positions.

Though there are true full-rest positions of hands on knees (at start) or hands in lap (at end), these seem like outliers that won't be useful as a base position, since none of the Illustrator gestures operate in those spaces.

Measures

Given the above clues, I created some measures to help explore what body relationships might be highly correlated to the structure of the music. Basically, I am assuming that if my abstract visualisations at least clearly carry some of large scale articulators of phrasing, the details within will "take care of themselves".

Proximity to Rest Position (PRP): Clayton (2007) used distance of hand from a rest position in his analysis. I am taking a similar approach, measuring the distance from each finger tip to the neutral position and summing those distances. [Aug 10: However, I might consider using a different term, like neutral position, since Clayton appeared to use rest position to apply to true resting of hands on the legs.]

I did this in Blender by creating an Empty at the rest position and an Empty to represent the distance. I animated the Z axis of the latter with a Driver:

Finger Tip Proximity (FP) and Thumb+Finger Tip Proximity (FTP): The finger tips are in close proximity in both the invocation and the neutral position. So it could be of value to simply measure the distance between the finger tips.

Plus, I took a more refined measurement of hand tip proximity is to take the distance between the finger tips and between the thumb tips, and average them.

As it turns out, close finger tip proximity is often closely related to the beginning of phrases. However, finger tip proximity also closes in fairly often during other mid-phrase events.

Shoulder to Hand Proximity (SHP): The close-to-body versus far-from-body contrast also seems important. As the arms straighten, hands move away from the body. So a simply measure of the degrees of far-from-body is the distance between the shoulder and the hand. This measure sums the left and right shoulder-to-wrist distances.

Hand Closedness (HC): This sums the left and right finger-tip to wrist distances, to provide a measure of the open versus closed state of the hand.

Video

The above video shows the first 18 phrases, with graphic representation of the above measures. This is enough to convince me that Proximity to Rest Position is an excellent candidate to focus on. (But how?)

I could spend a lot of time now analyzing the relationships of these measurements to the music. But that probably will not actually help me attain my immediate goals, so I might have to set that aside for another time.

References

Clayton, M. (2007) "Time, Gesture and Attention in Khyāl Performance". Asian Music v.38 n.2.

Tofail Motion Capture Mapping Test "simple-03"

This test makes me confident that the hierarchical motion mapping idea is worth pursing:


This is a very simple mapping of the upper body motion of Tofail Ahmed as he sings an khyal alaap in raag Bhairavi.

The spheres follow the middle finger and thumb endpoints, with the camera in the viewer position facing Tofail.

The planks are linked in a parenting hierarchy and receive local rotations of bones in the skeleton. (The capacity to do this is easily missed in Blender: when identifying the target object of a Copy Rotations constraint, one indicate the armature, then a second drop down will appear that lets one indicate the target bone.)

Also, to smooth out issues I was having with sudden shifts in the thorax and pelvis rotations, I substituted in the head at the thorax point and made that the root. So the line is head : clavicle: humerus : radius : <hand : finger > <thumb>.

An empty provides parenting to the root, and the empty is rotated 180° on each axis across the whole performance. This provides a foundation of slow, continuous motion to match, if you will, the fundamental drone. The head block, then, applies the rotations from the head as an offset to this base angle.

I'm intrigued by the juxtaposition of the direct position mapping (the balls) with the hierarchical mapping (the planks).

Indian classical music + gesture + visual music...

So I inaugurate this blog with snippets of work from an ongoing research project I launched in Autumn 2011. This project investigates the idea that new approaches to audiovisual composition (a.k.a. visual music or video music) can be inspired by the spontaneous gesturing of Indian classical vocalists.

Some Indian vocal performers’ hand and arm gestures seem to provide a kind of parallel discourse that reveals dimensions of what is being expressed musically. Can this spontaneous gesturing inform the creation of a satisfying "fluid counterpoint" relationship between complex sound and image gestalts?

The term "fluid counterpoint" may change in the course of the investigation. For now, it serves as a working concept. The term "fluid" here suggests looseness and continuums versus rigid boundaries and discrete, separable objects and events. "Counterpoint" indicates the interlinking of independent lines of activity to establish a greater whole. This interlinking is hierarchically organised in both the vertical (instance) and horizontal (time) dimensions.

The first part of the project involved the audio recording and motion tracking of a khyal performer, Tofail Ahmed. My intent is to map the motion tracking data to abstract visualisations, with the expectations of achieving this ideal of "fluid counterpoint" between the image and the music. In other words, rather than seeking an analytical solution to the "fluid counterpoint" challenge — or using my own artistic intuition and trial and error as I have in previous works — I am wagering that the spontaneous gesturing of the musician can itself act as the intermediary between the music and visual domains. 

If so, this could provide clues to the construction of real-time visualisation control systems based on body tracking, allowing coherent multidimensional control of visual systems by a "visualist/interpreter" listening to music, for example. 

In character animation and motion tracking, we speak of the hierarchy of the skeleton -- the descending line of inherited motions. I am working with the hypothesis that respecting the hierarchical organisation of joint movement is a key to mapping the movement data to abstract visualisation in a way that will have rich perceptual impact. 

There are reasons to be cautious here. 

Yes, the musician's gestures may satisfactorily evoke for the viewer aspects of the fully embodied unfolding human meaning underneath the music. But that doesn't mean that objective motion measurement can be readily mapped to another visual organisation and still carry deeply embodied semiotics of closed versus open, pointing away and pointing in, ascent and descent, tension, release, passing trajectories, etc. Further, the objective data is almost guaranteed to not carry the essence of culturally-defined symbolic gestures, or that essence is only going to survive mapping that is customised to that purpose.

Further, there is a risk of a strange twist on the uncanny valley — where an mapping of the motion data to the abstraction results in an image that is disturbing as clearly biological motion jerks around masses in a way that has none of the characteristics that give the original gesture it meaning. It should seem abstract, but instead it seems like an all-too biological monster.

So, as a person critical of much music-to-image mapping, I may just find more reasons to remain a critic.