Kinetic Narratives is an ongoing research project that explores motion authoring and real-time animation through hand gestures and computer vision. The core idea is simple: can we manipulate 3D objects in ways that feels "natural", using our body and no suits, no headsets?

The idea was to use computer vision frameworks to map kinetic motion to digital environments. For that I used open source libraries like:

• OpenCV
• MediaPipe
• OSC/UDP

Below there is one of the earliest prototypes I made in Unity using cvzone's RGB-object tracking module to extract 2D coordinate data and convert that into 3D world-space coordinates.

"Jose, and how do you get the third axis? Because a screen only has 2 coordinates."

Right, we don't actually have a 3rd axis so we need to infer that somehow. And we do that by calculating the proximity of my physical object sampled in relation to the position of the webcam. If you're curious about the technical details, you can check my GitHub repo on that.

The perception of control over digital assets felt really good which led me to understand how important is to work with low-latency modules like OpenCV and UDP communication to send data from Python to engines like Unreal, Unity or TochDesigner. I believe this infrastructure can shine in areas like interactive animation and live VFX pipelines. (did someone said Virtual Production or Interactive Installations?)

I’m currently expanding this system to include spatial interaction between multiple 3D assets. Imagine an environment where we can set different behaviors on our 3D avatars based on hand gestures. I propose a motion authoring system that uses no controllers but physical cues, embodied gestures and simple props enabling real-time interaction with digital assets.

Kinetic Narratives is a proof of concept that follows the same curiosity as other experimental tools and authors like Autodesk's' project Reframe and Murtanza youtube channel. In this context, the body is not just a controller or tool but but a performative and conscious actor for creative expresiveness.