Grading video with gestures. Is it useful? Not really.. Fun? Certainly, and whats more important here is the understanding of physical interfacing and getting things up and running quickly.
In part two we’ll be looking at adding a more favourable user experience with a nice interface & higher level gestures to make the experience a little less trivial. For now lets look at getting basic input from the real world and mapping that to anything you wish. I’m using a HD projector here for the full minority report experience.
You will need a Microsoft Kinect for Windows, i’m using a version 1, but a Kinect 2 is even better. Install Kinect Runtime & the Kinect Developer Toolkit. The toolkit includes Kinect studio which allows you to record yourself, play back & loop the data stream in to TouchDesigner. This will make your life easier as you start working with the Kinect to save you jumping up & down to test your movements.
Boot up TouchDesigner, or download it here. Awesomely they provide a free non-commercial licence, perfect for getting to grips with the software. If you’ve never used a node-based program before, i’d advise to watch a few TD 101 videos on their website so this doesn’t seem too daunting, but its actually super simple to use.
Lets get started:
Start a new project & double-click in the node view, this brings up the node palette, select ‘kinect’ from the CHOP tab. (TIP: TD will isolate highlight node titles if you start typing the name of the node you want. If you type ‘ki’ you will see that everything else but kinect is greyed out. This saves a lot of time quickly spotting the node you want, but it takes a while to learn all the names for more obscure functions).
A CHOP is short for ‘channel operator’, similarly TD has texture operators ‘TOP’s, surface ‘SOP’s, and so on. A CHOP contains channel information; discrete values similar to a container for variables. A lot of real world physical input comes in to TD as a CHOP and image inputs similarly as TOPS.
Grab the corner of the CHOP tile that has appeared and resize it to make it a bit bigger. Click on the newly created node to bring up its properties on the right hand side. You will need to select your device from the dropdown & wait for a connection.
Jump in front of the kinect now. You will see a number of horizontal bars start moving in the center of the Kinect CHOP tile. Wave your arms and legs around & you will see a vague correlation between your movements and these sliders. Each of these bars represents a single computed attribute about your skeleton, your head position, rotation, hand height, depth and so on…By isolating some of this sea of information we can read your inputs calculated by the kinect as single values such as your hand height for example.
To isolate hand hight we need to pull out that information from the Kinect CHOP. To do so we must add a select node. Double click on the background again and find the ‘select’ CHOP. A second tile will appear on the node view, position it just to the right of our Kinect CHOP. We must wire the two together to allow information to pass (cook) from the input to our select node. Think of it as plumbing. To do this you must find the tile’s outlet located on the right of the Kinect CHOP tile. Click and drag across to the left hand input of the select CHOP & you will see a virtual wire which represents this connection.
A select CHOP allows us to filter and isolate a specific channel from the list of many data streams the kinect gives us. To do this we must tell the select node the name of the channel or channels we wish to isolate. Each channel has a name assigned to its value & for the kinect each value has a very logical name, but what are they? You can read them from the small tile if you can make it big enough, but thats a bit of a pain if there’s a lot of channels. However If you middle mouse click on any CHOP you will get a list of available channels, rather like a menu.
Lets pull out the left hand height, this is an Y axis translation. Find its channel name from the kinect CHOP
Copy this value into the ‘select’ ops channel names box (located in its properties) to filter all but this channel. Now if you rase your left hand you will see this value move in the select node preview tile. From here we must take that value and map it to a parameter to make it do something.
Lets bring in a movie file to perform our grading transformations on, or even a still image if you like. You can drag a file from windows explorer in to the node view & TD will create a movieIn TOP for you automatically. We are now working with pixels rather than channels here so this type of data is known as a texture operator (note the tile’s purple colours to easily identify the op type).
The non-commercial version of TD will restrict the resolution to 1280×1280 so you may get a warning on some media but it will be reduced automatically for you. To modify the image, lets go and add another node, find the HSV op in the same method as before and wire it just after the video file. If you click on the HSV tile and bring up the properties you will see the rollout on the right hand side. Grab the saturation slider and move it up and down. The preview tile will reflect these changes (or hit the blue circle below the preview to set it as the background). But how do we map our select value to this slider? Well firstly we need to make the number scale match. Bring the slider all the way up & down and take note of the value range, you’ll see it goes from 0-1. Then take note of the values when you bring your hand up and down in the kinect, around -0.5 to +0.5. We need to scale our input range to match that of the saturation slider. To do this we use an Op called ‘math’ Find it and wire it in just after the select node. In the range tab in its properties you can then set the input and output values. Enter -0.5 & 0.5 in the min max, then set the output to be scaled to 0 to 1. Now you will see the channel slider transformed to this new scaled value.
To connect this value to the saturation slider we need broadcast our number by a method called ‘exporting’. Exporting allows you to send a value remotely to another operator in your flow. Select your math op and click the little star in the bottom right of the tile, then drag the channel text in the center of the tile and drop it over the HSV op. A list will pop up asking you where you want to send your exported value. Choose Export CHOP: saturation, and that’s it. Open the properties & you will see the slider now should be connected with your arm. You will also see a dotted line on your flow representing the connection between the nodes.
Do the same for any other parameters you want to map to a data channel, be careful to read the correct ranges from the sensor data, and the range to output via your math CHOPS.
Add a ‘view’ operator from the operator menu (located in the COMP tab) then open its properties and drag your final node on the the reference operator box, this allows you to map your final image to another monitor, in my case its set to output to monitor 3, a projector. Make sure you select fit to destination and borderless in the properties, then toggle ‘open’ to make it live.
In the next part we will work on a visual UI overlay and improving the usability with a visual gesture ‘ratchet wheel’ to allow much finer & less fatiguing control. We will also look at interpolating the kinect data for a slicker user experience.
Have a look at this quick video showing were we want to go next, the UI is added & smoothed to make the experience feel more controlled, as it averages the jittery data its also much easier for the user to perform fine control. Remember to use the best quality footage you can, TouchDesigner will playback MXF’s or image sequences happily if you can get your hands on some nice flat footage with plenty of info data in.
The hue angle isn’t particularly useful for the real word but in the next tutorial we will develop this for a graded/ungraded wipe like in this video. Then finally we will look at full UI/UX with a polished mode selector for the hand functions.