Drawing in Space and Her Favourite Positions

'Drawing in Space' is an experiment I've wanted to try for a while: using OpenCV and a webcam to draw in 3D.  Essentially you wave a coloured square of card around in front of the webcam and my code converts the path your hand takes into a 3D Drawing.  First it filters the image based on a colour you have chosen, selecting all pixels within a colour range determined by the mean and standard deviation of the patch chosen in the calibration phase.  It then finds contours in the filtered image, simplifies these reducing the number of edges, checks if any contours are square (concave, under a certain contained area, and with all internal angles within a given tolerance).  From all the squares found it then chooses which is the most likely square to follow on from the last based on the sum of the square of the magnitude of the delta in each point's position between frames (trying different permutations of the order of corners to account for rotations and mirroring).  If it can't find the square it motion tracks the corners from the last frame.  If any of these are missing it interpolates the data, and if all of them are missing it tracks strong features in the last image (in full colour, unfiltered) and uses a simple nearest neighbour algorithm to associate each corner of the last square with a nearby tracking point (if it was clever it would use some kind of error detection here to eliminate anomalies and then a delaunay triangularisation to interpolate points, but it needs to be lean to run at all).  If all of this fails, the script uses a third order differential on the last few points to extrapolate the next position, filling in the gaps for a few frames until data resumes.   The script also adjusts the colour it is searching for frame by frame to account for changes in light conditions.  Optionally you can also use the chessboard tracking instead of a coloured square.

The drawings are displayed in anaglyph as you draw, and can be exported as OBJ files.  I need some options to adjust the convergence point and so on, but its mostly there now.  A beta version of the code is online here (right click, save target as).  You need matching python 2.7, numpy 1.6.1, and opencv-python 2.4.0 compiled against each other.  If you're running Windows you can get numpy and opencv packages here.




'Her Favourite Positions' is an animation of 186 Portraits and 37 Full Body Poses from my Life Drawing classes at the RCA.  I matched them up frame by frame in AfterFX, and did a little warp correction here and there to get the proportions generally accurate.  I'm quite amazed that this even worked at all and that you can see a ghostly generic photofit face through the mess of lines.

Copying Poses with Constraints in Blender

Often I find that I need to use the copy and paste pose buttons to get a smooth transition when animating a ChildOf constraint.  Blender's built in copy and paste pose buttons ignore constraints so I coded my own.  Basically select the bones you want, run the 'copy pose to buffer' script, then change frame to after your constraint influence has been changed and run the 'paste pose from buffer' script.  This way you can get nice smooth transitions across those constraint changes...

Converting MonkeyJam Files to Spreadsheets with Python

I've timed out the 2D blocking for all 42 shots of my 'Crows' short in MonkeyJam.  Today I realised it would be great to have this accessible as real paper X-sheets which I could print and annotate as I work through the shots in 3D.  MonkeyJam beta's print feature seems pretty unstable and sometimes just generates junk, so I managed to hack together this quick python 3 script which will read the MonkeyJam '.xps' file and spit out a .csv file which you can then open in OpenOffice or Excel (or just open in notepad and copy and paste into a pre-formatted spreadsheet with all the right font sizes, column widths etc).

Demo Video

Here's the (almost) completed addon, and here's a demo video showing a real test case.  Note how I'm working in the 3D view the whole time and in the background the script is adjusting all the F-curves while I work.  Obviously you don't need to have the graph editor open, but its nice to remember what animating used to be like!

Quaternion Beziers in the 3D View

Quaternions might have great benefits for key-framing rotation (no gimbal lock and so on) , but I have to admit that when it comes to the 2D graph editor I don't stand a change of working out what effect of grabbing the handle of a key on a Quat's W channel will have on my animation.  Generally I'll adjust the key in the 3D view and leave the bezier handles well alone set to auto (or vector if I want a snap) because independently changing any of the W, X, Y, or Z channels is unlikely to yield any great improvements for my animation!

Today I expanded my add-on to draw the handles for Quaternion and Euler rotations, and to show the path of the tail of the bone (which is probably what we want to track when we're adjusting rotations).  The timeslide tool still works but I need to do a little more coding (and get my head around what a normalized quat is) before grabbing the bezier handles in the 3D view is possible.

EDIT: http://www.pasteall.org/19807/python - now with quaternion and euler editing!  Editing preview shown below.  At 1500 lines it outweighs the mushroomer, the crowd sim and the autowalker as my new biggest project... not bad for 3 days coding!

F-curves in the 3D View continued...

Today's progress... 3D Bezier F-curves are now editable in the viewport for unconstrained child bones with inherited rotation and asynchronous/independent curves!  I learnt a lot from studying Crouch's script which has a larger but different feature set and currently works with objects not bones (mine currently works with bones and not objects).  Hopefully my script shown below, will help push the discussion along and go some way towards bringing this feature to blender in a finished state before it hits maya!



The script is available here but be warned its in heavy beta, so back up your animation first!  Feel free to bug report in the comments below.  I've coded an on screen help (which will start when you enter the 3D f-curve edit mode) as the shortcuts had to be unusual to save messing with the keymaps.

UPDATED: http://www.pasteall.org/19807/python

There's extensive discussion of the autodesk technology preview which sparked all of this here (though its not really a new technology at all as apparently DigitalFish Reflex had it a few years back).

In other news an animator whose work I've been following recently is Dustin Grella.  If your mind is buzzing as much as mine with all that discussion on 3D fcurves you'll probably find it refreshingly low tech... here's Bag for a Banana Peel.

Editing F-curves in the 3D View

So after feeling really proud of myself for getting asynchronous f-curves represented in the 3D view in blender with live updating, I read of Crouch's concurrent efforts to bring the same functionality to blender and realised I'd been beaten to it!  Crouch's script only works with objects and has a very cool visualisation system for speed and acceleration as well as the 'time beads' functionality that maya's technology preview provides.  It doesn't represent asynchronous curves in the same way I've done below, but then maybe I'm going over the top.

Tonight I managed to get the script working with child bones - it has to cache the parent space in advance.  The great thing about open source is that I can borrow and learn from crouch's code.  My work in progress code is linked here but be warned its still quite messy and for interest's sake only!  UPDATE: http://www.pasteall.org/19807/python

"Spiders"

For info on how the auto-walking, crowd simulation and 'scribbler' work see my notes on scribd http://www.scribd.com/doc/47412229/Crowd-Simulation-and-Auto-Walking-Algorithms

A crash test dummy for my work in progress collection of scripts for auto-walking and flock simulation in blender. The crowd simulation system used for these was programmed from the ground up in python (not using blender's built in boids), as was the auto-walker (though I took a glance at the old 2.4x insect walker but decided to go down a different route for more flexibility with different numbers of legs etc.)

I also coded the 'scribbler' and the motion trails in python (thanks to this unofficial port of Python Image Library to Python3 which enabled me to use PIL inside blender to read pixels). Inspiration for the scribbler's algorithm comes in parts from mr doob's harmony​ and zefrank's original scribbler.

The sound is a combination of my own python code (sampling differentiated fcurve data in blender) and audiopaint.

I also had a bit of time to type up some notes on the 'Mushroomer' Algorithm as well based on some work I presented at the blender conference with the rest of the 'tube' project team: http://www.scribd.com/doc/47412460/Mushroomer-for-Blender-2-5-Algorithm-Description

Success at Last

First efforts to combine the data generated by my boids simulator with my auto-walking algorithm were fraught with difficulty.  'Noisy' fcurves had loads of jitter on them due to some of the fuzzy approximation methods used by the boids sim to keep the simulations fast.  Added to this as I was doing much of the calculations for the boids rotations using matrices or quaternions rather than Eulers, rotations would often flip from +/- 360 to 0 and back again.  I wrote a quick 'filter and smooth' script which effectively corrects Euler flips and blurs the data generated by the sim to allow the walking algorithm to produce more regular leg movement.



Here's the first crash test dummy... not particularly exciting but at least it works now!  Running is yet to be implemented and the walking algorithm falls apart if the distance the boid has to move in any given frame step is too large.  Writing sub-frame steps would easily correct this however and would probably be faster than any more intelligent algorithms.

Stay tuned for more demos and hopefully in the near future some source code for you to disect.

Leg Sequencing for the Auto-Walker

I've successfully implemented leg sequencing for the auto-walker I'm working on for the 'tube' project.  Its not a 'strict' method like many other auto-walkers use.  The legs can drift in and out of sequence if the bug turns a corner or so on, but the algorithm will then speed up or slow down each leg to reach the desired leg sequence order.

Hopefully by the end of the week I'll have stable enough source to release it!

Progress on the auto-walker

I spent most of yesterday just hitting dead ends.  My code became far too complex and it was almost impossible to find bugs.  I went for a full re-factor of the walking algorithm and now I'm getting much smoother motion.  It takes a few frames of pre-roll to get the legs in order - notice some jittering in the first second or so as legs are hurriedly placed to stabilise the bug.  By stabilisation I mean that if all the legs on one side of the bug (or in any given 'leg group' set by the user) become lifted then the algorithm will hurriedly place the leg which has been airborne the longest to keep the bug upright.  Still no body motion, but I've made a preliminary start on leg sequencing.

Boid Brain Implementation

After spending two weeks working on LESS THAN THREE, then taking a few days off, I'm now back at work again continuing my efforts to code a new in-house boids system for the tube project in Python.  This will all be open sourced in due course once I've tidied up the code.

I spent a day or so bug-fixing and clearing out some redundant defs, then I implemented a 'flock metrics' system.  Rather than considering the entire collection of boids to be the flock (as the built in boids system does), in my boids system the flock is defined as all boids within a certain radius (the 'sight distance') making computations much faster (and more realistic for ground based creatures).  Each boid needs to know a set of statistics ('flock metrics') about the flock before it can make a decision on what to do (in the 'boid brain').  It needs to know how dense the flock is, how many boids are in the flock, where the centre of the flock is, who its closest neighbour is, and what the rate of change of all of these factors is.

For example if the flock is very dense but it is becoming less dense then the boid can stop panicking.  If the flock is not dense but it is increasing in density the boid may wish to align its direction vector with other boids in the flock so as not to cause collisions.  If the flock becomes severely dispersed a boid may wish to head for the centre of the flock for protection and so on.  This decision making is called 'emotion modulation'.  The boid considers the flock metrics and makes decisions about what to do automatically.  Emotions such as panic can change very quickly when a boid becomes blocked in by its neighbours (this is called ' high volatility' in the boid script) whereas emotions such as centring (heading for the flock centre) may need more gradual adjustments (low volatility) as boids slowly notice they have been left behind.



The next step is to allow the user to control emotions with 'emotion effectors' in the same way the boids can be physically directed with 'direction effectors' (shown in the above video).  The emotions implemented in the above video are panic, sleep, herd (copy neighbours directions/alignment), centre (head for the centre of the flock - traditional boid/shoal/flocking behaviour) and cluster (approach but keep a safe distance from closest neighbour).  The last emotion left to implement is not so much an emotion but a signal to other boids which is repel/attract (a boid can signal other boids to move closer or further away).

As you can see in the above video there is still a slight issue with collision stickiness which needs to be solved.

Fingers crossed I'll be able to start work on the autowalker pretty soon.

Controllers Implemented in Pythonic Boids for Blender

While college was on I hardly had any time at all to work on this.  I found a spare three hours this morning and implemented 'controllers'.  Currently you can instruct the boids to follow or to be attracted/repelled/directed by controllers.  I also implemented a way of detecting the orientation of the ground plane from the cross product of the last two direction vectors (with necessary error checking, direction flipping etc) and replaced all of the orientation interpolation code with quaternion slerp.  This will obviously need some more careful coding to make sure that the boids don't suffer from quaternion flip - hopefully I can get the smooth interpolations of quats with the twistability of eulers (i.e. >360 degree angles).



I've also been messing with low-poly stuff seeing as that seems to be in vogue this year what with 'Between Bears' rocketing up the vimeo charts.  After walking in the woods this evening in low light I was trying to work out how to get the sense of a curved ground plane from only the striation of the tree trunks.  Kept running into anti-aliasing issues though - the opposite type from usual - smooth fades from black to white wasn't quite what I had in mind.

 

Python for Exaggerating and De-intensifying Poses

Sometimes my poses are just too strong and the animation ends up being very exaggerated and hard to follow, and sometimes they're too weak and need amplifying.  Joshua Leung's new push and relax features for pose bones in blender do a great job of handling in betweens and making them more or less similar to the neighbouring keys.  Often I find that before I even get to the inbetweening phase of animation I want to edit my key poses in a similar manner.  Especially with the spine and neck I want to be able to relax the bone's locations and rotations towards the rest pose (the same way I would smooth vertices when I'm modelling).  Here are two more python scripts I threw together this morning while working on another walk cycle...

Smooth Pose


import bpy,mathutils


amount=0.05


a=mathutils.Quaternion((1,0,0,0))
for each_bone in bpy.context.selected_pose_bones:
    each_bone.location=each_bone.location*(1-amount)
    if each_bone.rotation_mode=='QUATERNION':
        each_bone.rotation_quaternion=each_bone.rotation_quaternion*(1-amount)+(amount*a)
    elif each_bone.rotation_mode!='AXIS_ANGLE':
        each_bone.rotation_euler[0]=each_bone.rotation_euler[0]*(1-amount)
        each_bone.rotation_euler[1]=each_bone.rotation_euler[1]*(1-amount)
        each_bone.rotation_euler[2]=each_bone.rotation_euler[2]*(1-amount)

Exaggerate Pose


import bpy,mathutils


amount=0.05


a=mathutils.Quaternion((1,0,0,0))
for each_bone in bpy.context.selected_pose_bones:
    each_bone.location=each_bone.location*(1+amount)
    if each_bone.rotation_mode=='QUATERNION':
        each_bone.rotation_quaternion=each_bone.rotation_quaternion*(1-amount)-(amount*a)
    elif each_bone.rotation_mode!='AXIS_ANGLE':
        each_bone.rotation_euler[0]=each_bone.rotation_euler[0]*(1+amount)
        each_bone.rotation_euler[1]=each_bone.rotation_euler[1]*(1+amount)
        each_bone.rotation_euler[2]=each_bone.rotation_euler[2]*(1+amount)

At the moment the scripts only support quaternions and eulers as that's all I use in my rigs.  If you're an axis angle person please enlighten me as to what on earth its for!  Here's the walk cycle (crits welcome):



On another note I started hacking away at blender's source code last night trying to code smoother deforms for b-bones, which in my opinion look great until you turn on mesh display when it somehow manages to fold up the mesh in all sorts of nasty ways if you're using it for something wide (like a torso) rather than something skinny (like an arm).  While I haven't made much useful progress yet (other than beginning to understand the built in math libraries) I did stumble upon this very nice explanation of matrices, vectors and coordinate systems which I wish I'd found ages ago when I started diving into python in blender.

Quick Walk Cycle and Python to Mirror a Cycle

I started out aiming for John Travolta's Saturday Night Fever walk to staying alive but I quickly ended up with far too many keys and too snappy and exaggerated an action for a cycle.  I realised I needed to change the rig a bit to get the movements I needed so I stripped it right down and worked up this basic walk cycle just to test the changes work.

I also wanted to work with only half a cycle and be able to have a script mirror it for me.  I coded this quick python timesaver.  Basically just select all the bones you have animation on and enter the start and end frames into the script.  Eg. You have a stride on frames 1 to 21 and you want the mirrored stride to be placed on frames 21-41 so you enter start_frame=1 end_frame=21.  Be warned the script will purge any modifiers you have on fcurves (its lazy at the moment) and replace them with a simple cycle modifier so your pair of strides (on 1-41 in the example) continue looping forever.

import bpy

start_frame=1
end_frame=21

def set_frame(sf):
    bpy.context.scene.frame_set(frame=sf)
    bpy.context.active_object.update(scene=bpy.context.scene)
    bpy.context.scene.update()

for each_fcurve in bpy.context.active_object.animation_data.action.fcurves:
    for each_modifier in each_fcurve.modifiers:
        each_fcurve.modifiers.remove(each_modifier)
each_frame=start_frame
while each_frame<=end_frame:
    set_frame(sf=each_frame)
    bpy.ops.pose.copy()
    set_frame(sf=each_frame+(end_frame-start_frame))
    bpy.ops.pose.paste(flipped=True)
    print('copying '+str(each_frame)+' to '+str(bpy.context.scene.frame_current))
    bpy.ops.anim.keyframe_insert_menu(type=-4, confirm_success=False, always_prompt=False)
    if each_frame==end_frame:
        break
    set_frame(sf=each_frame)
    bpy.ops.screen.keyframe_jump(next=True)
    each_frame=bpy.context.scene.frame_current
#kill fcurve mods and add a cycle mod
for each_fcurve in bpy.context.active_object.animation_data.action.fcurves:
    each_fcurve.modifiers.new(type='CYCLES')       

Bone Roll in Blender

It's well known that setting bone roll to Z-axis up in blender often doesn't work well, if at all.  If you find yourself having to set bone roll manually using Ctrl+R then this script might help you out.  Tab into bone edit mode, select the bones you want to roll, copy and paste the below script to a text editor, set your target roll vector as new_z_axis_vector in the script below (at the moment its set for z up) and hit run script.

import bpy,mathutils,math

new_z_axis_vector=mathutils.Vector((0,0,1))

for each_bone in bpy.context.selected_bones:
    new_x_axis=each_bone.y_axis.copy().cross(new_z_axis_vector)
    if new_x_axis.copy().angle(each_bone.x_axis)>90:
        new_x_axis=(-1)*new_x_axis
    previous_z_angle=each_bone.z_axis.angle(new_z_axis_vector)
    roll_adjust=each_bone.x_axis.angle(new_x_axis)
    each_bone.roll+=roll_adjust
    if each_bone.x_axis.angle(new_x_axis)>0.01:
        each_bone.roll-=2*roll_adjust

Of course the correct rigging practice is to make sure the bone roll of all the bones in a chain is such that the z axes vectors of all the bones are in the same plane so you might want to set new_z_axis_vector to something other than (0,0,1) or world z, for example a leg IK chain might need a value of (0,1,0) to make all the bone z-axes point forward.

Getting Renderman Shader Editing on Windows

With renderman exporters for blender 2.5 on the horizon I thought I'd get a head start by learning the renderman shading system.  Sadly on windows its not quite as easy as it should be get a shader editor going.  The original Shaderman runs fine but its a little dated, and I'd previously tried getting shaderman.next to run to no avail.  Instead I tried Sler which like shaderman.next is written in python.  Some of the links to sler's dependencies are  broken so I tried searching around the internet and installed the dependencies as listed on sler's download page. Sadly, according to this blog post on superjazz it seems that many of the recent compiles of gtk are incompatible with recent compiles of python.  If you're getting lots of these errors you've probably hit the same problem I did:

ImportError: DLL load failed: The specified module could not be found.

To get round this you need to install an older version of python.  On top of that you need GTK with glade (which for some reason doesn't always seem to be included) - if you're getting errors about a missing libglade-2.0-0.dll then you've only got plain GTK without the glade.  The exact files which worked for me are:

Sler-v0.2beta
python-2.5.4
gtk-dev-2.12.9-win32
pygtk-2.12.1-3.win32-py2.5
pygobject-2.14.2-2.win32-py2.5
pycairo-1.4.12-2.win32-py2.5
aqsis-setup-1.6.0 - for preview rendering

Just unzip Sler to a clean folder then inside the bin subfolder create a new batch file with something like the following to set your path and start sler:

set PATH=C:\Program Files\GTK2-Runtime\bin;C:\Python25;C:\Program Files\GTK2-Runtime\lib;C:\Program Files\GTK2-Runtime\;C:\Program Files\Aqsis;C:\Program Files\Aqsis\bin

@set GTK2R_PREFIX=G:\Program Files\GTK2-Runtime\bin

@echo set PATH=%GTK2R_PREFIX%;%%PATH%%

@set PATH=%GTK2R_PREFIX%;%PATH%

python runsler.py

pause

Best of luck, let me know if you have any problems.

Boundaries for Python Boids and Using the NLA for Animation Layers

Thanks to Campbell's bug fix for the shrinkwrap constraint, my python coded boids (not blender's built in types) can now stay constrained to a surface, albeit very slowly.  The execution of my python code is pretty rapid, considering the number of adjacencies which need to be taken into account, and I have some tidy ups planned which should make it even faster.  The downside is that I need the code to know the visual location (not actual location) of each boid after shrinkwrapping, and then apply this to the actual location.  Logically if I did object.location=object.matrix_world.copy().translation_part() you would expect this to update the visual location.  However it seems that (somewhat sensibly) blender doesn't always update constraints after every line of python which could make the visual location of an object differ from its actual location.  Trying to tell blender to update the world matrix to reflect changes in the object's location seems to be the main problem, especially when the object has keys (so just setting the current frame won't work).  The best option seems to be to key the location (and oddly enough using object.keyframe_insert('location') doesn't update the matrix, but ops.anim.keyframe_insert(....) does, but is much much slower) then read back the matrix.  This unfortunately is extremely slow, but does seem to work.

UPDATE: documented a workaround inspired by some of Bassam's suggestions.

Some python work which on the other hand has been more successful is a workaround to use NLA strips as animation layers (in the same way they would work in maya).  To use the script key the base animation in and then snowflake the action in the NLA window.  Then add a new NLA strip on a higher layer and set the mode to additive (to do this you'll probably have to create an action with a garbage key which you can later delete, link the action to the object then snowflake it).  Set your new top layer NLA strip to add mode then tab into it (tweak mode as its now called).  Start posing and whenever you want to insert a key use this script http://www.pasteall.org/16791/python instead of the 'I' key.  This makes sure that the new keys are relative to the layer below.  In good news aligorith confirmed that this behaviour will soon be built into blender and you won't need my script for much longer!

Shark Swim Cycle

Tried to get layered animation working in blender but the NLA didn't seem to work the way I wanted it to (like animation layers in maya) so I gave up for the morning and tried to animate a swim cycle.  That didn't go too well either - I'd build the flippers on the model far too low down and they looked unnatural in what should have been their rest pose by their side.  My spline IK rig which was meant to make animating the sine wave easier ended up making it harder as I had far too few bones controlling the spline to get a smooth enough wave.  Added to that the tail could have done with a few more bones to smooth out the motion.  The looping destroys the effect of the fake caustics and volumetrics somewhat, but never mind, I hate default grey (or 'Material.001' as its otherwise known).

Python Boids in 3D

I'd only tested my code out in 2D up till now, even though I'd been coding all the vector maths in a way I hoped would work in 3D.  I ran a simulation above and to my pleasant surprise it worked perfectly first time.  Sadly trying to shrinkwrap/constrain the boids to a surface (a mesh marking the limits of where they are allowed to wander) I ran into a new bug in blender.  Its in the tracker, fingers crossed that by the next time I have a moment to work on the code all will be resolved!