Drawing in Space and Her Favourite Positions

'Drawing in Space' is an experiment I've wanted to try for a while: using OpenCV and a webcam to draw in 3D.  Essentially you wave a coloured square of card around in front of the webcam and my code converts the path your hand takes into a 3D Drawing.  First it filters the image based on a colour you have chosen, selecting all pixels within a colour range determined by the mean and standard deviation of the patch chosen in the calibration phase.  It then finds contours in the filtered image, simplifies these reducing the number of edges, checks if any contours are square (concave, under a certain contained area, and with all internal angles within a given tolerance).  From all the squares found it then chooses which is the most likely square to follow on from the last based on the sum of the square of the magnitude of the delta in each point's position between frames (trying different permutations of the order of corners to account for rotations and mirroring).  If it can't find the square it motion tracks the corners from the last frame.  If any of these are missing it interpolates the data, and if all of them are missing it tracks strong features in the last image (in full colour, unfiltered) and uses a simple nearest neighbour algorithm to associate each corner of the last square with a nearby tracking point (if it was clever it would use some kind of error detection here to eliminate anomalies and then a delaunay triangularisation to interpolate points, but it needs to be lean to run at all).  If all of this fails, the script uses a third order differential on the last few points to extrapolate the next position, filling in the gaps for a few frames until data resumes.   The script also adjusts the colour it is searching for frame by frame to account for changes in light conditions.  Optionally you can also use the chessboard tracking instead of a coloured square.

The drawings are displayed in anaglyph as you draw, and can be exported as OBJ files.  I need some options to adjust the convergence point and so on, but its mostly there now.  A beta version of the code is online here (right click, save target as).  You need matching python 2.7, numpy 1.6.1, and opencv-python 2.4.0 compiled against each other.  If you're running Windows you can get numpy and opencv packages here.




'Her Favourite Positions' is an animation of 186 Portraits and 37 Full Body Poses from my Life Drawing classes at the RCA.  I matched them up frame by frame in AfterFX, and did a little warp correction here and there to get the proportions generally accurate.  I'm quite amazed that this even worked at all and that you can see a ghostly generic photofit face through the mess of lines.

For Your Spare Time...

Since I'm currently writing this on a laptop that was manufactured no more recently than 2001, I'm unable to tidy up the films I made this term into an uploadable state, and won't be able to until I get in front of a computer that's less than 6 years old, which might be not for a week or so.  In the meantime if you need some alternative programming, which leaves you asking 'good God, what program did she use to make those?', and 'where can I find a tutorial on how to produce something with that kind of cutting edge graphic quality?', I can strongly recommend Stuff & Nonsense, by my sometime housemate and collaborator Julz.  Its rad.  Interrobang using the ask box.

Stop Motion and Memory Drawings

 Two of my daily memory drawings for the drawing elective I'm doing at the RCA.

Thursday, 11:30pm. Two deer caught in my torchlight on my cycle home.

Wednesday, 12am. Occupied bus shelter.

And some work in progress building stop motion ball joints (with drilled out brass flats and some ball headed bolts), and a twisted wire, foam and latex fox.  Used a blowtorch and a thread tap for the first time this weekend.  Exciting stuff.

Re-boarding the Crows

So I'm re-boarding the Crows for what I think is the 8th time in its exceedingly long history.  I think I'm nearing final... only 28 more boards to work on today then this revision will be ready for comment again.  Huge thanks to Julz, Marie and Kimmo who have been untiring in their suggestions and criticisms so far.  Character model sheets are also almost complete.

I also found a great review of David O'Reilly's External World here which I thought I should share.

The Computer Drawing Experiment...

Since implementing the 'Scribbler' for my earlier 'spiders' demo video I've started to think more and more about how I draw.  I wanted to see if I'd be able to write an algorithm which would draw as efficiently as humans do.  Firstly I adapted the scribbler to only draw with one long line and combined this with my optical flow distortion code, again using OpenCV for optical flow analysis and Blender as my drawing interface.  The scribbler would need to know where it had been recently so it didn't draw over those parts again and instead headed off to complete the rest of the image.  Given a selection of random near by points to scribble over to next it would need to decide which one was best based on how dark the point needed to be, how close the point was, and if it had been there before, and how many times.

Anyway this was the first result... a bit busy, not very accurate, and not much economy of line.  Very clearly not drawn by a human, or at least not by a sober one.



Next I decided to pre-process the images in Photoshop with an edge-detect filter.  I was going to write my own fast difference of gaussians (or even simpler a difference between neighbours) but I wanted to check the principle worked first.



Straight away much more economical with the lines, and to some extent more accurate.  Since edges or lines essentially have direction and fills don't (ie. you can shade a fill by moving the pen in any direction you want until you reach a border but you can only run a pen along a line in one of two directions) I could use some code to make the line prefer to not change direction and keep heading straight ahead.  This saves some of the over and back-ness of the free scribbling algorithm but isn't perfect.  My next step is to get the scribbler to intelligently work out which way the edge (or line) that it is currently drawing is curving, and to anticipate where to look next.  I think that when we draw we tend to look ahead a fair bit and queue up drawing instructions so our eyes are always a few centimetres ahead of our pen.  If our eyes get lost and have to search then our pen pauses for a moment.

Next I compared the two computer drawing algorithms with my own human 'blind' rotoscope.  The video is here if you want to see it.

Despite my algorithm's improvements it still seems like it misses important features in the image and doesn't prioritise what to draw.  I think that rather than looking for the darkest areas or the sharpest edges it probably needs to find a few high contrast points in the image and link them via as many different edges as are available.  I then built a homemade gaze tracker with my bike helmet, a webcam, AfterEffects, an OpenOffice Spreadsheet, Python and Blender.

The calibration here isn't perfect but this next video demonstrates an important fact; that the eye makes big jumps across the image and doesn't smoothly follow any lines, unless they are moving.  It also doesn't scan around anywhere near as much as I was expecting.  It avoids the edges of the image - I assume because these can be seen in periphery anyway and the film-maker probably hasn't put anything of interest there.  I'm also quite amazed by how much the moving street scene causes the gaze to get pushed out the right edge of the screen - the scribbler also suffered from that quite a bit on the moving images.  When the gaze goes out of the film window that's usually a blink, not a calibration problem.

Well that's by some stretch the longest post yet and almost makes up for a quiet month on the blog.  Now for some maths...

Optical Flow and Rotoscopy

Been feeling pretty ill the last few days. Managed to drag myself to the Broadcast Video Expo yesterday at Earls Court. Would love to go again tomorrow to watch some of Arri's lighting seminars, and the DaVinci Resolve colour grading workshops, but probably won't be feeling good enough yet so it'll be another day at home storyboarding and tidying up odds and ends.



Last week I rotoscoped over the intro to 'Down by Law' with one long continuous virtual line using blender. At college Felipe suggested I could do a bit more with the data and maybe find some way of using the video to distort the drawings. I wrote a python script using the OpenCV bindings for Python 2.7 to analyse the optical flow of the original film footage. I based the script on the lkdemo using CalcOpticalFlowPyrLK. The script dumped out a load of text files containing the feature tracking info.

In blender I wrote a second script which warped my 'drawing' (made up of vertices and edges) on a frame by frame basis using the nearest tracked feature point to drag the line around. Effectively the script puts the line's control points (vertices) into voronoi cells which are shifted by the tracked features (like a virtual earthquake). I added in very simple outlier detection checking consistency with the flow of neighbouring points along the rotoscoped line. I had wanted to convert the point cloud of feature track points to a mesh using delaunay triangulation, and then use the animated mesh to deform the string of vertices, but it would probably have been excessively slow in python!

Anyways thanks to Felipe for pushing me that bit further. Meanwhile progress on the scribbler continues - I've added in three more drawing styles (two of them shown below) and a depth of field control. Next step is to work on edge detection (I've got down on paperware how it should work) and adding a bit of stability to make sure no essential parts of an image get left undrawn on any one frame.

For The Crows, all over again

Nearly two years ago I started a project which I still wonder if I'll ever finish.  Whenever I'm on a long train journey I start to rework it.  I've started to formalise the storyboard at last.

FULL WIP STORYBOARD

Crits are always welcome.