Since implementing the 'Scribbler' for my earlier
'spiders' demo video I've started to think more and more about how I draw. I wanted to see if I'd be able to write an algorithm which would draw as efficiently as humans do. Firstly I adapted the scribbler to only draw with one long line and combined this with my optical flow distortion code, again using OpenCV for optical flow analysis and Blender as my drawing interface. The scribbler would need to know where it had been recently so it didn't draw over those parts again and instead headed off to complete the rest of the image. Given a selection of random near by points to scribble over to next it would need to decide which one was best based on how dark the point needed to be, how close the point was, and if it had been there before, and how many times.
Anyway this was the first result... a bit busy, not very accurate, and not much economy of line. Very clearly not drawn by a human, or at least not by a sober one.
Next I decided to pre-process the images in Photoshop with an edge-detect filter. I was going to write my own fast difference of gaussians (or even simpler a difference between neighbours) but I wanted to check the principle worked first.
Straight away much more economical with the lines, and to some extent more accurate. Since edges or lines essentially have direction and fills don't (ie. you can shade a fill by moving the pen in any direction you want until you reach a border but you can only run a pen along a line in one of two directions) I could use some code to make the line prefer to not change direction and keep heading straight ahead. This saves some of the over and back-ness of the free scribbling algorithm but isn't perfect. My next step is to get the scribbler to intelligently work out which way the edge (or line) that it is currently drawing is curving, and to anticipate where to look next. I think that when we draw we tend to look ahead a fair bit and queue up drawing instructions so our eyes are always a few centimetres ahead of our pen. If our eyes get lost and have to search then our pen pauses for a moment.
Next I compared the two computer drawing algorithms with my own human 'blind' rotoscope.
The video is here if you want to see it.
Despite my algorithm's improvements it still seems like it misses important features in the image and doesn't prioritise what to draw. I think that rather than looking for the darkest areas or the sharpest edges it probably needs to find a few high contrast points in the image and link them via as many different edges as are available. I then built a homemade gaze tracker with my bike helmet, a webcam, AfterEffects, an OpenOffice Spreadsheet, Python and Blender.
The calibration here isn't perfect but this next video demonstrates an important fact; that the eye makes big jumps across the image and doesn't smoothly follow any lines, unless they are moving. It also doesn't scan around anywhere near as much as I was expecting. It avoids the edges of the image - I assume because these can be seen in periphery anyway and the film-maker probably hasn't put anything of interest there. I'm also quite amazed by how much the moving street scene causes the gaze to get pushed out the right edge of the screen - the scribbler also suffered from that quite a bit on the moving images. When the gaze goes out of the film window that's usually a blink, not a calibration problem.
Well that's by some stretch the longest post yet and almost makes up for a quiet month on the blog. Now for some maths...
