Google’s Summer of Code 2010 comes to an end for me today. It has been a great time working on awesome projects like PyMT and Movid. My task was to enhance PyMT’s text input methods. One of the joys of this task was that it allowed me to work on a relatively wide scope of things. Here’s a brief list of what I worked on:
- I added a new spelling provider to PyMT that abstracts from individual spellchecking libraries. That means you can use your favorite spellchecking library, which is important considering that PyMT is cross-platform.
- I added two actual spelling providers that implement this protocol: One enchant spelling provider (usable after installing enchant) and one provider using OS X’s native AppKit spellcheckers (so you get that out of the box on OS X).
- Mathieu once wrote a basic virtual keyboard with spelling suggestions which I adapted, cleaned and merged.
- PyMT obviously already had some text input widgets, which I improved (e.g. MTTextArea).
- I began working on a version of MTTextInput with added spellchecking (like OO.org with red lines drawn for incorrectly spelled words), but that needs some more love.
- One of the more concrete objectives of my task was a Swype-like keyboard for PyMT. I created a prototype for that, see the video below.
- Another concrete objective was a split keyboard (split into two parts, one half for the left, one for the right hand) that adjusts to your hand’s properties (e.g. size). To achieve this, a substantial amount of changes was needed to our vision tracking application (Movid):
- For the keyboard to adjust to the user’s hands, a handtracking algorithm was needed that I implemented for Movid. It detects the fingertips of the hand as well as the hand’s center. These are just seen as a certain type of ‘blobs’ internally.
- These blobs need to be tracked over a sequence of frames from the camera. Additionally, we also want to find simple touches (without all the hand information). For that, I added and integrated BlobFinder and BlobTracker modules that obey a common format so they’re easily interchangable.
- When your camera senses a blob on the touch surface, the application needs to perform a mapping to get the blob into screen coordinates. We do that using a calibration module, which I had started before SoC. I finished it and merged it back into our master branch.
- As an extra feature, I added a PyMT module that you can use to calibrate your tracker from within your client application, eliminating the need to switch applications. I also added a Flash GUI for the calibration so that you can easily do it on any remote computer via our web interface.
- To actually send the handtracking to the client application, Mathieu added a TUIO2 module to Movid. I started a PyMT input provider for TUIO2. Both of which is work in progress, but I believe we’re the first project to adapt TUIO2 (there’s not even a reference implementation yet).
- The result of that can be seen in the second video below. Also, make sure to read the vimeo description.
- Other than that we now also provide portable binary packages for PyMT 0.5 for both Windows and OSX. I created the OSX package, so it’s no longer a major pain to install. You just download and run it.
- And, of course, many more fixes!
Some of that is already in PyMT 0.5. All of the Movid stuff will be in the first release. In future releases we shall see much improved versions of these prototypes and hopefully even context aware word suggestions.
Here are the two promised videos, if you’re reading this through a planet, please go directly to my blog.
Prototype WipeToType Keyboard for PyMT from Christopher Denter on Vimeo.
Ergonomic multitouch keyboard prototype from Christopher Denter on Vimeo.
Thanks to all the people who made this possible. Thanks Google, Christian, Pawel, Mathieu and Thomas, for being (a) fantastic mentor(s). It has been a great pleasure and privilege to work with you in GSoC 2010 and I sure will continue to work on both projects.