Hi people, Researchers at the User Interface Design Group, MIT Computer Science and Artificial Intelligence Lab (CSAIL) have developed a system for scripting applications by incorporating screenshots. Check out this press release: < http://web.mit.edu/newsoffice/2010/screen-shots-0120.html>, and this story from Network World: < http://www.networkworld.com/news/2010/012110-sikuli-scripting-language.html
.
From the project website:
Sikuli is a visual technology to search and automate graphical user
interfaces (GUI) using images (screenshots). The first release of Sikuli contains Sikuli Script, a visual scripting API for Jython, and Sikuli IDE, an integrated development environment for writing visual scripts with screenshots easily. Sikuli Script automates anything you see on the screen without internal API's support. You can programmatically control a web page, a desktop application running on Windows/Linux/Mac OS X, or even an iphone application running in an emulator.
You can watch a demo, download the code, and find more information at < http://groups.csail.mit.edu/uid/sikuli/>. And you can find the original and forthcoming papers here: < http://uid.csail.mit.edu/projects/sikuli/sikuli-uist2009.pdf> and < http://groups.csail.mit.edu/uid/projects/sikuli/sikuli-chi2010.pdf>. Joseph.