How to Operate a Computer without Using Your HandsBy Wil Forbis
September 15, 2010 (updates are noted with date)
IntroductionThere are various reasons why a person would want to operate computer without using their hands. The most likely is that one or both of their hands is damaged (e.g. carpal tunnel, RSI (repetitive strain injury), paralysis) and they have to temporarily or permanently restrict the interaction of their hands with the computer. Or they may want to increase their workplace efficiency by performing many of their normal computing functions in a hands-free manner, leaving their hands available to do other things like dial phone numbers, page through documents or juggle kittens. I've spent several years exploring the various options for hands-free computing and decided to write this article as a way of offering my thoughts on the subject.
My story in short: after years of computing (mousing and typing), playing guitar and general hand abuse, I developed severe repetitive strain in both my forearms. This debilitated me to such a degree that I couldn't type or use a mouse for any meaningful period. As a result I had to investigate ways to interact with the computer that did not require hands. The primary solutions I discovered were:
I'll discuss each of these in more detail below. But first, a sober warning. You cannot replicate every aspect of hands powered computing with hands-free solutions. (Theoretically, you can, but the experience would be maddening.) With hands powered computing, you have two standard tools for input: the keyboard and the mouse. The keyboard is primarily utilized to type in text and secondarily to interact with programs by using keyboard shortcuts or using keystrokes to navigate up and down menus. The mouse is, of course, used to click on various sections of the graphical user-interface. Many programs will claim to be 100% keyboard controllable but, while this may be technically true, the experience of being limited to the keyboard is frustrating. The solutions I use are acceptable for "text heavy" computing --- word processing , writing e-mail, using spreadsheets, HTML and web programming etc. (If you are a hard-core programmer, I think it is feasible that you could do your job entirely hands-free, though you would definitely take some hits to your productivity.) I find my solutions adequate for web browsing as well. These solutions are less acceptable for mouse intensive computing like using Photoshop, music composition software like Digital Performer or Garage Band, or animation tools like Flash Developer (or whatever it's called these days.)
Voice Dictation Software
Most people have an understanding of the basics of voice dictation software. You speak (into a microphone) and the words you say appear on the screen. You can also use your voice to issue commands that open up particular files or programs, or even to click a particular point on the screen.
The voice dictation software I use is, without a doubt, the industry leader: Dragon NaturallySpeaking produced by Nuance. There are different products, but most people can get by with the two lower end options, now called "Home" and "Premium." Home is about about a hundred bucks, Premium twice that. You can buy them online, or at your standard Best Buy type store. At the time of this writing, Nuance is about to release version 11. I'm currently using version 9.5, (Update May 9, 2011: I'm now using version 10 and generally consider it an improvement. I've heard strongly negative things about the current Dragon release, version 11.) because I have zero confidence in any software company's ability to adequately test its product before they release it. (Plus, as you will read, I'm pretty happy with what I've got.) I use the version for Windows, but they do have options for the Mac, which I cannot claim to have any experience with.
There are other voice dictation software tools that can be found on the web, though I know little about them. Both the current versions of Windows and the Mac OS have some kind of built-in voice dictation functionality which I presume to be underwhelming, though I'm open to being convinced otherwise. But, even a cursory examination of the available products will make clear that Dragon NaturallySpeaking is your best bet.
I use NaturallySpeaking for three types of hands-free functions: typing, running applications, and navigating the file system and the Web. I provide some detail on each of these below.
This is really what voice dictation was designed for. I do most of my writing in Notepad and Dragon does a pretty good job of translating what I say to words on the screen. When you first install the software, you go through a training phase where it acclimates to your voice. I find that even if you skip the training (not that I recommend it), you get pretty good recognition right out of the gate. So, if I say "I would like to order a Thai iced tea," that's what shows up on the screen.
You can also use voice commands to navigate through your text. If I want to insert a word 10 words back, I will say "move back 10 words" and the cursor will do so. If I want to move up six lines, I say "move up six." If I want to move down five paragraphs, I say "move down five paragraphs." I also use a lot of the standard search functionality to navigate through documents. If I know the word "Aardvark" is near a section I want to edit, I will launch the Find tool by speaking the keyboard shortcut and then run the search, as in "Press Ctrl F, Aardvark, Click Find, Next." (More details on the "Press" and "Click" commands below.)
The program doesn't always flawlessly transcribe what I dictate. When I spoke the word "Aardvark" above, it translated it as "artwork." Nonetheless, accuracy is very high and when Dragon makes a mistake, you have the opportunity to train it to better understand what you're saying. You can also create custom commands for frequently used text. For instance, you can, create a command called "My Address" and every time you use it it will print out something like...
3452 Smith street
At this point in my usage, I believe that voice dictation "typing" is actually superior to regular typing. (This varies by individual, of course. If you were an incredible typist before starting to use voice dictation, you may never get up to your original speed. I was always a pretty crappy typist.) One obvious advantage is that as long as Dragon "hears" the right word, it's going to spell it correctly. (Consequently, my spelling has become much worse.) I also find that the process of writing with voice dictation forces me to think through what I'm saying in such a way that my writing is clearer and more precise.
Hands-Free Application Use
If using computers was only about producing text, voice dictation would make the process blissfully simple. Of course, there's more to it than that. Some of our time with computers is spent opening up applications like Word, Excel, Firefox, Photoshop etc. For the most part, we don't use typed text to perform these operations, though one can use keyboard shortcuts. Once the programs are open, we spend time trolling through pulldown menus and submenus and clicking various icons. How does one use voice dictation in these scenarios? Well, it ain't always easy. It is true that anything you can do with the keyboard, you can do with voice dictation. If you're not a "power user" you may be unaware of how much of the functionality of a computer program can be accessed using keyboard shortcuts like "CTRL-P" (to print) or "ALT-F"* (to open the File menu in many Windows programs). Dragon has a "Press" command that can be used to access this functionality. So, to print a webpage I'm looking at, I say "Press Control P" and it launches the print dialog. When I first started using voice dictation, I printed out all the keyboard shortcuts for a program I use quite often, the web design tool Dreamweaver, and taped them to my computer screen. It wasn't always pleasant, but I was able to do quite a bit of web work with no hands.
* To be completely clear aboutwhat I'm describing here, "CTRL-P" means hold down the Ctrl key while simultaneously pressing the P key. "ALT-F" means hold down the Alt key while simultaneously pressing the, you got it, F key.
You can, of course, also use the Tab, Enter, Shift, Delete, Backspace, up and down arrow keys and any similar keys with the "Press" command.
Many programs, however, have functionality accessible only by clicking a button or icon, not by using a keyboard shortcut. This is shoddy software interface design, and any programmers related to the product should be flailed, but it's an imperfect world. For these situations, Dragon has a "Click" command. This command has the effect of a mouse click on whatever icon or button you name. For example, if I'm looking at the main screen in Outlook and have an e-mail selected and then say "Click Reply," a new window with a "reply" e-mail pops up. After I dictate my reply, I can then say "Click Send" which has the effect of clicking the Send button on the e-mail toolbar.
There are a couple problems with the "Click" command. It tends to work reliably in standard Windows applications like Word or Outlook, less so in third-party apps like Photoshop or Dreamweaver. Secondly, some applications will have buttons or icons that don't actually have a name and you can't say "click the funny icon in the left-hand corner that has a ball surrounded with what appears to be winged lizards with halos."
What do you do when the "Click" command has failed you? At this point we turn to the mouse grid. Before describing how it works, I will refer you to this link which has a quick animation demonstrating the functionality. Basically, saying "mouse grid" overlays a 3 x 3 grid onto the screen. Each grid square has a number, and saying that number redraws a smaller 3 x 3 grid in just that square. You can repeat this process an infinite number of times, making the square smaller and smaller and focusing in on your target. It sounds pretty inefficient, but in reality isn't that bad. You quickly learned to mentally map out the screen to specific grid of coordinates so you can look at a point and say "mouse grid 348" and your mouse will move to that point. Having said that, there's no way you could play a video game or use a mouse intensive application like Photoshop with this method.
There are other programs offering mouse grid functionality, like Voice Finger, reviewed here. I have never tried it. (Update Oct 9th 2010. I have taken a closer look at Voice Finger. Currently it only works with the Windows Speech Recognition built into the Vista or Windows 7 operating system (not Dragon) but looks like the most functional mouse grid tool I've seen.) Additionally, Ergoarchitect has a very interesting looking product called ShowNumbers that identifies anything that can be construed as clickable on the screen and overlays a number on top of that space. You then click the control by saying something like, "click 52." Price is $32, though there's a version that utilizes a physical footpad at around $120. (update: October 5, 2010. Since first writing this article, I have purchased ShowNumbers and am quite happy with it. I strongly recommend it for increased efficiency over simply using the mouse grid.)
I use the term "navigation" to refer to the process of finding and opening a file or application on your computer, or navigating the Internet in a browser. In this area, Dragon is certainly not as efficient as using your hands, but it's functional. Being that we can do most of this navigation using keyboard shortcuts, it's easy enough to apply voice dictation. It's also easy enough to interface with the operating system to navigate. If you were using your hands to open a specific file you would probably go to the start menu, go to My Computer, drill down through the file structure to find the folder with your file, open your file. This process can be translated to speech as "click start, my computer, local disk C, press enter, Dell (presuming you have a folder called "Dell"), press enter, service guide.pdf, press enter." (The end result of this being the opening of a likely unhelpful Dell service guide.)
Let's say you wanted to open up Internet Explorer and go to a quality website like acidlogic.com. The full speech command would be, "click start, Internet (wait for Internet Explorer to launch), go to address (this is a specific Dragon command which puts focus on the address bar of the browser), acid no space (to run words together you have to use the "no space" command) logic.com, press enter.)
Of course, some intelligent organization of your computer (for example, putting shortcuts to often used files on your desktop) can cut down your navigation time tremendously.
Like any software program, Dragon has its flaws. A user of the program has a collection of files called a user profile which contains their custom commands and information Dragon has learned about that user's speaking style. For whatever reason, this profile often gets corrupted on my machine, requiring a reinstall of the software. Fortunately, I have an uncorrupted backup profile which has most of my information. (update May 9, 2011: as mentioned above, I've switched to version 10, and that has thus far eliminated my user profile corruption issues.) Dragon NaturallySpeaking is also something of a resource hog and will sometimes become unstable if it has to share RAM with a lot of other heavy hitting programs. (This will vary system to system, of course.)
One can't underestimate the value of a good microphone when using Dragon (or any voice dictation software.) I use the KnowBrainer Hands-Free Headset Microphone which is of good quality and is quite affordable ($50.)
It's worth noting that there are a number of third-party add-on tools that can be used in conjunction with Dragon NaturallySpeaking. The main advantage of these tools is that they provide additional commands not available with out-of-the-box Dragon. Probably the best known is the KnowBrainer product. I have to confess that I haven't purchased it myself, but it looks very interesting and promising. I will also say that the KnowBrainer speech recognition forums are fantastic --- possibly the best Internet forum I've ever used for any subject. I've gone in there with several baffling problems, and all of them have been courteously and graciously solved by forum members.
There's also a tool called Vocola which appears to be something of an add-on language framework for Dragon. Haven't tried it myself, but it is positively reviewed. (Update: October 9, 2010. I have installed Vocola and have been playing with it. Very useful for automating a lot of recurring navigation tasks. I definitely recommend it. Installing vocola involves installing several different programs; there's a downloadable package at the ergoarchitect site that walks you through the full install.)
Update: October 5, 2010. I've just stumbled across a product called Utter Command which looks comparable to KnowBrainer and Vocola. I haven't used it, but it looks worth exploring. Cost is in the $300+ range.
Update: November 1, 2010. Here's an additional product which can work with any version of Dragon. It's similar to other products listed in this section in that it expands the range of available commands. Pricing varies depending on certain packages, but a couple hundred bucks will get you going. My usual caveat: I haven't actually tried it. Voice Power Ultimate.
Update: April 7, 2011. Tasti appears to be a cheaper, less functional version of other voice dictation programs. It focuses on specific tasks such as searching the Web and playing music. It does allow for creation of customizable commands. It may be worth investigating.
One final note about voice dictation. Just as you can overuse your hands while mousing and typing, you can overuse your voice while dictating. I haven't had any problems but some people have, including this fellow. It's something to keep in mind.
The bulk of my hands-free computing has been using voice dictation. However, as explained above, using voice dictation to navigate around and click buttons is cumbersome. I ended up buying a mouse controllable by foot to have an easier time mousing. The version I have --- the FootTime mouse available from bilila.com ($149) --- is a mouse slipper you wear on your right foot, accompanied by a plastic box with buttons. To move the mouse on the screen, you drag your mouse slipper across a giant mouse pad under your feet. To do mouse clicks, you click the buttons on the box with your left foot, a little bit like tapping the brake while driving. (You can also do mouse clicks with voice dictation, of course.)
I don't entirely agree with the design of the product which requires that you lift your leg up slightly to move the mouse on the pad. I think a better design would be to essentially take a rollerball mouse, flip it up side down, then attach a slipper to that, and your leg could roll the slipper around without having to fight the effects of gravity.
The FootTime mouse itself is a decent enough product. It connects by USB port to the computer. I've had a couple of the lasers in the mouse slipper die, but I was living in Los Angeles where the product is manufactured, so I just drove over to the company office, and the main guy (whose name I've forgotten) gave me a new one.
Obviously the idea of a foot mouse has certain flaws. Your foot just doesn't have the fine motor control of your fingers, so you can't do the kind of delicate mousing operations sometimes needed while doing graphic design in Photoshop. But it's adequate for most situations, and faster than using voice dictation to navigate around the screen.
There is one big drawback to a foot mouse. When I first got it and started using it extensively, I very quickly got muscle strain in my leg, much more painful than any of the pain I had experienced in my hands. I had to lay off it for several days, and even now I am careful about its use. But it can be helpful.
Other Hands-Free Mice
There is another foot mouse I discovered while researching this article called the No Hands Mouse. It's not a slipper, but a foot pad which allows you to direct the mouse by pushing in various directions. More information at www.footmouse.com. It's worth watching this demo video of the product if only to hear the appalling "No Hands Mouse" theme song.(This discussion at speechcomputing.com has discussion on the NoHandsMouse.)
Ergoarchitect.com has the StepAssist Foot Pedal which duplicates the "mouse click" functionality of a mouse.
Of course, a hands-free mouse doesn't have to involve the feet. There are other mousing devices using the eyes, the chin, and various parts of the body. Their efficiency is clearly not comparable to using a hand mouse, but might be worth investigating depending on the degree of your disability. I have not tried any of these myself, and cannot vouch for their quality. I merely offer them as areas of further exploration.
The Hands-free touchpad from www.handsfreetouchpad.com is a "chin powered" product which looks rather dubious to me, and destined to create neck issues. I leave you to make your own judgment. Demo video here.
The Smartnav mouse is a tool which allows you to "steer" the mouse using small head movements, or, I believe, movements from any part of your body. I don't think I would recommend this for someone with RSI, but it might be useful for a person suffering from more debilitating problems such as arm or body paralysis. Demo videos here.
(This discussion at speechcomputing.com has discussion on the Smartnav.)
Update October 30, 2010. The Magic Trackpad --- reviewed at Wired --- essentially provides two large trackpads that can be placed at opposite ends of a keyboard and manipulated by hand. Obviously it's not "hands-free" but might be of interest to some people. I expect a PC version will be coming around shortly.
Update November 10, 2010. I just stumbled across the DataHand. It's a complete re-visualization of how keyboard can be operated. It's certainly not hands-free, but is worth a look.
Update November 23 2010. Here's an interesting article about the development of a voice powered mouse. To my knowledge, it's not currently on the market, but it seems likely it will be soon. Probably most useful to people with disabilities more severe than RSI.
Update: November 23, 2010. File under "Questionable." Microsoft patents technology to use "electromagnetic waves to detect the motion of the feet" for foot computer interface.Update March 28, 2011. Smartfish Technologies has two interesting looking products. One is the Whirl Mouse which moves on a joint axis and keeps the wrist in motion. The other is a "intelligent" ergonomic keyboard called the Engage Keyboard. I'm tempted to get the Whirl Mouse, and if I do, I will report back here. In general I recommend you peruse their site.
The Future of Hands-Free Computing
In a perfect world, no physical activity would be required for us to interact with the computer; the computer would simply and flawlessly read our thoughts. Well, that world is a ways off, but quite a bit of progress has been made in brain powered computing. As you may be aware, the human brain fires off electrical signals with every activity. The electrical signals transmitted when picking up a cup of coffee are different than those transmitted when petting your dog. By tracking the signals using a "brain helmet" (similar to what anyone who's ever had an EEG test has worn) software can order a computer or robot to perform different actions.
More info at these links.
Brain powered gadgets are here (January 2011)Technique for Letting Brain Talk to Computers Now Tunes in Speech (April 7, 2011)
I'm interested in any information anyone has in relation to the topic of hands-free computing, particularly as it relates to issues like repetitive strain and carpal tunnel syndrome. The best way to pass it on is to leave a comment on this post on my blog related to this very topic. You can also e-mail me at my "public" e-mail address, acidlogic at hotmail.com but note that I don't check it very often since it's a magnet for spam (since it is my public e-mail.)