Of Physical and Virtual Mobile Keyboards

Nearly three years ago, I watched as Steve Jobs presented a slide of images of existing smartphone physical keyboards, from the Blackberry to the Treo, and then reasoned afterwards that a touch keyboard held the advantage of being adaptable to any situation. This reduces clutter, saves space, and allows for a larger screen without the need to add physical bulk (even if it's just millimeters) required to produce a slide-out keyboard.

Yet, a few years onward, there are people who still insist that a hardware keyboard is an advantage to a soft keyboard like the touch keyboard.

Tiny Plastic QWERTY buttons

I'm no stranger to the tiny plastic buttons comprising the QWERTY keyboards on phones. I expressed real interest in a Blackberry or Treo in 2005, and I composed emails on my father's Blackberry Curve throughout 2008, one of them being 476 words / 1941 characters long without spaces. These keys hurt my thumbs during any extended typing, and this is speaking as someone with relatively slender fingers compared to the average person.

The iPhone keyboard, on the other hand, has never given my fingers any pain or strain, even with extended typing sessions. I don't have to press down hard on each little plastic button far smaller than size of a thumb or fingertip. Now, how hard you have to press down on a key varies from physical keyboard to physical keyboard on a phone, but in order to maintain slim form factors, they typically won't be anywhere near as easy to press as a keyboard on the desktop (where physical keyboards do make absolute sense).

Large Touch Key Sizes and Dynamically Resizing Landing Areas for Keys

Besides, the area for a touch key for an iPhone, at least, is larger to begin with. And beyond that, apparently the iPhone's touch keyboard guesses what the next likely characters are, and enlarges the invisible touchable area beneath the visible key to give the user a more forgiving margin of error for that key. For example, if I'm typing "G", the letter "I" will have a slightly larger tappable area because it is statistically likely to follow, whereas the letter "Q" will not.

Auto-Correction

In fact, the guesswork itself of the next probable character is nothing new, and has existed in the form of predictive text (such as T9). I could press 7(pqrs) 3(def) 2(abc) 3(def) to get "r-e-a-d" instead of pressing 777(prRs) 33(dEf) 2(Abc) 3(Def) with the delays between each numeric pad key. This was a form of auto-correction, which also wasn't new when the iPhone came onto the scene, but it's easy for people to forget the important role auto-correction plays when it comes to touch keyboards. This isn't your typical shopping mall kiosk where the touch keyboard is just a literal software port of the physical keyboard.



It saves you time by automating capitalization, word completion, and punctuation. This all factors in when talking about the speed aspect of the keyboard.

Insertion Points, Selection, and Cut/Copy/Paste

But it doesn't end there. If a user wants to insert a text cursor in a random location of a long line or paragraph, and select and cut/copy and paste a portion of the content, the touch-based selection holds a huge margin in ease and speed versus a D-pad or trackball. On a Blackberry Curve with a trackball, this requires moving the cursor incrementally character by character, and line by line. The physical setup on current phones doesn't allow you to quickly jump across huge blocks of text with a tap, and or maintain a simple selection above and below the fold of the current viewport.

The touch-based setup also allows the developers of the operating system to write and add aides that assist with text selection. For example, in the case of the iPhone, tapping and holding down on an area of text places the cursor there and displays a magnifying glass over the selection. This is immensely helpful with accuracy at high speeds of movement across lines of text, and more so with small text.

Foreign Languages and Special Characters

With physical keyboards, for the most part, what you see is what you get. If you want special characters, you had better hope they're hidden inside a symbols modifier key, but otherwise, you're out of luck. The iPhone's touch keyboard appears to also have special characters behind a numbers/symbol button on first glance, but you can additionally hold down particular letters to place special characters without ever leaving the QWERTY view. For example, you can access "ó" by holding down the letter "o", "¿" by holding down the "?" key, or "€" by holding down the "$" key. The list goes on, and this is only for the U.S. English keyboard. Each of the international keyboards has its own version of this too, which brings me to the other special character advantage - foreign languages.



I've just illustrated how quickly accented letters could be added, which comes useful for romantic languages. But what about non-Latin-based languages? What about, say, Traditional Chinese?

This is where essentially all physical phone keyboards fall short entirely. Again using the iPhone as an example, you can use international keyboards, exclusively, in dual-mode with another, or in even higher multiples. In my case, I have the U.S. English keyboard in dual mode with the Traditional Chinese keyboard, which uses gestures to allow the user to literally write out the entire character on a virtual pad. The same guides apply - there's its own form of auto-correction with its guesswork of what character you're about to write. It can usually guess the character before you've fingered all the strokes, and it uses the best guess by default unless you tell it otherwise. The predictive text is also there, as it guesses the next likely character to follow the one you've just input.

You could even write your strikes in a relative mess, or even in simplified Chinese, and it would still figure it out. And the appropriate punctuation and everything else you'd want is waiting there right where you need them.

Adaptive and Future-Proof

I can easily concede that the lack of tactile feedback on a touch keyboard like the iPhone's is no minor drawback, and it has appeared that manufacturers have attempted various ways to compensate, ranging from the iPhone's audio click cue to the Blackberry Storm's click screen (though a bit confusing with the flat/depressed split mode) to the Android phones' vibrational tap feedback. From what I have seen in the patents, such as the pop-out surface bumps, the progress of the research looks promising, and I believe that it's simply a matter of time for tactile feedback to be perfected on a touch key.

Yet, even when considering all these features, the characteristic of virtual keyboards that is most appealing is how future-proof it is. Not only does the touch keyboard adapt to any situation, but when there comes a need for something new in the future, perhaps a new currency symbol or an entirely new method of input, it's ultimately just a software update away with the phone you already have.



* Footnote: There are phones that sport physical keyboards with touch screens, though typically a touch keyboard has either been nonexistent or hacked in, or tossed in as a poorly executed afterthought.

In-Browser Search Engine Switching

When I first used Firefox under the Firebird name, one of my favorite features was the ability to quickly add and switch search plugins for other sites. In the case of Firefox, you could type one query, and any other search engine or site search was just a click away. Or for keyboard shortcut aficionados, ctrl+k/cmd+k > ctrl/cmd + up/down > enter.

Safari didn't offer this feature, but years back, I discovered a third party Safari plugin called Inquisitor, at the time the work of an independent developer. Among the features it offered, it also allowed users to also add and switch between search engines with a single query.

But what I loved most was how easy he made it to add search plugins. You see, for Firefox, I wrote several search plugins starting at the end of 2004 and beginning of 2005, using the Sherlock format. Some of these (Yahoo! Movies, Yahoo! Widgets) have been replaced by OpenSearch versions uploaded by other people, but some of the early ones remain in case you want to see what I'm talking about (Cal Berkeley plugin from February 15, 2005).

With Inquisitor, on the other hand, we could simply use a variable representing the query within the URL parameter used in any given site search. For example, if I searched IMDB for "Memento", the URL ended up looking like this: "http://www.imdb.com/find?q=memento;s=all". At that point, I would be able to replace the "memento" search query with a variable in the Inquisitor settings to get this: "http://www.imdb.com/find?q=%@;s=all", where %@ just happened to be the variable used by Inquisitor.

Suddenly, I could add just about anything site within seconds, from Finance quote searches to torrent sites to corporate intranet searches.

It didn't cross my mind that someone could easily top this, but Google did just that with Chrome. When typing a domain like imdb.com into the hybrid URL-search bar, the right side of the bar hints that you can hit the "tab" key to type a search query for a search within that site (in this case, imdb.com).


Most major dedicated search engines try to facilitate site-specific searches these days, but for times when you want to perform a site search, the browser has evolved to help get you there.

Password Masking

Caught in the ongoing tug between ease-of-use and security is password masking, a point of contention in the past with some of my colleagues working more closely with security related issues. Whether security and usability necessarily have to be inverses of each other is something to leave for another post, but what's clear is that it's certainly the case with our current form of masking typed text strings in password fields. There are three major types of password masking I've seen.

1. Full masking: Every alphanumeric character or symbol is represented by an asterisk or dot, effectively masking them.
2. Partial masking: All characters are masked, except for the last typed key. (Example: iPhone OS)
3. Invisible: No asterisks, dots, or replacement characters of any kind will display. (Example: Unix environments)

Full Masking

Full masking is the most common technique, and while it tells you where are you in the password you've typed so far, it also gives observers that information too. With this technique, the only way an onlooker can grab those passwords is by a combination of physically watching the keys typed and educated guesses based on what the asterisks hint about the password (its length, whether slowed down typing to enter numbers or symbols, and so on). Remember, I'm talking about what can be attained visually and audibly, as that is the point of password masking. Areas like keylogging, plaintext passwords, and such are another area of concern entirely. Now, when it comes to full masking, it generally works fine until something is causing typos, which masking will hide. This includes common mistakes like leaving the caps locks key on, or missing a shift modifier key, or general typos with commonly misspelled words or lengthy randomized text strings. It's worth noting that the caps locks issue is sometimes addressed by detecting that it's on, and subsequently warning the user when it is.

To deal with this issue, it seems that sometimes entirely masked passwords come with an option to toggle the asterisks on and off, such as with the WEP/WPA key fields in OS X. In other words, it's an override option to temporarily remove masking at the discretion of the user.

Partial Masking

But what happens when full masking carries over to a device where typos are far more frequent, such as a mobile device? The user could slow down immensely, or type at regular speed and hope that the login won't lock down or throw a CAPTCHA form after a couple invalid attempts. The iPhone OS addresses this by masking all characters in dots, except for the last typed character (for a couple seconds). It's an improvement, but at the expense of anyone peering over your shoulder seeing each last character. Anyone keeping an eye the entire time can thus see your entire password in the clear, and at a readable pace considering that even the fastest typists on mobile keyboards are a huge margin from the fastest on the desktop keyboards. I have mixed feelings about this, but then again, even fully masked, typed keys on touch keyboards display their character in a tab above the area obscured by the tapping finger. So any watchful person can still catch on that way, regardless of whether the password field itself is fully or partially masked. This is evident on such touch keyboards as the ones on iPhone OS and Android.

Invisible Masking

In a UNIX environment, you'll notice that password prompts give no feedback for what you're typing or what you've already typed, ironic given that this environment is where strong complex passwords are common. I've seen this confuse many, many users, and it's a commonly asked question that won't go away. Eventually, most people get accustomed to this, and it becomes just about as easy to use as full masking - for most cases, that is. But when it comes to lengthy randomized passwords, entering passwords becomes a snail-paced task, during which keystrokes become easier to observe and follow. (This is unless you happen to be a god at rapidly typing 40-character alphanumeric, mixed-cased passwords interjected with symbols with and without modifier keys.)

The Locks at Every Gate

As we raise the number and complexity of locks on a gate, we also construct higher and higher hurdles for intruders to overcome, but also for the people who have to encounter these security measures everyday, every hour, or even every few minutes. These measures typically work fine, but serve users poorly in intense cases to the point of inducing people to find less secure workarounds ranging from writing passwords to copy-pasting a password in the clear. (Try typing a 30 character alphanumeric, mixed-case Wi-Fi WPA2 key on a mobile physical or touch keyboard, and see if you aren't tempted to copy-paste too.) The less visual and acoustic cues there are, the more it slows everything down. You'll get used to it, but we just need to vary where to draw the line on a use-case basis because ultimately the most inconvenienced person is not the sinister characters - it is you.

Blast from the Past: SDI and MDI

This is an entry I wrote on March 20, 2006 touching upon SDI and MDI:

Warning: This is a usability and interface topic. You may quietly exit through the back doors. No hard feelings. Otherwise...

I know that Adobe Acrobat 7 has been out for quite a while now, but I figure that I need to get the word out wherever I can. The following is a problem that's been bothering me since Acrobat 6.

Notice this. Earlier versions of Adobe Acrobat used a multiple document interface (MDI), where all documents resided within a single parent window. The problem was that they forgot to add "tabs" for easy navigation between the documents in this multiple document interface.

I wrote a complaint in the official forums a while back, and in version 7, it seems that they finally tried to solve the problem by switching to a single document interface (SDI), where each document has its own window on the Windows Taskbar. But the Adobe Acrobat team forgot something again. If you exit any given document with the Microsoft Windows [X] button (the red one in Windows XP), every single document closes. The expected behavior, based on other applications written for Windows, is that only that one document should close (not all of them).

Or perhaps the Acrobat team has a good explanation for this behavior? (I certainly can't think of one.)

My original entry: http://gordeonbleu.livejournal.com/20578.html

Inline Autocompletion

Inline autocompletion is a common part of search bars, but for the longest time, autocompletion was anchored to the beginning of the URL in a web browser address bar. In the middle of last year, Firefox included an "awesome bar" in version 3, which allowed us to type: "lunar" to bring up a past history or bookmark of "http://en.wikipedia.org/wiki/Penumbral_lunar_eclipse", whereas other browsers required typing, "en.wiki..." (not even flexible enough to allow "wikipedia" to yield results). Over a year onward, and this still hasn't spread to other browsers.

Inline autocompletion

Anchored autocompletion

Seeing as most web browsers haven't integrated their search and URL bars entirely as Chrome has, this is one handicap of most browsers that maintain discreet address bars, as they miss out on one of the top usability benefits of unanchored autocompletion - lessening the requirement on the user to remember URLs.

Browser Sniffing on the Mobile Web

Browser sniffing holds a level of stigma in the web development/design world, as we have been spending years and years creating cross-platform, cross-browser sites that use more robust techniques of singling out browsers, rendering engines, or platforms as a last resort through our knowledge of what's supported - CSS conditional comments, JS object detection, and various tricks and (if needed) hacks both client and server side. It has long been our practice to create a solid separation of presentation, content, and functionality in a way that degraded gracefully (or more recently, progressively enhanced).

This has worked well on the desktop platform, from desktop workstations to notebooks to tablets.

The Age of Rich Mobile Computing

So the question becomes, what to do with the mobile platform. For years, there have been very basic mobile-specific pages for basic phones with tiny viewports and browsers that couldn't handle much more beyond HTML, with notable omissions of support in areas like CSS and Javascript.

But ever since the iPhone brought fully-featured mobile browsers to the mainstream, there has been this huge trend of companies creating iPhone-tailored sites designed for the width of their viewports, and guaranteed to work on W3C standards-compliant browsers, including Webkit-based ones like MobileSafari, and soon after the ones on the Android and webOS platforms, which currently also have devices with similar viewports. Consequently, these iPhone-tailored sites generally automatically work well with most modern smartphones with full browsers, effectively creating a second tier of mobile sites for smartphones.

For the iPhone initially, it seemed both sensible and insensible simultaneously that people were creating sites that fixed themselves to a specific viewport width. One of the great abilities of MobileSafari on the iPhone was that zooming on small screens was finally easy with the pinch gestures, so coupled with the full HTML/CSS/JS browser, there would be little reason not to experience the same full website used on the desktop platform. Even considering this, however, people still designed sites that negated the pinch zooming.

Still, with a smartphone version of the site, you would ideally be served the same information you need in a way that didn't require any zooming at all, because the user is still otherwise pinching his way into a zoomed out preview of the page that he can't initially read. But sometimes smartphone sites aren't thought out as well, and information or features end up missing.

In some cases though, desktop versions of sites are so mobile-unfriendly (Facebook desktop site) that we pretty much have to rely on mobile sites or native applications to access them on mobile devices.

But there's a lingering question that we may all face in the coming years - at what point is a device considered a "mobile" device and not a desktop computing device?

Drawing the Line at "Mobile"

In late 2008, I had this discussion in an iPhone development forum where I wanted to know how to allow a user to use a link to exit a mobile page back to the full desktop site, without cookies and without having the desktop site's mobile browser detection causing an infinite cycle between the desktop and mobile sites. In other words, just as many sites were doing, I was automatically directing all appropriate mobile devices to the mobile site, but wanted to offer an option to return and remain at the original version.

We came up with a solution, but not before having a heated discussion over whether browser sniffing on the mobile platform was a forgivable exception. My point was that most phones beyond the iPhone did not possess easy pinch zooming, and magnifying with a trackball one square block at a time was not a pleasant experience. Beyond that, there were functions most mobile devices could not yet handle so smoothly, such as click and drag, and general mobile-specific features like data detection (phone numbers, addresses for maps, etc).

But the best counter argument I heard that makes me reconsider is that the line between desktop and mobile computing is not as clearly defined as we think, especially if you think about the smaller 7" netbooks in the middle of the spectrum with screen resolutions in the 800x480 pixels.

It's possible this gap could shorten and fill over time with in-the-middle devices like these and other devices with smaller resolutions. For comparison, the iPhone family of devices sports a 3.5" 480x320 pixels screen.

It's also possible this gap might remain as sparse as it is today, if we assume that the 7" netbooks are roughly the smallest non-niche form factor the mainstream is willing to bear on a desktop platform with a two-handed QWERTY keyboard, and that today's smartphones with thumb-based QWERTY keyboards (hard and touch-based) are in the upper bound for mobile to remain pocket-friendly.

Ideally, we do want our fluid layout pages to scale down well to the current mobile category of devices, and not to serve and maintain special mobile versions of our sites. As it stands, the gap still holds, and for the sake of the user experience in the present day, this is how many of us will approach it. We'll see where the future takes us from there.

Thoughtful Charts and Real-Time UI Feedback in Google Finance

I'll have to admit that I've never been a frequent user of the advanced options of Yahoo! Finance, so the information that Google Finance provides is just right for finance users like me who just want the fundamentals, and Google Finance has provided excellent usability for that.

One of my favorite features that came out of Google Finance was the extended hours trading on charts of logged in users. Real time quotes in after hours trading was already common in both Yahoo!'s and Google's finance products, but visually displaying the pre-market and after-market movements diagrammatically was important because it displayed the history of the price movement per share of a stock, whereas beforehand, you only had the current trading price during active after-hours, or the last price after that session had ended. It has also been a particularly useful way to guess the trend to be at the opening bell.



A couple days ago, they introduced a tiny detail that I believe users will find helpful - any digits changing in the displayed price will flash green or red momentarily, depending on whether it's an upward or downward change. To put it in perspective, the norm before this was to only color-code the difference in points or percentages, not any part of the price itself. This is a small step, but it reminds me of the days in the late 90's when CNBC turned their navy blue change indicators and values to color-coded green and red values, a novelty that has long since become a standard.



Beyond that, still among my past favorite interface details are - showing news indicators on the charts at the point where news was announced, and transactions tracking to monitor gains and losses. The next step would be to somehow make this site scalable on a mobile viewport, the Flash support on various mobile devices notwithstanding.