Posts Tagged ‘touch screen’

Touch screen interface considerations

February 24, 2010 5 comments
When moving to a new hardware platform such as a touch sensitive mobile device, it is important to design the UI with the constraints of the device in mind.  Early Windows tablets failed in large part due to the OS not being suited to touch interaction; icons were too tiny to press on without a stylus.  I have heard the same complaint about early generation Windows Mobile phones; the user in question said he had to literally press tiny squares on the screen where the up and down arrows were in the scroll bar.  The UI widgets from desktop programs were ported over to these new platforms without giving thought to the decreased screen resolution, screen size, and finger dexterity that a mobile platform provides.

The iPhone and its multitouch gestures provide very intuitive, natural ways of interacting with the reduced screen real estate.  Scroll bars are no longer controls that must be held and dragged; rather they are small, unobtrusive widgets that appear only while a page is being dragged.  The entire page becomes the draggable area, in keeping with the idea of maximizing the size of items that need to be interacted with.  Coupled with the pinch gestures for zooming in and out of webpages, the browsing experience on the iPhone is almost flawless.  There are two main problems stemming from the metaphor of dragging the entire webpage: the lack of hover and drag and drop.

Hover behavior

Given the fact that touch screens only pick up your inputs when you are touching the screen, there is no notion of hovering (the browser doesn’t display a cursor where you last touched).  Hovering over elements in a webpage usually has one of two effects:

1) Showing a tool tip

2) Displaying additional information in a section of the browser; e.g. mousing over an embedded link reveals the target of the link in the lower left corner while using firefox.

In very few cases, mousing over or hovering over elements actually acts like a click; see for an example.

Number 2 is not a big concern; when you long press on an embedded link, a screen pops up revealing the target of the link:

Number 1 is more of a concern.  If your site is designed to be viewed on a mobile browser, you should ensure that that you have a version of the CSS that does not require you to mouse over or hover over elements to reveal information.

Drag and drop

Drag and drop is a common desktop OS UI metaphor.  It can be slightly tricky for users to handle due to the dexterity required, especially if the drag and drop targets are small; most digital natives don’t have any problems with it.  It is less common on the web than traditional desktop fat clients, but there are certainly webpages that use it.  For instance, WordPress uses drag and drop to configure sidebar widgets:

The problem is that these movable elements cannot be selected and moved on the iPhone; they select when pressed but as soon as you move your finger, they deselect and instead the page moves.  While this would be a problem in and of itself, drag and drop is also used in common UI widgets such as sliders.


Sliders are an excellent UI controller for selecting an approximate value from a bounded range of continuous values.  Look at‘s elegant dual slider control for narrowing down price:
With a single control, we get two functions:
  1. Displaying the current range of values
  2. Allowing user to narrow down that range into a smaller subset.

For instance, if user only wants sneakers that cost between 50 and 100 dollars he just drags both ends of the slider bar:

This illustrates a drawback of the slider approach: it is imprecise when the range of available values is large and the control is small on the page; the slider would have to have subpixel precision in order to make user

The frustrating thing is that the iPhone has BRILLIANT slider support in its iPod app.  They get around the problem of precision by making use of the vertical dimension that normally is ignored.  When your finger is right on the slider, the slider follows your finger one to one.  When you move your finger down the screen, the sensitivity is turned down and it takes a greater horizontal change to elicit a change in the position of the slider (changing from “Hi-Speed Scrubbing” to half speed to quarter speed.)

White dot represents where the finger is held down

The drawback of this interaction is that it is novel; I discovered it by accident and I’m sure a lot of iPhone users don’t even know about it.  However, it is a wonderful solution to the problem of imprecise fingers; it could be used in traditional desktop apps as well.

Sites with interactive javascript graphs frequently use sliders; see for instance OKCupid’s blog post on why you should date older women:

On the iPhone, it is impossible to move the slider to interact with the graph.

The problem is that there is no standard HTML way of specifying a slider; instead sites use a mishmash of Javascript to get that effect.  (Read a tutorial how).  If there were a standard HTML component, it’s likely that the iPhone could handle it more gracefully.  For instance, the iPhone changes the standard way dropdown menus are implemented

Standard desktop view of a dropdown control

iPhone view of control – a finger-friendly wheel widget pops up, occupying half of the screen.

So one solution to the problem of lack of slider support would be to have a standard HTML widget for sliders that the iPhone could then interpret and render in a more useful way (e.g. with the vertical dimension added like on the iPod slider controls).

Another potential solution would be to make a gesture that indicates a mouse drag.  There is already a way to select text, which is done in a desktop environment via the same mechanism as drag and drop (press, move mouse, release mouse button).  I could see this working in one of two ways.

1) As a multitouch gesture.  One finger is held as still as possible indicating that the background should be anchored, while the other finger proceeds to press on the draggable component and move it around the screen.  The drawback of this is that the pinch gesture currently works even when one finger is held still; thus this gesture would overload that of pinch.  I know I personally pinch by moving both fingers, but some people might not.

2) When a draggable entity is long-pressed for long enough (1, 2 seconds?) browser enter different mode where motion events do not scroll the page but instead move the draggable item.

If you have a multitouch enabled browser, you can see how dragging might work in this proof of concept page. Unfortunately the pinch gesture does not work correctly – it’s much jerkier than a standard webpage.  Their implementation suggests a third alternative, immediately entering drag mode when a draggable item is touched.  The problem with this is that, if there were multiple draggable items on the screen it would be very difficult to actually grab a portion of the screen in order to pan around to new areas.

This way of handling drag and drop is used by games on iPhone, such as Words With Friends.  Note in the left picture the scroll bars indicating that the board is being moved.  On the right the finger is moving while over the E tile, so the E is dragged along with the finger.  Moving the finger off the screen is equivalent to releasing the mouse button in a standard drag and drop action; the tile is laid down in nearest square.


The iPhone has introduced some great UI metaphors such as pinch and zoom as well as making the entire area of the page draggable for easy scrolling of text.  In the realm of the web browser specifically, these advances prevent two common desktop OS UI metaphors from working, that of hovering over elements and dragging and dropping elements.  The problem of hovering is not insurmountable; web developers need to be aware of this drawback and code their websites in such a way that information is not lost if you cannot view tooltips or otherwise hover over elements.  The second problem, that of an inability to select an element, drag it, and drop it, is more pressing.  There are certain tasks that are absolutely impossible to complete while using an iPhone browser, such as anything requiring you to move a slider control.

Hopefully the new version of the iPhone OS can remedy the drag and drop problem.

Categories: UI Tags: , , ,