Specifying Gestures By Example

by Dean Rubine

Summary

The paper proposes a system of creating a gesture recognizer automatically from example gestures. This system is incorporated into GRANDMA (Gesture Recognizers Automated in a Novel Direct Manipulation Architecture) which allowed the gestures to be used in a direct manipulation environment. The paper describes the interactions needed to control a drawing application, GDP. Of note is that GDP is not sketch recognition, the user utilizes a gesture to create a shapes starting point and then finishes the creation of the shape by dragging (rubberband). This system is single stroke only; the limitation avoids a segmentation issue and single strokes coincide with a single tensing and relaxing muscle movement. The author notes that 15 examples per gesture class is adequate. Also the author describes how gesture handlers are created for each gesture, flaunting the object-oriented nature of the project. The strokes are recorded by taking in x and y positions in addition to the time. A set of 13 features is calculated for each stroke. The feature hold that a small change in the shape should correspond to a small change in the feature value. The sample estimate is determined by the average of the feature for a class. At around 50 training examples the performance gained plateaus, also as the number of possible classes increases the recognition rate drops, but not beyond 90%. Two extensions to the system are proposed; one in which the system does eager recognition, and multi-finger recognition.

Discussion

The paper makes mention of multi-finger recognition which surprised me a little. Such devices have become popular recently, with Jeff Han's TED presentation and the popularity of the iPhone, but this paper was written in 1991. Sixteen years is quite a gap for a idea incubated in research to begin appearing in consumer products.

The algorithm is quite nice in that it doesn't limit features in being added, and also that the feature values appear to be weighted. As I read this paper I kept thinking about Palm's Grafitti language and the similarity in design. It did seem odd that the delete 'x' gesture used the start of the gesture not what object the gesture overlapped to determine what to delete.

Citation

Rubine, D. 1991. Specifying gestures by example. In Proceedings of the 18th Annual Conference on Computer Graphics and interactive Techniques SIGGRAPH '91. ACM Press, New York, NY, 329-337. DOI= http://doi.acm.org/10.1145/122718.122753

Sketchpad : A Man-Machine Graphical Communication System

by Ivan Sutherland

Summary

This paper is about Sutherland's graphical interaction system, it should be noted that the paper was published in 1963. The paper opens with a great quote "Heretofore, most interaction between man and computers has been slowed down by the need to reduce all communications to written statements that can be typed; in the past, we have been writing letters to rather than conferring with computers." The paper opens with an example of drawing a hexagon shape: the drawing is accomplished with a light pen and a variety of buttons and knobs. The light pen is used to draw the line, by way of rubber bad, and the other inputs are used to constrain the drawing. Throughout the paper Sutherland makes many references that are similar to object oriented programming concepts (such as "instance" and inheritance). He also notes that not only is the drawing stored, but information about its creation (constraints is stored). Changing the basic shape, if instances of that shape have been used ripples through the document. The paper notes four broad areas in which such a system is useful: storing and updating drawings, gaining scientific or engineering understanding of operations that can be described graphically, topological input for circuit simulators and highly repetitive drawings. The paper describes the ring structure, light pen and display technology that was used in the system. Also discussed is the abstraction, the recursive merging and deleting. Of note is the discussion of how constraints can be put onto already created drawings. The paper lists patterns, linkages, dimensional lines, bridges, artistic drawings and electrical circuit diagrams as domain examples.

Discussion

Sutherland, could for see the issue I would have when he put the sentence, "It is only worthwhile to make drawings on the computer if you can get something more out of the drawing than just a drawing" into the conclusion. This is one of those papers that I always get a weird feeling about. Sure, 1963, and it was an incredible break through. He did amazing things. My concern is that for one reason or another these ideas did not catch on. This wasn't some nameless researcher, Sutherland is widely respected. He was at MIT, he had funding. It is because the possible uses are limited? Have we become so use to keyboard/mouse interaction that what was once considered a more natural input technique (pen) is no longer natural?

Citation

Sutherland, I. E. 1964. Sketch pad a man-machine graphical communication system. In Proceedings of the SHARE Design Automation Workshop DAC '64. ACM Press, New York, NY, 6.329-6.346. DOI= http://doi.acm.org/10.1145/800265.810742

Introduction to Sketch Recognition

by Tracy Hammond and Kenneth Mock

Summary

The paper discusses pen based computing and sketch recognition with particular attention paid towards educational environments. Background is given on the creation of the field, specifically Ivan Sutherland's early contributions. Tablet PCs are described as well as the distinction between passive (that which requires contact) and active (that which requires a special stylus) digitizers. Also described are different pen input devices (tablets, UMPC, drawing tablets) and the software that can be used with such input means. The paper focuses then on this type of input in educational situations. How lectures can be annotated, edited, and prepared using pen input is discussed.

The paper then moves into a discussion of sketch recognitions and lists domains - art, music, chemistry, mathematics, finite-state machines, UML, COA - in such recognition makes sense. The paper also gives a brief overview of FLUID (LADDER and GUILD), and comments on how sketches can be tied to higher-level functionality. The authors describe how shape definitions are created in LADDER using primitive shapes and constraints. Also noted are two challenges with pen-input: 1) recognization systems being able to handle varied input, 2) administrative concerns. The authors then discuss two case studies of tablet use in classrooms. The paper concludes noting the increase in TabletPC's adoption in educational markets.

Discussion

Both of the case studies take place in Math classes: it would have been better to have each case study focus on a different subject. The paper gives a good overview of current pen input devices, those devices' roles in an education environment, and what is happening in sketch recognition. The work was a bit fluffy, but the author openly admits that.

Something that concerns me about sketch recognition is that it seems heavily domain specific. I have difficulty moving beyond the "that's neat" feeling into viewing it as something that can have a fundamental shift in the way people interact with computers.

Play this game, how would you want to interact with a computer if anything was possible? Paper has some great benefits, but far too many of them I feel are physical not the user interface. By physical I mean the lightness, the contrast. In pen input I lose some of those nice physical qualities but I gain the nice digital ones (the ability to edit, the ability to reproduce, the ability to collaborate over a large distance).

Introduction

Name: Brian David Eoff

Year: PhD 3rd

Email: firstname.lastname@gmail.com

Academic Interest: User Interface, HCI, IR, Semantic Web, Personal Information Management,Ubiquitous Computing, Information Discovery

Relevant Experience: Hammond's HCI class, Choe's AI class, Worked with LADDER and GUILD before.

Why Are You Taking This Course: To explore an alternate way for people to interact with computers.

What do you hope to gain?: Be exposed to new ideas and paradigms of interaction, to stimulate thought about how a user interacts with a machine, and trying to make that process more natural.

What do you think you will be doing in 5 years? Finished Ph.D or fled academia and started a small company with friends.

What do you think you will be doing in 10 years? Living in Seward, Alaska. In a compound. Preparing a small revolutionary army.

What are your non-academic interests? Reading, Basketball, Swimming, Sleeping.

Tell a fun story about yourself: As a young boy I really wanted to be a soldier (GI Joe hooked me hard). For some reason I associated the word "Vietnam" with being a soldier. At a dinner party one of my parent's friends asked me what I wanted to be when I grew up. I responded I wanted to be Vietnam. They laughed. And I never became a soldier.