RelatedWork.tex

\chapter{Related Work}
\label{chapter:related-work}

Existing sketch tools can be classified into two categories: (1) those that interpret sketches with the purpose of turning them into representations that adhere to strict notational conventions, and (2) those that support sketching activities in general. Johnson et. al. \cite{Johnson} provide an extensive review of all of the sketch tools contributed until 2008. Here we make a selection of the tools and approaches most relevant to Calico, and especially include more recent work.

\section{Tools that Interpret}
\label{relatedwork:1}

The  tools within this category focus on assisting designers by transitioning their sketch into representations that adhere to strict notational conventions. For instance, sketched boxes and lines between boxes may be turned into UML diagrams, or informally drawn user interface widgets into an interactive user interface. While we purposefully chose not to interpret sketches in Calico, these tools remain relevant as background work.

Many tools that interpret sketches focus on generating prototypes for user interfaces. One such tool is SILK \citep{Landay}, which allows users to generate a GUI prototype for walkthroughs from a sketch, and then generate the corresponding Java code that implements the GUI. SILK was the first tool of its kind, and stressed both the importance of working in a low-fidelity environment, and the value of rapid iterative design work. Particularly the fact that sketches can be turned into usable prototypes that, by walking through them, can give near immediate feedback to the designer. SILK differs greatly from Calico in focus. It is limited to UI design, and focuses on only a few design behaviors. Another tool, DENIM \citep{newman2003denim}, allows users to create low-fidelity mockups of web sites that the user, again, can walk through to get a sense of the flow of the web site. DENIM builds on the concepts of SILK by both supporting interpretation of sketched content and navigation between web pages (as opposed to user interfaces). DENIM further contributes the notion of multiple levels of hierarchy for navigating sketches, in which the user zooms out to navigate the network of web sites, and zooms in to view sketched content on individual web pages. DENIM's use of multiple levels of levels of abstraction for navigating web pages, and linking web pages, is similar to Calico's intentional interfaces, but is highly specialized for designing web pages. Further, DENIM still only supports a handful of design behaviors (i.e., refinement, impromptu notations, and moving between levels of abstraction).

Other tools target multiple domains, such as the tools InkKit \citep{Plimmer} and Freeform \citep{chung2005inkkit}, which created usable software systems from sketches. The design of these two tools stresses the importance of retaining the sketchy look-and-feel of what is drawn, which is based on the insight that visually beautifying content too early can be harmful to the design process \citep{Shipman}. Unlike previous tools, InkKit and Freeform interpret different types of sketches, such as user interface and ER diagrams. InkKit specifically uses different windows within a space, each that is configured to interpret one pre-defined notation. Sketches in one window are explicitly linked into another window, which may be used to create a storyboard, or to populate a widget (such as a list populating a dropdown menu). Sketches are interpreted to generate a working prototype system that, once again, can be walked through. By interpreting sketches from multiple perspectives, these systems provide rich prototypes for feedback at a low cost of effort for the designer. These systems are similar to Calico in that users can sketch a system from multiple perspectives, however these tools restrict users to using a pre-determined set of notations. Moreover, from a design perspective, these tools only support a subset of design behaviors, which including using different kinds of represenations, refinement, mentally simulating over sketches and moving between different perspectives, which are limited to the set of possible interpreted notations.

Another group of interpretive sketch systems includes those that recognize UML. The earliest such tool was Knight \citep{damm2000tool}, which allows  designers to create partial UML diagrams by selecting a portion or all of their sketch to be interpreted as UML diagrammatic elements. Knight was inspired by studying software designers at the whiteboard, and contributes the observation that software sketches begin vague, but become refined over time. The Knight tool allows designers to sketch and convert their sketches into a beautified UML representation by manually triggering the recognition process. Both Calico and Knight allow users to refine their drawings from plain whiteboard sketches into objects at a later point, however Knight automatically interprets UML notations for the user and offers advanced notational support, while in Calico users must manually refine sketches and also have limited support for a wide range of notations. Knight also supports a subset of the design behaviors, namely refinement and moving between a limited set of perspectives.

The tool SUMLOW \citep{chen2008sumlow} further advances on the work of Knight by allowing interpreted sketches to retain their sketchy appearance after interpretation (citing InkKit as inspiration), and allows the mixing of different UML diagrams (i.e., class diagram, sequence diagram, use case diagram, etc.). While both Knight and SUMLOW make UML diagrams a less viscous representation by allowing users to simply sketch, the retainment of the sketchy appearance after interpetation introduces a level of provisionality that was not available in other sketch interpretation tools. Both Calico and SUMLOW are similar in this philosophy. Hosking and Grundy later integrated SUMLOW into a full software design environment, Marama-Sketch \citep{Grundy}, which designers can use to sketch diagrams in a broad variety of notations. Marama-Sketch brought tighter integration between sketched models and actual code, a quality which Calico itself does not have. At the same time, Calico still supports many more design behaviors that SUMLOW and Marama-Sketch do not, notably moving between alternatives, levels of abstraction, mental simulation, and collaborative behaviors.

Numerous other UML-oriented sketch tools have been created, generally following similar strategies (for a survey, see \citep{Johnson}). 

Another approach to sketch recognition is to build a general purpose framework for interpreting sketches. LADDER is such a tool \cite{hammond2006ladder}. It provides an extensible, general purpose language which can be adapted to many notations, including software design notations, biology and chemistry notations, and military symbols. Of the many interpretive frameworks (e.g., SATIN \cite{Hong}, LADDER \cite{hammond2006ladder}, InkKit \cite{Plimmer}, SketchML \citep{Avola2009}), most target a single, or small set of notations. Based on the observation that many frameworks use different engines, yet that each still suffers from mixed interpretations of sketched content, Dixon provides an approach that queries multiple engines simultaneously, and ranks the results by their probability of success \cite{dixon2008whiteboards}. These tools vary greatly from Calico, given that they focus on sketch interpretation, while Calico explicitly chooses not to recognize sketches.

Other tools allow sketches of mixed notations to coexist in the same area as well, though interpretation is left to an explicit manual choice by the user rather than automated recognition. An example of such a system is Patches, in which the user can create sets of irregular shapes called ``patches'' \citep{Kramer}. These can be moved around, be made translucent, and, most importantly, a patch can be assigned an interpretation that designates it to become a list, outline, table, or other element. The functionality, as well as look-and-feel, of the scraps functionality in Calico is similar to the Patches system. However, they differ in their goal, where Patches views its functionality as providing lenses over existing sketches, and uses patches to offer transparent layers on top of each other. Calico, on the other hand, uses scraps as building blocks to create more complex representations. More so, Calico's primary aim is to support a wide range of design behaviors, while Patches suports just a limited set (i.e., different kinds of diagrams, refinement, impromptu notations, and moving between perspectives). Patches later inspired the overall functionality of the sketch framework SATIN \citep{Hong}, which was later used to create the tools DENIM (previously mentioned) and Designer's Outpost \footnote{ mentioned in the next section}. 

\section{Systems that Support Sketching Activities}
\label{relatedwork:2}

The second type of sketch system is aimed at supporting the general design activity. These systems help organize sketched artifacts during meetings, make them retrievable, and help manage and maintain the plethora of sketched design content that gets generated during design sessions. 

Colab, created for the Liveboard system at PARC, was the first system to work with this hardware, and acted as a meeting tool \citep{Stefik}. It had a simple, uncluttered interface with a mode selector and a visible set of saved sessions in the fashion of a filmstrip. The next system developed for the Liveboard was Tivoli \citep{Pederson}, which used the filmstrip metaphor to manage sketches in a side panel. Tivoli also organized sketches into chunked items for easy moving, and had additional functionality for creating empty space. Tivoli did not attempt to recognize any of the drawn elements, but rather provided intelligent support for common tasks, such as creating lists and checking off items in lists. After Tivoli, Dolphin \citep{Streitz:1994:DIM:192844.193044} introduced a client-server architecture for group collaboration and allowed users to create links between canvases. Dolphin was the first whiteboard system to provide hyperlinks between content in canvases, which would not appear again in other tools until the creation of DENIM ten years later. These three systems show a remarkable foresight in their underlying ideas. Chunked items in Tivoli are a precursor to scraps, and hyperlinks in Dolphin operate similar to tags in Calico. Distributed operation also was supported. Compared to these systems, Calico has made significant strides in refining these early concepts, making them truly usable, and enhancing them (for instance through intentional interfaces, which supports spatially opposed to temporal organization of sketched content). Moreover, Calico succeeds in bringing them together in a single system.

Many other tools provide methods for managing the presence of multiple sketches within a fixed space. PostBrainStorm \citep{guimbretiere2001fluid} uses the border areas as zones where sketches auto-shrink, thereby minimizing space use by elements that are temporarily set aside. PostBrainstorm further allows users to place a physical picture on a whiteboard, and have the same picture automatically appear in the system. While Calico does not support such physical object scanning, the importing of existing photos showed itself to be an important feature in the use of Calico. 

Flatland \citep{mynatt1999flatland} automatically groups sketches into clusters, provides group manipulation, and moves them out of the way when more space is needed based on where the user starts drawing. Flatland was not the first to provide clusters (e.g., Patches), but Flatland does make them the center focus, and offers rich support for organizing lists, creating maps, etc., as well as allowing users to move backwards and forwards in the history of clusters. Scraps, in comparison, do not support as many domains at this time, but offer the benefit of being stackable and relatable. Moreover, they form the basis for supporting numerous design behaviors while Flatland only supports a few (i.e., different kinds of representations, sketching only as much as needed, and impromptu notations).

Range \citep{Ju} extends the approach taken by Flatland by using a person's physical proximity to the board as a trigger to create space. It also presents past sketches to the designer when they move physically away from the board in order to provide inspiration for creative work. 

Overall, many of these systems attempt to support the designer at the whiteboard in general activities such as clustering objects and making space without the designer manually requesting this space. Calico does not provide such ubiquitous support in a single whiteboard, but instead allows users to manually organize their sketches using intentional interfaces. Such manual organizing of space has value in one's own design process, as the act of organizing is a large part of the design process \cite{Nickerson}.

A third set of tools specifically targets the organization of multiple sketches and canvases. Bellamy et al. \cite{Bellamy:2011:STI:1985793.1985909} presents an ideation tool for building user interfaces. Users sketch the states of a user interface, which are presented back to the user as a set of storyboards. Additionally, the history of the user's exploration is presented back to the user as a tree that includes any undo's. Other systems target the visualization of separate canvases, such as IdeaVis \cite{geyer2012ideavis}, which presents a visualization for a network of whiteboard canvases. The IdeaVis system captures hand-drawn sketches using a camera, and presents them back to a co-located sketch team using an ``interactive hyperbolic tree visualization'', in which users can navigate a network of canvases in which the visual size of canvases becomes larger when in close proximity to the active canvas. Their studies showed that automatically making relevant canvases large helped users maintain focus to important sketches. This system is most similar to Calico's intentional interfaces in that they both visualize a network of canvases, however intentional interfaces maintains a fixed size for canvases. At the same time, intentional interfaces supports types of relationships as well as the ability to manually organize the canvases. Calico, also, supports many additional design behaviors, such as different kinds of representations, refinement, mental simulations, and collaborating over sketches.

Another set of tools targets the collaboration aspect of electronic whiteboard systems. Designer's Outpost \citep{klemmer2001designers} bridges the gap between physical Post-It notes and digital content, transitioning content of Post-It notes into digital artifacts that then can be organized in various ways. The core functionality of this system allows users to create links between digital Post-It notes, group Post-It notes into clusters, and zoom in and expand individual Post-It notes to work with them in great detail. Designer's Outpost provides novel methods of interacting with remote users, such as showing a user's shadow on the whiteboard in order to communicate gestures and body language. Calico, in comparison, does not communicate body language, but instead uses the fading highlighter to communicate transient information. Moreover, Calico's aim is different. While the focus of a design session in Designer's Outpost is on the hierarchy of Post-It notes, which leads to strong support for brainstorming ideas, Calico's focus is on the canvases themselves, and provides support for design behaviors such as using different kinds of representations, refinement of sketches, and impromptu notations.

TEAM STORM \citep{Hailpern} presents another approach, which addresses the issue of public and private spaces. Users can create sketches on their own devices, and have the option of sharing their sketches in the public space, in which other users can interact with and provide feedback on sketches. Further building on the concept of a shared collaborative space, the GAMBIT \cite{Sangiorgi:2012:GAM:2305484.2305527} system addresses the issue of using multiple devices together, including large displays, laptops, tabletops, and phones. In studies of GAMBIT, researchers observed a trend in sessions in which users first build mental models of their domain, construct scenarios, and lastly sketch prototypes using GAMBIT. In comparison, sketches in Calico are always public, but users can move to a separate canvas to work asynchronously. The support exists though in less formal ways. At the same time, Calico has additional functionality through which it supports a much wider range of design behaviors. 

Tele-board \cite{Gumienny2013}, similar to the previous tools, is a distributed whiteboard and Post-It Note tool. Similar to Designer's Outpost, it allows users to generate sticky notes from remote locations, group them, and review whiteboards in a history viewer. In their three month study at globally distributed offices, they found that remote users found the system more beneficial than user in the company's central office. Tele-board and Calico share the similarities that they both support designers moving between perspectives, which Tele-board does using a slidesorter, switching between synchronous and asychronous distributed work, designers explaining sketches to one another, however Tele-board does not support different kinds of representations, or bringing together work.

Lastly, Camel \cite{cataldo2009camel} integrates a whiteboard system into Eclipse, allowing users to post UML diagrams onto poster boards. In doing so, Camel provides software teams value by supporting code reviews and design walkthroughs in an informal, sketchy manner: developers can simply write and draw the code being presented.


%%% Local Variables: ***
%%% mode: latex ***
%%% TeX-master: "thesis.tex" ***
%%% End: ***