Discussion.tex

\chapter{Discussion}
\label{chapter:discussion}

In the previous chapter, I reported on the experiences of three different groups using Calico. In this chapter, I bring these observations together to answer the research questions posed in Chapter \ref{chapter:research-question}.

The rest of the chapter is organized as follows. In Section \ref{discussion:strengths-and-weaknesses}, I review the strengths and weaknesses of each feature, drawing upon how they were used across all three field evaluations. Section \ref{discussion:cog-dim}, takes a step back from the observations and examines the theoretical potential of Calico's features by performing a cognitive dimensions analysis, revealing insight into why features did and not work. Section \ref{discussion:overall-strengths-weaknesses} then examines the overall strengths and weaknesses of Calico, examining the benefits and drawbacks of using Calico in a work environment. Section \ref{discussion:interviews} discusses the feedback from the users, their personal experiences, and what they saw as important in their experiences. Section \ref{discussion:summary} then summarizes the contributions of this chapter.

\section{Strengths and weaknesses of each feature}
\label{discussion:strengths-and-weaknesses}

In this section, I examine each feature on its own merits to discuss their strengths and weaknesses. Table \ref{chapter:discussion:strengths-weaknesses} provides an initial summary of this section, and each feature is examined in detail in the subsequent subsections.

\begin{center}
\begin{longtable}{|p{3cm}|p{6cm}|p{6cm}|}
\caption{The set of design behaviors and the features that supported them}\\
\hline
\textbf{Feature} & \textbf{Strengths} & \textbf{Weaknesses}\\
\hline
\endfirsthead
\multicolumn{3}{c}%
{\tablename\ \thetable\ -- \textit{Continued from previous page}} \\
\hline
\textbf{Feature} & \textbf{Strengths} & \textbf{Weaknesses}\\
\hline
\endhead
\hline \multicolumn{3}{r}{\textit{Continued on next page}} \\
\endfoot
\hline
\endlastfoot
\hline
Scraps \& 

connectors &
%strengths
1. Flexible nature is useful in creating many representations relevant to software

2. Gestures enable quick manipulation of hand drawn sketches and scraps without mode switching

3. Existing hand-drawn boxes could be refined into scraps

4. Bubble menu is self discoverable
 &
%weaknesses

1. Requires training to use properly

2. The press-and-hold gesture for select and moving scraps was slow for continuous arranging of a large number of scraps

3. Could not change scrap color or border, text scraps were ``visually heavy''

   \\
\hline
Palette &
%good 

1. Bootstrapped design sessions by importing existing artifacts into multiple canvases

2. Used to build set of reusable icons in storyboards

3. Used as a global clipboard to copy content across canvases in order to juxtapose them

&
%bad 

1. Hard to find scraps in palette when it has many items (more than twenty)

2. Users could not add plain sketches to the palette, must be turned into a scrap first

\\
\hline
Intentional 

interfaces &

1. Clusters provided a simple metaphor to separate content of different projects

2. Linking canvases allows a narrative to be constructed from a set of canvases

3. Made free space to sketch immediately available because they can immediately jump to new canvas or copy a previous one

&
%bad
1. Moving between the canvas view and the cluster view was not a smooth action, canvases were too small to visually distinguish after moving

2. Users had a difficult time understanding where a new canvas appeared in the cluster view

3. Difficult to juxtapose content across canvases

 \\
\hline
Fading 

highlighter &
%good
1. Supported explaining sketches and mental simulations

2. Unlike traditional pointers, fading stroke allows for transient annotations such as arrows, underling, circling, etc.

&
%bad 
1. Strokes made with highlighter are anonymous

\label{chapter:discussion:strengths-weaknesses}
\end{longtable}
\end{center}

\subsection{Scraps}
Scraps, on the whole, were a relatively successful feature in Calico, and a significant improvement of their Calico Version One counterpart. Scraps replaced the lasso functionality to manipulate sketches, which enabled users to manipulate content without changing modes. Scraps also depicted software representations such as software components, process flows, and user interfaces. They sometimes were used to represent lists, though not often. While there are opportunities to provide better support, such as the ability to change its fill and border color, scraps saw a significant amount of use across all groups. 

A strong quality of scraps was their flexibility in representing diagrams. While scraps were not always used in favor of regular whiteboard sketching, they were used often and yielded benefits such as helping diagrams evolve more gracefully and arranging content. Actions which become natural when using scraps, such as annotating it with a color patch, underline, are not straightforward in other mediums. The same actions could be done with drawing tools, but arranging sketches would be more complex. Alternatively, one could design using a more formal tool which would allow for easy arranging of sketches, but the tool may not be able to freely add annotations or symbols. The flexibility afforded by scraps led to the unexpected behaviors in the experiment, such as when the interaction design group tagged them using color and the research group color coded their scraps as well. In comparison to Calico Version One, which saw a relatively narrow set of representations created with scraps in experiments, Calico Version Two yielded a much greater variety of representations created with scraps. It is possible that the longer term evaluation led to more time to become accustomed to using scraps; however, it is more likely that the revised functionality of scraps led to this improvement.

What scraps lacked, however, was more expressive power in their visual features. Users could not change the blue color of scraps and their outline, which many users requested the ability to do. Users in the studies commented that they wished to create text scraps without the blue boarder, as they felt that it made the sketch ``visually busy''. In another session, a user wanted to create text scraps with text of different colors in order to encode meaning, but instead used hand-written text in order to use colors. While these and other features may have been useful, their absence did not obstruct the sessions as users found ways around them, and users remarked that these sessions were provisional.

Scraps gestures, such as creating, selecting, and moving, were a strong and weak point for scraps, depending on the situation. Particularly with long-term users, scrap gestures became a powerful tool to quickly move and copy content. When working with box-and-arrow diagrams for software, they allowed content to be rearranged quickly. When working with lists, they allowed individuals to adjust spacing in handwritten text or make more space. In user interfaces, they enabled rearranging of elements to simulate using the sketched interfaces. Unfortunately when used to categorize large number of scraps, as the interaction designers did, the press-and-hold gesture to select and move scraps were perceived as too slow. The interaction designers, in this case, requested a separate mode specifically targeted at moving scraps.

A weakness of scrap gestures was that they required some practice to become fluid. Gestures such as the landing zone to create scraps were enabled quick manipulation, but new users required some training before they could trigger the gesture reliably. 

Scrap gestures in Calico Version Two, however, was a large improvement on the scrap gestures in Calico Version One, which were not discoverable and were reported as unpredictable. In contrast, the bubble menu in Calico Version Two was reported as being straightforward and discoverable. Further, the landing zone used to create scraps was self discovered by users, while the equivalent action in Calico Version One was also difficult to discover. 

Lastly, the ability to convert regular hand-drawn sketches into scraps, i.e., refine them, further benefited teams. This addressed a significant issue in Calico Version One, which was that individuals often did not use scraps because they would first hand draw a box, and later draw a scrap around the box, resulting with a hand-drawn box inside a scrap. While not used often, users at field sites did use it for both refining existing sketches into scraps and as a recovery mechanism when they failed to trigger the ``landing zone'' gesture correctly.  Users further refined their scraps by using list scraps, which organized scraps into a linear compact list. 

\subsection{Palette}

While the palette significantly improved on the palette of Calico Version One, it still remained one of the weaker features in Calico Version Two. This was due most in part because the palette did not noticeably support impromtu notations for software oriented groups. However, while not performing strongly in this specific design behavior, the palette succeeded in other aspects, such as supporting impromtu notations for storyboards and acting as a global clipboard to transfer content between canvases.

The palette performed strongly in supporting users in three scenarios. First, the palette served to bootstrap a design session for interaction designers by importing several dozen images from outside of Calico. The imported images drove the design session which spanned several days. Second, the palette actually did serve to store a set of graphical icons that were improvised by the interaction design group in their design sessions. Members from the interaction design group found these icons useful in creating storyboards. Third, all groups used the palette to juxtapose new sketches against previously created ones. They did so by using the palette as a global clipboard to copy old sketches to the canvas with the newly created sketch.

Some aspects of the palette could have used further improvement. Users had a difficult time locating items in the palette when it contain in excess of twenty scraps. The interaction designers imported dozens of images of faces, which were difficult to distinguish because of their small size in the palette. Also, users wished to add sketches to the palette without first making the sketch into a scrap. Users reported that making the sketch into a scrap made it too heavy, e.g., they simply wanted the sketch without a scrap's blue background and border.

A potential weakness of the palette was that users designing software systems did not use it to perform the fourth design behavior, inventing impromptu notations. However, this was not a weakness of the palette, but instead caused by two factors: 1) users found it faster to redraw certain sketches because it was faster, and 2) other Calico features were redundant with the palette. With respect to the first point, users often only needed to sketch the name of a component, which could be created much more quickly manually using scraps than finding the component in the palette. With respect to the second, the availability of the copy functionality in numerous forms lessened the need for the palette. For example, when users sketching a software system wanted to create an alternative or move perspectives, they often used the ``copy canvas'' button in intentional interfaces, or used the ``copy scrap'' button in a scrap's bubble menu.

\subsection{Intentional interfaces}

Intentional interfaces was a relatively successful feature in Calico. It advanced on the concept of the grid from Calico Version One, and satisfied numerous design behaviors in doing so. It provided qualities that were not available in the grid interface, such as names for canvases, relationships, and order of work performed. Intentional interfaces, however, had some weak points as well. Navigation was sometimes clumsy in the cluster view, juxtaposing sketches across canvases was difficult, and understanding the relationships between canvases was sometimes difficult for new users.

Many of the grouping mechanisms in intentional interfaces improved upon the grid interface of Calico Version One. While the grid was well received by users of Calico Version One, users increasingly desired ways to separate and categorize their content. Intentional interfaces addressed this problem in several steps. First, clusters provided a generic method to divide canvases into topics, which helped partition content between different projects in Calico.

Second, linking canvases provided another level of organization of canvases, while at the same time offering an ordering to the canvases. Providing an ordering to the canvases was shown to be beneficial in practice as it allowed a set of canvases to compose a narrative. A sequence of canvases could show a walkthrough of a software architecture as well as the exploration of the design space across multiple perspectives, abstractions, and alternatives. Further, grouping canvases by linking them helps users better recall design sessions and understand the content of their sketches several months afterwards.

A third benefit of intentional interfaces was the immediateness of free space. In comparison to the regular whiteboard, users felt more at ease in creating more sketches because they could always create more space and return to previous sketches. The presence of both the ``new canvas'' and ``copy canvas'' buttons made moving to a new sketch a ready-at-hand action, which they used to explore different perspectives and new alternatives.

Users encountered a few issues using intentional interfaces as well. First, the movement between the canvas and the cluster view was not a smooth transition for the user. When loading the cluster view, the perspective always zoomed out, which made the individual canvases too small to distinguish. Second, users had a difficult time understanding the shape of the cluster when creating new canvases from within a canvas itself. One member of the OSS group described the shape of the cluster as ``wizardry'', in which it was a mystery to him as to where new canvases would appear. Third, users could not easily compare contents between canvases.

\subsection{Fading highlighter}

	
The highlighter feature worked well for a particular set of scenarios. The fading highlighter stands on its own as users did not combine its use with other features, but it was useful in scenarios that the other features did not support. Particularly, it supported users in mentally simulating over their work, and also in explaining concepts to others. 

The second strength of the fading highlighter was that it enabled transient annotations in verbal walkthroughs and explanations. Traditionally, verbal explanations of sketches include either a simple pointer, like a baton or laser pointer, or the speaker must permanently mark up a sketch. The fading highlighter provides a compromise of the two approaches, allowing the speaker to use symbols that convey more meaning, such as drawing arrows between sketches, underling them, and circling key points for emphasis.

While the fading highlighter addressed these situations well, it could still be improved to provide better support the two design behaviors that it targets. First, it could provide better support for explaining sketches by providing better awareness features. For example, the strokes could somehow show who is making the stroke. Second, mental simulations could be better supported if the sequence of strokes were somehow recorded. Many of the mental simulations explain how data is moved between components. If this explanation could be captured, and later played back, particularly with audio explanations, the design rational within the design sessions could be played back.

\subsection{Summary}

Each feature brought forward a set of unique advantages not available in the other features. Scraps were strong in supporting the kinds of sketches designers drew. The palette acted as a global clipboard and made sketches reusable across canvases. Intentional interfaces provided many ways to organize, partition, and make free space for sketches. The fading highlighter supported verbal explanations of sketches well. While each feature had usability issues, such as navigating in intentional interfaces and scraps requiring training to use properly, the features provided a net benefit to their users.

\section{Cognitive dimensions analysis}
\label{discussion:cog-dim}

Having examined the use of Calico's features \textit{in practice}, this section now examines the \textit{theoretical} support of those features by performing a cognitive dimensions analysis. The cognitive dimensions framework exposes the affordances that a notation or medium offers. Analyzing Calico's features through this framework provides a basis to determine how those features help or hinder the notation that is implied within Calico. In order to perform the cognitive dimensions analysis, I look at each dimension, and discuss factors that either increase, decrease, or, all things being equal, provide equivalent support for designers as compared to the whiteboard. These findings are summarized in Table \ref{table:discussion:cognitivedimensions}.

%Cognitive dimensions is the golden standard of what Calico should be. Is it possible.. not if they did it. CD exposes to what uses a notation or tool leads.

%Table \ref{table:discussion:cognitivedimensions} summarizes the findings in this section. In most cases, Calico's features provided benefits to one or more cognitive dimensions. Intentional interfaces played a large role in finding relevent canvases in designs diffused across canvases. Scraps and the palette made sketches less viscious and easier to manipulate. Overall, Calico improved on the regular whiteboard for the following dimensions: abstraction, hard mental operations, premature commitment, provisionality, secondary notation, viscosity, and visibility. Calico also had some shortcomings for some of the groups. These can be attributed to inefficient support for the cognitive dimensions of consistency and error-proneness. Support for all other dimensions, including closeness of mapping, diffuseness, hidden dependencies, progressive evaluation, and role expressiveness received moderate support.

\begin{center}
\begin{longtable}{|p{3cm}|p{4cm}|p{4cm}|p{4cm}|}
\caption{CDs Analysis of Calico}\\
\hline
\textbf{Cognitive Dimension} & \textbf{Factors that cause an increase over the whiteboard}& \textbf{Factors that are equal to the whiteboard}& \textbf{Factors that cause a decrease to the whiteboard}\\
\hline
\endfirsthead
\multicolumn{4}{c}%
{\tablename\ \thetable\ -- \textit{Continued from previous page}} \\
\hline
\textbf{Cognitive Dimension} & \textbf{Factors that cause an increase over the whiteboard}& \textbf{Factors that are equal to the whiteboard}& \textbf{Factors that cause a decrease to the whiteboard}\\
\hline
\endhead
\hline \multicolumn{4}{r}{\textit{Continued on next page}} \\
\endfoot
\hline
\endlastfoot
Abstraction	
& %increase
Scraps help visualize abstractions and make them manipulatable
& %equal
You can draw anything
& %decrease

\\
\hline
Closeness of 

mapping	
& %increase
Can manipulate notations at the level of scraps and connectors
& %equal
Can draw in any notation
& %decrease
Annotations, like cardinality, do not move along with connectors
\\
\hline
Consistency	
& %increase
Can save and refer back to previous sketches, on whiteboard they may have been erased
& %equal
Must be socially enforced
& %decrease
Difficult to socially enforce when content is diffuse and requires hard mental operations, as opposed to whiteboard where content is all in one space
\\
\hline
Diffuseness	
& %increase
Calico breaks up the whiteboard space across several canvases
& %equal
Small designs stay on one board
& %decrease

\\
\hline
Error-proneness	
& %increase

& %equal
No error checking
& %decrease
Manipulating and automated copying of existing sketches may prevent errors caused by redrawing sketches
\\
\hline
Hard mental operations	
& %increase
Design may be diffuse in Calico and details of sketches scattered across many canvases
& %equal
Small designs are contained within one area
& %decrease
Content can be imported into Calico using image scraps; content can be positioned side-by-side using scraps; palettes allow users to copy content for reference across canvases; navigation button allows user to return to previous content quickly
\\
\hline
Hidden dependencies	
& %increase
Content may be diffuse, making hidden dependencies hard to find
& %equal
Small designs have all potential references in one area
& %decrease
Intentional interfaces helps manage diffuseness by grouping content together to make hidden dependencies easier to find
\\
\hline
Premature commitment	
& %increase

& %equal
Medium is considered provisional; rejected decisions can be preserved by crossing them out rather than erasingor deleting them
& %decrease
Easier to explore alternatives by copying existing ones using scraps, palette, and intentional interfaces
\\
\hline
Progressive evaluation	
& %increase
All sketches are preserved and can be returned to; fading highlighter allows evaluting without marking the original sketch
& %equal
Users must manually judge the ``rightness'' of a sketch by manually reviewing it
& %decrease
Diffuse designs are harder to progressively evaluate; fading highlighter marks during an evaluation cannot be later referred to as regular marks could be on a whiteboard
\\
\hline
Provisionality	
& %increase
Users can make copies and explore alternatives without losing original ideas; more sketches can be reviewed across canvases in intentional interfaces; fading highlighter allows sketching over design without permanently marking it; canvases can be tagged as alternative using intentional interfaces
& %equal
Medium is considered provisional by users; anything can be changed at any time
& %decrease
More difficult to evalute sketches diffused across canvases; marks from progressive evaluation no longer remain and cannot be referenced
\\
\hline
Role expressiveness	
& %increase
User has unlimited space to fully express all parts of a design
& %equal
User is limited by the space available
& %decrease
Designs spread across canvases have higher diffusion, lower visibility, and may require hard mental operations to reference other canvases
\\
\hline
Secondary notation	
& %increase
Second notations can be attached to scraps and move with scraps; custom shape of scrap can be used; color of connectors can be used to capture meaning
& %equal
Users can sketch in any secondary notation
& %decrease
Annotations outside of a scrap do move with a scrap and the annotation is left behind
\\
\hline
Viscosity	
& %increase

& %equal
Users can sketch anything they want; a full whiteboard is less likely to be changed because it contains many important decisions
& %decrease
Scraps make sketched content not ``locked in'' such that can easily be moved and manipulated; gestures make manipulating scraps a fluid action
\\
\hline
Visibility	
& %increase

& %equal
Sufficiently small designs are entirely visible on the whiteboard
& %decrease
Designs diffused across multiple canvases have very low visibility, leading to hard mental operations
\label{table:discussion:cognitivedimensions}
\end{longtable}
\end{center}

\subsection{Abstraction}
Abstraction refers to Calico's ability to support representations at different levels of abstraction. Since Calico at its most basic supports plain sketching, it inherently allows maximum flexibility in what can be represented, much the same as the traditional whiteboard. 

Calico, however, improves on the regular whiteboard because users can manipulate those abstractions at the level of scraps or, when using intentional interfaces, entire canvases. On the whiteboard, the designer can only draw and erase strokes. In Calico, they have scraps to depict abstractions, which support quick movement of the drawn abstractions, implicit grouping by stacking scraps, and have arrows that remain attached to the scrap as it is moved. Intentional interfaces enables the user to explore different levels of abstraction across multiple canvases, and supports the user in navigating between these levels of abstraction by linking canvases. 

When the content of a design session is suffuciently small, the whiteboard may have a better expereince than Calico because all abstractions are available in one space, while in Calico they may be diffused across several canvases. When on a single whiteboard, the designer can immediately look at all levels of abstraction at once. If the content of the design grows beyond what can be sketched on the whiteboard, then Calico becomes more useful, as it can capture all sketched content across several canvases, whereas on the whiteboard that content would need to be erased, or scattered across multiple whiteboards, posters, or pages. 

\subsection{Closeness of Mapping}
%You can draw in any notation. Factors that increasing is that you can manipulate notations at the level of scraps and connectors. factors that decrease it is annotations like cardinality... they don't move along with connectors.

%Very similar to the whiteboard. Compared tot he whiteboard, all that is preserved. Now I've given people the way to manipulate them as wel.

Closeness of mapping refers to how closely representations map to the referred concepts. Calico is similar to the whiteboard in this regard in that designers are at liberty to sketch any representation. If the designers decide that there is a more suitable representation, they are able to create a new sketch that depicts that representation simply by drawing its corresponding shapes. These shapes, then, can be turned into scraps when so desired. 

When using scraps, Calico has the positive consequence that representations can be manipulated as objects, which may more closely map to the mental model of the designer than plain strokes would. Further, where on the whiteboard representations may need to be erased and redrawn to create a sketch that closely matches the desired notation, scraps are reused by copying them. 

A negative consequence of using scraps is that certain visual visual annotations, do not move along with scraaps and connectors. This may cause sketches to become less closely mapped to their concept as cardinality annotations may become disassociated from their target sketch. This may cause additional work as users will need to manually move annotations with scraps, or simply leave the annotation behind after the scrap is moved. 

%
%- people could get very close to what they are representing. Importing screenshots into the environment is a big one.
%- interaction designs could manipulate their objects directly, no indirect names
%- software people could import snippets of code
%- researchers imported screenshots of user interfaces, source code, etc.

\subsection{Consistency}
Consistency refers to the degree to which features of structures and syntax are used the same way throughout. Calico does not provide an automated method to enforce consistency. As with sketching, maintaining consistency between elements and sketches is up to the users to ``socially'' enforce \cite{Petre2013BookChapter}. If all content can be condensed to a single whiteboard, the designer can refer back to previous sketches to check for consistency. On Calico, since the designer is encouraged to use multiple canvases, it may be more difficult to manually verify consistency across all canvases.

However, if the design is too large for a single whiteboard, Calico would provide some help over the whiteboard. Where on the whiteboard sketches would need to be erased, the designer may save them in Calico, and refer back to them to verify consistency between old sketches and new.

%- consistency brought by calico comes from copying, none beyond other, unless enforced by personal discipline.
%- palette supported this, but seldom used this way
%- multiple perspectives enforced consistency,
%- researchers used multiple documents, hard to maintain consistency across those

\subsection{Diffuseness}

%There's a cost with Calico - it makes things more diffuse. By taking the original whiteboard, cutting it into pieces, I've actually made it a little bit harder (speaking of Calico). 
%
%Counter argument is that people work with it different. They partition it differently. The jury is out. 
%
%Can't say there's moderate support for diffusement, instead say there's moderate support for reducing diffuseness.
%
%I don't have as much as much space in an individual Calico canvas as the whiteboard. If everything can be sketched on the whiteboard, it's superior because I can see everything at once. However, if the design grows beyond the space of a whiteboard, diffuseness becomes obliviousness on the whiteboard because what is not immediately visible has been erased. Calico incrementally becomes more important because diffuseness on the whiteboard has become obliviousness, sketches are no longer exist, but in Calico they do. However, in Calico there is now the navigation problem among these things. The structure of intentional interfaces is one way we address that. It is unclear... it doesn't erase diffuseness, it doesn't moderately support it, but what it does do is acknowledge it, and we have built features to address that. More research is required to determine how well the features combat diffuseness, the support is there.

Diffuseness refers to how much the meaning of sketches is spread out across multiple sketches. Calico potentially increases diffuseness because the potential for an unlimited number of canvases may encourage designers to break up their designs across multiple canvases. On the whiteboard, designers are forced to limit their design sketches to the limits of the whiteboard.  If everything can be sketched on the whiteboard, then it is superior because designers can see everything at once. 

However, if the design grows beyond the space of a whiteboard, Calico's functionality helps in dealing with diffuseness. On the whiteboard, diffuseness becomes obliviousness because what is not immediately visible has been erased. Calico incrementally becomes more important because, while on the whiteboard sketches no longer exist because they have been erased, in Calico they remain. Yet, this raises a new issue, which is the problem of how to navigate between sketches in Calico. The structure of intentional interfaces is one approach to managing the diffuseness. It is unclear if intentional interfaces effectively addresses diffusenss, as it does not remove it. What it does do is acknowledge it, and gives the user tools to work around manage it themselves. More research is required to determine how well the features combat the consequences of increased diffuseness.

%Further, enabling one to copy canvases and transfer content using the palette encourages diagrams to be diffuse. However, intentional interfaces mitigates these issues by helping to navigate designs diffused across different canvases. By providing links between those  canvases, users can more easily find content immediately relevant to the current canvas. Intentional interfaces does not provide a fully comprehensive solution to diffused diagrams, but it does moderately support it in a lightweight fashion.

\subsection{Error-proneness}

Error-proneness refers to the degree to which a notation induces ``careless mistakes'' \cite{Petre2013BookChapter}. As with plain sketching, Calico does not provide any safe guards against errors. Individuals are free to improvise and switch between notations, which may increase their tendency to perform errors if there is no clearly defined standard notation. 

While Calico does not alert the user of when they have performed an error, Calico's features do automate certain actions which may reduce the likelihood of errors. Scraps and connectors, for example, provide flexibility in creating representations by enabling the moving, resizing, and rotating existing sketches. They maintain the structure of objects as they are moved, which may help when working with box-and-arrow diagrams. In comparison to plain sketching, scraps and connectors help prevent errors by making additions to representations, like process flow diagrams, less tedious so they do not require diagrams to be redrawn when repositioning elements, as one would have to if sketching at the whiteboard. On the other hand, designers recognize that there is value in redrawing a sketch from scratch because each time a sketch is redrawn, its contents are re-evaluated \cite{petre2009insights}, and copying existing sketches removes this opportunity.

%The research group benefited from the use of scraps, which helped avoid errors by removing much of the tedium in working with their state diagram. When working with their state diagram on the whiteboard, they reported that it had become too large to manage. After moving to Calico, they had more flexiblity in managing their space. They were able to create more space by moving text-scraps, which moved all connectors with the scrap as well and retain the shape of the diagram. Further, the flexiblity to create connectors with custom paths, as opposed to straight lines, made connectors more legible. 

%Overally, scraps and connectors helped prevent errors by making adding additions to the state diagram less tedious, and not requiring diagrams to be redrawn to reposition elements.

%No safeguards against errors, other than what was socially enforced by the designers.
%
%- repeated elements helped avoid using the wrong elements, and reviewing content in meetings helped catch errors


\subsection{Hard Mental Operations}
%By making it electronic, and breaking it into pieces, you gain, but you lose something too. On the whiteboard, when people start to erase, you gain.

Hard mental operations refers to information that needs to be referenced or is nested. Similar to the cognitive dimension of diffuseness, in a simple or small design, both the plain whiteboard and Calico are equal in that all parts of the design are visible and there may be little need to reference outside information. However, in larger projects, Calico may have more hard mental operations because the design may be diffused across several canvases requiring the user to reference sketches outside the immediate space. Users may need to use the intentional interface feature to move between canvases to reference other sketches, or use the palette to move content so that it can be referenced in the active canvas. For a large design on a plain whiteboard, content may need to be erased, and Calico may provide better support because that content would be preserved.

Calico further reduces hard mental operations in other sutiations. It allows users to both import content and copy content to the canvas they are currently in. First, image scraps allow users to import existing artifacts into a canvas to refer to while sketching. For example, a developer may import source they are working on, or a diagram created in another tool. Second, scraps allow users to move content so that it is positioned side-by-side to what they are currently working on. Third, the palette allows users to copy content between canvases so that they can reference old content in the new space. Fourth, the navigation button in intentional interfaces allows users to return back to the most recently visited canvas. While this may still quality as needed to refer to parts of the design outside of the immediate visibility, the feature reduces the effort needed to do so.

%In the case of the OSS group, the developers needed to reference information that was spread across several canvases. Members from their team reported that they did not need to do so often all members were already familiar with the depicted system. When they did need information to be available more immediately, they copied content onto the target canvas. The developer working with source code did so by pasting screenshots of his code into his canvas. In the case of the interaction designers, they pasted pictures of people they interviewed, however they needed to reference their notes from their interviews. The researchers, however, had a greater need to perform hard mental operations because of the scale of their state diagrams. The researchers who originally developed the system eventually internalized the diagram to a degree that they no longer needed to reference it, however the researcher that needed to be onboarded created additional diagrams, such as tables, in order to help him mentally step through the state diagram. In his case, he mitigated the need to reference other canvases by copying the pieces of the state diagram that he needed into his own canvases.

%Overall, the ability to copy content within Calico somewhat reduced the need to reference other canvases. Scraps, intentional interfaces, and the palette were helpful for both importing screenshots and copying content to the immediate working area.

%How often did designers need to reference elsewhere and nest?
%
%- the OSS group had to incur hard mental operations by referencing across diagrams because of multiple levels of abstraction
%	- they moved to high levels of abstractions, and use cases
%	- hard to interpret what was there because had to keep track of everything
%- the interaction design group had to do lots of hard mental operations because interviews not represented. Also placing along axies, and mult. perspectives
%- researchers experienced a lot of hard mental operations because of the terseness of the diagrams (paths, etc.)

\subsection{Hidden Dependencies}

%On the whiteboard, everything is there. I can glance over on the whiteboard, but I can't in Calico.

Hidden dependencies refers to representations that are dependent on one another, but the dependencies are not visible because they are either not declared or not in a person's field-of-view. As observed in past studies \cite{dekel2007notation}, sketches that cross several spaces result in having dependencies on other spaces that are not explicitly stated. Sketches that begin in one canvas in Calico may extend onto another canvas, or across mediums, such as their own computer or paper, if the users are using Calico as a complementary tool. This could occur, for example, when canvases depict different perspectives or abstractions. 

The intentional interfaces feature in Calico helps in declaring and finding these hidden dependencies on other sketches by providing features that group canvases of related content. Clusters in intentional interfaces help group canvases into sets of similar topics. Linking canvases into chains provides a second level of grouping that help designers in finding related sketches. These features do not solve the issue of revealing all hidden dependencies, but they do improve working with them by by making those dependencies easier to find.

\subsection{Premature Commitment}
A notation or representation that causes a user to \textit{prematurely commit} means that it requires a person to make a decision before they have the information they need \cite{Petre2013BookChapter}. Sketches and content produced in Calico are typically used to support conversations or thinking through a design, and thus there is less of a concern to prematurely commit because all content is provisional. The sketchy appearance of content within Calico visually re-enforces the notion that content is provisional. Also, the ease of changing content, such as drawing an ``X'' symbol over rejected content, allows for decisions to be rejected without permanently discarding the idea. In this regard, Calico is equivalent to the whiteboard which also has these qualities of provisionality. 

However, Calico further reduces premature commitment in comparison to the whiteboard by making sketches more easily copied and reproduced. Features such as scraps, the palette, and intentional interfaces make it easier to explore alternatives. For example, while on a whiteboard, a user may be hesitant to erase a large figure or deviate because they do not want to lose previous work. In Calico, they can simply create a copy of it in a new canvas, or continue their work in a new blank canvas.

%Members from all groups reported not feeling pressured to commit to decisions within Calico. The OSS group reported that they valued the ideas and tasks generated from using Calico rather than the sketches themselves. They reported that they sometimes immediately revisited sketches after meetings in order experiment with alternatives before returning to their desk to implement a new change. The OSS group did email snapshots of sketches from Calico to themselves, but reported that these emails were archival and served to remind them of the ideas generated. The ideas were not considered committed until they were implemented into the system. In the research group, the members used the sketches from Calico in support of implementing their software system and writing an academic conference publication. One of the researchers remarked that nothing in Calico was permanent until ``it was written in the [academic conference] paper''. 

%Overall, the provisionality of content and the ease with which it can be changed, i.e., low viscosity, lowered the tendency for content to be prematurely committed in Calico. However, there is very little gain in this regard over the whiteboard. The ease of copying content reduces the tedium in exploring alternatives, but this was only marginally beneficial for the groups with respect to preventing premature commitment.

%Calico provided a sketching look and feel to elements. Users consciously commited to decision by converting handwritten text into rectangular scraps and sometimes formal connectors.

%- there is not a sense of commitment to what is on the board.
%- research group turned to calico/whiteboard because it gave the sense of freedom from commitment

\subsection{Progressive Evaluation}
Progressive evaluation refers to the ability to run a simulation or obtain feedback on a representation that is only partially complete in order to determine its correctness. While Calico does not directly support progressive evaluations, it does support users in manually judging the ``rightness'' of a sketch by supporting the reviewing and explanation of sketches. Much like the whiteboard, designs may be manually reviewed and discussed by close inspection. 

With large designs, Calico provides some additional benefit with intentional interfaces, which aids users in navigating between canvases to review them. However, as the sketch grows too large, it becomes progressively more difficult to manually evaluate all sketches. Further, the collaborative aspect of Calico allows remote designers to obtain feedback, during which the fading highlighter would help during group reviews to discuss content. In local designs, the fading highlighter further helps progressive evaluations by making the sketches cleaner, evaluative marks are not left behind using the fading highlighter. However, this could be considered a loss in comparison to the whiteboard as well, as designers could potentially refer back to these marks to remind them of their evaluation.

%The intentional interface feature supported the OSS group in reviewing their contents by allowing them to zoom out and switch between canvases quickly. The OSS group reported that they would switch to the cluster-view, zoom in to the canvases they had been working on, and pan while discussing them. During group meetings, they further used the fading highlighter during discussions and evaluations of components, drawing paths taken by data between components during their explanations. The researchers, similarly, engaged in verbal dialog to review their designs. The researchers engaged in review of their designs, particularly the state diagram that summarized their software system. They stepped through the state diagram to verify its correctness, and also used it in support of evaluating the completeness of their software system. They used the sketch to keep track of what was and what was not implemented, and updated their state diagram with this information.

%Overall, Calico supported individuals in performing a manual evaluation of their designs. In this regard it could be considered similar to the whiteboard, but intentional interfaces provided the benefit of reviewing multiple canvases, and also the quality that Calico was an online shared resource that was shared by everyone. 

%- all groups used intentional interfaces to review, summarized, and proceed. different perspectives

\subsection{Provisionality}
Provisionality refers to the quality that indecision or options can be expressed \cite{Petre2013BookChapter}. Given that sketching is the primary mode of operation in Calico, individuals are free to break away from formal notation to declare multiple alternatives. Users can always erase a sketch to change a value, or simply cross out a sketch so that a design can be rejected without visually removing it from the design space. Calico goes further in this supporting provisionality by supporting the quick generation of alternatives using duplication features such as scrap copying, canvas copying, and the palette. With intentional interfaces, uses can explicitly express entire canvases as being an alternative by tagging it as such within intentional interfaces.

%The OSS group used the fading highlighter to propose ideas about sketches that they were not sure of yet. After discussing a box-and-arrow diagram using the fading highlighter for more than twenty minutes, they changed elements within that diagram. They did so by drawing a large ``X'' over the component, and wrote the name of an alternative component next to the component with an ``X'' on it. In another instance, members in the OSS group used intentional interfaces to create a copy of a canvas, and tagged the canvas an ``alternative''. They did so on multiple occassions, such as during a group discussion, and also in individual design sessions. The research group expressed options in another fashion. In their case, they performed a major refactoring of their system, effectively diverging from the design in their previous sketches. Rather than deleting these old sketches, they moved to another cluster to sketch the design of their refactored system. They considered the contents of the old cluster as an archive of the first version of their system. 
 
%Overall, Calico provides some improvements in provisionality over the whiteboard. Within a canvas, Calico is much the same as a whiteboard in that the user has the flexibility to sketch alternative names. Across multiple canvases, intentional interfaces helps manage options at a larger scale by providing tags to label canvases as alternatives. Further, the ability to copy and create new canvases and clusters encouraged more sketching, and possibly exploration of alternatives.

%Everything in Calico was considered provisional and outside the formal specfications. Provisionality was reduced by copying content onto another medium. Teams walked away from designs, and return to them in order to be reminded.

\subsection{Role Expressiveness}

Role expressiveness refers to a reader's ability to see how the parts fit into the whole design, and how those parts relate to one another \cite{Petre2013BookChapter}. In other words, a reader can independently interpret what each part in a sketch does without further information. Given that users have the flexibility to sketch any notation they need, the limiting factor in both the regular whiteboard and Calico is the space available to create representations. With enough space, the role of each part can be better understood if all details are visible. On the whiteboard, this means that a sketch must be sufficiently small. In Calico, this means that there is a virtually unlimited amount of space, but the user must then content with a high amount of diffusion and low visibility. Intentional interfaces provides the means to manage the space needed to explore a design across multiple sketches to depict all parts of a design.

With Calico, there is the potential to address this issue using intentional interfaces by using several canvases to include all parts of a design. However, there is a further issue. Within the context of the informal design activities that Calico supports, designers typically only draw as much as they need to, which results in partial diagrams with minimal detail. As such, designers may create representations that have poor role expressiveness due to minimal detail. Very often, the original designer may need to be consulted in order to understand their sketch. However, content in sketches may be easier understood by looking at related content and hidden dependencies, which the intentional interfaces feature may help the designer do by navigating between canvases linked into chains. 


\subsection{Secondary Notation}

Secondary notation refers to the use of formalisms that deviate from primary established notations. Because of the freeform nature of sketching within Calico, users have the flexibility to overlay additional detail in sketches and use improvised notations as needed. Within Calico, users may choose to represent concepts using scraps with connectors as an alternative to basic sketching, in which case they can use the shape of scraps, as well as the colors of connectors to overlay meaning. Users may add annotations to scraps, such as drawing a symbol in the corner of a scrap, text content in the scrap with a color that has an assigned significance, or by using connectors in creative ways.


\subsection{Viscosity}

Viscosity refers to a medium's resistance to change. Similar to a regular whiteboard, the contents in Calico exhibit a low viscosity because content can always be erased and redrawn in any manner. 

Calico has three features which potentially reduce viscosity in comparison to plain sketching: content is moveable, gestures make manipulating content faster, and content can be copied. First, all sketched content can be moved using scraps, as opposed to the whiteboard where it must be redrawn. The capacity to move content means that sketched content is not ``locked in'' like a whiteboard, but instead content is less viscous because it can be adjusted. Second, with sufficient familiarity with scrap gestures in Calico, moving content can potentially be done quickly with minimal effort. A user may be less likely to modify content if accomplish that task requires a great deal of effort. In Calico, users do not need to switch modes or move their hand away from sketches, allowing them to move content fluidly without distracting them away from plain sketching. 


\subsection{Visibility}

Visibility refers to how easily all parts of a design are visible, or at least how easy it is to juxtapose two parts of a design \cite{Petre2013BookChapter}. As with a whiteboard, a design is highly visible so long as it is small enough to fit within a single space. In Calico, designs that are diffused across multiple canvases will have much less visibility than designs that are limited to a single canvas. The intentional interfaces feature attempts to address this issue by making it easier to navigate between canvses, and also to zoom out to the cluster view and view all canvases at once, albeit at a low-detailed perspective.

%Also, juxtaposing parts of a design is easier in Calico due to scraps, the palette, and intentional interfaces. Scraps allow users to reconfigure a diagram such that two parts can be placed next to one another by moving them or copying them. The palette allows parts of the design to be copied to other canvases to compare them. Intentional interfaces increases the ease of moving between parts of the design, and organizes the canvases so that it is easier to step through a design. Further, two canvases can be placed next to one another in the cluster view for comparison.

\subsection{Summary}

Overall, the major benefits of Calico over the whiteboard are that intentional interfaces allows more content to be preserved whereas on the whiteboard it would be erased, and the fluidity of manipulation offered by scraps. The benefits make content less viscous, more provisional, promote secondary notations, role expressiveness, and reduce error proneness. What is lost in these features is that designs may be more diffuse, and because of that diffuseness, require more hard mental operations, have less visibility, more hidden dependencies, and be harder to progressively evaluate. For each benefit gained through features, Calico loses the simplicity of the whiteboard, but attempts to mitigate these losses through the other features.

\section{Overall strengths and weaknesses}
\label{discussion:overall-strengths-weaknesses}

Having examined the features from a practical perspective in Section \ref{discussion:strengths-and-weaknesses} and at a theoretical perspective in Section \ref{discussion:cog-dim}, this section examines the overall strengths and weaknesses of Calico as a whole.

\subsection{Strengths}

\textbf{Strength: It supported all design behaviors}

All design behaviors were performed by users in the field at least once using Calico's features. While not all Calico groups performed every design behavior using the feature, for example only the research group used intentional interfaces to retreat to previous designs, they did use the features in their own work. Some features were used to perform design behaviors often, such as navigating between canvases using intentional interfaces, and creating low-detail detail diagrams using scraps. 

Some features were used rarely, but served an important purpose in those moments. The OSS group only engaged in refining their sketches from plain sketches to UML diagrams once, but in that moment it allowed them to transition from a freeform sketch to a more formal one. The research group also brought their work together using scraps to tag what they did not and did not implement in source code, but using scraps enabled the tags to remain attached to the scraps as the diagram was modified. Collectively, the features were flexible enough to address all design features without requiring the users to shift their context or shift to different modes.

\textbf{Strength: Supported them when they needed it as opposed to forcing it upon them.}

While all features were used to support their respective design behaviors, the designers only used those features when it benefited them. As the second design behavior suggests, designers will only draw, or use formal notations, as much as they need, e.g., they will only write a name to represent a component rather than the entire details of that component. This behavior extends beyond simply what they sketch, and onto their use of Calico's advanced features: designers will not use the advanced features when they do not need them.

Regardless how minimal the effort may be, each feature brings with it some cost that the user must pay in order to gain the benefit of that feature. Scraps enable rapid interaction with sketched content, but require the user to think in terms of objects and manually group them. Intentional interfaces enable the user to have more free space, but requires the user to think in terms of multiple canvases with low visibility. A user may not need the benefits offered by these features and may simply want to sketch without structure imposed on their work. A strength of Calico was that it allowed them to do that, but, when ready, to also convert regular sketches into scraps using the press-and-hold gusture, or partition canvases across several canvases by pressing the ``copy canvas'' button and gruop them using intentional interfaces.

%--> introduce quotes
%--> users could use modeling, but used free sketching. Many features were not used simply because they did not need them!
%--> refinement only happened once or twice, but when they used it, they needed it

\textbf{Strength: Supports a wide range of design}

All groups engaged in different forms of design, yet Calico supported all teams in performing their own work. For the OSS group, this meant designing new components and refactoring source code in a mature open source software system. In the case of the interaction designers, they were engaged in early phase requirements gathering by processing interviews which will be used to build personas for interaction design. In the third case, the research group, the design involved refactoring and designing the first version of a software tool in a globally distributed team. 

These groups used Calico for their designs, but created their own culture in using the tool and created different diagrams. The OSS group created narratives explaining how data is moved through their system using multiple canvases. The interaction design group organized and tagged scraps, and copied canvases to generate multiple perspectives. The research group carried on their sessions remotely, and sketched dense diagrams using scraps. Calico supported the groups during the entire lifecycle of their sketches, which includes brainstorming, sharing, refining, and archiving \citep{walny6069462}. The groups adapted Calico to their existing workflow by sketching or importing diagrams, working, and later archiving their work by emailing or referring back to old sketches in Calico.


%--> three different teams in three different situations
%
%--> It fits on top of existing design processes and tools

\textbf{Strength: Can create a variety of diagrams}

Across the design sessions, Calico showed that it could be used to create a variety of different types of diagrams. Some diagrams, such as box-and-arrow diagrams, benefit greatly from scraps because they could more easily be moved, related, and were space efficient. Other types of sketches, such as the image scraps placed along one and two dimensional plots benefit from being able to tag the elements with colors and move them along the plot to organize them. For complex diagrams, users can use plain sketching, but with scraps, they have a general purpose tool to group them and juxtapose them by moving them next to one another.


%--> they were juxtaposed
%--> they were different

\textbf{Strength: Presents a minimal hurdle for first use}

A strength in Calico's core design is that users can use its most basic features without training. New users unfamiliar with Calico may begin using it by dragging their finger across the board to create a stroke. The minimal overhead allows new users to begin working without the overhead of creating a new project, reserving a space, or investing themselves in extensive training. This is helpful when sharing or explaining a design to new users, who can begin participating in a design without any training. 

%Many of the feature are discoverable.
%people can walk up to it and use it

\textbf{Strength: Interaction is fluid}

Calico uses a minimal set of modes for manipulating content, which leads to more fluid experience than traditional digital sketching tools. Plain sketching is fluid because the designer is interrupted, at most, to reach for a new pen and can engage in a flow \cite{csikszentmihalyi2009creativity} with their content. The combination of scrap gestures for creating scraps and the bubble menu take a step towards the ideal standard of fluidity of the whiteboard by requiring fewer steps than traditional digital drawing tools to manipulating content. Feedback from the OSS group and the research indicated they prefered Calico's scrap interactions for manipulating content, but found it less precise than traditional drawing tools, while the interaction designers preferred traditional drawing tools for interaction.

\textbf{Strength: The basic interaction mechanisms are strong}

Overall, Calico contains a strong basic set of interactions. While Calico does not contain advanced drawing features such as shadows, blur, etc., nor does it have advanced modeling notations such as different types of arrows, shapes of boxes, etc., the groups did not find Calico lacking in basic features. Users valued the basic set of features in Calico, such as the eraser, undo/redo, colors, stroke thickness, and the set of Calico's advanced features. They found Calico's features sufficient to carry out their sessions, and found Calico's features natural to use after a period of time, in particular scrap gestures and the bubble menu. The same could not be said for the interaction scheme of Calico Version One, and users reported finding mode switching found in regular drawing tools awkward when used on the whiteboard.

\textbf{Strength: It allows for creative uses that were not expected}

A consequence of designing a flexible environment was that the groups used Calico in ways that were not envisioned in the original design of Calico. The use of Calico by a member of the OSS group to refactor their source code was a deviation from what was expected. Calico was envisioned to support the sketching and manipulation of abstractions, but the OSS group member found the writing on the electronic whtieboard good enough to write pseudo code and found the use of colors helpful as a secondary notation. The use of colors to tag scraps in both the interaction design group and research also came as a surprise. The use of Calico by the research group to keep track of what they had and had not implemented further corroborates Newell's notion that sketching serves as an external memory \cite{Newell}, but that specific use was not anticipated.

\textbf{Strength: Free space is always available}

Users could explore new ideas in Calico without having to erase old content in Calico. As mentioned in the previous cognitive dimensions session, this brings the advantage of allowing users to fully express their designs, but brings with it the problem of diffused diagrams that cause low visibility, hard mental operations, hidden dependencies, etc. While these negative consequences are concerns that should be addressed further, the value provided by easily available free space was an overall positive outcome for groups. During informal design sessions, the overhead incurred by needing to summarize designs within a single whiteboard may lead to ideas not being recorded. It is better that a designer need to search for a sketch they created, rather than not having the sketch at all.

%A limiting factor in many sessions is the amount of free space that designs have to sketch.
%
%Of course, the more sketches that people create, the more diffuse their diagrams are.

\textbf{Strength: Narratives can be created using canvases}

The OSS group and one member from the research demonstrated that intentional interfaces is useful for creating narratives from sequences of canvases. Similar to how designers may refine sketches by converting lists into UML class diagrams, members from the two aforementioned groups manually reorganized the order of their canvases using canvas links in order to create a story from their sketches. These groups later shared their sketches, and compared the act of stepping through these canvases to giving a Powerpoint presentation.

\textbf{Strength: Sketched content is always available}

A quality of Calico that members from the OSS group and the research valued was always being able to access their canvases. Group members may not always have access to the physical equipment, and being able to access the content on their own computers was helpful. A member from the OSS group stated:

\begin{myindentpar}{1cm}
\emph{``The fact that someone can work with their own tablet or computer, like [a member of the team] did with his alternative view, is something really powerful to do. If you try to do that in real life... we're blessed with two whiteboards, but some companies may only have one. It makes it harder to do that. Especially when someone is already at the whiteboard discussing something, and you want to bring in an alternative perspective but you need to wait until they're done doing whatever they're doing. That definitely makes groupwork easier.''}
\end{myindentpar}

The remote member of the research group further reported that being able to access the content from the weeklong intensive design session was helpful. 

%Important in the shared OSS setting. digital documents made content always available, but this made content freely available in its physical equivalent setting.

\textbf{Strength: The system is distributed}

Having synchronous collaboration be a fundamental feature was a strong point of Calico in the environments of the OSS and research groups. For the OSS group, this led to the scenario that members could sit around the electronic whiteboard and participate from a couch with a tablet in their lap. Members provided feedback in meetings informally by handing tablets back-and-forth and sketching over the diagrams, having their annotations displayed on the large electronic whiteboard. With multiple tablets, multiple team members could talk simultaneously without a single arbitor at the whiteboard blocking content, which is what traditionally occurs at the regular whiteboard \cite{Shih2009}. For the globally distributed research group, the consequence was that remote members participating using Skype could directly contribute by manually sketching their ideas, rather than remaining a passive contributor.

\textbf{Strength: Users are not locked in to synchronous work}

A person at the whiteboard that wants to write something, but cannot, may ``spin their wheels'' and ignore conversations until they can externalize their throught \cite{Olsonb}. A strength of Calico is that users can branch off into their own canvas, either by using creating a new canvas or copying the one they are actively using, and explore their own ideas independently. In meetings of large groups, this frees members to generate their own alternatives when inspiration strikes them, and later request everyone to join his canvas so that the designer may explain their alternative design.

%--> copy your work and do my own thing, then call you over
%--> synch and asynch, and how easily that's handled

\subsection{Weaknesses}

\textbf{Weakness: Juxtaposition sketches on different canvases is difficult}

A weakness in Calico is that sketches across canvases are difficult to juxtapose. Feedback from users indicated that they wished to refer to content from other canvases. In order to refer to past sketches, users either used the palette to copy sketches so that they are immediately available for reference, or they used the navigation buttons to rapidly jump between previous canvases and the canvas they are currently in. These other features were used to compensate for the inability to place sketches side-by-side.

%--> People want to put things side by side, but right now they can't.


\textbf{Weakness: Users that do not invest in organizing canvases will experience difficulty in later locating their work}

In previous work, there is much evidence that the Calico grid helps in design. It provided an intuitive interfaces to organize and partition work between canvases. We believe that  intentional interfaces is a step forward from the grid in the right direction for supporting design, however this feature has not had the opportunity to be polished and iterated on to improve its usability. One weakness in this iteration of intentional interfaces is that users that do not actively manage their canvases will likely have difficulty in knowing where their sketches will be located in the cluster view. Users that manually manage their canvases do so by linking their canvases using tags. Users that do not manage their canvases create radial clusters, with no name, that are difficult to identify. Future iterations of intentional interfaces in Calico will need to address this issue.

\textbf{Weakness: Reviewing a large number of canvases is tedious}

Moving across several canvases using the cluster view is a slow experience using Calico that can become tedious. A consequence of creating diffuse diagrams with many canvases is that the cluster view will need to adjust the size of canvases in order to display all canvases. The resulting effect is that images becoming too small to distinguish when there are a large number of canvases. Other features help to mitigate this problem, such as using the navigation buttons, and using the breadcrumb bar. However, members in the OSS group reported that, despite the slow interaction, they prefered to use the cluster view with previews to review sets of related canvases.

%In the current implementation, it makes images too small (intentional interfaces).

\textbf{Weakness: Does not follow traditional interaction patterns, requires users to learn new gestures to manipulate objects}

While the basic sketching functionality of Calico can be used by new users without training, the advanced feature set of Calico may take new users some time and practice to become proficient. A further issue is that users may approach Calico with a pre-conceived set of affordances that is based on their experience with other touch based devices. For example, members of the OSS group expected the large electronic whiteboard to use the same gestures as their mobile phone, while the interaction designers expected it to behave like Microsoft OneNote. Calico introduced another set of interactions that the users needed to learn. 

\textbf{Weakness: It stops short of something that becomes an actual notation}

While Calico supports users in creating approximations of formal notations such as UML, it stops short in delivering the behavior and expressive power of a formal modeling tool. A strength of Calico is that it supports working with informal notations and refining notations into more formal approximations by creating scraps and linking them with connectors. However, a designer may reach a point in which they are ready to add formal detail to their diagrams, such as double headed arrows, cardinality, and other formal relationships. While Calico does not currently provide support for the more formal elements, such support has been observed as being designed in late-stage designs, after a designer has committed to their design and begin recording lower-level, detailed design decisions.

%--> While we can do this and this, and seem like a UML diagram, the people will actually want UML functionality. We need to find a way to fold that in
%--> additional notational detail
%--> they want it near the end. It is a UML diagram, it is an interface diagram
%--> Up until I was doing something, then I'm doing a different type of design. While Calico is very good at making that transition from just sketching (words on paper) to actual diagram, it falls short in giving you the full notational support for that diagrm
%--> this issue plays out in a different sense. Meetings happen at different levels, in some meetings the transitions happen

\textbf{Weakness: The setup is not lightweight}

A problem observed in the setup of Calico was that, in order to use Calico, the system needed to be prepared and maintained by an administrator. While this is not specifically a weakness of Calico itself, it is highly relevant to Calico because, based on my observations, individuals will not use Calico if there is a delay in using the system. Instead, they will turn to using a normal whiteboard or a pad of paper. 

In practice, the large electronic whiteboard needed a dedicated system that was always running so that each group could begin using Calico simply by turning the attached projector. In order to launch Calico on a personal tablet, the tablet required the installation of the Java Virtual Environment, which at times delayed meetings when new members needed to install their software. However, once group members performed the initial setup of Calico on their own machine, connecting to the Calico server became a more lightweight action.

% what else do I see
% feel free to remind people why this matters

\section{Interviews with users}
\label{discussion:interviews}

%In the previous sections, I found that Calico's features adequately support the broad set of design behaviors and discussed the strengths and weaknesses of those behaviors. In this section, I corroborate those findings with a deeper discussion of the features. I do so by first performing a cognitive dimensions analysis of Calico's features at a hypothetical level. I then review the feedback from the interviews.

% The interviews is mostly okay, just need to make it consistent

%//I've already made my interviews, now it's just my opinions
%It's clear that each group is using Calico very differently. They're finding different strangths and weaknesses. Interactions designers... not having an infinite canvas. When push came to shove, they made a lot of use of Calico. 
%There is the open source group, they didn't make big use of scraps, but they did represent code. Not our intention, but big use. One of the members used code because he had a large whiteboard.
%The research group. One of the guys did a lot of prep work.
%//the important part is that there is a lot of different uses of Calico, and Calico held up. All designs were created. 
%//Design group stated that not having an infinite canvas was a hurdle
%//what we got from the researchers were comments like this:
%//what we got fromt he oss group were like this:
%//if you pull out real quotes, it's stronger.
%//roll context into discussion. 
%
%Characterizing each session

In the previous section, I examine the capability of Calico's features to support design from the perspective of a cognitivie dimensions analysis. In this section, I discuss the feedback received on the features from the participants interviewed in each group.

From the interviews and observations, what is clear is that each group used Calico very differently. Each group used Calico to support different activities, with different amounts of people, and in different settings. These groups subsequently found different strengths and weaknesses in Calico.

The interaction designers used Calico to simulate activities that would have normally involved physical artifacts over a physical whiteboard. As such, many of the comments that came from the interaction design group related to Calico's strengths and weaknesses as they compared to the physical artifacts that the interaction designers normally used. For example, a strength that the interaction designers found was its ability to import images as image scraps and arrange them as they would have on a physical whiteboard. Explaining why this was important, one member of the interaction design group stated, 

\begin{myindentpar}{1cm}
\emph{``It's really hard to say `that person or this person'. Much easier to speak to a face.''}
\end{myindentpar}

 They found that Calico's strengths were those that enabled actions not easily done with physical artifacts. For example, replication of artifacts encouraged the interactions designers to explore more perspectives than they would have, had they used actual physical artifacts. Calico's features further allowed them not only to arrange the images themselves, but also the sketches generated around those images, in which they could arrange and copy written text to organize the whiteboard. 

However, they found some experiences in Calico still not on par with using the physical whiteboard. They viewed the gestures of moving scraps too slow, where one member of the interaction group stated that 

\begin{myindentpar}{1cm}
\emph{``the overhead in manipulating [scraps] was too much.''}
\end{myindentpar}

 Further, they found the space available in canvases being too small compared to the boards they normally use, stating that

\begin{myindentpar}{1cm}
\emph{``[we] [were] blocked by the physical limitations of the [electronic] board.''}
\end{myindentpar}
 
Despite these obstacles, they overcame such obstacles, making dozens of canvases with sketches and scraps. They made real progress in a real design task within their professional work.

The OSS group took a different approach to using Calico, making use of it in group design discussions of three or more people, and also personal sessions. A strength of Calico in this setting was to increase their capability to share their meeting space. For example, intentional interfaces encouraged members to brainstorm more, where one member of the group states, 

\begin{myindentpar}{1cm}
\emph{``we're more willing to draw any random thing on there because we know that we can erase it, or go to a new canvas at any point. I would say more random ideas get thrown on there.''}
\end{myindentpar}

The OSS group in particular took advantage of free space to create dozens of canvases. For example, they consecutively chained canvases within intentional interfaces to generate sets of canvases that together formed narratives, such as how data steps through software components, or how a user interface behaves. 

Some of their use was also unexpected. The sessions involving source code came as a surprise, as I did not expect the developers to copy and paste screenshots of actual code into Calico, but these sessions became some of the most interesting, as they involved a great deal of impromptu notations to step through source code.

%``I had something where I needed to get my thoughts written out as far as I was trying to design...''
%``It was really useful to switch... I used a lot of color. It kind of helped me identify, like, which kind of objects are important. The other thing was that I wanted to look at the XML code that was relevant to this diagram, so I went on my machine, logged in over the web browser, took a screenshot of my code, and pasted it in to the canvas. That way when it came in over here on the board, I could look at the code from the screenshot, and do my work based on that. So that was very helpful.''
%``Having it in one page, one screen, was helpful.''
%``I ended up taking a different direction in the strategy I was working. So I just copied the canvas, I made an alternative, yeah, so that was nice. I made another screenshot of different code.''

The research group presented a third unique setting in which meetings were distributed, had a member that did a lot of design up front before the group design sessions began, and generated very large process diagrams that described the entire behavior of their software system. As opposed to the other groups, the research group members were more isolated from one another, and used meetings to review their work, and plan what to do next. The research group viewed one of Calico's strengths in their setting as being able to carry on meetings with remote members, where one member stated: 

\begin{myindentpar}{1cm}
\emph{``I think the biggest benefit was crossing space and time. Being able to pause and resume...the main benefit was that after I left, we were able to reference all those things that we created while I was there pretty easily.''}
\end{myindentpar}

They viewed Calico as more beneficial than photos of a whiteboard, stating that 

\begin{myindentpar}{1cm}
\emph{``you can't go back and edit that.''}
\end{myindentpar}

Further, one member of the research group prepared his work ahead of time for meetings, stating:

\begin{myindentpar}{1cm}
\emph{``I could have done this in a powerpoint as well, [but] it's much better [to have done it in Calico] because later when I show it, I can have people changing it, and in powerpoint you cannot sketch and draw.''}
\end{myindentpar}

This particular member further found the visual structure of intentional interfaces helpful when reviewing work from past meetings, stating 

\begin{myindentpar}{1cm}
\emph{``if you're designing a complex thing with stages and you're trying to tell a story, you can say: okay we've tried that, would you like to see all this path we went through?''}
\end{myindentpar}

Calico served as a virtual meeting space for the members that helped maintain continuity of meetings across the duration of the project.

%--- christian
%``we had one whole design that was thrown away''
%
%``designs get very complex... you want to keep a history of what you've done, the branches that you've pruned. Having a structure is essential. If you're designing a complex thing with stages and you're trying to tell a story, you can say: okay we've tried that, would you like to see all this path we went through? If you don't have the structure you'll have to create it somewhere else. If it's already here...''

The takeaway is that each setting introduced a unique way of using Calico. Each group had a different set of needs, a different setting and setup of people, and yet, Calico held up to a significant portion of their needs. Each group followed through with their design sessions and completed work in actual projects without abandoning their use of the tool. That is not to say that it was a perfect match for all groups. One of the teams, the interaction design group, did say that they would not continue to use Calico because its features did not match their needs, such as an infinite canvas and the system having difficulty with the number of images they used. This may either mean that their needs would be better suited by another tool, or further research is needed within the Calico environment.

Further, from the interviews, three observations stood out: (1) Calico did not interfere with their normal design process of the OSS and research groups, and despite difficulties, the interaction design group performed non-trivial design sessions, (2) Calico deployments still have issues that need to be addressed that are intrusive, and (3) that Calico led to positive changes in their design habits.

First, the OSS group and the research group reported that Calico and its features did not prevent them from carrying out their design as they normally would have, while the interaction design group reported that Calico somewhat got in the way of their normal design activity. The OSS group reported that they did not feel any loss of expressive control in using Calico in comparison to the whiteboard, and reported that they normally would have performed many of the same activities on the whiteboards available in the immediate area. The research group reported that they formerly carried out their designs on the whiteboard, and transitioned to using Calico for the same activities. The interaction design group reported that it closely approximated the whiteboard, but they found that the quality of sketch input forced them to write large and that the system lagged at times, which was distracting during their sessions.

%Did it prevent you from doing what you normally would have done?
%- quality of sketching on whiteboards was largest compromise. Tablets helped, but was not the same (1,2,3)

Second, there were some issues that were commonly intrusive among the three groups. Nearly all groups reported that the large electronic whiteboards diminished the quality of their own handwriting, forcing them to either write slower, or write larger. The OSS group connected to their Calico server using either tablets or their own computers, which they reported was preferred in producing clearer handwriting due to precise input. All groups used text-scraps to produce legible text as well. All groups also reported that launching Calico on their own machines was sometimes a hurdle. The OSS group reported that it was less of an obstacle for them because of the availability of tablets preloaded with Calico. The research group reported trouble setting up Calico on the laptops of new members of their group.

%Features that detracted
%- going against the grain... people are accustomed to their existing tools.
%- too slow to bring up
%- turned to word documents to supplement activity

Third, all groups reported some positive changes to their existing design habits. All groups reported that they had a sensation of having more free space, and reported that they created more sketches than they normally would have as a result. The research group reported that they created more complex sketches as a result of scraps and connectors, which helped them address a deeper level of complexity in their design. Both the OSS group and the research group reported that, since they did not feel a need to delete unused sketches, they returned to old sketches more often.

\section{Summary}
\label{discussion:summary}

Having presented my analysis of the experiences reported in Chapter \ref{chapter:evaluation}, I now return back to the research question in Chapter \ref{chapter:research-question}. The goal of Calico was to create a minimally invasive, small set of features that worked together to address the full set of design behaviors. Rather than create a new feature for each design behavior, I created a small set of four features that address all fourteen. What I have is a small set of features which work together to address all design behaviors.

Addressing the first part of the research question, the observations made thus far show that Calico's features were indeed minimally invasive in real world  design activities. Minor issues did exist that users found invasive, for example limited space in a canvas, poor quality of handwriting on the large electronic whiteboards, and slow movement speed of image scraps for the interaction designers. Despite these issues, all users were able to use Calico to perform actual work in real design sessions, and users saw Calico as improving their capability to design in comparison to the regular dry-erase whiteboard.

Addressing the second part of the research question, most of the features did coherently work together. Table \ref{table:discussion:minimally-invasive} succinctly summarizes the contributions of each feature to the set of fourteen design behaviors. Each feature supports multiple design behaviors, and they clearly are integrated, and few of the features stand alone. Scraps and the palette are very much integrated, where both work together in design behaviors 2, 4, 6, and 14. Scraps were also integrated with intentional interfaces, both supporting design behaviors 5, 6, 7, and 9. The fading highlighter, however, stands apart from the other features, supporting design behaviors 8 and 13. While it stands alone, it is powerful in what it does, and does so in a minimal and lightweight fashion.

\begin{center}
\begin{longtable}{|p{4cm}|p{4cm}|p{8cm}|}
\caption{The set of design behaviors and the features that support them}\\
\hline
\textbf{Design Behavior} & \textbf{Supporting Feature} & \textbf{Design Principles} \\
\hline
\endfirsthead
\multicolumn{3}{c}%
{\tablename\ \thetable\ -- \textit{Continued from previous page}} \\
\hline
\textbf{Behavior} & \textbf{Supporting Feature} & \textbf{Design Principles}\\
\hline
\endhead
\hline \multicolumn{3}{r}{\textit{Continued on next page}} \\
\endfoot
\hline
\endlastfoot
1. They draw different kinds of diagrams	&Scraps \& connectors	& + Supported lists, box-and-arrow, UI diagrams well\\
\hline
\multirow{2}{4cm}{2. They produce sketches that draw what they need, and no more}&Scraps \& connectors	& + Represented software components as scrap and connectors, very little detail included. \\
\cline{2-3}
&Palette 	& + Reused low detail scraps they created across canvases\\
\cline{1-3}
3. They refine and evolve their sketches over time	&Scraps and connectors& + Plane sketches could be made into scraps, have detail added to them, and add connectors\\
\cline{1-3}
4. They use impromptu notations	&Scraps \& connectors	& + Color took on meaning in connectors

+ Scraps were tagged/underlined with color\\
\cline{2-3}
	&Palette	& + Visual icons were created using scraps and used repeatedly\\
\hline
%	&Scraps and connectors &Elementary visual looks and behavior can be randomly composed	&\\
%\hline
\multirow{2}{4cm}{5. They move from one perspective to another}&Scraps \& connectors	& + Different notational conventions can be mixed and matched on a single canvas
\\
\cline{2-3}
	&Intentional Interfaces	& + Users can explicitly request a new canvas to work on a perspective

+ Canvases are linked and ordered, helping find related perspectives	
	\\
\hline
6.      They move from one alternative to another	&Scraps \& connectors	& + Different alternatives can be quickly constructed by copying and moving and otherwise manipulating scraps and connectors\\
\cline{2-3}
	&Palette	& + Different alternatives can be quickly constructed by reusing elements from the palette and composing them differently \\
\cline{2-3}
	&Intentional Interfaces	& + Users can explicitly request a new canvas to work on a different alternative

+ Canvases are linked and ordered, helping find related alternatives	\\
\hline
7.      They move from one level of abstraction to another	&Scraps \& connectors	& + Different abstractions can be quickly constructed by copying and moving and otherwise manipulating scraps and connectors	\\
\cline{2-3}
	&Intentional Interfaces	& + Users can explicitly request a new canvas to work on a deeper level of abstraction

+ Canvases are linked and ordered, helping find related canvases of different levels of abstraction	\\
\cline{1-3}
8.      They perform mental simulations	& Fading highlighter	& + Users can use the highlighter to mark up their diagrams without editing them	\\
\hline
9.      They juxtapose sketches	&Scraps	& + Uses can move perspectives, alternatives, and abstractions next to one another by moving scraps\\
\cline{1-3}
10.  They review their progress	&Intentional Interfaces	& + Users can step back and examine their progress and process, overall and in parts, in the intention view\\
\hline
11.  They retreat to previous ideas	&Intentional Interfaces	& + Users can choose to enter one canvas in the intentional view or make a new canvas at any time\\
\hline
12.  They switch between synchronous and asynchronous work	&Intentional Interfaces	& + Users can choose to enter one canvas in the intention view, or make a new canvas and work separately\\
\hline
13.  They explain their sketches to each other	& Fading highlighter	& + Users can use the highlighter to draw attention to certain parts of a canvas	\\
\hline
14.  They bring their work together	&Scraps \& connectors	& + Scraps can pick up sketches and drop those sketches onto other scraps, merging them\\
\cline{2-3}
	&Palette	& + Designers can place sketches from different canvases into palette and later merge them into a single canvas
\label{table:discussion:minimally-invasive}
\end{longtable}
\end{center}

%Do these features fit together, and do I believe that they fit together
Addressing the third part of the research question, the set of four features did indeed support all design behaviors. Scraps and the palette supported designers in the kinds of sketches they created (design behaviors 1 -- 4). They allowed diagrams to ``grow organically'', have rich coversations over the runtime behavior of components, and build relationships over unrelated sets of data. Intentional interfaces, scraps, the palette, and the fading highlighter helped support the way designers navigated between sketches (design behaviors 5 -- 11). Intentional interfaces helped designers find relationed perspectives, alternatives, and levels of abstraction, and did so by attaching related canvases with tagged links and providing an order to them, which also helped to review their progress and retreat to previous ideas. The fading highlighter further supported designers in mentally simulating over box-and-arrow diagrams. Lastly, Intentional interfaces, the fading highlighter, scraps and the palette helped designers in collaborating with one another (design behaviors 12 -- 14). Intentional interfaces helped designers in moving between synchronous work and opportunistically branching off to asynchronous work. The fading highlighter enabled designers to explain their sketches to one another from remote computers. The palette further allowed designers to bring work together from different canvases, and scraps also help to coordinate bringing work together.


%Analysis of grid versus intentional interfaces. We reduced the number of features that used to be there where each serve multiple purposes.

%\section{Limitations and threats to validity}

%--> What do these sessions allow me to say? The sessions are not directly comparable.

%
%In this section, I address potential issues which may lead to threats in validity.
%
%- Was not able to observe studies in person.
%- People interviewed may not have remembered correctly, or incorrectly interpreted logs.
%- All uses of Calico may have been idiosyncratic, and specific to the culture observed.
%- Observed different phases of the lifecycle for each group.

%%% Local Variables: ***
%%% mode: latex ***
%%% TeX-master: "thesis.tex" ***
%%% End: ***