Evaluation.tex

\chapter{Evaluation}
\label{chapter:evaluation}

In the previous chapter, I presented the final version of Calico with features that address all fourteen of the design behaviors, as well as examples of how the features may support the design behaviors in practice. In this chapter, I present an evaluation of Calico Version Two's ability to support those design behaviors through an ``in the field'' qualitative evaluation. In this chapter, I simply report the experiences from those who used Calico Version Two in the course of their own work and map their use onto the design behaviors. In Chapter \ref{chapter:discussion}, I reflect on these experiences.

Within the evaluations of Chapters \ref{chapter:calico-version-one}, I presented the study of software designers in a controlled setting, designing a solution for a hypothetical problem. This setting provided the evidence to suggest that scraps, the palette, and the grid can indeed support a subset of the fourteen design behaviors. However, in moving forward to evaluate the final version of Calico, it is prudent to consider the shortcomings of an evaluation conducted within a controlled environment. First, it is possible that the short duration of the laboratory studies does not give participants sufficient time to become accustomed to Calico's features. Longer-term use will allow groups to receive more practice, and learn how to use the features more effectively. Second, group behaviors such as switching between synchronous and asynchronous work and explaining their sketches are more pronounced in groups of three or more. Design behaviors based on group activities do indeed happen with only two designers, but the support Calico provides may have a more noticeable impact on collaborative work in larger groups than two. Third, Calico may have other benefits for group projects that may not appear in our analysis framework. For instance, Calico may help group meetings start more quickly because the sketches persist between one meeting and another.

Therefore, in this chapter, I report on a longer-term, qualitative study of Calico in use at software companies to not only evaluate its support for the fourteen design behaviors, but also understand its role within the context of a real-world design environment. In order to conduct this evaluation, I performed a multi-week qualitative study at three sites, including a commercial open source software company, an interaction design company, and a distributed research group. In this chapter, I report on my findings of how Calico is used over the long term at these three settings, how users incorporated Calico into their own work, and how it affected the way they design.

After reporting on each site, I report on the design behaviors as they occurred collectively across all three sites. My focus is to examine if the design behaviors occurred  while using Calico, and if so, what features supported those behaviors beyond how they would manifest themselves on a regular whiteboard.

The rest of this chapter is organized as follows: Section \ref{chapter:evaluation:overview} describes the setup of the study, the participants, and the data collections used. Sections \ref{chapter:evaluation:deployment1} - \ref{chapter:evaluation:deployment3} presents the results of the field studies framed within the research questions proposed put forward in Section \ref{chapter:evaluation:overview} and Chapter \ref{chapter:research-question}. More specifically, Section \ref{chapter:evaluation:deployment1} describes the findings at the commercial open source software company. Section \ref{chapter:evaluation:deployment2} describes the findings at the interaction design software company, and Section \ref{chapter:evaluation:deployment3} describes the findings with the distributed research group. Section \ref{chapter:evaluation:design-behaviors} collectively reviews the three field sites for evidence of design behaviors and how they were supported by Calico's features. 

\section{Method}
\label{chapter:evaluation:overview}

I evaluated Calico at three field sites, including two commercial companies and a geographically-distributed research group. At each site, I verified that a Calico installation was setup correctly, which included a large electronic whiteboard, a server instance of Calico, and access to pen-based tablets that could also access Calico. All participants were trained in the use of Calico.

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=8cm,keepaspectratio]{./figures/Evaluation/setting-opensource}
  \caption{The physical setup at the commercial OSS company.}
  \label{fig:evaluation:setting-opensource}
\end{figure*}

The first group is a commercial open-source software company, located in southern California, that develops software within the healthcare industry. In this text, I collectively refer to the members of this company as the OSS group. A software team of five people within the company, including four software developers and one team lead, used Calico in their project. In this deployment, depicted in Figure \ref{fig:evaluation:setting-opensource}, a Hitachi Starboard FX was installed on site, as well as three ASUS EEE121 tablets for use with the whiteboard. Within the same space, there were two couches present in front of the board, and a regular whiteboard next to the couches. All group members were given a tutorial on how to use Calico Version Two, and shown how to launch Calico and connect to the server from their own machine.  For the duration of the study, the team was engaged in architecting and implementing the next version of their project, which was written using Java software. The board itself was physically adjacent to their desks. This group was evaluated over a four week period.

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=16cm,keepaspectratio]{./figures/Evaluation/setting-interactiondesign}
  \caption{A tutorial of the usage of Calico given to the interaction design group.}
  \label{fig:evaluation:setting-interactiondesign}
\end{figure*}

The second group was an interaction design firm located in northern California. In this text, I collectively refer to the users from this company as the interaction design group. In this deployment, depicted in Figure \ref{fig:evaluation:setting-interactiondesign}, a Hitachi Starboard FXDUO88 was installed on site. Upon installation, the members of the company were invited to a tutorial and shown how to launch Calico from their own machines and connect to the boards. While no tablets were installed, as was done with the OSS group, the company did already have pen-based tablets which could launch Calico. Within the company, two individuals used Calico to support their work. In contrast to the OSS group, the individuals at the interaction design company were not responsible for code, but instead used Calico to perform interaction design activities, such as creating user personas using notes from interviews and creating storyboards. The pair of individuals used Calico over a five day period to process their interviews.

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=16cm,keepaspectratio]{./figures/Evaluation/setting-researchgroup}
  \caption{The physical setup of the research group.}
  \label{fig:evaluation:setting-researchgroup}
\end{figure*}

The third group was a distributed software research group located at the University of California, Irvine on the west coast and Carnegie Mellon University on the east coast. In this text, I refer to this group as the research group. This deployment, depicted in Figure \ref{fig:evaluation:setting-researchgroup}, mixed existing users of Calico at UC Irvine, and people new to Calico at CMU. Two members were located on the west coast and had access to two adjacent Hitachi Starboard FXDUO77 machines, as well as tablet based machines. One member was located on the east coast with access to an HP tablet machine with an electronic pen. This research group collaborated on a research project for a massive online development environment for developing Javascript projects. This group was tasked with designing the front end and writing the software for their project. While they used Calico for a long period of time, only the most recent seven months are evaluated. During that time, the CMU member visited UCI for a week of intense work with Calico; though it saw frequent use thereafter. While apart, skype was used to coordinate their weekly meetings, and was used alongside Calico.

\subsection{Data collection}

The observations presented in this chapter are primarily based on data collected from interviews and usage logs generated by Calico. The evaluation period for each site was at least four weeks, and the participants at each site were free to use the system as much, or as little, as desired. Given the long duration of the study, and that usage of Calico by users at each site was opportunistic rather than planned, we did not use video.

In order to review the design activity performed using Calico, I examined the usage logs produced by Calico. Each action taken by users was recorded into a history file, including all drawing activity, navigation, usage of features, and accessing content from remote machines. Each user action included a timestamp, name of the machine that was running the Calico client, action performed, and an image of the canvases at the time the action was performed. 

Based on examinations of logs and the content drawn, I conducted semi-structured face-to-face interviews. I conduct two semi-structured interview with the OSS group, one with the interaction design group, and one with the research group. In each interview, I began by asking ``What was your most vivid design experience with Calico?'' From this question, participants typically walked through the designs they created in Calico, physically pointing, mimicking their usage of features, and describing the contents of, as well as the activity leading to, their designs. During these walkthroughs, I asked for clarification of content within their designs as well as activity that I observed from their usage logs.

In subsequent questions, I specifically targeted the first two research questions. First, in order to evaluate whether Calico could be considered ``minimally invasive'', I asked participants if they needed to make any compromises in using Calico, what obstacles or surprises they encountered in creating their designs and if they struggled with the features. I also inquired how design sessions would have gone had they not used Calico. 

Second, in order to evaluate if Calico has a ``coherent set of features'', I asked participants to report their experience on using the features in general, what features they found helpful, and if the features clashed with one another. 

Due to the sensitive nature of some of the data collected, measures were taken to ensure that data was securely collected. Of the three sites visited, those in charge of two of the sites (the OSS group and the research group) provided consent to use the data collected for academic publication. The third site (the interaction design group), however,  was under a strict NDA agreement, and all images, logs, and raw data were not permitted to be taken off-site. In this case, usage logs were processed and reviewed on-site, and only personal notes were permitted to be taken off-site. Furthermore, for the purpose of publication, relevant images containing sensitive content were redrawn such that key insights, such as usage of notations, are preserved, but the content itself is anonymized.

\subsection{Data analysis}

Based on the logs and interviews, collected data from each session was reviewed in two phases. First, I reviewed interviews for each group for responses pertaining to the overall design activity that occurred, their general feedback on their experiences using Calico, and feedback on specific feature usage. In the second phase, I reviewed both the usage logs and interviews and categorized my observations into the following three categories: (1) how they navigated between canvases, (2) the representations they used, and (3) how they conducted group work (mimicking the three categories of design behaviors). 

%Lastly, the interview notes and logs were examined for any surprises that do not fit within the research question posed.
The results from the three field sites are partitioned across Sections \ref{chapter:evaluation:deployment1}, \ref{chapter:evaluation:deployment2}, and \ref{chapter:evaluation:deployment3} due to the variety of behaviors observed. The observations across all three sites are aggregated in the overarching discussion presented in Chapter \ref{chapter:discussion}.

\section{Results: Deployment at a commercial open source software company}
\label{chapter:evaluation:deployment1}

Of the entire duration that Calico was installed at the OSS group, the developers reported using Calico in three extensive design sessions. Across all three sessions, the developers were engaged in developing the next version of a healthcare ``message processing'' tool. 

\subsection{Overview of design sessions}

In the first design session, depicted with excerpts of the canvases in Figure \ref{fig:ossgroup:session1}, the developers turned to the whiteboard to sketch out a user interface for creating custom handlers for different types of messages. In the second design session, depicted with excerpts of the canvases in Figure \ref{fig:ossgroup:session2}, the developers turned to the whiteboard to design a set of slides that will be used to explain the software architecture of their system. In this case, all members already understood the architecture, but wanted to create a representation ``that was easy to understand''. In the third design session, depicted in Figure \ref{fig:ossgroup:session3} with excerpts, while coding, a developer turned to Calico in order to refactor source code that needed to be updated for their upcoming software release.

In the following, I group my observations of the developers' usage of Calico based on how they navigated between canvases, the representations they created, and how they used Calico to manage group work.

\begin{figure}%
  \centering
  \subfigure[Exploration of entities using box-and-arrow diagrams] {
      \label{fig:ossgroup:session1:a}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session1/canvas1}
   }
  \subfigure[Hierarchical perspective of component structures from Figure \ref{fig:ossgroup:session1:a}] {
      \label{fig:ossgroup:session1:b}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session1/canvas2}
   }
  \subfigure[Refined mock-up for user interface in Figure \ref{fig:ossgroup:session1:a}] {
      \label{fig:ossgroup:session1:c}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session1/canvas3}
   }
  \subfigure[Alternative user interface mockup to Figure \ref{fig:ossgroup:session1:c}] {
      \label{fig:ossgroup:session1:d}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session1/canvas4}
   }         
  \subfigure[Mock-up of use cases created after discussion of Figure \ref{fig:ossgroup:session1:d}] {
      \label{fig:ossgroup:session1:e}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session1/canvas5}
   }   
   \caption {Representations used by the OSS group in their first design session to design a user interface.}
   \label{fig:ossgroup:session1}
\end{figure}%

\begin{figure}%
  \centering
  \subfigure[Minimal sketch of software architecture for ``Messaging Processing'' and ``Event Bus''] {
      \label{fig:ossgroup:session2:a}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas1}
   }
  \subfigure[Less abstracted perspective of Figure \ref{fig:ossgroup:session2:a}] {
      \label{fig:ossgroup:session2:b}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas2}
   }
  \subfigure[Different perspective of ``Event Bus'' in Figure \ref{fig:ossgroup:session2:b}] {
      \label{fig:ossgroup:session2:c}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas3}
   }
  \subfigure[Lower level of abstraction of Figure \ref{fig:ossgroup:session2:c}, ``Alert'' is a child class of ``Event Listener''] {
      \label{fig:ossgroup:session2:d}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas4}
   }
\subfigure[Different perspective of Figure \ref{fig:ossgroup:session2:a}, Software architecture of the ``Alert'' event listener] {
      \label{fig:ossgroup:session2:e}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas5}
   }
  \subfigure[Lower level of abstraction of ``Action Group'' in Figure \ref{fig:ossgroup:session2:e}] {
      \label{fig:ossgroup:session2:f}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas6}
   }
  \subfigure[Different perspective of Figure \ref{fig:ossgroup:session2:e}, components summarized in list] {
      \label{fig:ossgroup:session2:g}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas7}
   }   
  \subfigure[Use case diagram encompassing Figures \ref{fig:ossgroup:session2:a} to \ref{fig:ossgroup:session2:g}] {
      \label{fig:ossgroup:session2:h}     
      \includegraphics[width=6.8cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session2/canvas8}
   }   
   \caption {Representations used by the OSS group in their second design session in which they worked on representation that details the software architecture for handling different types of messages.}
   \label{fig:ossgroup:session2}
\end{figure}%

\begin{figure}%
  \centering
  \subfigure[First iteration of source code] {
      \label{fig:ossgroup:session3:a}     
      \includegraphics[width=14cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session3/canvas1}
   }
  \subfigure[Second iteration of source code] {
      \label{fig:ossgroup:session3:b}     
      \includegraphics[width=14cm,keepaspectratio]{./figures/Evaluation/ossgroup/Session3/canvas2}
   }
   \caption {Representations used by the OSS group developer to refactor their code in the third reported design session.}
   \label{fig:ossgroup:session3}   
\end{figure}%

\subsection{Feature usage}
\label{chapter:evaluation:deployment1:part1}

\subsubsection{General feedback}

On the whole, the individuals that did use Calico reported that they did not find the features to be invasive to the basic sketching activity. At a bare minimum, all individuals said they were able to use the large electronic whiteboard as they would a regular whiteboard. Individuals reported some drawbacks with the hardware itself, in which the electronic whiteboard hardware experienced lag so they needed to write slowly to create legible representations. However, the pen-based tablets were much more precise. They also reported some surprises in the usability of the tool, for example they expected the forward and backward navigation buttons to step through the chain of linked canvases in the intentional interface, instead of the visited canvases. Also, the members of the OSS group expected greater functionality in changing colors of scraps and text-scraps, but fell back to plain sketching to use colors.

When asked if they made any compromises in order to perform their activities using Calico, users responded that they did not need to make many compromises. The only compromises that they reported were writing slower and larger on the electronic whiteboard. If they did not use Calico, they reported that they would have moved to a meeting room with several whiteboards rather than the space immediately adjacent to their desks. One participant reported that in one activity they would have written on the window using erasable markers, but instead chose to use Calico. They also reported that they normally would have taken pictures of their work, but instead used the email feature of Calico.

\subsubsection{Feature specific}

When asked about the features as a whole, the members of the OSS group reported that they did not feel that the features conflicted with one another. Additionally, both interviews and usage logs confirmed that all features were indeed used at least to some extent, with the exception of the palette, which was very rarely used. The following captures the general feedback from the OSS group concerning each feature.

\textbf{Basic sketching and features.} Aside from the issues with the responsiveness of the large electronic whiteboard mentioned above, OSS group members reported that the basic sketching features were (e.g., changing pen color, changing line width, undo, and redo) helpful. Sketching on regular tablets was responsive, and developers from the OSS group stated that synchronous sketching was helpful in meetings, allowing members to work together, or hand tablets back and forth between each other while designing with the large electronic whiteboard. They reported frequent usage of multiple colors and the undo/redo functionality.  They did, however, mistake the ``clear canvas'' button for the ``eraser mode'' button several times, leading to the canvas being accidentally cleared, but this action could be undone with the ``undo'' functionality. Two of the developers in the OSS group reported some difficulty adapting because upon using the device they default to gestures they use on other devices such as pinch-and-zoom, which are not available in Calico. Another member of the OSS group requested additional colors be made available as well.

\textbf{Scraps.} The participants found the use of scraps helpful on occasion. One member from the OSS group wanted to create text-scraps with different colored text, but instead turned to writing by hand instead of typing when this was not possible. Further, when using text scraps at the large electronic whiteboard, members from the OSS group found it cumbersome to switch between the bluetooth keyboard and the large display, and stated that they would have preferred an on-screen keyboard. Writing content was sloppy, so where possible, they created text scraps to make content more legible. Otherwise, OSS group members reported that scraps were sufficiently helpful to move content and represent objects. Among the three design sessions observed, scraps were used to represent objects in two of those sessions, and in the other session, were used simply to manipulate plain sketched content.  In another design session, one of the members of the OSS group encountered difficulty in using the ``shrink-to-contents'' functionality for scraps, which creates a rectangular scrap around sketches if present, or simply a rectangle if no content was present. The OSS group member reported wanting to create a rectangle, but the ``shrink-to-contents'' button instead created a smaller rectangle around the existing sketch. Lastly, a member of the OSS group requested that scraps be resized with a corner anchor, without locking their aspect ratio.

\textbf{Palette.} The palette received light usage by the OSS group. The palette was used in one of the three design sessions, as a few scraps were used across several canvases. When asked how often they reused scraps, the OSS group collectively replied ``rarely''. They reported that they typically went to a new blank canvas when discussing their software system, after which they archived the canvases used by emailing them to the entire group. With respect to reusing scraps with the palette, they stated that it was faster to simply redraw their representations.

\textbf{Intentional interfaces.} OSS group members responded that they found the ``chaining'', i.e., linking canvases using tags, as supported by intentional interfaces to be helpful in capturing individual design sessions. They used the ``new canvas'' and ``copy canvas'' buttons several times with tagging to create chains, and all design activities were done in a single cluster. When drawing within a canvas, one OSS group member reported that they exclusively used the breadcrumb bar to navigate between canvases. Another developer from the OSS group preferred to use the cluster view, but reported being annoyed that the cluster view changed their zoom perspective in an unexpected ways. In their words, the view ``kept jumping around''. As a result, the users reported a lot of unnecessary panning in the cluster view. Two other developers in the OSS group also reported that they found the forward and backward navigation buttons confusing. They expected these buttons to navigate to the next canvas in the chain within the intentional interface, not the most recently visited canvas. One developer in the OSS group asked for a minimap of the cluster view to be directly visible in the canvas. One member of the OSS group called the organization ``wizardry'', since they could not see how it changed.

\textbf{Fading highlighter.} The OSS group reported that the fading highlighter was useful within group sessions. They used the fading highlighter in two of the three sessions, and reported that they used this feature to explain designs.

\subsection{Canvas navigation}

\textbf{Using intentional interface chains to capture design sessions.} The OSS group reported that they organized each design session by ``chaining'' canvases within the cluster view, as depicted in Figure \ref{fig:ossgroup:clusterview}. Each design session was linked together using tags, typically with either the ``perspective'' tag or the ``alternative'' tag. Each new design session began as a new canvas within the center of the radial circle, and extended outward using canvases linked as chains. A member of the OSS group reported that panning through their chains of canvases was helpful in reviewing the sketches that they created.

\begin{figure}%
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/Evaluation/ossgroup/ossgroup-clusterview}
   \caption {Cluster view of canvases. The three major sessions were grouped by ``chaining'' canvases.}
   \label{fig:ossgroup:clusterview}   
\end{figure}%

\textbf{Using canvases to explore perspectives.} The OSS group reported that they used several canvases to explore their designs across multiple perspectives. They explored different perspectives both by using different types of representations, such as user interface mockups and software architecture diagrams, and also by creating sketches that depicted different parts of the system. 

In their first design session, depicted in Figure \ref{fig:ossgroup:session1}, they created sketches using different types of representations to explore the components of the user interface and its behavior at runtime. In Figure \ref{fig:ossgroup:session1:a}, they used box-and-arrow diagrams alongside low-detailed user interface sketches to brainstorm major components of the user interface and their relationships to one another. In Figure \ref{fig:ossgroup:session1:b}, they shifted to a hierarchical list to record the structure. In Figures \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:d} they depicted the contents of Figure \ref{fig:ossgroup:session1:b} as it would appear to the end user. In Figure \ref{fig:ossgroup:session1:e}, they sketched out three different use cases, where ``CH'', ``CH2'', and ``CH3'' represent three different results for input entered by the user in the scrap ``All Channels''.

In their second design session, depicted in Figure \ref{fig:ossgroup:session2}, they explored different parts of the same design across the different canvases in order to document the flow of information through the system. For example, Figure \ref{fig:ossgroup:session2:a} shows ``message processing'' sending information to the ``Event Bus''. Figure \ref{fig:ossgroup:session2:c} shows how information is passed from the ``Event Bus'' scrap onto the ``Event Listener'' scraps. Figure \ref{fig:ossgroup:session2:d} further shows how information is passed from the ``Alert'' scrap, which itself is an instantiation of an ``Event Listener'', onto the ``Email'' and ``Channel'' scraps. The OSS group members sketched out each step in order to thoroughly explain the rules and logic used for passing information between components. 

\textbf{Stepping through a design using levels of abstraction. } The OSS group used multiple levels of abstraction to aid in explaining their architecture. They used it both to progressively step into the detail of their design, and also to back away to once again see the bigger picture.

When stepping progressively into their design, they made use of the ``copy canvas'' feature to repeat a canvas, but replace parts of their sketches with more detailed versions to step into lower levels of abstraction. The second design session in Figure \ref{fig:ossgroup:session2} shows two such cases. First, Figure \ref{fig:ossgroup:session2:b} shows a lower level of abstraction of Figure \ref{fig:ossgroup:session2:a}. Between Figures \ref{fig:ossgroup:session2:a} and \ref{fig:ossgroup:session2:b}, ``message processing'' is replaced by scraps labeled as ``Channel'' , which represent the components that process the messages.  In Figure \ref{fig:ossgroup:session2:b}, each colored connector represents messages of a specific type that are passed from a ``Channel'' to the ``Event Bus''. Second,  Figures \ref{fig:ossgroup:session2:c}, \ref{fig:ossgroup:session2:d}, \ref{fig:ossgroup:session2:e}, and \ref{fig:ossgroup:session2:f} show a sequential step-through in levels of abstraction. The ``Event Listener'' scrap in Figure \ref{fig:ossgroup:session2:c} is represented as the ``Alert'' scrap in Figure \ref{fig:ossgroup:session2:d}, where it is shown in much greater detail. The ``AG1'' scrap in Figure \ref{fig:ossgroup:session2:d} is represented as the ``Action Group'' scrap in Figure \ref{fig:ossgroup:session2:e}, where it is elaborated in much more detail. The ``Action Group'' scrap in Figure \ref{fig:ossgroup:session2:e} is again represented in ``Action Group'' in Figure \ref{fig:ossgroup:session2:f}, showing further sub components for ``Action Group''. While creating these scraps, the OSS group used the palette to copy elements between canvases, such as ``Event Bus'' and ``Action Group''.

After their stepwise exploration leading to Figure \ref{fig:ossgroup:session2:f}, the OSS group members sketched a high level picture of messages passing through the system in Figure \ref{fig:ossgroup:session2:h}. Not shown in the figure, the OSS group members created several copies of the contents of Figure \ref{fig:ossgroup:session2:h} in order to explain how different types of messages may pass through the system. In their session, they returned to previous canvases when they had questions about particular components.

\textbf{Copying canvases to generate alteratives. } The OSS group sometimes used multiple canvases to create alternative solutions to existing designs. For example, a member of the OSS group reported that, while working with their team members on the user interface in Figure \ref{fig:ossgroup:session1:c}, they ``did not quite agree with the design'' and deviated from the group to create their own version. To do so, they used the ``copy canvas'' button, selected the ``alternative'' tag in the intentional interface tag panel, and created a new user interface, the result of which is shown in  \ref{fig:ossgroup:session1:d}. 

In a third design session, one of the OSS group developers used multiple canvases to help them in iterating on past designs. After finishing the design in Figure \ref{fig:ossgroup:session3:a}, the OSS group member actually implemented the designed changes in the software, to later return to the large electronic whiteboard to verify that their source code correctly implemented the original pseudocode in Calico. In this second iteration, they created a copy of the original canvas and pasted a screenshot of the source code that resulted from the previous design session, where they continued to refine the pseudocode.

\subsection{Representations}

The OSS group created several types of representations in their design sessions.

\textbf{Box-and-arrow diagrams. } Across the design sessions, they created what they termed ``box-and-arrow'' diagrams. They did not make any direct reference to notations such as UML class diagrams or ER notation, but rather viewed their sketches as rudimentary abstractions of their existing software system. The first was used for brainstorming, and the second for explaining an architecture.

In the first design session, depicted in \ref{fig:ossgroup:session1:a}, they created a set of box-and-arrow diagrams to brainstorm the components of a user interface. The design session began by first listing out the relevant entities in the software, which were the text scraps ``Source'' and ``Destinations''. In this session, the OSS group experimented with different combinations of actions, which were reflected with further text scraps associated with connectors at the bottom. They used connectors to represent the data sources that the `Source'' and ``Destinations'' scraps could pull from and send to.

In the second design session, the OSS group used a combination of scraps, connectors, and plain sketches to represent software components and the passing of data. When asked why they sometimes represented components as plain sketches and other times using scraps, they reported that they used scraps to represent components that they were actively designing and using. The OSS group reported that elements in sketches, such as ``In'' and ``Out'' in Figure \ref{fig:ossgroup:session2:a} and LLP in \ref{fig:ossgroup:session2:h}, represented outside information. Creating items such as ``Event Listener'' in Figure \ref{fig:ossgroup:session2:c} allowed the components to be reused and moved. For passing data, they simply used colors to represent particular types of data that were passed. For example, between Figures \ref{fig:ossgroup:session2:a} and \ref{fig:ossgroup:session2:b}, the OSS group used four different types of colors to show the types of data that are passed from a ``Channel'' to the ``Event Bus''. In Figure \ref{fig:ossgroup:session2:c}, the OSS group repeated the use of colors to show how the different types of data were handled.

\textbf{User Interface Mockups. } Across the sketches in Figure \ref{fig:ossgroup:session1}, the OSS group used a mix of plain sketching with scraps to design a component and its interface across several canvases. In both Figure \ref{fig:ossgroup:session1:a} and \ref{fig:ossgroup:session1:b}, they sketched the software entities alongside a user interface mockup. In Figure \ref{fig:ossgroup:session1:a}, the OSS group reported that they created three low-detailed user interface mockups while ``throwing ideas out on the board''. They iterated over their user interface across multiple canvases, two of which are depicted in Figures \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:b}.

%- While developing these, they sketched a low-detailed user interface within a scrap at the top of Figure \ref{fig:ossgroup:session1:a}, in which the ``plus'' symbols can be toggled by the user and the resulting configuration is displayed to its immediate right. The right side of Figure \ref{fig:ossgroup:session1:a} represents a quick draft of the user interface proposed in the top left. In Figure \ref{fig:ossgroup:db1b}, the developer created another perspective of the elements in Figure \ref{fig:ossgroup:db1a}, but instead organizing the entities using a hierarchy, and drafted another user interface. This was about throwing ideas out on the board.

\textbf{Lists. } The OSS group used lists in two instances. First they used lists as a method to hierarchical organize the user interface in Figure \ref{fig:ossgroup:session1:b}. Second, they used lists as a mechanism to describe the functionality of a software architecture in detail in Figure \ref{fig:ossgroup:session2:g}. The OSS group reported that they used lists to summarize their design sessions and to reflect on designs they recently created. However, they did not create many lists because they found that the board reduced the aesthetic quality of their writing.

\textbf{Use case sketches. } In order to capture the behavior of their designs at runtime, they created primitive use case scenarios. For example, in the final canvas of their first design session, Figure \ref{fig:ossgroup:session1:e}, the OSS group sketched the behavior of the user interface when different channels were checked. This representation depicts the results of three different use cases for the interfaces shown in Figures \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:d}. The left scrap in Figure \ref{fig:ossgroup:session1:e} represents the input panel, and the three scraps on the right hand side each represent the result of a different use case, in which either ``Ch 1'', ``Ch 2'', or ``Ch 3'' is tapped on the left scrap. A member of the OSS group created these three previews by drafting an initial scrap on the right hand side, and copying it multiple times, adjusting the values for each respective ``Channel'' selected. 

In the second design session, they created several copies of the contents in Figure \ref{fig:ossgroup:session2:h} to explore different use case scenarios. They used different use case scenarios to walk through input messages of different types and frequency, as well as using different event listeners to handle those messages.

\textbf{Source code. } In the third design session, one of the members of the OSS group used the large electronic whiteboard to help him in refactoring his code. Depicted in Figure \ref{fig:ossgroup:session3}, the OSS group member needed to refactor his code in order to process an XML file that contains new fields in the next version of their software tool. In order to carry out his design session, he opened Calico on his own desktop, copy and pasted a screenshot of the XML code that he needed to process, and continued his design session at the large electronic whiteboard. In his exit interview, he needed ``a space to think freely''. While sketching, he used several custom annotations and colors to depict the process flow of different software components through the XML structure. Black vertical bars were used to represent a call stack, which he drew to help him in understanding how far the callstack goes in a recursive call. The red arrows were used to represent how the call stack moves through the data.

Overall, the OSS group member reported that using different colors helped him see things ``at-a-glance''. He stepped back from his pseudocode several times, experimented with walking through the code in different orders, and used a different color for each walkthrough. The final design of the first iteration is shown in Figure \ref{fig:ossgroup:session3:a}. After implementing the design in the code base, and upon discovery that an issue was not addressed in his first iteration, the OSS group member copied and pasted the latest Java source code to once more step through their design again to address the discovered issue.

\subsection{Collaborative work}

\textbf{Preparing work ahead of time.} The first design session was a shared group session that spanned the greater part of a working day. In the morning, one of the OSS group members took a tablet to their desk and worked out a design, and later presented his design in a group meeting with four other developers in the OSS group, where they continued to work together on the design for several hours afterwards. A subset of the sketches produced in this design session are depicted in Figure \ref{fig:ossgroup:session1}. In this design session, they were designing how to handle data as it is passed through each ``channel''.

\textbf{Explaining designs to the group.} In the second design session, the team members made heavy use of multiple tablets, as well as the fading highlighter. When the group meeting began in the afternoon, the OSS group made heavy use of the fading highlighter to explain the behavior of the system shown in Figure \ref{fig:ossgroup:session2:a}. Having worked out much of the design independently earlier that day, a member of the OSS group used the fading highlighter to explain the state of the design thus far. 

\textbf{Spontaneous asynchronous work.} During the design session, some of the design members would privately move to another cell, create a design, and call other members of the group to visit their cell. For example, while the group discussed the data that was passed from ``Event Bus'' in Figure \ref{fig:ossgroup:session2:d}, one of the OSS group memebers wanted to understand how the ``Alert'' component worked in the greater context, and moved to the canvas depicted in Figure \ref{fig:ossgroup:session2:h} to sketch a much more abstract representation to show how data flowed through it. After sketching it, they called the other designers over, and made frequent movements back and forth between these canvases to compare them.

%The first design session, depicted in Figure \ref{fig:ossgroup:session1}, was performed over two separate days. In this design session, the developer was creating a tool in which an end-user may create several ``channels'' for processing data, and for each channel, a ``source'' is chosen and piped to any number of ``destinations''.  The developer created a set of sketches specifically to help himself plan the component that he was developing.
%
%Across the sketches in Figure \ref{fig:ossgroup:session1}, the developer used a mix of plain sketching with scraps to design a component and its interface across several canvases. Within both Figure \ref{fig:ossgroup:session1:a} and \ref{fig:ossgroup:session1:b}, the developer sketched both the the software entities alongside a user interface mockup. The design session began by first listing out the relevant entities in the software, which were the text-scraps, ``Source'' and ``Destinations''. In this session, the developer experimented with different combinations of actions, which were reflected with further text-scraps associated with connectors at the bottom. While developing these, they sketched a low-detailed user interface within a scrap at the top of Figure \ref{fig:ossgroup:session1:a}, in which the ``plus'' symbols can be toggled by the user and the resulting configuration is displayed to its immediate right. The right side of Figure \ref{fig:ossgroup:session1:a} represents a quick draft of the user interface proposed in the top left. In Figure \ref{fig:ossgroup:db1b}, the developer created another perspective of the elements in Figure \ref{fig:ossgroup:db1a}, but instead organizing the entities using a hierarchy, and drafted another user interface.
%
%The developer used both the copying of scraps and the copying of canvases to mock up iterations of the user interface. The developer created two alternative solutions for the user interface in Figures \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:d}, the second of which was created by copying a canvas, and selecting ``alternative'' in the intentional interface tag panel. In the final canvas, Figure \ref{fig:ossgroup:session1:e}, the developer sketched the behavior of the user interface when different channels were checked from the user interfaces in Figure \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:d}. The left scrap in Figure \ref{fig:ossgroup:session1:e} represents the input panel, and the three scraps on the right hand side each represent the results of different use cases, in which either ``Ch 1'', ``Ch 2'', or ``Ch 3'' are tapped on the left scrap. The developer created the three previews by drafting an initial scrap on the right hand side, and copying it multiple times, adjusting the values for each respective ``Channel'' selected. 


%The second design session was a shared group session that spanned the greater part of a working day. In the morning, one of the developers took a tablet to their desk and worked out a design, and presented his design in a group meeting with four other developers, where they continued to work together on the design for several hours afterwards. A subset of the sketches produced in this design session are depicted in Figure \ref{fig:ossgroup:session1}. In this design session, they were designing how to handle data as it is passed through each ``channel''.
%
%With respect to how the designer organized the work across canvases, they reported that they used different canvases to represent different perspectives and views onto the same component. In Figures \ref{fig:ossgroup:session2:a} and \ref{fig:ossgroup:session2:b}, the developers created representations that built on the logic of passing data through ``channels'' from the the first design session using Calico, but now handle the data by sending it to an ``Event Bus'' scrap. The developers used the copy feature to create Figure \ref{fig:ossgroup:session2:b}, which represents a more detail view of Figure \ref{fig:ossgroup:session2:a} and was tagged as ``perspective'' in the intention view. The developers further extended their design of the system by adding the ``Event bus'' scrap to the palette, and reused the scrap in Figures \ref{fig:ossgroup:session2:c}, \ref{fig:ossgroup:session2:d}, and \ref{fig:ossgroup:session2:e}. From the canvas in Figure \ref{fig:ossgroup:session2:e}, the developers added the ``Action Group'' scrap to the palette, and copied it to another canvas, where they worked that component in greater detail. After having designed the interactions between the components of the system, they summarized the content in the canvas depicted in Figure \ref{fig:ossgroup:session2:g}. In Figure \ref{fig:ossgroup:session2:h}, the developers sketched an example of data passing through the components from the previous sketches, i.e., the ``Channel 1'' scrap, being processed, and sent to another channel.
%
%With respect to the representations used, the developers used a combination of scraps, connectors, and plain sketches to represent software components and the passing of data. When asked why they sometimes represented components as plain sketches or using scraps, they reported that they used scraps to represent components that they were actively designing and using. The developers reported that elements in sketches, such as ``In'' and ``Out'' in Figure \ref{fig:ossgroup:session2:a} and LLP in \ref{fig:ossgroup:session2:h}, represented outside information. Creating items such as ``Event Listener'' in Figure \ref{fig:ossgroup:session2:c} allowed the components to be reused and moved. For passing data, they simply used colors to represent particular types of data that was passed. For example, between Figures \ref{fig:ossgroup:session2:a} and \ref{fig:ossgroup:session2:b}, the developers used four different types of colors show the types of data that was passed from a ``Channel'' to the ``Event Bus''. In Figure \ref{fig:ossgroup:session2:c}, the developers repeated the use of colors to show how the different types of data were handled. 
%
%With respect to designing with the group, the team members made heavy use of multiple tablets, as well as the fading highlighter. When the group meeting began in the afternoon, the developers made heavy use of the fading highlighter to explain the behavior of the system within Figure \ref{fig:ossgroup:session2:a}. Having independently worked out much of the design independently earlier that day, a developer used the fading highlighter to explain the state of the design thus far. During the design session, some of the design members would privately move to another cell, create a design, and call other members of the group to visit their cell. For example, While the group discussed the data that was passed from ``Event Bus'' in Figure \ref{fig:ossgroup:session2:d}, one of the developers wanted to understand how the ``Alert'' component worked in the greater context, and moved to the canvas depicted in Figure \ref{fig:ossgroup:session2:h} to sketch a much more abstract representation to show how data flowed through it. After sketching it, they called the other designers over, and made frequent movements back and forth between these canvases to compare them.





%For what activities did they turn to the whiteboard?
%
%Used a tablet while.
%
%Developer worked it out for himself first. Used a tablet rather than the large whiteboard. Afterwards, brought in another person to help them.
%
%Multiple people were using the board at the same time for different activities.
%
%User needed to get their thoughts written out. Was going to design the next version of their software. Was attempting to come up with a migration path from the old version to the new version. Needed to work out issues. Found color to be useful, to identify importance. Wanted to look at XML code that was relevant. Pasted code that he was working on into the board.
%
%Took a different strategy in direction. Made an  alternative, made another screenshot of different code. Used breadcrumb bar to navigate between canvases. 
%
%Wanted a text box to be able to draw.
%
%Another user was used to interface from apple devices, wanted pinch and zoom.
%
%While the main group was discussing something, one person branched off and developed a new alternative. When he was ready, he proposed it to the whole group. They could have done the same thing with two whiteboards, but then they would have to move around. Being able to work simulataneously is nice.
%
%One user was interested in having.
%
%More willing to draw anything since they know that they can create a new canvas, delete it, or return to a previous.
%
%icons are confusing.
%
%Used

\section{Results: Deployment at an interaction design company}
\label{chapter:evaluation:deployment2}

Of the entire duration that Calico was in use at the interaction design group, the interaction design group reported that they used the board on occasion for notes, and used the system to conduct one major on-going design activity. Usage logs indicate that the design session took place across five days, where usage went on throughout the work day. The two members of the interaction design group conducted the design together.

In order to preserve confidentiality in this section, the topic and content of the material have been obfuscated, and the images have been reproduced. I use generic references without mention of the target domain, and have recreated the images to align with this hypothetical task, but still represent what they drew.

Also, it is important to note that in this group, during their five day design session, they used an earlier version of Calico Version Two that did not yet include the intentional interfaces feature, but instead included the grid. The interaction design group did, however, use the intentional interface feature later in a one hour design session. All experiences reported for this group should be assumed as using the grid, unless the intentional interface feature is explicitly stated.

\subsubsection{Overview of design session}

The design session that took place involved processing a set of fifty interviews that the interaction design group conducted, and building a set of user personas based on those interviewed. The interaction design group reported that they typically would have processed the interviews by printing the faces of the fifty people interviewed onto note cards, and working with the note cards in a physical space. They reported that they could have simply written names, but found speaking to images of the people they interviewed to be more evocative and effective. In the large open space, they would organize the note cards by physically grouping them together according to emergent categories, writing notes on the back of the note cards, and taking pictures of the groupings. 

With Calico, they saw an opportunity to perform the same activity, but preserve work that would have been lost every time they re-arranged the note cards. Instead, they performed this activity by importing images of the fifty individuals that they interviewed into the palette, and dropped these onto the canvas as image scraps. In Calico, they viewed the work they did as non-destructive and preservative, where a particular arrangement of the image scraps could be copied and reorganized an indefinite amount of times. If inspiration struck them regarding an earlier arrangement, they were always able to return to a previous arrangement of the image scraps and continue with the most recent work later.

\subsection{Feature usage}

\subsubsection{General feedback}

Overall, the participants reported that the work they performed in the system was useful, but ultimately found it difficult to adapt their work habits to the structure of the system. Representing images as scraps provided advantages such as reducing the overhead in creating physical printouts for each person they interviewed, reusing them across several canvases, and allowing them to return to past explorations. The interaction design group, however, reported that they had a culture of using OneNote, a paginated sketch-based application, on pen-based windows machines, which led to a clash between the expected functionality of Calico and Calico's actual functionality. For example, they expected functionality such as OneNote's lasso mode for selecting and moving content, and subsequently found Calico's scrapping functionality confusing and difficult to adopt. Also, they found Calico's Grid and Intentional Interface system of managing canvases not as fluid to use as OneNote's system of vertically scrolling through paginated sketches. Further, they reported that their ability to write legibly was limited by the large electronic Starboard hardware, which resulted in larger content that was sketched more slowly than expected. They would have preferred the ability to rest their wrist on the board to write comfortably. Ultimately, however, they worked around this limitation to perform their design work.

The interaction design group reported that using Calico did indeed cause some changes in how they designed. First, they reported that, while different from OneNote's paginated system, they felt the sensation of a greater amount of free space to sketch, and as a result, created more sketches and content than they would have otherwise. Within each canvas, however, they reported desiring more free space, either through zooming and panning of content, or paginated spaces such as within OneNote. 

Additionally, the interaction design group had several feature requests. They found the use of modal dialogs, such as email confirmation dialogs and text entry, to be disruptive during their design sessions. They asked for highlighting functionality that is more analogous to its real-life counter part, i.e., persistent and shading the area behind a text with color, which they reported as using often for tagging items in their sketches. They further reported the sudden switch between sketching an object, and creating scraps to be disorienting, and requested separate modes for drawing, creating, selecting, and moving scraps. They further found the set of graphical icons for buttons and menus to not be straightforward, and requested additional descriptions.

Despite these issues, they produced twenty one canvases of content. Sixteen of those canvases made significant use of scraps, six of which used made use of multiple list scraps. One canvas even used connectors between scraps.

\subsubsection{Feature specific}

\textbf{Basic sketching and features.} Participants reported some difficulty in accurately sketching with the system. They reported that it was difficult to produce legible handwriting. For example, when attempting to use print, Calico would produce cursive text. Other times, Calico would lag and sometimes lose details of their writing.

\textbf{Scraps.} The participants reported that the functionality that scraps enabled, such as moving sketched contents, was helpful, but reported some difficulty in using scraps. The participants reported that the interaction for moving scraps felt slow, particularly when moving scraps to categorize them in tables or quadrants. Further, the system began to slow down when using a large number of image scraps, thirty or more, within the same canvas. This is partly why they requested separate modes.

It is interesting to note, though, that early versions of Calico Version Two had such separate modes, which was received with very mixed reactions when deployed to students in a software design class. Students frequently had the ``mode switch problem'': they did not know which mode they were in and made mistakes as a result. This was one of the reasons we reduced the number of modes dramatically.

\textbf{Palette.} The participants found the palette helpful for loading pre-existing graphics. The participants imported all fifty images of the people they interviewed into the palette, and used the images from the palette across several canvases. However, they found it difficult to find items in the palette when they loaded all fifty images into it, as the palette displays saved scraps as small snapshots, making them difficult to discern.

\textbf{Grid / Intentional interfaces.} During the majority of their usage, the interaction design group did not have intentional interfaces, but instead had the grid. They reported that they often moved to the grid perspective in order to get a high level perspective of their work. One of the designers reported that ``it's important for me to see everything at once''. The interaction design group used intentional interfaces for a brief period, but had difficulty adjusting to using it as a method of moving between canvases. They had trouble using the breadcrumb bar to navigate between canvases, and found the preview of the canvases within the cluster view too small. This, in a way, is not surprising given the brief exposure.

\textbf{Fading highlighter.} The interaction design group reported positive feedback about this feature, however, they seldom used it in practice. They reported the feature would be highly useful in working with larger groups, but all of their sessions took place with at most two people. Further, they found the name confusing as it did not function as they expected a regular highlighter would, in which a marker changes the color behind written text. Instead, they colloquially referred to it as the ``John Madden mode'', in reference to the sports announcer.

\subsection{Canvas navigation}

\textbf{Using perspectives to build personas.} The interaction design group used multiple canvases in order to organize their interviews by different perspectives. The people that they interviewed did not necessarily interact with the target software system the same way, but rather had different roles in interacting with the system, such as the customer or vendor, where each experienced different parts of the system. The interaction design group used various canvases for exploring the different groups, their dynamics with one another, and the emergent categories within each group. They performed this exploration by creating multiple copies of a template canvas that contained images of all fifty of the people they interviewed onto other canvases in order to explore different categorizations. For example, they organized the people they interviewed by location, finances, knowledge of technology, and similar stories. 

\textbf{Backing out to review work.} The interaction design group underwent cycles of intensive activity within a particular canvas, and bursts of consecutive movements between several canvases. In their initial exploration, they organized their image scraps along various one- or two-dimensional plots. After working in these canvases, the usage logs showed that the interaction design group moved between a zoomed out view and the different perspectives across canvases in rapid succession, which they reported as reviewing their designs in progress.

\begin{figure}%
  \centering
  \subfigure[One-dimensional plot with scattered notes] {
      \label{fig:ixdgroup:session1:a}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep1}
   } 
  \subfigure[People interviewed organized using a two-dimensional plot] {
      \label{fig:ixdgroup:session1:e}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep2}
   }
  \subfigure[Three-dimensional organization juxtaposed against two-dimensional organization] {
      \label{fig:ixdgroup:session1:f}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep3}
   }
  \subfigure[People interviewed organized by table, Euler diagrams, and colored tags] {
      \label{fig:ixdgroup:session1:b}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep4}
   }   
  \subfigure[People interviewed organized into spatial clusters with labeled topics] {
      \label{fig:ixdgroup:session1:c}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep5}
   }
  \subfigure[Story-driven flow chart of people's experiences] {
      \label{fig:ixdgroup:session1:d}     
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/ixd-group/rep6}
   }   
   \caption {Various visual representations used by the interaction design group to create personas. \textit{Note: content in sketches does not depict actual work by the interaction design group, but instead are fictional examples of types of representations that they used.}}
   \label{fig:ixdgroup:session1}
\end{figure}%

\subsection{Representations}

The interaction design group created several representations in order to help them categorize their data. 

\textbf{One-dimensional plot.} When the designers began their design session, they created a one-dimensional plot, as in Figure \ref{fig:ixdgroup:session1:a}, to categorize the people they interviewed by a particular quality. They began writing names of individuals along the plot, grouping those individuals into lists, and marked the plot's horizontal axis with qualities at intervals between the lists of names. Additional text was written at each interval, or at edges, describing additional information about the qualities.

\textbf{Multi-dimensional plot.} In Figures \ref{fig:ixdgroup:session1:e} and \ref{fig:ixdgroup:session1:f}, the interaction design group used a two-dimensional plot to further organize their image scraps. The interaction design group first began with a one-dimensional plot that depicted a range of behaviors within a particular quality. They then placed the images and names of the individuals they interviewed along this plot. Afterward, they converted the sketch into a two-dimensional plot by drawing a line down the middle, and then further grouping the images along the new plot. Within the canvas depicted in Figure \ref{fig:ixdgroup:session1:f}, they juxtaposed a three-dimensional triangle plot next to a two-dimensional plot, both of which categorized a different set of scraps.

\textbf{Organization by plots, tags, and Euler notation.} After copying scraps onto the canvas, they used several techniques to simultaneously overlay  different layers of categorization. Within Figure \ref{fig:ixdgroup:session1:b}, the interaction design group used three types of categorization. First, they used a table. When the designers first entered this canvas, they initially dropped image scraps of those that they interviewed onto the canvas to view, and immediately afterwards drew a one-dimensional line and partitioned it into segments, similar to the canvas in Figure \ref{fig:ixdgroup:session1:a}. This time, however, they extended the one-dimensional line into a table by drawing long vertical lines and categorized the scraps within these spaces. Second, they used Euler diagram notation to depict emergent groupings by circling a set of image scraps and writing the name of the category next to the grouping. For example, the two image scraps in the top left have a yellow circle around them with the text ``NO CONDITION''. Third, they used color tags to denote another level of grouping. On the far left side, they created a legend with mappings between color and a category name. They then tagged the image scraps with color. The result was a set of image scraps that were organized using a table, Euler diagrams, and color tags.

\textbf{Spatially clustering objects.} In Figure \ref{fig:ixdgroup:session1:c}, the designers allowed the categories to emerge from the individuals they interviewed in a bottom-up fashion. In this canvas, they first wrote the topic of the canvas, which was ``Design Behaviors'', and grouped the faces of the people interviewed into categories based on their experiences, and then wrote the name of the grouping, such as ``ENJOYS TO EXPLORE'' or ``SEEKING OTHER TOOLS''. The interaction design group first wrote the topics using the regular pen, then later converted the topics into text scraps. The interaction design group further copied and pasted their notes from that session, and annotated with a dash to indicate that it was an attribute of a cluster. The interaction design group similarly spatially clustered their image scraps in Figure \ref{fig:ixdgroup:session1:d}, but additionally used arrows, and in one case formal connectors between scraps, to associate an image scrap with more than one group.

\textbf{Story-driven flowchart. } In one particular instance, the interaction design group walked through the customer's experience by drawing a story, as in Figure \ref{fig:ixdgroup:session1:d}. From the stories they gathered in their interview, they drew the different events that occurred during an average customer's purchasing experience, and noted possible options that may occur at each point. In order to construct the story, they reused elements they had stored in the palette, such as the image of a stick figure to indicate actors, and various icons to represent events, such as a phone to call a customer after a pending order was fulfilled. For particularly interesting events and corner cases in the story, the interaction design group placed the image of the person they interviewed to add to the story. 

\textbf{Storyboards.} The interaction design group created storyboards that demonstrated user interactions in a step-wise fashion. The interaction design group drew boxes that contained a picture of each step in a chain of actions, a set of descriptions for that image, and call-outs explaining pieces of the drawn picture.

\subsection{Collaborative work}

\textbf{Sharing control.} The interaction design group reported that a motivator for using Calico was to share control of content on the whiteboard. They further reported that they typically fall into the roles of either producing content at the whiteboard while the other participates from their laptop and provides design critiques of the content produced. When they did use Calico, the interaction design group, for the most part, continued their traditional roles, where the non-sketching interaction designer provided critiques while developing personas. However, while developing storyboards, the interaction designer taking on the non-sketching role was pro-active in typing out notes for each storyboard frame, entering both text and copy-pasting his pre-existing notes from an excel spreadsheet.

\section{Results: Deployment at a distributed software-based research team}
\label{chapter:evaluation:deployment3}

The research group used Calico over a period of five months in their meetings and design sessions. The high-level goal of their research was to develop an online solution for crowdsourcing the writing of software programs. Prior to using Calico, the team had already developed a software prototype, and used a whiteboard to discuss their designs, which consisted of process flows. They reported that their design had become too complex to perform solely on the whiteboard, and believed that Calico could help them evolve the design while still maintaining the informality of the whiteboard. The research group turned to Calico to ``work out the details of the process flow diagram once and for all''. At the beginning of the five month period, the remote member of the team flew in locally for a five day intensive session in which they created a grand design of the entirety of the system. They continued their collaboration remotely the remaining time. 

\subsubsection{Overview of design sessions}

The research group created several dozen canvases in their months of usage, using it to record meeting notes, design the underlying software architecture, and sketch out the user interface. The research group reported creating two major design diagrams, which consisted of a process-flow chart of the entire system, depicted in Figure \ref{fig:researchgroup:a}, and a second flow-chart for testing user-generated code in Figure \ref{fig:researchgroup:b}. Both were created during the initial five day session, and were continuously updated afterward. In creating the process flow in Figure \ref{fig:researchgroup:a}, the research group first sketched out the system as it existed at the date of the meeting. They then continued designing the parts of the system that did not exist yet. After four months of usage, they chose to perform a major refactoring of their system and the generated diagrams were no longer current. However, the research group reported that they continued to reference the old figures.


%Original design predated calico. Originally had it as a sketch in his office. ``What kind of process flow diagram was this?''. Ben towne came to town... they had the design machine, but it wasn't really worked out or designed in any sense. The motivation was that they wanted to sit down and work out the details of the process flow diagram once and for all. The design was their effort to sit through all the different sort of states that could be there. All the transitions. They spent a few days doing edits on their diagrams. Canvas1 was a process flow diagram for functions. They have another state machien for tests. The code already existed... so all through the week they would look back at the diagram, and figure out what was implemented and what wasn't.
%
%From the interview, the researchers created two major designs using Calico. The first was a process-flow chart taken 
%
%They wanted a way to add a layer so they could easily annotate the diagrams without messing up what was there. They instead kept track of everything in their heads...
%
%The diagram was already becoming very busy, it became hard to change it. It became an archival view

While two members of the research group were concerned with designing the system in its entirety, a third joined the research project with the intention of focusing on one specific aspect of the project, specifically how debugging is performed by the end-users of the system. Having joined the project after a prototype was already created, this additional group member needed to be onboarded \ref{cherubini2007let}. Throughout this process, he worked with the other members to become acquainted with the project by using Calico to sketch components of the infrastructure relevant to his tasks, and further build his project on top of the existing pieces of the system. In his role, he created sketches pertaining to his part of the system (Figure \ref{fig:researchgroup:c}), and additional sketches to help him understand the system (Figure \ref{fig:researchgroup:d}).

%- Another major member of the project joined four months into the project.
%- Extended system into his aspect.
%- used calico for onboarding.

\begin{figure}%
  \centering
  \subfigure[Process flow that described the main functionality of the software system] {
      \label{fig:researchgroup:a}
      \includegraphics[width=14cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas1}
   }  
  \subfigure[Process flow representing flow of how user submitted programs are tested] {
      \label{fig:researchgroup:b}
      \includegraphics[width=14cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas2}
   }
   \caption {Representations created by the research group during their one-week intense session.}
   \label{fig:researchgroup:1}   
\end{figure}%

\begin{figure}%
  \centering
  \subfigure[Process flow representing content from Figure \ref{fig:researchgroup:b} juxtaposed against a table representing another view on the process flow] {
      \label{fig:researchgroup:c}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas3}
   }
  \subfigure[Screenshot of live system used to design future versions] {
      \label{fig:researchgroup:d}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas8}
   }
  \subfigure[Members of the research group copied code into Calico to understand and revise system] {
      \label{fig:researchgroup:e}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas6}
   }
  \subfigure[User interface mockup of next version] {
      \label{fig:researchgroup:f}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/researchgroup/canvas7}
   }      
   \caption {Representations created by the onboarded researcher that were used to better understand the existing software system.}
   \label{fig:researchgroup:2}   
\end{figure}%

\subsection{Feature usage}

\subsubsection{General feedback}

Overall, the research group found Calico to be a tool that supported them in carrying out their meetings. Prior to using Calico, the group used whiteboards to conduct their meetings, and would email pictures taken of the whiteboard among the group for those that were remotely located. They found that Calico allowed them to create more complex diagrams than they would have on the whiteboard. They further reported that for much of their design process they preferred to create the diagrams in Calico rather than a formal tool because they wished to maintain the informal feel of the whiteboard, and the ability to draw around the diagrams as needed. They did, however, recreate their major diagrams in Calico in more formal tools in order to document their designs.

All members reported that they felt more comfortable sketching lots of diagrams because of virtually unlimited space. All group members valued having their old designs readily accessible, and with some framing of their context due to intentional interfaces. One group member reported that he wanted ``to keep a history of what [I]'ve done, the branches that [I]'ve pruned. If you're designing complex things with stages, [you] need to tell a story''.

One of the members reported that it served as a lightweight thinking tool for him. The third reported that he typically used lightweight note-taking tools such as Evernote, which helped his individual thinking process, and later shared his work with the group. He reported that Calico offered more flexibility in what he could create, and found the flexibility helpful in preparing content before sharing it in group meetings.

%`we've already tried that, would you like to see where we've gone? It's already here''.

They reported some difficulties with Calico. Group members reported that they sometimes had difficulty launching Calico, or setting it up on a new machine was sometimes an obstacle with collaborators. Also, while sketching was adequate, the drop in writing quality was sometimes distracting. They often overcame this by using text scraps, but this was not always the case. 
%They stated that Calico lacked necessary formal objects, which they compensated for by created detailed sketch objects and copying them.

%- sort of like evernote. He uses it like he would evernote, because he's most of the time alone, then shares it with the other group memebers
%- ``designs get very complex''. 

%In interviews, I asked users if using Calico and its features were invasive to their design activity...

%When asked if they made any compromises in order to perform their activities using Calico, 

%their goal was to just draw something up. They thought about it in terms of a freehand drawing space.

%- 


%- sketching is a bit off
%- moving things is one of the largest positives


%- one member likes to work in structures

\subsubsection{Feature specific}

%When asked about the features as a whole...

\textbf{Basic sketching and features.} The research group reported that the basic sketching features were adequate for their design sessions, although they reported that the electronic whiteboard made their handwriting messy. They further reported that some of the menu buttons were not clear in their functionality. They expected the ``navigate backwards'' and ``navigate forwards'' buttons to move back and forth in links within the intentional interface rather than navigating the history of visited canvases. They further thought the ``clear canvas'' button was the eraser mode button, causing them to accidentally clear their canvas, which they recovered by pressing ``undo''.

\textbf{Scraps.} The research group made frequent use of scraps across their design sessions. They found scraps and connectors useful because both made otherwise static elements interactive, such as when working with process flows. They reported feeling less afraid to create complex diagrams because the elements could be moved.  They could create space by moving scraps, and also categorizing objects, such as when placing them in tables, like in Figure \ref{fig:researchgroup:c}. 

They encountered some difficulties using scraps. Scraps were difficult to select when many were overlapping. Scraps could not have their text changed, which required the research group to delete scraps and their connectors when text needed to be changed. They further desired the press-and-hold selection gesture to trigger more quickly.
%Encountered problem when they had multiple scraps in one area and tried to select it. They had layered annotations, and that made selecting things a problem.
%- issues where they couldn't edit the text
%- adding connectors was a pain, because editing required deleting, and recreating it.
%
%- wrestled with trying to draw sometimes, selecting
%
%Connectors very important. Referencing arrows very important. Nice that arrows could lock on, and arrows would move while scraps would drag.
%
%Creating and moving scraps sometimes left you with a line. It works some of the time, but it's not predictable whether it's going to work or not going to work.
%
%Adding text is not intuitive, stumbled upon the keyboard shortcuts by accident.
%
%- preferred scrap interaction over lasso 
%
%ngs within a canvas
%
%- overall he enjoyed his experience
%- he wanted a way for press-and-hold to be faster

\textbf{Palette.} The research group did not use the palette in their sessions. Rather, when using repeated elements, they copied canvases and deleted non-relevant content. They also did not use repeated elements in their sketches, but rather always sketched new content, or reused old sketches.

\textbf{Intentional interfaces.} The research group made heavy use of the breadcrumb bar to move between canvases. They found that using the breadcrumbs was the easiest way to move between canvases. They reported that canvases were too small to visually identify in the cluster view, where canvases in a circle with thirty or more canvases became very small. Identifying clusters by name in the breadcrumb bar was easier in comparison.

Two members of the research group also did not use tagging with intentional interfaces because they found the system of tagging to be confusing. They found grouping canvases by clusters helpful because it allowed them to separate content between members. However, within clusters, they did not link canvases. The third researcher, in contrast, used tagging to link canvases into linear chains, each of which represented a topic. He reported that many canvases by themselves were difficult to return back to and read without context, but if viewed one after the other, one could more readily understand the concepts in juxtaposition to the previous canvas. On the other hand, he found the act of tagging canvases as ``alternatives'' or ``abstractions'' awkward because he was not sure what those meant in the context of his sketches.

%- he did not use tagging because he didn't really understand it.
%- useful if he could tag pieces of this
%- A problem is the performance
%- difficult to read things out of context. But if they look at canvases one after the other, it makes it easier to understand concepts that extend the previous ones.

\textbf{Fading highlighter.} The members made light use of the fading highlighter. They reported that, while it was useful on occasion, they preferred to talk out loud and make pointing gestures using their hands. In remote sessions, the group member from CMU relied on video communication to see these non-verbal cues. Though they found the fading highlighter helpful, they oftentimes forgot that it was available. They reported that they would have used it more if it had been easier to invoke.

\subsection{Canvas navigation}

During sessions involving the design of the process flow in Figure \ref{fig:researchgroup:a}, the research group reported that they seldomly moved to another canvas. Rather, they used additional tools such as Google Docs to record requirements and design decisions, or simply talked out loud. All sessions using Calico also involved the use of Skype and Google Docs.

\textbf{Using perspectives to support onboarding.} The research group member who underwent onboarding used canvases of several perspectives in becoming acquainted with the software system. This member used a mix of user interface mockups, freehand sketches, tables, and other diagrams to step through the implementation of the software already written and the design of their own new features. This initial exploration involved importing an image of the interface into Calico,  and stepping through the interface to inspect items pertaining to debugging user generated code. From these sketches, he switched to other canvases in which he generated the process flow in Figure \ref{fig:researchgroup:b}, which was another perspective of Figure \ref{fig:researchgroup:a}, but distilled to only include details relevant to his project. Later, while again exploring the process flow in Figure \ref{fig:researchgroup:c}, the group member created several copies of the canvas, preserving the process flow at the top of the canvas, and creating different tables, lists, and mock-ups below while experimenting with different algorithms.

\textbf{Exploring alternative process flows.} The same researcher that underwent onboarding drew several alternatives for his part of the system. He created several copies of the canvas in Figure \ref{fig:researchgroup:c}, which consisted of a process flow diagram juxtaposed against a table. Between these alternatives, the process flow depicted at the top of the canvas did not change. Instead, the table below served as a template to experiment with alternative paths for components. The group member frequently referenced his previous canvases to compare against other alternatives.

%Having recently used diagrams would have been helpful. Mostly used 1 or 2 diagrams, at most 4 to 5.

%- began with mockups. in every statement that used a callee function, they logged before, and after.

%- simplification of large diagram. From the large sketch, created a separate sketch that was a distillation of his own parts

\subsection{Representations}

The research group primarily used Calico to work on process flows, however they additionally used other types of representations.

\textbf{Lists.}  Not depicted, lists were used on occasion to record bullet points from meetings or during brainstorms. The research group used a mix of both written lists, which were used as scratch notes, and text scraps, which were rapidly created during brainstorm sessions. On the whole, lists were not used often because the research group took notes in Google Docs, and used Calico to supplement written documents.

Additionally, in some complex diagrams, the research group reported keeping track of ideas using lists. For example, in a sketch similar to Figure \ref{fig:researchgroup:c}, a group member reported writing down questions he had about states of the diagram, which he would investigate later. Similarly, the top of Figure \ref{fig:researchgroup:d} contains additional notes and questions while brainstorming a user interface.

%- came up with some questions while creating their design, noted them on the side.

\textbf{Timeline.} Not depicted here, timelines were used to record target dates for important dates, such as conference deadlines.

\textbf{Process flow.} The research group used process flows extensively, as the design of how tasks flowed from one person to another are critical to their design. As such, a lot of detail was put into their process flows in order to capture the details of the system.

The research group used process flows as their representation of choice to design their system. During their intensive five day meeting, they attempted to model the entirety of their software onto one canvas, calling it a ``a view onto their system''. As the design grew, they encountered issues with available space within the canvas, but did not wish to break up their design across multiple canvases. The research group reported that they ``thrashed about how much detail they should put in the diagram''. As their design grew, they shifted from using plain sketches with drawn arrows, to using scraps with connectors. They found text scraps easier to read than their own handwriting and more space efficient. Also, they found the ability to have connectors remain attached to scraps as they were moved very useful, as it allowed them to make space for more states, or place similar states in close proximity to one another. After the five day session, they reported a resistance to changing the diagram because of its complexity. They ``knew where everything was because it had always been there'', and when they did make additions, those parts of the diagram ``grew organically'' by occupying vacant white space without displacing the positions of the scraps around it.
% - it's a view onto their system, and they think about the correspondence between all of their views

They encoded additional information into their diagrams by using color, as depicted in Figure \ref{fig:researchgroup:a} . One research group member stated that they ``needed some way to encode what was going on'', where some scraps represented states, some represented tests, and some as points of branching. For example, black connectors represented a state transition with no information passed, blue represented the passage of tokens, and so on. The research group reported some difficulty in recalling the meaning of all colors for connectors. They further distinguished the scraps representing states that were not yet implemented by tagging them with color. In the upper right of Figure \ref{fig:researchgroup:a}, they wrote ``Red: not yet implemented'' in red, and tagged scraps with this quality by underlining them with red. 

%Tagging.
%- annotated stuff with categories (what was implemented, what was not)
%- blue represents sending a whole artifacts.
%- used pen color to declare different types of connector
%- needed some way to encode what was going on. Some boxes were states, some are tests, some are particular branches

When the research group described the diagram, they saw it not as a complex network of individual states, but rather as a set of higher-level structures that represented different parts of a complex work flow. That is, they implicitly grouped parts of the diagram into larger components. Their discussions took place over different regions of the diagram, which they understood among themselves to represent components and verbally referred to them by name, but did not record within the process flow because they were not sure how to represent them. When talking about features they requested a way to declare ``these are the typical paths'' or ``these two paths are similar''. When their discussions took place, they discussed specific workflows within the diagram, and discussed what happens in those workflows. The research group struggled with how to abstract away these workflows, and created additional diagrams, such as diagram depicted in Figure \ref{fig:researchgroup:b}.

%Paths
%- 
%- ``layering'' was an important concept.  They wanted to declare ``these are the typical paths'' or ``these two paths are similar'',
%-  ``annotate that particular paths have a particular meaning''.
%- were trying to encode different workflows, what happens in different situations
%- became difficult to keep track of all the paths
%- thought about ways to abstract what was going on

%Figure \ref{fig:researchgroup:1}
%- didn't know how to use connector feature

%- simplification of large diagram. From the large sketch, created a separate sketch that was a distillation of his own parts

\textbf{Table.} One of the research group members used tables to help him process sketches while onboarding. He created tables such as the one depicted in Figure \ref{fig:researchgroup:c}, and juxtaposed them against simplified sketches of process flows in order in support of a detailed examination. The process flow in Figure \ref{fig:researchgroup:c} contained several paths that could be taken, and the group member created tables to capture a particular workflow. He did so by listing each state in the row header, transitions along the column headers, and placing scraps in the intersecting cells. He created several such tables to capture different workflows.

One of the group members reported that they created the table in Figure \ref{fig:researchgroup:c} to examine a set of corner cases. The table served as an aid to help them think through the process flow. The group member reported that they used the table to simulate an algorithm, thinking through ``how this would be executed''. They reported that scraps were helpful in being able to move things. They reported that they normally would have created this diagram on a whiteboard, but in comparison, Calico provided ``lots of space'', and made their table ``very clean'' because they did not have to redraw things. 

%``Dynamic execution of [his] diagram''. Would have enjoyed being able to play states.

One of the group members, in contrast to the others, used Calico more frequently than the others to help him design while working alone. He would prepare several canvases of sketches prior to a meeting, sketches including lists, tablets, process flows, etc., to explain his designs. He expressed that he preferred to make his designs within Calico both because the other members were already using Calico, and because sharing sketches in Calico was faster because it was already a shared space.

%- Examined corner cases. Came up with different solution. Created mockups. Looked at dependencies at subcalls.
%
%- worked alone, and used canvas to help himself think through. Used table to help him think through the points of failure in the process flow.
%
%- Looked for corner cases
%
%- u

\textbf{User interface mockups. } The research group created prototypes of the user interface late in the design cycle, after having already created a working system that they could interact with. They used mockups during discussions to suggest changes to the user interface, and when designing new web pages. During these sessions, they imported screenshots of their current interface into Calico as a visual reference, and sketched interfaces without the use of scraps.

The onboarded group member created user interface mockups within Calico in order to help him in thinking through his design. In his task, he used a combination of screenshots and sketched mock-ups to first walk through how existing end-users perform their task and submit their finished work. From these sketches, he created a process flow to capture the interaction of the user and the flow of information.

%- began with mockups. in every statement that used a callee function, they logged before, and after.
%
%- difficult to think through the problem
%- created mockups of what they wanted the user to create
%- ``similar to building blocks''.
%- they wanted to create a visual frontend for building a user interface
%- problem with people programming things as event listeners... harder to build his tool around that

%- broken up into different structures. Some with arrows.

\textbf{Spatially clustered text scraps. } In two meetings, the research group generated a set of text scraps as part of a brainstorm, and later grouped them into similar topics. They did not use formal arrows, but rather used grouping symbols, such as Euler diagrams, and drew arrows between the groups to indicate different types of relationships. 

\textbf{Dendrogram.} One researcher created a horizontal dendrogram to explore a set of questions generated from other sketches. The researcher began with a set of four questions, and from those questions, expanded into a tree of questions to explore a problem. They drew arrows to link items between the questions, as well as several freehand annotations.

\textbf{Source code.} In several of the meetings, one of the group members pasted screenshots of source code into Calico to discuss his implementation. He pasted both psuedo code and actual Javascript code. In meetings, he drew call outs from the screenshots to explain each method call, as in Figure \ref{fig:researchgroup:e}. 

\subsection{Collaborative work}

The research group conducted meetings using the two electronic whiteboards as well as their own laptops. During co-located meetings, they sometimes loaded both boards with Calico, but visually viewed different canvases on each whiteboard, or used the second board to display Google Docs. They reported that only one person usually sketched, while another took notes on in Google Docs, and any remaining participants simply talked. 

%During meetings, one person mostly sketched.
%- They used both boards. Second board had either second diagram (canvas 2), google doc, or skype face. Almost always the case where they edited just one of the canvases.
%- They mostly talked through particular scenarios. One person was in charge of drawing everything.

\textbf{Helping remote members ``feeling connected''.} In distributed meetings, remote members reported feeling ``more connected'' when using both Calico and video chat software. In these meetings, members often used another display loaded with either Skype or Google Hangout. The remote participant reported that the ideal setup included a camera that pointed at the boards with Calico, because it allowed him to observe the body language of the speakers. They reported that this setup was particularly helpful during the early phases of their collaboration because people often pointed to objects on the boards, gestured to content at the board from their seat, or gestured in free space while explaining an idea. The combination of both being able to see the body language of remote members and being able to manipulate that same content in real-time culminated into the sense of ``connectedness''.

%Remote member felt connected.
%- When person left, they could reference things pretty easily.
%- particularly useful use case was when the camera was in the back, and pointed at both the calico board and the skype board, and it showed where people were pointing at. They had body langauge, what they were pointing at, etc. 
%- Body langauge is important in the early phases, because people are pointing, gesturing, using hands in their explanation of meaning.

\textbf{Continuity between design sessions.} The research group also reported that Calico made it easier to stop and resume design sessions. One member reported that the ``biggest benefit was crossing space and time'', in which Calico provided a consistent virtual space to conduct meetings prior to the five day visit, during, and long after. All canvases remain in a consistent location that all group members can always access. They did not need to worry about cleaning up any materials in their space or sharing whiteboard space with other research groups conducting meetings in the same space. 

%Starting and stopping.
%- ``Biggest benefit was crossing space and time''. Being able to ``put things on pause and resume''. Content was mobile across rooms, and able to stop and start activities again next time because content was where they left it.

\textbf{Providing an archival reference.} After the diagrams became out of date, the research group reported that they used Calico as an ``archival reference''. The research group originally thought that they would continuously interact with their sketches, but instead used their sketches as visual references. The team eventually referenced the sketches less and less as they had internalized the contents of the sketches and could reference freely without the need to have the sketches in front of them. However, they reported that they valued having the sketches in Calico rather than static photographs of whiteboards. They reported that it gave the sketches ``a sense of permanence'' because they could return and make adjustments to the sketches at any time, which they did. After undergoing a major refactoring, the content of the sketches in Calico became outdated, however, they did reference them still at times while moving forward. They reported that it was ``good to have a reference of what the architecture was like in the previous version''. They went back to past designs to see what they did before. When looking back at previous designs, they noted that ``it would have been nice to have annotations as to why this [previous design] wasn't going to work''. They remarked that while their sketched designs were most useful during implementation, they ``would turn back to the diagram because they forgot how they implemented something while writing''.

%Used Calico as an archival reference.
%- The team internalized their sketches from Calico, and pulled up sketches occasionally as an archival reference.
%- Thought it would be more interactive, but it wound up being visual reference.
%- It was more about having this shared representation that was visible, and stuck around longer.
%- Did have advantage over picture because they could tweak it.
%- It wasn't finalized until it was in the paper
%- It was more helpful during implementation, but would turn back to the diagram because they forgot how they implemented something while writing.
%- refactoring did not get reflect in Calico. However, they did reference it again while moving forward. It was good have a reference of what the architecture was like in the previous version. They went back to past designs. 
%- ``It would have been nice to have annotations as to why this wasn't going to work''.
%- It would be more useful to be able to add notes within the context of sketches, to declare rationale of a specific spot.

%- they created one whole design, that was later thrown away. The design persisted, but they said that it was thrown out. Interesting that the designs still remained

\section{Design behaviors}
\label{chapter:evaluation:design-behaviors}
In this section, I turn to examining the design behaviors across all three sessions. I report on whether they occurred or not, and whether the features of Calico were used as intended. This is not an exhaustive review, i.e., I am not reporting on every instance of each design behavior occurring. Rather, I am looking for proof of existence. This breaks down into three parts: (1) did I see proof of the designers performing the design behaviors, (2) did they use the features of Calico to perform them, and (3) if so, which features.

%Start this section with a table. Summary of findings. Talk about each row of the table, because each row is a design behavior, in a little more detail
%Each sub section: design behavior... explanation. Now, rather than defending each one. 

\subsection{Design Behaviors Summary}

Based on evidence from usage logs, as corroborated by the interviews, all design behaviors were observed as occurring at least once. More specifically, all 14 design behaviors occurred in the OSS group, 9 of the design behaviors occurred in interaction design group, and 13 of the design behaviors occurred in the research group. With respect to the usage of features, 13 of the 14 design behaviors for the OSS group were performed using Calico's features, 5 of the 9 design behaviors for the interaction design group were performed using Calico's features, and 12 of the 13 design behaviors were performed using Calico's features for the research group.

Table \ref{chapter:evaluation:designbehaviors-table} summarizes the features that supported each design behavior, which included scraps \& connectors, palette, intentional interfaces, and fading highlighter. Referring back to the table of features in Chapter \ref{chapter:calico-version-two} (Table \ref{table:calico-version-two:designbehaviors}), each feature was designed to support a specific set of design behaviors. In Table \ref{chapter:evaluation:designbehaviors-table}, all non-targeted features are shaded in gray; that is to highlight when features were used in support of behaviors they were not explicitly designed for.

A summary of the design behaviors as they occurred is presented as follows:

\begin{enumerate}
	\item \textbf{They drew different kinds of diagrams.} All groups performed this behavior using scraps, as well as with plain sketching as on regular whiteboards without using Calico features.
	\item \textbf{They produce sketches that draw what they need and no more.} All groups created scraps that were either sparse in details or notationally rich, depending on their needs. They also did so with plain sketching when not using scraps.
	\item \textbf{They refine and evolve their sketches over time.} The OSS group refined sketches into box-and-arrow diagrams using scraps. The interaction design group refined one-dimensional plots into both two-dimensional plots and tables using regular sketching.
	\item \textbf{They use impromptu notations.} Both the research group and the interaction design group created custom notations using scraps and connectors. The interaction design group created custom icons using the palette. The OSS group created their own visual language using plain sketching by creating connectors with custom colors, shapes, and textures.
	\item \textbf{They move from one perspective to another}. All three groups used intentional interfaces to move between canvases of different perspectives. The OSS group and the research group benefited from intentional interfaces because chaining canvases provided an ordering for the canvases, which helped members of each group remember the meaning of their sketches within canvases. All groups also moved between perspectives within a single canvas.
	\item \textbf{They move from one alternative to another.} The OSS group and the research group used intentional interfaces to copy canvases and label them as alternatives. All three groups discussed alternatives verbally without using Calico.
	\item \textbf{They move from one level of abstraction to another.} The OSS group and the research group used intentional interfaces to step between canvases of different levels of abstraction. The OSS group used scraps to represent the software components they ``stepped into'' and ``out of''.
	\item \textbf{They perform mental simulations.} The OSS group and the research group used the fading highlighter to walk through the flow of data in their designs. The interaction design group created elaborate sketches that walked through the story of a user using regular sketching, but did so without any of the advanced features of Calico.
	\item \textbf{They juxtapose sketches.} The OSS group and the research group used scraps to position sketches next to one another for reference. The research group used the palette to copy content across canvases so that they could reference it. All groups also drew sketches of different perspectives in the same area to juxtapose them.
	\item \textbf{They review their progress.} All groups reported that they used intentional interfaces to review the sketches they have created (and the grid in case of the interaction design group). The OSS group and the research group also created handwritten lists that summarized the designs that they recently created.
	\item \textbf{They retreat to previous ideas.} The OSS group and research group both used intentional interfaces to return to past designs and design decisions. The research group also returned to designs that were several months old.
	\item \textbf{They switch between synchronous and asynchronous work.} The OSS group and the research group had members that used intentional interfaces to deviate from the group discussion to their own canvas to reference content or create alternatives. However, though it did occur, it occurred relatively rarely in the research group. 
	\item \textbf{They explain sketches to each other.} The OSS group and research group used the fading highlighter to explain designs. The OSS group reported finding this very useful to discuss data that passed between software components. 
	\item \textbf{They bring their work together.} The OSS group created new canvases that combined work from previous canvases.
\end{enumerate}

\begin{center}
\begin{longtable}{|p{5cm}|p{5cm}|c|c|c|}
\caption{The set of design behaviors and the features that supported them}\\
\hline
\textbf{Design Behavior} & \textbf{Supporting Feature} & \textbf{OSS} & \textbf{IxD} & \textbf{Res} \\
\hline
\endfirsthead
\multicolumn{4}{c}%
{\tablename\ \thetable\ -- \textit{Continued from previous page}} \\
\hline
\textbf{Design Behavior} & \textbf{Supporting Feature} & \textbf{OSS} & \textbf{IxD} & \textbf{Res} \\
\hline
\endhead
\hline \multicolumn{4}{r}{\textit{Continued on next page}} \\
\endfoot
\hline
\endlastfoot
\hline
1. They draw different kinds of diagrams&Scraps \& connectors &X &X &X  \\* 
\hhline{|-|-|-|-|-|}
%\hhline{|~|-|-|-|-|}
2. They produce sketches that draw what they need, and no more&Scraps \& connectors &X &X &X  \\\hhline{|-|-|-|-|-|}
3. They refine and evolve their sketches over time&Scraps \& connectors &X & &  \\\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{4. They use impromptu notations}&Scraps \& connectors & &X &X  \\\hhline{|~|-|-|-|-|}
&Palette & &X &  \\*\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{5. They move from one perspective to another}&Scraps \& connectors &X & &X  \\\hhline{|~|-|-|-|-|}
&Intentional interfaces &X &X &X  \\*\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{6. They move from one alternative to another}&Scraps \& connectors &X & & \\*
\hhline{|~|-|-|-|-|}
&Intentional interfaces &X & &X  \\*
\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{7. They move between levels of abstraction}& Scraps \& connectors &X & & \\*
\hhline{|~|-|-|-|-|}
&Intentional interfaces &X & &X  \\*
\hhline{|-|-|-|-|-|}
8. They perform mental simulations&Fading highlighter &X & &X \\\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{9. They juxtapose sketches}&Scraps \& connectors &X & &X  \\*
\hhline{|~|-|-|-|-|}
& \cellcolor[gray]{0.8}Palette & \cellcolor[gray]{0.8}& \cellcolor[gray]{0.8}& \cellcolor[gray]{0.8}X  \\*
\hhline{|-|-|-|-|-|}
10. They review their progress&Intentional interfaces &X &X &X  \\*
\hhline{|-|-|-|-|-|}
11. They retreat to previous ideas&Intentional interfaces &X & &X  \\*
\hhline{|-|-|-|-|-|}
12. They switch between synchronous and asynchronous work&Intentional interfaces &X & &X  \\*
\hhline{|-|-|-|-|-|}
13. They explain their sketches to each other&Fading highlighter &X & &X \\\hhline{|-|-|-|-|-|}
\multirow{2}{5cm}{14. They bring their work together}& \cellcolor[gray]{0.8}Scraps \& connectors & \cellcolor[gray]{0.8}& \cellcolor[gray]{0.8}& \cellcolor[gray]{0.8} X \\*\hhline{|~|-|-|-|-|}
&Intentional interfaces &X & & 
\label{chapter:evaluation:designbehaviors-table}
\end{longtable}
\end{center}

\subsection{Kinds of sketches software designers produce}

\subsubsection{Design Behavior 1: They draw different kinds of diagrams}

Across all three field deployments, the developers and designers created several representations using different types of notations, or approximations of notations. All groups used lists, the OSS group used box-and-arrow diagrams, the interaction design group made heavy use of one- and two-dimensional plots, and the research group used process flows and tables. Most of the canvases across all of the sites used only one type of notation, but a few of the canvases within each group also contained a mixed set of representations using different notations. This difference compared to the experiments in Chapter \ref{chapter:calico-version-one} and \ref{chapter:notation-paper} came about because, unlike the controlled experiments in which the entire design session occurred at the whiteboard with no other materials, the designers in-the-field turned to Calico for specific tasks, such as when the OSS group created sketches to plan the explanation of an architecture, which may lead to canvases with more targeted content. Longer term use of Calico may lead to using it for more in-depth tasks, designers may supplement it with other tools such as word documents, their own computers, and so on. Designers may also need to create fewer sketches since they may have internalized many of the representations they sketched in previous meetings, which the research group reported to be the case for them.

The OSS group and research group created many different types of representations using scraps. The OSS group used scraps to create box-and-arrow diagrams and user interface mockups while brainstorming the elements and look of a GUI, as depicted in Figure \ref{fig:ossgroup:session1:a}, and mixed lists with user interface mockups in \ref{fig:ossgroup:session1:b}. In each group, creating the representations in the same space allowed them to evolve both concurrently, where the box-and-arrow and list representations allowed the developers to consider the components from a structural perspective, and the mockups allowed them to consider the design from the end-user's perspective. They reported that, by depicting these elements as scraps, it made the elements easier to move and resize around the canvas, and therefore felt more like entities in their eyes. For the research group, one member of the research group mixed different types of representations in one of his canvases in order to help him think. In Figure \ref{fig:researchgroup:c}, the researcher used a table in order help him step through a process flow, and used scraps to move items around the table.

The interaction design group, outside of image scraps and text scraps, seldom used scraps to depict major concepts in their canvases, but did create different types of representations as they would have on a regular whiteboard. They used scraps to depict very basic concepts, such as image scraps to show the faces of those they interviewed and text scraps to produce legible text when their handwriting itself was not legible. However, when creating complex representations, they simply sketched as they would have on a regular whiteboard. The interaction design group placed graphs of different types next to one another, such as the one-dimensional and two-dimensional plots in Figures \ref{fig:ixdgroup:session1:a}, and the triangle graph and two-dimensional plot in \ref{fig:ixdgroup:session1:c}. 

%- Same canvas? Not often... Figure in OSS group. Figure in Interaction design group. Figure in research group (christian's!)

\subsubsection{Design Behavior 2: They produce sketches that draw what they need, and no more}

All groups engaged in this behavior, in which they rarely actually completed their sketches in all detail, and used full notational conventions sparingly. Evidence of this came from both the amount of formal conventions used in the sketches, and the disparity between the design as drawn in Calico and the designs as explained in the interviews.

First, all groups varied the amount of notational conventions used between sketches, even when those sketches were of the same representation type. The OSS group, for example, expressed most of their software architectures using only boxes-and-arrows, and labeled the connecting arrows in very few instances. In most cases, the OSS group discussed the design verbally and only added detail to their diagrams in order to have something to point at during discussion. The interaction design group used varying amounts of notational convention, sometimes labeling the axes in their plots in great detail, as in Figures \ref{fig:ixdgroup:session1:a} and \ref{fig:ixdgroup:session1:d}, other times including very little detail as in Figure \ref{fig:ixdgroup:session1:b}. The research group also engaged in this design behavior when working with their process flows, in which only some states contained entry and exit conditions, and many arrows lacked labels. 

Second, it became evident through the interviews that the sketches made in Calico omitted many important details of designs sessions. In interviews, members from the OSS group and the research group had difficulties identifying the meaning of some of the sketches they created, however they recalled the overall objective of the sketches, which they deemed more important than the individual details. These sketches were used to support activity while ``in the moment'', where sketches were used to accomplish an immediate goal such as brainstorming, explaining, or, in the case of the interaction design group, categorizing people interviewed. For example, a single developer in the OSS group created a sketch to explore use cases of a user interface (Figure \ref{fig:ossgroup:session1:e}), but only used the bare minimum detail to do so. The sketch itself is a distillation of previous sketches, Figures \ref{fig:ossgroup:session1:c} and \ref{fig:ossgroup:session1:d}, and does not use notations to suggest that it is a use case diagram. The source code in Figure \ref{fig:ossgroup:session3} is also a sketch used in an individual session with minimal detail. The developer invested the time to write pseudo-code, but used arrows with minimal explanation of their meaning. 

In the above cases, all groups used scraps to perform this design behavior, and the OSS group and interaction designer group used the palette as well. The OSS group reported that they used scraps to represent ``important entities'', which served as a low-detail representation that they could point to during design discussions and move around the canvas. They reused these low-detail entities across several canvases using the palette as well. The interaction design group used image scraps of people, which they saw as the lowest level of detail in their sketches. They additionally saved low-detail icons such as a telephone and computer to the palette, and placed them in their sketches. Their sketches did not explain these icons, but their meaning was known throughout the design discussion. The research group also used scraps to represent states in process diagrams, which they manipulated quite heavily. Across all groups, content that was not the subject of manipulation often did not become scraps.

The fading highlighter also played an important role in supporting low-detail diagrams for the OSS group. This group used the fading highlighter extensively to discuss components, draw flows of data, and discuss details of software components without permanently adding more details. By using the highlighter instead of actually drawing more detail, they explicitly chose to preserve sketches at a low level of detail.

% externalize an entity so that they can externally examine it, but sketching done in the examination includes a scarce amount of detail. 
%
%The sketches used to support individual thinking had few details and little use of formal notations.
%
%Of the sketches used to support personal thinking
%
%Sketches to support personal thinking. In gui, created something because it didn't look right. Didn't include much detail of rationale. In code, got important part on board, but didn't write beyond that, used arrows to help him think.
%
%Sketches to support group brainstorming. In group brainstorm, had more details.
%
%Sketches to support explanation to group. Explanation, had much fewer details.
%
%With respect to notational convention, we saw very little of that present.
%
%- minimal drawings in oss group. ``they only draw what they need'', very true for much of the OSS group. Diagrams support discussions, no need to declare details since they won't be executed! The things that get written down are the major entities. Objects for OSS group. pictures for interaction design group. state transition names for researchers
%
%- notational convention is almost non-existant. These are not deliveries, but rather, representations provide framing for conversations (thinking of OSS group).

\subsubsection{Design Behavior 3: They refine and evolve their sketches over time}

Two of the three groups exhibited this design behavior. In the case of the OSS group, they first sketched the names of entities, and these sketched names eventually became text scraps with connectors. The result of these sketches is depicted in Figure \ref{fig:ossgroup:session1:a}. In this sketch, the developers began with simple sketches, and increased their notational convention and amount of detail over time. In the case of the interaction design group, they began with pictures of faces, and across several canvases, categorized these pictures using visual structures such as tables and plots. Within Figure \ref{fig:ixdgroup:session1:b} in particular, the interaction design group evolved their representation by beginning with a single dimensional line, adding categories on that line, and transforming it into a table. The interaction design group did not set out to create the table, but rather created the table after categorizing the faces. Other representations also began as a one-dimensional plot, such as Figure \ref{fig:ixdgroup:session1:e}, which was later refined into a two-dimensional table.

The research group did not exhibit this behavior. This was because the research group began their major design sessions by copying existing diagrams from the whiteboard into Calico, and did not start design sessions from scratch, as was the case for the OSS group and the interaction design group in the sketches they refined.

The OSS group used the scraps feature to refine an existing sketch, but the interaction design group did not. Scraps helped the OSS group refine their sketches because scraps support the creation of box-and-arrow diagrams. The interaction design group refined their sketches into more complex plots, which scraps do not support.

\subsubsection{Design Behavior 4: They use impromptu notations}

All three groups performed this design behavior in which they created a visual language that was particular to their own respective designs. Each group made consistent use of their own devised notations, meaningfully using connectors, tagging, and self-drawn icons to layer meaning on structured diagrams.

The interaction design group and research group used scraps to create their own notations. Both groups used Euler diagrams to group scraps by drawing circles around sketches (used within Figures \ref{fig:ixdgroup:session1:b} and \ref{fig:researchgroup:b}, among other diagrams not shown). Each group tagged scraps using colors, where the interaction design group used color patches in Figure \ref{fig:ixdgroup:session1:b}, and the research group used colored underlines in Figure \ref{fig:researchgroup:a}. The research group, unlike the interaction design group, used color coded connectors with their scraps, which represented specific types of transitions between process flows. In the case of the colored connectors, the data types were informally known by the people who sketched them, but their meaning was not apparent from just the sketch alone. 

The OSS group also created their own notations, but did not use scraps to do so. Like the research group, the OSS group used colored arrows to represent different data types. However, the OSS group improvised representations in ways that regular Calico connectors could not, such as by using shape and texture to give meaning to their sketches. The arrows in Figure \ref{fig:ossgroup:session2:b} used a dashed texture to represent data, as opposed to solid arrows which were method calls between components. In Figure \ref{fig:ossgroup:session2:c}, the OSS group used colored boxes to represent events that were broadcasted to multiple components, but originated from the same source. Further, the member from the OSS group that created the psuedo code in Figure \ref{fig:ossgroup:session3} improvised several types of arrows that served as placeholders for concepts during his walkthrough. Interestingly, the member from the OSS group could not identify the meaning of his own improvisations afterwards, but reported that they helped his thinking during design.

The interaction design group further used the palette to create their own set of scrap icons. They created symbols such as a phone, a computer, and other symbols to represent actions, which they used used in their storyboards by placing the icons next to actors who could perform those actions.

%(examples include Figures \ref{fig:ossgroup:session1:a}, \ref{fig:ossgroup:session2:d}, \ref{fig:ixdgroup:session1:a}, \ref{fig:researchgroup:session2:b})

%- definitely, show the pictures from the source code thing. Show further notations from calico
%
%- yes, reappropriated, but only from plain sketches into boxes...

\subsection{How they use the sketches to navigate through a design problem}

\subsubsection{Design Behavior 5: They move from one perspective to another}

All groups shifted their focus among multiple canvases with different perspectives in their design sessions. Both the OSS group and the research group, in working with source code, moved between perspectives such as the user interface, software architecture, lists of requirements, source code, etc. The interaction design group, in contrast, used Calico to build personas from interviews, and moved between canvases that categorized the same data, but using different categorizations and visual structures (such as tables, one- and two-dimensional plots, etc.). 

The OSS group made heavy use of intentional interfaces in moving between perspectives. The OSS group chained canvases in their sessions, which provided them the benefit of ordering canvases such that a chain of canvases could convey a story. Sometimes this order reflected the chronology of their exploration in the design space, where the earliest canvas in a session was on the inner circular ring. However, canvases were sometimes inserted within the chain when they returned back to previous sketches and deviated to a new idea. Overall, having the canvases linked into groups helped them remember ``how the session played-out''. In some cases, members from the OSS group could not identify parts of their sketches, but were able to recall their meaning by examining the surrounding canvases in the chain.

The research group also used intentional interfaces, but to a lesser degree than the OSS group. In the research group's intensive five day session, they reported that they preferred to maintain a single diagram so that all components of their system were always visible in their meetings. They instead preferred to examine other perspectives using their own computer within Google Docs. One of the members of the research group, however, did use intentional interfaces to move between multiple perspectives in his own design sessions. This member used perspectives to help in learning the system. He moved between canvases containing large diagrams and others containing screenshots of the user interface in order to understand how the different pieces worked together. 

In the case of the interaction design group, it was difficult to judge their performance with the fifth design behavior because they had little time to experiment with intentional interfaces in comparison to the other groups. During their five day research session, they used the grid to conduct their design session, in which they used the grid extensively to produce several different perspectives and they did move between those different perspectives, sometimes even rapidly switching between several canvases. During their one hour brief session in which they used the version of Calico with intentional interfaces in order to create storyboards, they limited their session to a single canvas, and ignored the intentional interfaces feature altogether. They reported that it was important to them to be able to see their sketches in their entirety the whole time. 

While the three groups used intentional interfaces to different degrees for moving between canvases, all three groups found the ``copy canvas'' ability useful. The interaction design group in particular used a template canvas to begin explorations of new perspectives. For each canvas, they set a generic topic, such as ``Design behaviors'' (Figure \ref{fig:ixdgroup:session1:c}), from which sub-categories emerged from their grouping of content. Both the OSS group and the research group used copied content to examine elements juxtaposed against other representations, such as tables and mockups.

%- all sw groups used it in the classical sense
%- interacitn designers explored emergent dimensions of data
%- researchers used it for exploring the system.

\subsubsection{Design Behavior 6: They move from one alternative to another}

Two of the groups did use multiple canvases to explore multiple alternatives, but the interaction design group did not. Within the OSS group, the alternatives were often generated as a result of conflicting opinions during discussion. Multiple members had conflicting opinions, which inspired some members to copy a canvas and generate their own interpretation. In another case, a member of the research group returned to the same diagram of a process flow at a later point after having implemented it, and created several alternative copies of the canvas, in which he explored different process flows. Across both the OSS group and research group, the members used tags to label their canvases as alternatives. Also common in both groups, the members requested better tagging functionality to declare which alternative they ultimately chose, but also did not want to discard unused alternatives, because they valued them as records of past design explorations.

In the case of the interaction design group, they did not explore alternatives using multiple canvases. Unlike the other two groups, the goal of the interaction design group in their sessions was to build a better understanding the people they interviewed within a particular context. Instead of drawing multiple alternative sketches, they negotiated alternatives verbally, for example, by proposing different ways of categorizing content in a canvas, or if a person belonged to one category or another.

\subsubsection{Design Behavior 7: They move from one level of abstraction to another}

Similar to the role of perspectives, members from the OSS group and the research group moved between levels of abstraction to help focus their attention within the design. The OSS group moved between levels of abstraction in their sessions to step through the flow of data within their software architecture. In stepping through components, they copied canvases, expanded scraps to dive into more detail, and created new canvases with scraps that represent a high level view of their architecture. In the case of the research group, one member shifted a level of abstraction to break down diagrams and focus on relevant parts. He reported creating a copy of a process flow that depicted the entire system, erasing all components not relevant to his task, and designed a subpart of the system. In the case of the interaction design group, they instead performed a bottom-up approach, working from dozens of image scraps, and creating high level abstractions that described the people in the photos. All observed groups created canvases with text, either handwritten or using text scraps with list scraps, that summarized the contents of other canvases, which they referred back to while designing components. 

The OSS group and research group used intentional interfaces to move between levels of abstraction, and the OSS group additionally used scraps for this purpose as well. The OSS group used it the most fluidly, where they used intentional interfaces to continue their work onto another canvas without interruption. When ``jumping into'' a component, they copied a canvas, tagged the canvas using intentional interfaces, and reused the scraps that were copied from the previous canvas.  The OSS group used scraps to represent the components that they ``stepped into'' and ``stepped out of''. When ``jumping out of'' a component, they created a blank canvas and sketched the component. The research group did not use intentional interfaces as often, but reported it as useful for moving abstractions, and the interaction design group did not use it at all for this purpose, neither in their sessions with the grid nor their session with intentional interfaces.

%- yes, true for oss group and researchers
%- requires a certain amount of complexity

\subsubsection{Design Behavior 8: They perform mental simulations}

All groups reported that they mentally stepped through their sketches, both verbally in groups and on their own. Both the OSS group and the research group mentally simulated the flow of data within their system. The OSS group displayed sketches of their architecture on the large electronic whiteboard with Calico in meetings, and discussed the same sketch for long periods of time, both gesturing at components with their hands and using the fading highlighter from a tablet that was remotely connected to the same canvas. The research group performed similar activities, but rather than drawing entities and software components, they used process flows, and walked through the states of data as it traveled through their system. The interaction design group, instead, walked through user scenarios, and sketched flow diagrams that described the story of the user. The interaction design group also mentally walked through the stories of the people they interviewed by sketching elaborate work flows, such as in Figure \ref{fig:ixdgroup:session1:f}.

The OSS group made made heavy use of the fading highlighter in this activity, the research group used it some. In one particular meeting, four members of the OSS group discussed a Calico sketch for 30 minutes using the fading highlighter, as depicted in Figure \ref{fig:designbehaviors:mental-simulation-fig-b}. The fading highlighter allowed members to use a tablet to mark up the sketch on the large electronic whiteboard from their seat, allowing them to circle items as in Figure \ref{fig:designbehaviors:mental-simulation-fig-a}.  The research group also used the fading highlighter in their meetings, but not as often. They reported that they internalized their process flow diagrams, and did not need to perform as many detailed walkthroughs of their design. The fading highlighter was reported as being useful when a person had a tablet in their hands, and the screen was broadcast to everyone else. They also stated that they would have used the fading highlighter more in their walkthroughs had they remembered that the fading highlighter existed.

\begin{figure}%
  \centering
  \subfigure[Example of single fading highlighter stroke] {
      \label{fig:designbehaviors:mental-simulation-fig-a}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/designbehaviors/mentalsimulation}
   }
  \subfigure[Composite of 10 minutes of fading highlighter use] {
      \label{fig:designbehaviors:mental-simulation-fig-b}
      \includegraphics[width=7.5cm,keepaspectratio]{./figures/Evaluation/designbehaviors/mentalsimulation-composite}
   }
   \caption {The OSS group sometimes spent as long as 30 minutes walking through the data flow of a diagram using the fading highlighter.}
   \label{fig:designbehaviors:mental-simulation-fig}   
\end{figure}%

In contrast to the other groups, the interaction design group did not use the fading highlighter. There are two possible reasons for this. First, the fading highlighter was most useful when using multiple devices, but the interaction design group preferred to use only the electronic whiteboard. Second, it is possible that the interaction design group preferred to sketch the mental simulation as a workflow instead of using the fading highlighter. The members of the interaction design group were both skilled in sketching visual icons expressing ideas in walkthroughs, and could readily create these icons without interrupting their work flow. Figure \ref{fig:ixdgroup:session1:f} depicts an example sketch that includes such icons.

\subsubsection{Design Behavior 9: They juxtapose sketches}

While not performed often, the three groups did occasionally set their representations side-by-side within the same canvas, i.e., \textit{juxtapose} them. In the majority of sketches created, the users preferred to create multiple canvases, and compared content informally by moving back and forth between canvases. However, they did juxtapose content within the same canvas for specific purposes, such as early exploration and reference.

The OSS group and the research group used scraps to help them juxtapose several different perspectives at once during very early phase exploration. In creating a new user interface, the OSS group created lists, box-and-arrow structures, and low-fidelity interface mockups within the same space, as seen in Figure \ref{fig:ossgroup:session1:a}. During this early phase brainstorming, the box-and-arrow diagrams evoked imagery of possible user interfaces in the OSS group members, which they sketched out in the periphery using scraps. The OSS group moved the content around the canvas using scraps, and also placed the objects side-by-side. The research group similarly used mockups alongside software structures, but instead began from the user interface and wrote pieces of code. 

The OSS group and research group also juxtaposed sketches in order to have a reference while creating a newer sketch. One OSS group member imported source code as an image to reference while designing psuedo code. A research group member also imported screenshots of the application's user interface while designing his part of the system. He further copied pieces of the process flow and used an adjacent table to step through the diagram, as in Figure \ref{fig:researchgroup:c}.

The interaction design group also juxtaposed their sketches, but did not use scraps to do so. They created multiple charts within the same space, finding that sometimes their data did not fall within one type of categorization, but two different types, as in Figure \ref{fig:ixdgroup:session1:f}. 

\subsubsection{Design Behavior 10: They review their progress}

Evidence that users reviewed their progress during their design sessions was more difficult to distinguish, but all groups reported performing this activity to some degree. Nearly all participants summarized or distilled the essence of an activity to a bullet point list in a canvas, which they would occasionally reference and update. The OSS group seldom marked up canvases with annotations, but instead created canvases with lists, in which they would note details about software components and entities from their past conversations. The research group would sometimes create bullet point lists to guide their meetings, but most often turned to Google Docs to review their agenda within their meetings. The interaction design group used a single canvas with several notes from their interviews, to which they referred when moving between different perspectives. 

Intentional interfaces served a strong supporting role when reviewing progress in all groups. Usage logs showed that members from all groups would occasionally rapidly move back and forth between several canvases, or move to a bird's eye view to review several canvases at once. Interviews confirmed that they were reviewing progress across canvases and moved to this view to ``take it all in'', respectively. All groups reported that the cluster view was not detailed enough to compare components, but instead it served to anchor discussions and allow those present at meetings to gesture at the canvases while they talked.

\subsubsection{Design Behavior 11: They retreat to previous ideas}

Retreating to previous ideas was a behavior that was only observed in multi-week, long-term design sessions. Both the OSS group and the research group  continued to use Calico within the same project, and reported that they did not return to previous ideas until a later design session. Members from both the OSS group and the research groups reported that they created new alternative designs in later sessions, but returned to past sessions to refresh their memory on the past approaches that they took. In the case of the research group, they performed a significant restructuring of their software architecture, but one of the members reported returning to out-of-date sketches because it reminded him of the rationale for certain design decisions.

Intentional interfaces supported the OSS group and research group in retreating to previous ideas. The OSS group very rarely returned to past sketches, but they reported that although infrequent, they found it useful on occasion. The research group reported that intentional interfaces was useful to them in returning to designs that were several months old. The research group stated that they could identify their old design sessions from the cluster view based on the visual structure of canvases. The chaining of their canvases in the cluster view provided contextual cues, such as relationship of canvases and order of canvases in chains, that helped the research group members remember the meaning of their sketches. 

\subsection{How they collaborate on them}

\subsubsection{Design Behavior 12: They switch between synchronous and asynchronous work}

The OSS group and the research group switched between synchronous and asynchronous work in their meetings, while the interaction design group did not. During their respective meetings, members from the OSS group and the research group deviated from the ongoing discussion to their own canvas to explore an opportunistic thought, and later share their insight with the group. The interaction design group, in contrast, have an established work culture of working in pairs to brainstorm and reflect on each others' work. This led to the interaction design group not exhibiting this behavior.

Members from the OSS group reported that they deviated to asynchronous work at least once every design session, and did so using the intentional interfaces feature. 
In one instance, a member from the OSS group disagreed with the layout of a user interface in Figure \ref{fig:ossgroup:session1:c}, and used the ``copy canvas'' feature to begin the process of creating the alternative in Figure \ref{fig:ossgroup:session1:d}. After creating his alternative, the member presented his changes back to the group on the large electronic whiteboard. In another instance, another member of the OSS group wanted to unify the diagrams diffused across several canvases, and deviated from the group discussion to draw the system in a specific use case, which is depicted in Figure \ref{fig:ossgroup:session2:h}. Once the member brought this use case of the group's attention, they created several copies of this canvas to examine use cases with different inputs. In both of these examples, intentional interfaces helped the group members navigate between these canvases by chaining them.

The research group also engaged in this design behavior using intentional interfaces, but to a lesser degree than the OSS group. In their design sessions, they navigated to other canvases in order to reference past sketches, and also to update sketches in other canvases. The members of this group sometimes requested the other group members to join them in another canvas, but the remote member reported difficulty in navigating to that canvas using intentional interfaces. In contrast to the OSS group, the research group did not report branching off to create alternatives. This was possibly due to the setting. Members in this group did not have as much face-to-face time as members of the OSS group who shared an office, which led to members focusing more in meetings. The members of the research group met regularly one to two times a week, which they used to update each other on their progress, and discuss issues they encountered. 

The interaction design group did not engage in this design behavior. During their sessions, one member of the group controlled the board while they worked together to address the design problem. One member of the group took the lead in creating sketches, while the other offered his feedback verbally.

%- an enabler
%- most important in oss group
%- ixd didn't do it at all
%- res less so
%
%- true in group design sessions

\subsubsection{Design Behavior 13: They explain their sketches to each other}

All groups encountered situations in which they needed to explain their sketches to one another. The members from the interaction design group worked very tightly together, and most explanations came from one designer challenging the design decisions of the other designer. Members of the OSS group worked more independently, where some used Calico on their own, and later presented their designs to receive feedback from other developers in the group. The members of the research group operated much more independently, in which several days would elapse before they coordinated their efforts and would present summaries of their latest work to one another. In nearly all cases, explanations were primarily carried out by pointing, gesturing in the air, and freely speaking to one another.

The highlighter was the most helpful feature to support this design behavior, however, most teams reported often forgetting to use it ``in the heat of the moment''. The interaction design group did not use the fading highlighter, preferring to simply speak out loud. The OSS group used the fading highlighter for extended periods of time when stepping through an explanation of a design that spanned several canvases. They reported sometimes being slowed down during explanations when moving between canvases, requiring that they announce what canvas they move to so that everyone else can move to that canvas as well. The research group also found the highlighter helpful, but reported often forgetting that the feature existed.

\subsubsection{Design Behavior 14: They bring their work together}

Groups very rarely brought their work together, or merged their work from separate canvases, after performing asynchronous work. The interaction design group did not perform asynchronous work and did not have an opportunity to exhibit this behavior. The research group used Calico to present their work in weekly meetings, but they did not combine their work. The OSS group, however, did bring together separate ideas, but they did so by creating new canvases. 

The OSS group created new canvases to combine work using intentional interfaces. In two separate occasions in which members from the OSS group moved between synchronous and asynchronous work in the same meeting, they generated copies of the same canvas that were variations of one another, but they did not merge these canvases. Instead, they created a new canvas, linking it to the previous canvases using tagging, and sketched a use case scenario that summarized the content from previous canvases. 

%- not so much in the real world... everything that is sketched are suggestions. Bringing work together is too much work. They may summarize though.

\section{Summary}
\label{chapter:evaluation:summary}

In this chapter, I presented my observations of Calico Version Two in use at three locations. The commercial open source software company provided insight in how a software team may use Calico to support ongoing with work among developers while coding. The interaction design group provided insight into how interaction designers may use Calico to help them create personas and storyboards. Lastly, the research group provided insight into how a set of distributed researchers may use Calico in support of long-term collaboration in the development of a software system for a period of several months. In the next section, I bring these separate uses together by discussing their collective lessons and insights in the context of Calico's support for design behaviors.

%%% Local Variables: ***
%%% mode: latex ***
%%% TeX-master: "thesis.tex" ***
%%% End: ***
%
%\subsubsection{Design Behavior 1: They draw different kinds of diagrams}
%
%\subsubsection{Design Behavior 2: They produce sketches that draw what they need, and no more}
%
%\subsubsection{Design Behavior 3: They refine and evolve their sketches over time}
%
%\subsubsection{Design Behavior 4: They use impromptu notations}
%
%\subsubsection{Design Behavior 5: They move from one perspective to another}
%
%\subsubsection{Design Behavior 6: They move from one alternative to another}
%
%\subsubsection{Design Behavior 7: They move from one level of abstraction to another}
%
%\subsubsection{Design Behavior 8: They perform mental simulations}
%
%\subsubsection{Design Behavior 9: They juxtapose sketches}
%
%\subsubsection{Design Behavior 10: They review their progress}
%
%\subsubsection{Design Behavior 11: They retreat to previous ideas}
%
%\subsubsection{Design Behavior 12: They switch between synchronous and asynchronous work}
%
%\subsubsection{Design Behavior 13: They explain their sketches to each other}
%
%\subsubsection{Design Behavior 14: They bring their work together}