Conclusions.tex

\chapter{Conclusions and Future Work}
\label{chapter:conclusions}


In this dissertation, I have presented a sketch-based environment to support software designers during the early phases of design. I have done so by building a set of minimally invasive features that together support the set of fourteen design behaviors by software designers. Specifically, designers across a range of disciplines used scraps to create many different types of representations in their work. They used the grid and intentional interfaces to navigate between their sketches, and, importantly, intentional interfaces helped designers to recall the context in which they were created. Lastly, they used intentional interfaces and the fading highlighter to collaborate within their team, both in local and distributed settings.

My analysis evaluated the design sessions of 39 individuals, comprising of 16 computer science graduate students at the whiteboard, 16 computer science graduate students using Calico Version One, and 7 professionals across three software-related organizations using Calico Version Two. From these design sessions, 1) I have built and tested an initial prototype, Calico Version One, and 2) built the second iteration, Calico Version Two, and performed a field evaluation at three different professional organizations using Calico to conduct their own work. From these deployments, I have gained insight into how each feature supported the design behaviors in practice.

Scraps were a powerful feature that played a major role in supporting the kinds of sketches that designers create, and a minor role in supporting them to navigate and collaborate. Scraps were most useful in representing box-and-arrow type software representations in the OSS group and the research group. Most scraps used little detail, most times only having the name of the component they represented. The improvised notations were surprising as well, in which users tagged scraps and connectors with colors to signify meaning in their design sessions. Scraps also served to help navigate between sketches in the same canvas by helping to juxtapose diagrams, and move diagrams out of the way. Overall, scraps did not satisfy everyone's needs, such as the interaction designer group, but in most cases, they were helpful in supporting many of the design behaviors in practice.

The palette played a minor role in supporting the design behaviors, only supporting the use of impromptu notations and juxtaposing diagrams. While I expected designers to use the palette more in practice, designers preferred to reuse diagrams simply by redrawing them by hand, using the scrap copy feature, or copying a canvas. However, the scrap served the role as a global clipboard, and while it was very rarely used, it served an important purpose: to easily move content across canvases, which designers did to reuse particular sketches as a reference.

Intentional interfaces played a large role in supporting navigation between sketches and collaboration. It gave all groups that used it a sense of boundless space that encouraged them to sketch more, and which they used to create different perspectives, alternatives, and levels of abstraction. Clusters provided a means to separate projects, and canvases linked using tagging provided a means to relate canvases to one another. Most importantly, linked canvases provided an ordering to canvases, which users in the OSS group and the research used to create a narrative from their sessions. The organization of intentional interfaces further served as a useful metaphor to organize collaboration, which occurred in both co-located and distributed settings.

The fading highlighter only supported two design behaviors, mental simulations and explaining designs, but it supported these design behaviors in ways that the other features could not. The fading highlighter was sometimes not used because users forgot that it was there, but when it was used, it served as an aid to verbal walkthroughs. It was more helpful than a simple pointer because the speaker could draw arrows, and could do so without modifying the original sketch. In particular, it was most helpful in multi-device or distributed settings, allowing listeners to look to the large electronic whiteboard while another person sketched from their device. 

Together these four features supported all fourteen design behaviors, and did so while being minimally invasive to the design activity. The features worked together to support the design behaviors, and did so while maintaining the flexibility and fluidity of the regular whiteboard. 

\section{Summary}

My contribution from each of the chapters are as follows:

 \begin{enumerate}

   \item \textbf{Chapter \ref{chapter:motivation} (\nameref{chapter:motivation})} introduced the definition of design behaviors in detail and presented the grand set of design behaviors that I aimed to support. 

   \item \textbf{Chapter \ref{chapter:research-question} (\nameref{chapter:research-question})} posed the topic that this dissertation addressed: \textit{What minimally invasive, coherent set of features can be design that is sufficient to effectively support these behaviors?} I layed out the method by which I approach this topic and how I was to evaluate this approach.

   \item \textbf{Chapter \ref{chapter:calico-version-one} (\nameref{chapter:calico-version-one})} presented the first prototype of Calico, which was built toward testing the feasibility of the approach. It did so by first testing a subset of the design behaviors, specifically design behaviors 1, 2, 3, and 5. To address these behaviors, I introduced three features: scraps, the grid, and the palette. The approach was evaluated in a comparative study between Calico and the regular whiteboard, both with respect to the design behaviors performed as well as the design conversations that took place. In the qualitative analysis, I found that the four design behaviors were indeed supported by Calico, however, participants did not feel they were well supported by scraps. From the quantitative conversation analysis, I found that the design activity was carried out the same in Calico as it is on the whiteboard, and Calico may lead to more shared focus between participants and longer discussions of individual topics.

   \item \textbf{Chapter \ref{chapter:calico-version-two} (\nameref{chapter:calico-version-two})} introduced the second and final version of Calico in this chapter. These features were: the revised interaction for scraps, the intentional interfaces features that replaces the grid, the distributed nature of the architecture, as well as other features targeted at supporting the full set of design behaviors. I further present their implementation in detail, present examples of how these features support the fourteen design behaviors, and describe software implementation of this second iteration.

   \item \textbf{Chapter \ref{chapter:evaluation} (\nameref{chapter:evaluation})} presented the study of the final version of Calico ``in the wild'' at a commercial open source software company, an interaction design company, and a software research group. Each setting presented a unique set of representations, methods for navigations over sketches, and ways of collaborating. In the commercial open source software company, Calico supported a group of developers in designing the next iteration of their software in both individual and group design sessions. In the interaction design group, Calico supported a pair of designers processing a set of interviews to create personas in preparation for a software project. In the research group, Calico supported a distributed group collaborating in building a software system and also helped to onboarding a new member.

   \item \textbf{Chapter \ref{chapter:discussion} (\nameref{chapter:discussion})} took a step back and put my work in a broader context. I discussed the implications of my findings, reconnected my work to the broader design literature, and provided some findings and observations that are outside the evaluative framework of the design behaviors, but are worthwhile to highlight nonetheless. 
   
% discussed the implications of my findings, reconnect my work back to the research question in Chapter \ref{chapter:research-question}. From the interviews, I found that Calico was minimally invasive to those working with software representations such as box-and-arrow and state diagrams. However, the scraps feature and its system gestures was found to be confusing for those with existing experience with other sketch tools that have their own set of gestures. Most features within the tool were found to be cohesive with one another, which was judged by the number of features that support one another. Over half of the features were benefited by the other features, where basic sketching received the most benefit from others, and the fading highlighter received the least benefit. 
   
%   With respect to the design behaviors, all design design behaviors were observed within the three groups, and were supported in doing so with Calico's features. All groups used scraps to create different types of representations that were low-detailed, were refined over time, and used improvised notations. Two of the three groups, the OSS group and the research group, used intentional interfaces to navigate between their canvases, moving between different levels of abstraction, perspectives, and alternatives. They also used the fading highlighter to review their work and mentally simulated over their sketches in group meetings. The same two groups used intentional interfaces and the fading highlighter to collaborate over their sketches by explaining sketches and diverging into asynchronous work. The groups, however, did not bring their work back together, as the final design behavior suggested.
   
   \item \textbf{Chapter \ref{chapter:related-work} (\nameref{chapter:related-work})} reviewed the existing literature of tools that support software design on an electronic whiteboard, covering both tools that support sketch recognition and those that support the management of sketches.
 \end{enumerate}

\section{Directions for Future Work}
This dissertation illuminates several potential areas of study for future work. 

\subsection{Compositional notations}

The first area of future work extends from providing more domain specific support for scraps. While each group did not strictly use notations, they did create sketches that borrowed idiomatic pieces from various formalisms, such as UML class diagrams, process flow diagrams, etc. In many cases groups used scraps to represent these notations, such as when the OSS group used boxes-and-arrows and the research group used state diagrams, but scraps fell short in fully supporting these custom additions. For example, both groups added cardinality to connectors using plain sketching, but the cardinality was left behind when the scrap was moved. There is an opportunity in this case, and many others, to provide formal support for these elements. 

There are many ways to support these elements. One such case would be to change a scrap itself into specific types of scraps, such as a ``UML class diagram scrap'' or a ``user interface scrap''. List scraps is an example of such as approach, in which an existing scrap can be changed into list scrap to vertically order a set of items. However, this approach does not fully take into account the behavior of refinement of sketches observed in design behavior 3, and may prematurely commit a designer to using a formal notation, when they only want to borrow a part of that notation. This leads to the second approach, which is to allow designers to add parts of notations to existing scraps. I call this approach of adding parts of notations to scraps ``compositional notations'', in which a scrap may be ``composed of'' notational elements, such as a UML class diagram cardinality element, or a user interface element. By allowing the user to build the level of formality of their scrap piece-by-piece, the user can use only those elements that they need, and refine the appearance and behavior of their scraps over a period of time, without enforcing all of the rules of the eventual formal notation too early. 

The compositional notations feature is based on the observation that many formal notations, despite being different, share some parts of their notations. For instance, certain arrows are universally regarded as connectors in UML, Entity-Relationship, and activity diagrams, to name a few. As another example, shape identifies the type of element in both ER Chen notation and activity diagrams. My intuition, then, is to make available these shared, elementary parts as building blocks, so that they can be composed into
notations when and where the designer wants. 

%Based on my observations that designers in various situations do not begin at a notation, but rather refine a sketch by adding notational elements to it (design behavior 3), designers would be better served if they can build on the existing scraps they use.  
%
%Calico current supports this to a degree.
%I have made such a suggestion in a past position paper, which calls this refined support \textit{software modes}
%
%- lists
%- box and arrow diagrams
%- state diagrams
%- tables
%- user interfaces
%////////////
%- designing software
%- designing user interfaces
%- designing requirements
%///
%position paper

\subsection{Informal analysis of scraps}
%
%The second way to support is to provide lightweight feedback of the sket
%
%The second way is to provide an analysis of 
%
%Providing analysis over scraps... nick and alfredo paper
%
A second path of future work is to explore the automated feedback that could be given to a design in an informal setting. The obstacle in this case is that sketching does not lend itself to automated feedback as a sketch is typically not interpretable by anyone other than the designer(s) themselves. In the case of Calico, however, there is the opportunity to build on the scraps feature to bring feedback that is automatically provided by the environment. The key choice, however, is which kind of feedback. In the face of sketching not to be interrupted, it is clear that formal models should be ruled out -- they require far too much specification before they can provide useful feedback; they simply are best left for later phases of modeling. At the same time, one should ask whether the precision that typically comes with the formal models is necessary. More specifically, scraps could be assigned imprecise annotations that enable some interpretation of a sketch.

The hypothesis underneath this approach is that precise values and specifications are not necessary. Rather, I postulate that during early design it is necessary to have an idea of what might happen, to have an approximate answer, and that it is necessary to quickly vary the underlying parameters from which this approximation is derived such that the qualities that are sought can be explored with. 
This approach was investigated in more depth within a workshop paper that provided such support by building on top of Calico \cite{mottalightweight}. It posed four questions:

\begin{enumerate}
	\item What kind of analysis do we want to exploit? 
	\item What type of model is the user going to draw? 
	\item How will the tool acquire the necessary inputs for the models analysis? 
	\item How is the feedback presented to the user?
\end{enumerate}

While the research in the workshop paper did not investigate these questions in detail, it provided an initial prototype to address these questions. It presented a prototype built as a plugin to Calico. It utilized performance analysis because the domain is relatively simple in comparison to other formal approaches. It used activity diagrams, as opposed to other notations such as sequence diagrams, to model performance because past work showed that designers discuss message passing between software components by pointing at representations of the software components and entities. In order to acquire inputs to build the activity diagram, scraps could be tagged with generic values, such as CPU process, network event, database access, which provided generic time-based values for analysis. The feedback was presented back to the user by changing the opacity of the scraps, where the elements in a slow performing diagram became more transparent towards the end of the path.

%in which a user could sketch an informal activity diagram in Calico and receive have their activity diagram change opacity to reflect if the system described by the activity diagram was slow . This work posed 
%
%- research group may have benefited from logic checking in state diagrams
%- OSS group may have benefited from a performance analysis.
- 

\subsection{Contextualized sketching}

A third path of extending work is to embed outside information within Calico so that it can be explored in an informal setting. This area of research is motivated by the observation that much design takes place in the context of a system already under development [1-2], with the code base in existence taken into account. The sketches created during such maintenance design sessions often explicitly represent the existing code base in an abstract manner. Sometimes, as was the case in some of the design sessions of the OSS group, the sketch is even as simple as a few unlabeled boxes that verbally are identified, other times, the sketch contains a significant amount of detail duplicating the code base. In such cases, valuable time can be wasted drawing these diagrams, and there is always the risk that the diagram that is drawn does not faithfully represent the code as it currently exists.

To support these kinds of sessions, Calico could be extended to create a sketchy representation of a code base that can be interactively explored, manipulated, and augmented by designers. An initial idea is that a developer, directly in Calico, should be able to point to a configuration management repository, ask for a specific version of the code for one file, and then be presented with a scrap that represents that file and a set of outgoing lines to other scraps that represent code in the proximity of the main file under consideration. This set of scraps is augmented with a set of explosion points, small markers that when tapped expand the information available to the developers and let them navigate the code base (not unlike Code Bubbles [65]). Tapping on one of the auxiliary classes, for instance, might make it the center and bring up its connections. Tapping a class itself may bring up its methods, tapping a method might even bring up its code. This is of course but one way in which code could be represented. Other ideas involve reverse engineering techniques that automatically identify key methods in classes and just bring those up, using a zooming and panning technique like Code Canvas [66], using slicing to quickly bring up related code, and using higher levels of abstraction such a components or services.

\subsection{Design histories}

A fourth way of extending Calico is to provide a visualization of the design history. The usage logs generated by Calico can be used to reflect on one's own design history, as well as study the design process of professional designers. In addition to these, there are two further opportunities to extend this research, the first of which explores knowledge captured by tags in intentional interfaces, and the second of which juxtaposes views to compare design sessions.

In the first case, the tags captured within intentional interfaces can be used to build a more comprehensive design history. From the history of navigation alone, the usage logs can be used to create time-ordered sequences of the design changes. With tags from intentional interfaces, the time-order sequences can be annotated to highlight important events, such as when an alternative was explored, when designers moved levels of abstraction, and other events as well. 

In the second case, designs histories could be visualized in such a fashion that they can be compared. For instance, the nature of early sessions where the basis for a design is laid could be compared against later sessions where this basis is refined. In another case, this could be used in education to review the design sessions of students. In this latter case, a professor may want to review the design histories of students for homework, but reviewing the design histories of thirty or more students may not be feasible. Instead, the professor may benefit from a visualization that juxtaposed the design sessions of all students in the class so that general trends appear from the histories, for example, if the majority students fixated on one aspect of their design, or did not sufficiently explore alternative solutions.

Capturing design histories by reviewing the usage logs would allow individuals to reflect on the overall design process taken. Previous research has shown that novice and professional designs differ in how they arrive at a solution. For example, novice designs are more likely to fixate on a single problem, while professionals will provisionally make a decision and move on [cite]. Similarly, professionals may focus on two subjects at a time, and regularly rotate between pairs of subjects \cite{Baker2010590}. 
Calico can take advantage of this in two ways: 1) recording the process taken by professional designs in the field so that it can be studied, 2) recording the process of students so that the professor can provide feedback to the students.

With respect to recording the design process of professionals, a professor may use the recorded process as an examplar of official design. In past design courses, professors have provided students with case studies of professionals conducting design, such as videos of them at work. Selected portions of these design histories could be given to the students by the professors as examples. Previous proposals of tools have proposed similar methods of reviewing design histories, such as the Design practice stream (DPS) tools by Nakakoji \cite{Nakakoji6035659}, which plays back a history, and CogSketch \cite{Forbus1149}, which is used to understand how glyphs are used across all several domains.

With respect to recording the design process of students, Calico would enable professors to observe the design activity of students while outside of class. While professors can step in and provide guidance to students who become stuck in class, much work happens outside of class where they cannot provide guidance. The professors often can only see the final artifact or document that students submit. By having histories available, professors can review the histories to see if students are fixating on problems, adequately exploring alternatives, and so on.

It should be noted that the tools to review the usage logs generated by Calico are rudimentary. They provide a usage summary, as well as a set of screenshots of all actions taken, which can become highly verbose and be time consuming to review. There is an opportunity to improve this process so that professors and students can more effectively review their logs.

%%% Local Variables: ***
%%% mode: latex ***
%%% TeX-master: "thesis.tex" ***
%%% End: ***