Motivation.tex

\chapter{Motivation}
\label{chapter:motivation}

\section{Design Behaviors}

Before introducing the design behaviors that I aim to address in my research, it is useful to firmly define a design behavior. I define a design behavior as \emph{a recurrent, recognizable set of actions serving a single purpose within a design meeting.} This definition recognizes the breadth of activities that may occur during a design meeting, from phoning a colleague, to using sticky notes to brainstorm, to sketching some concepts.  For purposes of this dissertation, however, we scope our investigation to design behaviors that occur at the whiteboard, and specifically those pertaining to creating and modifying whiteboard content, navigating the content created, and collaborating in doing so.

I make four important observations regarding this definition:

\begin{itemize}
\item \textbf{\emph{recurrent}} -- A design behavior can be observed to happen consistently across many design meetings and across many designers. Performing a particular action once in a meeting, such as drawing a diagram of an intersection, does not make 'intersection drawing' a design behavior.  Repeated behavior, particularly as applied to different types of diagrams, across multiple design meetings is required. 
\item \textbf{\emph{recognizable}} -- A design behavior stands out as a coherent set of actions within an overall design meeting. The actions clearly belong together and can be distinguished from other groups of actions.
\item \textbf{\emph{set}} -- A design behavior necessarily unfolds over time with multiple actions that a designer undertakes. A single stroke or gesture does not qualify. 
\item \textbf{\emph{serving a single purpose}} -- A design behavior has a purpose in the overall exploration of a design problem and its potential solutions. The set of actions contribute to furthering the exploration.  
\end{itemize}

With this definition in hand, I examined both the software design literature and the broader design literature for design behaviors. What I found was that the general design literature is more mature than the software design literature, in that it has identified quite a few design behaviors that span across different design disciplines. For instance, designers across building architecture, engineering, and product design use constraints to guide their design thinking \cite{cross2007designerly}. As another example, designers in multiple fields use visual similarity between their sketches and the final product to help them envision the eventual physical artifact \cite{do1998right}.
  
The software design literature is only now beginning to catch up to the topic of design behaviors, with the emergence of studies that are beginning to look at software designers ``in action'' (e.g., \cite{baker2012guest,cherubini2007let,dekel2007notation,petre2009insights}). These studies, thus far, confirm the behaviors seen in other design disciplines, but at the same time do not confirm all of them yet, simply because the number of studies remains small. In the below, I include a subset of the design behaviors found in the general design literature, quite a few of which have already been confirmed in the software design literature (i.e., Behaviors 1, 2, 3, 4, 6, 7, 8, 9, 10, and 13) and some of which I believe will be confirmed in future (i.e., Behaviors 5, 11, 12, and 14). I feel confident in making this assumption because of my own informal observations of software designers in action, particularly in studying the videos of the SPSD 2010 workshop \cite{baker2012guest}. While I have not performed full studies of those videos expressly for the purpose of corroborating the general design literature, informally I have seen multiple instances of all of the behaviors I discuss in the below. 

I separate the fourteen behaviors I intend to support with Calico into three categories:

 \begin{enumerate}
 \item Kinds of sketches software designers produce
 \item How they use the sketches to navigate through a design problem
 \item How they collaborate on them
 \end{enumerate}

Each of these categories I detail below.

\section{Kinds of sketches software designers produce}
\label{chapter:motivation:kinds}

The first category deals with what the designers draw. The sketches that designers make at the whiteboard are typically not the goal in and of themselves and, as such, the types of sketches that designers make will vary depending on their current design activity. Sometimes, the sketches are used to help the designer in their thinking, by externalizing their ideas and thoughts onto the whiteboard \cite{lawson1994design}. Other times, the whiteboard is simply a medium to explain a thought, idea, or design in progress. The developer is not using it for problem solving, but instead to communicate information to a listener or collaborator \cite{eugene1992engineering}. Because of these different purposes, what designers draw is dependent on what they work on at what point during a design meeting. This leads to the following design behaviors.

\begin{enumerate}
\item \emph{They draw different kinds of diagrams.} In order to explore a design problem, software designers sketch many different types of diagrams, often within the same canvas [5,17]. They may sketch, for instance, entities and relationships, interface mockups, scenarios, architectures, and other kinds of diagrams \cite{cherubini2007let}. The freedom to sketch multiple kinds of diagrams in the same space is fundamental to supporting the exploration of the design space, as it enables designers to explore an issue from different angles, at different levels of abstraction, or even in different ways altogether. The sketches in Figure \ref{figure:motivation-designbehavior-kinds}, which contains a snapshot of the whiteboard from a design session in which two software designers are designing an intersection in a traffic simulator, present such an example. On the right, the designers used the sketch of a traffic intersection to help them in working through and refining the UML model of some of the entities in the simulation on the left. In turn, their work on the entities lead to revisions to the sketch of the intersection on the right. This behavior of working with multiple representations aligns well with the observed phenomenon that tools which restrict designers to use one notation hinder the design exploration and lead to fewer alternatives being considered \cite{shipman1999incremental}. 

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=8cm,keepaspectratio]{./figures/motivation-designbehavior-kinds.png}
  \caption{Diagram from a design session that includes UML diagrams, a map, and annotations.}
  \label{figure:motivation-designbehavior-kinds}
\end{figure*} 


\item \emph{They produce sketches that draw what they need, and no more.} Of the many sketches that software designers create, few are drawn in full detail. Software designers typically can get what they need from a quick and incomplete sketch, e.g., a barebones user interface or boxes-and-arrows sketch \cite{virzi1996usability}. The benefit of a low-detail sketch is that it can be created quickly, and modified easily, giving the designer more rapid feedback \cite{cherubini2007let,petre2009insights}. Further, providing too much structure too soon can create unconscious barriers to change, resulting in a less exploratory and a less broad search for candidate solutions \citep{wong1992rough}. This behavior further breaks down into two parts:
 \begin{enumerate}
 \item \emph{They only draw what they need with respect to the design at hand.} Low-detail sketches tend to “incorporate relevant information and omit the irrelevant” \cite{tversky2002sketches}, including only as much detail as necessary to advance the designers' thinking. When talking, they will only draw what they need to reinforce what they want to communicate \cite{petre2009insights}. When thinking, they will only draw the details relevant to the immediate issue to help them reason \citep{dekel2007notation}. For example, the elements in the left of Figure \ref{figure:motivation-designbehavior-kinds} differ in the amount of detail they contain, where some contain data attributes and others contain only a name.
 \item \emph{They use only those notational conventions that suit drawing what they need.} Sketches only include as much notational convention as the designer needs in a given situation \cite{petre2009insights}. For instance, if a sketch can express an idea using only boxes-and-arrows, then no more will be drawn, but if a sketch must represent a hierarchical relation, then a richer array of arrows will be present, typically following the convention of an existing formal notation.
 \end{enumerate}

\item \emph{Over time, they refine and evolve their sketches.} The level of detail that designers want in their sketches varies over time. Early on, they may need very little, but later they may need much more as they expand on their ideas \citep{ossher12flexible}. Figure \ref{figure:motivation-designbehavior-refinement} presents an example of such a refinement, which details the evolution of the diagrams from Figure \ref{figure:motivation-designbehavior-kinds} from a relatively simple list, to a complex box-and-arrow diagram. As designers discuss and work through an idea, they will evolve their sketches with additional details to capture their decisions. In general, as part of this refinement process, the sketches will contain more visual precision, and the designer will rework the design to fix any inconsistencies \citep{damm2000supporting}. This behavior, too, breaks down into two parts:
 \begin{enumerate}
 \item	\emph{They detail their sketches with increasing notational convention.} As a designer’s understanding of the design space matures, so does the representation that they use. As I already discussed, while the designer is aware of the full expressive powers of formal notations, they only borrow from those notations what they need at the time. However, as the designer progresses and the decisions firm up and additional, less-critical decisions are made, they tend to use more and more of the formal notational convention to represent their commitment to the chosen direction \citep{ossher2010flexible}. The formal elements in Figure \ref{figure:motivation-designbehavior-refinement} illustrate the decisions captured by the designers, where some arrows contain information about cardinality and some elements are contained in a bounding box to reflect their status as an entity in the model. Note that this move towards a more formal notation is not a strictly uniform activity, as different parts of the design may exist at different levels of maturity \citep{petre2009insights}.

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=8cm,keepaspectratio]{./figures/motivation-designbehavior-refinement.png}
  \caption{Diagram from a design session that was refined from a list into a UML diagram.}
  \label{figure:motivation-designbehavior-refinement}
\end{figure*}  
 
\item \emph{They appropriate a sketch in one notational convention into another notational convention.} Refinement of sketches does not always mean refinement toward and in a single notational convention. Sometimes, designers appropriate one kind of diagram into another \citep{dekel2007notation}. For example, the designers initially created the data model in Figure \ref{figure:motivation-designbehavior-refinement} as a list, then later added boxes, then boxes with arrows, and finally evolved the sketch into a UML class diagram. The designer in all likelihood did not plan this, but in working out their design in place, they re-appropriated the sketch to suit their needs \citep{mangano2012design}. They use what is readily available over re-creating a similar diagram, especially if they do not anticipate needing the original diagram later. 
 \end{enumerate}

\item \emph{They use impromptu notations.} Designers do not exclusively work with the notational convention they know (e.g., UML, ER, etc.), but also, at times, will improvise in the moment. The deviations that they make from standard notations, such as annotating UML diagrams with custom symbols, are deliberate additions that break convention to capture insights before an idea is forgotten. Beyond such annotations and minor deviations, developers also will sometimes adapt wholly new notations on the fly. These often relate to the problem domain that they are explaining, since few domain-specific notations exist, but shorthand is still needed to support the design process \cite{dekel2007notation}. For example, Figure \ref{figure:motivation-designbehavior-impromtu-1} uses freeform text to annotate what looks like a stripped down UML diagram rather than standard UML notation to include additional information. Further, Figure \ref{figure:motivation-designbehavior-impromtu-2} uses rectangular boxes with circles inside to represent a queue of cars waiting for a traffic light, circles with edges to represent traffic lights that govern each queue of cars, and a clock symbol to signify that events in the intersection occur according to a timer component. The designers created these notations on the spot to capture the ideas in their conversation. 

\end{enumerate}

\begin{figure}
  \centering
  \subfigure[UML class diagram] {
      \label{figure:motivation-designbehavior-impromtu-1}
      \includegraphics[width=8.5cm,keepaspectratio]{./figures/motivation-designbehaviors-impromtu}
%      \resizebox{.45\hsize}{.35\hsize}{ }
   }
  \subfigure[Traffic intersection] {
      \label{figure:motivation-designbehavior-impromtu-2}
      \includegraphics[width=6.5cm,keepaspectratio]{./figures/motivation-designbehaviors-impromtu-2}
%      \resizebox{.45\hsize}{.35\hsize}{ } 
   }
   \caption {Diagrams using impromtu notations}
   \label{figure:motivation-designbehavior-impromtu}
\end{figure}

\section{How they use the sketches to navigate through a design problem}
\label{chapter:motivation:navigation}

The second category pertains to how designers use sketches to navigate the design space. While designers may create many different sketches over the course of a design meeting that vary in detail, notation, and what they represent, there is typically a thread of thought that relates the sketches and the ideas represented in them to one another. 

\begin{enumerate}
  \setcounter{enumi}{4}
  \item \emph{They move from one perspective to another.} Software designers create many sketches through which they shift their focus between perspectives. The designer in Figure \ref{figure:motivation-designbehavior-perspectives}, for example, is working on the user interface component for a map, and may navigate to the two different views of the intersection to his left, or explore the UML model of its data model to the far left. The designer uses each new perspective to better understand how the parts of a design fit into the whole, asking questions such as: ``[what] if we look at it like this, from this angle, it fits together like this'' \cite{petre2009insights}. Each perspective presents a new way of looking at the same design, and what may be subtle in one perspective, may be more pronounced and easier to understand in another. 
  
  \begin{figure*}[tbh]
  \centering
  \includegraphics[width=14cm,keepaspectratio]{./figures/motivation-designbehavior-perspectives.png}
  \caption{The designer navigates between different types of diagrams and abstractions.}
  \label{figure:motivation-designbehavior-perspectives}
\end{figure*}  

 \item \emph{They move from one alternative to another.} In a sufficiently complex software design task, a software designer will generate sketches of competing solutions before committing to a particular choice \cite{zannier2007comparing}. Expressing alternatives as sketches rather than simply mentioning them aloud allows designers to manage their focus and more effectively explore alternative solutions \cite{myers2008designers}. Once created, designers can compare alternatives and weigh their trade-offs \cite{buxton2010sketching} (see behavior 5). They may shift their attention back and forth between alternatives and adopt ideas proposed in one into another, or synthesize the ideas of several alternatives into an entirely new alternative \cite{jones1992design}. For example, the sketches in Figure \ref{figure:motivation-designbehavior-alt} depict two approaches that could be used to model a traffic light, where the left sketch models the light as a single signal with a red, yellow, green, and left turn signal, and the two right sketches use two lights that are separate and make the left turn signal independent from the regular traffic symbol. Placing these diagrams side-by-side helps designers in visualizing and discussing the differences between the two. 

\begin{figure*}[tbh]
  \centering
  \includegraphics[width=5cm,keepaspectratio]{./figures/motivation-designbehaviors-alt}
  \caption{Two diagrams that designers used to discuss alternative approaches for modeling a traffic light.}
  \label{figure:motivation-designbehavior-alt}
\end{figure*}   

 \item \emph{They move from one level of abstraction to another.} Software designers move between different levels of abstraction, either by ``diving into'' parts of their design to explore them in more detail or by shifting ``back up'' to the higher-level representation. For example, the designer in Figure \ref{figure:motivation-designbehavior-perspectives} has navigated to the sketch of a map, which presents a zoomed-out view of the two intersections to his left. This shift in abstraction happens often in software design, as many of its notations are hierarchical in nature. A software architect may shift their focus from working out how software components interact with each other to choosing a component and working out how it functions, perhaps by drawing its internal architecture and diving in even further. This behavior typically leads to a multitude of sketches that together consider different abstractions simultaneously \cite{petre2009insights}. Many scenarios requiring shifts of abstraction have been documented, including the design of user interfaces \cite{da2001user}, web pages \cite{van2003design}, and so on.

 \item \emph{They perform mental simulations.} Software designers use mental walkthroughs to gain insight into the consequences of their design \cite{zannier2007model}. They may need to understand how information flows among components, or inspect their design by mimicking how an end user would interact with it. The software designers `interrogate' their design by testing it against hypothetical inputs and scenarios, often marking over their existing sketch while simulating. For example, while discussing the logic cars use in moving through intersections, the designer in Figure \ref{figure: motivation-designbehaviors-mentalsimulation} runs his finger along the path of the map (shown by the white arrow) to mentally simulate the path that a hypothetical car may take, and verbally walks through the logic the car uses while doing so. Through these mental exercises, the designers can bring to light their implicit assumptions and expose flaws in the design \cite{petre2009insights}. 
 

 
 \begin{figure*}[tbh]
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/motivation-designbehaviors-mentalsimulation}
  \caption{Designers sometimes use their sketches to mentally simulate their designs in use.}
  \label{figure: motivation-designbehaviors-mentalsimulation}
\end{figure*} 

 \item \emph{They juxtapose sketches.} In order to compare and contrast ideas, software designers will often juxtapose sketches across perspectives, alternatives, and abstractions \cite{petre2009insights}. A class diagram may be examined in parallel with a sequence diagram to aid the designer in determining how a message is passed between components. As another example, the designer in Figure \ref{figure:motivation-designbehavior-juxtapose} points to both a data model and a map to understand how a car object is passed between components as it travels through an intersection. The juxtaposed diagrams help the designer in reasoning how the design might work, using the knowledge gained from one diagram to help identify the omissions or mistakes in another, as well as any inconsistencies between them \cite{petre2009insights}. 
 
 
\begin{figure*}[tbh]
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/motivation-designbehavior-juxtapose.png}
  \caption{Designers juxtaposing two sketches by pointing.}
  \label{figure:motivation-designbehavior-juxtapose}
\end{figure*} 


 \item \emph{They review their progress.} Not all time spent during design is dedicated towards producing new content or verifying whether the design does what the designer intends it to do. At some point, the designers must take stock of what they have done. They momentarily take a step back, away from the design, and consider the progress that they have made and what they have yet to do \cite{mangano2012design}. They may return to the problem statement or list of requirements and mark off everything they have done to address it, they may generate a new list of issues that they further need to address, or simply just talk amongst themselves, to assess where they are. As an illustration, the designers from Figure \ref{figure:motivation-designbehavior-review} circled a particular subset of their requirements list and explicitly related the items related to that requirement that may have issues that are intertwined with another issue.
 
 \begin{figure*}[tbh]
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/motivation-designbehavior-review}
  \caption{While reviewing progress, designs may mark up their requirement lists.}
  \label{figure:motivation-designbehavior-review}
\end{figure*} 
 
 \item \emph{They retreat to previous ideas.} Periodically, designers may reach a stopping point in the exploration of their current set of sketches, such as when they become stuck or simply have exhausted an alternative. They then may choose to return to a previous state of the design (and its sketches) to start anew \cite{zannier2007model}. For example, an abandoned proposal for a time-based architecture may become a more lucrative option if an event-based architecture proves too costly in system memory usage. In returning to past ideas, the designer may bring new insights and a matured understanding from the exploration they just exhausted, which they can use to explore the past ideas further.
 \end{enumerate}

\section{How they collaborate on them}
\label{chapter:motivation:collaboration}

Software design is a highly collaborative activity, especially at the whiteboard where sketching and design exploration is almost always performed in collaboration with others. The behaviors in this section result from the collaborative aspects of working toward a single vision of the design that is shared by all parties.

\begin{enumerate}
  \setcounter{enumi}{11}
  \item \emph{They switch between synchronous and asynchronous work.} While much of the work that takes place at the whiteboard is typically synchronous, with all participants focusing on a single aspect of the design they are discussing, it is known that designers occasionally break away to explore an idea independently while the others continue with the main discussion \cite{dekel2005supporting}. This typically occurs when a sudden inspiration strikes, or when a designer wishes to develop a counterexample or alternative to what is being discussed now. The designers in Figure \ref{figure:motivation-designbehaviors-asynch}, for example, have split to independently work on different aspects of the design. The female on the left is exploring exactly how the timing of an intersection would work using a line graph, whereas the male on the right is working through a UML representation that explores the persistent aspects of the timing mechanism.
  
 \begin{figure*}[tbh]
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/motivation-designbehaviors-asynch}
  \caption{Designers sometimes break into independent groups to work out solutions, before later synchronizing their insights.}
  \label{figure:motivation-designbehaviors-asynch}
\end{figure*} 

 \item \emph{They explain their sketches to each other.} After any independent work takes place, and even in cases where one designer is drawing on behalf of the group and ``has the floor'', the designers must synchronize their mental models of the state of the design \cite{dekel2007notation}. This behavior relates to Behavior 9, but represents its collaborative version. They will need to verbalize their mental simulations to explain the consequences of a particular choice, clarify the meaning of a sketch, or even simply explain their assumptions or inspiration. While explaining, they may sketch on top of the existing diagram, using their marks to guide attention or add detail, or simply gesture over the diagram if they do not want to edit the sketches.

 \item \emph{They bring their work together.} Sometimes, as a result of asynchronous work, the designers need to integrate their ideas from separate sketches into a unified design. This may involve bringing parts of a sketch over, creating a new sketch that integrates both, or sometimes working on a third alternative that combines the best aspects of each, but requires a different underlying approach to make that work \cite{dekel2007notation}.
\end{enumerate}

\section{Summary}

Taken together, these fourteen design behaviors capture ways in which designers work at the whiteboard, ways that we seek to support in my work. Designers continuously exhibit these behaviors, making choices such as how much or how little they sketch out a design, how they navigate their focus from one sketch to the next, or how they collaborate over these sketches.

These design behaviors, of course, never stand in isolation.  Quite the contrary, any design meeting consists of an interleaved sequence of them.  In explaining a design, a designer may navigate between different perspectives, alternatives or abstractions, and create and refine diagrams of different detail in the process. For example, the designer in Figure \ref{figure:motivation-designbehaviors-interleaving} explains an idea to her design partner (Behavior 12) by mentally simulating a car moving along the map (Behavior 8), and points to different perspectives (Behavior 5) during the mental simulation to show how it affects various parts of the program. 
 
 \begin{figure*}[tbh]
  \centering
  \includegraphics[width=12cm,keepaspectratio]{./figures/motivation-designbehaviors-interleaving}
  \caption{The designer interleaves many design behaviors.}
  \label{figure:motivation-designbehaviors-interleaving}
\end{figure*} 

Given that these design behaviors occur naturally at the whiteboard, I aim to provide support for all fourteen design behaviors in order to improve the capability of the designer at the whiteboard. The design behaviors must be equally supported, where the support for any particular one should not impede the designer's ability to perform another. As such, I must take an incremental approach in exploring support for these behaviors to ensure any added features meet this criterion.

\chapter{Research Question}
\label{chapter:research-question}

Now that I have introduced this set of design behaviors, I return to what it would mean to build support for it. I first recognize that there is a spectrum of ways in which the design behaviors could be supported. On one end of the spectrum lies the whiteboard itself, a minimally intrusive, informal medium that supports only drawing and erasing of content. The naturally occurring behaviors are permitted, but not necessarily supported, which is the basic problem that this research is trying to address. On the other end of the spectrum is the formal design tool, a highly structured, heavy-weight environment with hosts of explicit features. Examples of such tools are Rational Rose \cite{Quatrani} and ArgoUML \cite{robbins2000cognitive}. Theoretically, they support a number of the behaviors, such as, for instance, Behavior 5 by providing multiple views on the same model, or Behavior 6 by maintaining many projects. They, however, do not nearly support all, and the ones that they do are not nearly as fluidly supported as necessary. 

Between the informal whiteboard and the formal tool lies a range of possible approaches that provide different blends of the strengths of each. This leads to my research question:

    \newenvironment{myindentpar}[1]%
     {\begin{list}{}%
             {\setlength{\leftmargin}{#1}}%
             \item[]%
     }
     {\end{list}}

\begin{myindentpar}{1cm}
\emph{What minimally invasive, coherent set of features can be designed that is sufficient to effectively support these behaviors?}
\end{myindentpar}

I am particularly interested in exploring this question in the context of an electronic whiteboard, that is: what kind of support can we implement that transforms the interactive whiteboard from a place where software designers just sketch like they normally would to a place where their design behaviors are explicitly enabled through the functionality of the tool they are using? Most tools, today, support one behavior or at best a few.

In presenting this research question, it is useful to examine its precise phrasing:

\begin{itemize}

\item \emph{\textbf{minimally invasive} -- I seek to build support that does not completely abandon the whiteboard experience. People go to the whiteboard for its fluidity and flexibility. Any solution that I design must preserve the feel of the traditional whiteboard as much as possible.}

\item \emph{\textbf{coherent set of features} -- I wish to arrive at a set of features that build on each other using a unified set of design principles and metaphors. Building an isolated feature for every behavior would not satisfy my research goal.}

\item \emph{\textbf{sufficient to support all of these behaviors} -- Each behavior should in some way be supported by the overall set of features, and support for any particular behavior should not come at the cost of another. }
\end{itemize}

Having put forward this research challenge, I must analyze the resulting effect of the support for the design behaviors as well. This is a difficult question to answer, particularly since the literature to date has documented that these design behaviors take place, but has not yet fully articulated their effects on the design process and design product. For instance, switching among perspectives (Behavior 5) seems to have a positive effect on the eventual design \cite{baker2010ideas}, and, in certain cases, it has been documented that the consideration of multiple alternatives (Behavior 6) also seems to lead to a better design \cite{buxton2010sketching}. However, even these studies are hard pressed to provide absolute answers. The first study does not examine the optimal length of time a perspective should be explored: is thirty seconds too short, thirty minutes too long, or what is the general distribution? Similarly, the second study does not talk about the number of alternatives: is three sufficient, should twenty be explored, or does it depend on the design problem in some way? These are questions that cannot be objectively answered at this time. 

For this reason, my evaluation must be exploratory. An obvious evaluation would examine factors such as the frequency of design behaviors, time spent working in each behavior, and how my solution impacts the interleaving of the design behaviors. A more comprehensive evaluation would attempt to correlate the appearance of these design behaviors with the quality of the process and the quality of the design by, for instance, relying on outside experts to rate the designs that are produced and correlating this quality with the frequency, interleaving, etc. of the design behaviors. While this would possibly lead to a conclusive statement with respect to the impact of my solution, it would be very challenging to perform this evaluation given that a typical design exhibits complex interleavings of many design behaviors, making achieving any statistical relevance difficult. 

My evaluation, therefore, will take a qualitative approach. By observing Calico in practice and by interviewing users of the tool, I will analyze the usage of Calico within the scope of the design behaviors and analyze the impact of the features on those design behaviors. Additionally, while the observations will likely pertain to the existing set of fourteen behaviors, I of course do not rule out that new behaviors may emerge, as enabled by the features of the software, that might be worthy of study. 

As such, the summary of the approach in the remainder of this dissertation is that I will describe Calico (two different versions) as well as the result of a qualitative evaluation of Calico in use at several local companies.

%%% Local Variables: ***
%%% mode: latex ***
%%% TeX-master: "thesis.tex" ***
%%% End: ***