Initial Prototype Design

We propose the use of SnapExplAIn as a tool to support the maintaining of cognitive flow during the process of design with AI-assisted process annotation. This system is developed to facilitate a deeper understanding of users' design processes in various contexts and scenarios, capturing both the evolution of design and the decision-making that contribute to the final product, while making mid-point annotations to improve communication and understanding.

The system design combines semi-automation (for design-process capturing) and Wizard of Oz (for AI analysis and generation) methodologies, ensuring both flexibility and depth in data collection. SnapExplAIn is built to accommodate both physical and computer-aided design, with its mechanism being centered around having access to the creation processes for midpoint specification and insight pulling.

Data Capturing Mechanism

SnapExplAIn operates as a plug-in or add-on to existing computer-based design tools, such as Maya or Adobe Illustrator, providing a seamless way to document the progression of digital design work. Users can capture screenshots by pressing a specific keystroke (Ctrl+S in the prototype) -- This design facilitates low-interruption documentation for recording key moments. The snapshots reflect the progression of the digital design process, enabling creators to gather visual documentation effortlessly. With this flexibility, SnapExplAIn can be used across various digital design environments, providing a straightforward method for documenting and analyzing processes. By providing a mechanism akin to pressing a "save" button in digital work, SnapExplAIn allows users to define capturing moments while minimizing disruption to the creative flow.

Automated Annotation Process

Once the mid-process snapshots are captured, they are analyzed by GPT-4. We first conduct prompt engineering to the GPT model for recognizing and describing process snapshots. In this phase, we use prompts like, "Please describe this process picture", and "Given this next step, revise the previous response and provide a new sequential description." The objective is to teach the model how to create meaningful descriptions and annotations that encapsulate the transitions between the processes' steps. Using a chained prompt approach, the model learns to understand the correlations and differences in the sequence of snapshots, enabling it to generate accurate descriptions of processes by learning about processes. We also prompt-tune the model to create annotations of desired depth and detailness. In the second phase, we tune the GPT model to respond to specific cues that simulate the sequence of snapshots. The process begins when we input the command "[Start]". We then feed in the snapshots one by one, with SnapExplAIn generating descriptions for each step, and updating the previous responses as needed to maintain a cohesive narrative. After processing all the snapshots, we end with the command "[End]", prompting SnapExplAIn to output a complete step-by-step annotation that accurately depicts the entire design process. This dual-phase training approach ensures that SnapExplAIn generates detailed and accurate annotations which precisely reflect the evolving nature of the creative process. It also allows for greater flexibility in the input and output stages, offering a dynamic way to document and understand the progression of the design process.

Pilot Study Design

The study began with a 25-minute tutorial to familiarize participants with the basics of the creativity support tool. This session covered essential technical skills and provided an overview of the expected workflow. After the tutorial, participants enter a 25-minute design phase where they created a specific object or artifact using the prototype. In this trial, all 5 participants were asked to create an airplane. Also, participants are allowed to use outside resources (e.g. online searching) for inspiration and reference for their designs.

In our study, participants in experimental conditions A and C were instructed to take mid-point snapshots, while participants in groups A and B were required to manually document their design process, using a free-form documentation method of their choice.

Upon completion of the design task, participants of each condition were asked to finish the NASA Task Load Index (NASA-TLX) survey for measuring their cognitive load during the design process. Additionally, we conducted semi-structured interviews to gather qualitative feedback. For subjects not using SnapExplain, we asked about general experience of the design process. For subjects using SnapExplain, the interview generally covered the following topics:

• General experience on designing with TinkerCAD, what went right and what went wrong.

• How would you describe the experience using SnapExplAIn, what is your preferred way for specifying the midpoints?

•  How did SnapExplAIn help (or, not help) during the design process?

•  (After showing SnapExplAIn's outputs) -- Would manual documentation or SnapExplAIn be preferred, and why?

• Does the automatically generated annotations of SnapExplAIn align with what was expected?

This evaluation process allowed participants to reflect on their design journey while providing feedback on SnapExplAIn's performance. The discussions helped identify areas for improvement, enhancing our understanding of how SnapExplAIn supports the creative process and guiding future refinements of the system created with a rapid/iterative prototyping approach.

(Annotated) Prelim Conclusion and Insights

In the pilot study, 5 participants were recruited to participate in the design sessions. The interviews indicate that while documentation is crucial in the creative design process and mitigates cognitive load for the users by eliminating process tracing, its own creation disrupts the creative flow. SnapExplAIn's automated approach, in contrast, was able to capture and illustrate key moments without breaking the creative flow, significantly reducing cognitive load. On the other hand, participants found SnapExplAIn useful for capturing and annotating key moments in the design process without interrupting workflow -- The automated annotations provided a way to document progression with minimal effort. However, some annotations lacked accuracy and context, suggesting a potential need for clustering and integrating the knowledge of deeper insights into the designer's intentions, especially in the earlier phases of the process.

Links

Tool Link (Currently under iterative/rapid prototyping): https://designvc.web.illinois.edu/