PLOTTHREAD
Make Storyline Visualization Design Easier and More Expressive Using
A Human-AI Co-design Authoring Tool
OVERVIEW
As a research assistant at Interactive Data Group, Zhejiang University in my undergraduate, I researched and designed computational authoring tools to empower storytelling visualization design. "PlotThread" explored how human-AI co-design can help users craft expressive storyline visualizations. Our paper was published to IEEE InfoVIS 🎉
DURATION
2019.08 - 2020.04 (7 months)
ROLE
UX Design
User Research
Front-end Development
Usability Testing
TEAM
1 Lead Researcher
2 Developers
TOOLS
Sketch
ReactJS
JavaScript
Adobe After Effects
Protopie
Background
Telling Stories with Visualizations: Storytelling and visual expression are integral parts of human culture. Storytellers, especially online journalists, have increasingly been integrating visualizations into their narratives.
Storyline Visualization: The storyline visualization is a prevalent storytelling medium widely used in presenting the evolution of complex relationships between entities, such as the interactions among characters in movies. In some cases storytellers would allow the storyline visualization to function in place of a written story. In general, storyline visualizations use lines to represent characters and groups of lines for interacting characters.
Design Considerations for Storyline Visualizations: To create an aesthetic and legible storyline layout, several design principles and optimization goals were proposed in Design Considerations for Optimizing Storyline Visualizations, IEEE InfoVis 2012, e.g, “Lines in the same group should appear next to each other, otherwise they should be far from each other”, “wiggles, crossings and while space should be reduced”, etc.
Hand-drawn Storyline Visualization Example: Lord of the Rings
Hand-drawn Storyline Visualization Example: Star Wars
Hand-drawn Storyline Visualization Example: The Moon and SixPence (left), The Justice League (right)
Hand-drawn Storyline Visualization Example: Lord of the Rings
Problem
1. Manual creation of storyline layouts is too difficult and labor-intensive
The design of storyline visualizations is a time-consuming task involving many trials and errors as users need to balance between aesthetic goals and narrative constraints during manual creation.
2. Existing automatic generation methods are not efficient or flexible enough
Though existing methods utilize optimization algorithms to automatically produce aesthetic and legible layouts, they are still limited regarding 1) efficient exploration of the storyline design space and 2) flexible storyline layout customizations.
Using a set of pen & touch interactions, people can flexibly and naturally specify their desired layout to their AI collaborators. Based on AI suggestions, users can efficiently create expressive storyline visualizations.
Final Solution
We designed PlotThread, a human-AI co-design authoring tool for storyline visualization, which involves an AI collaborator in the design process to reduce the human labor while inspire design creativity.
Outcome
Co-authored “PlotThread: Creating Expressive Storyline Visualizations using Reinforcement Learning”, which was published to InfoVIS 2020. Below is our video description:
Process
01
Research
02
Literature Review
Competitive Analysis
User Study
Technical Research
Design
03
Brainstorming
Concept Sketch
Low-fi Prototype
Idea Comparison
MVP Design
Iterations
04
MVP Development
Rapid Testing & Design Iteration
AI Design
05
Workflow Design
System Design
RL Models
Evaluation
Usability Testing
Design Iteration
AI Iteration
RESEARCH
Literature Review
To investigate existing methods and their limitations, we conducted literature review on storyline visualization and authoring tools. We found that:
-
Previous automatic generation methods relied on pre-defined story scripts and structured datasets, which is too technical and time-consuming for users without technical background.
-
Previous authoring tools do not allow users to flexibly change their narrative structures during authoring, which is not natural for storytelling.
Competitive Analysis
To find our design opportunities, I then compared three existing methods to create storyline visualizations based on their design flexibility and efficiency.
Strengths and Weaknesses
Though general design tools support free-form authoring experience that is more flexible, they are not as efficient as automatic generation tools.
Though existing post-editing tools are more efficient, they are not as flexible as manual creation.
Our Opportunity
Help storytellers create storyline from scratch.
Make the authoring experience both efficient and flexible.
USER STUDY
Hand-drawn storyline workshop
To understand users’ natural design workflow and behaviors when crafting storylines, I conducted a hand-drawn storyline workshop, where 12 participants from different background and of different levels of design skills were recruited.
User Design Workflow
By analyzing the video recordings and the interview quotes, I summarized the users’ typical design workflow and key findings.
Key Findings
User Feedback
To better understand the limitations of existing methods and find new opportunities to better support storyline visualization creation, I collected user feedback on the hand-drawn method, optimization-based automatic generation method, and the optimization-based post-editing method. Below are the lessons I learned:
DESIGN
Design Concept
Based on preliminary research, we brainstormed the general idea of the tool that we were going to build:
A digital authoring tool allowing for free-form exploration of narrative structures and efficient exploration of visualizations.
Instead of requiring users to create a pre-defined dataset, we want to support a free-form authoring process so that users can flexibly explore the narrative structures.
To help users explore alternative designs more efficiently while maintaining the legibility of the storylines, we decided to incorporate both optimization algorithms and user input into the creation process.
Brainstorming
We brainstormed on three dimensions: narrative structure construction, initial layout generation methods, and layout refinements. By evaluating the feasibility of different technical solutions, we came up with two ideas for minimum viable products (MVP) and 1 idea for future iterations.
MVP Concepts
Focusing on testing the idea user iterations in narrative structure construction, I came up with two MVP concepts and created the wireframes.
Prototype
To quickly test our ideas, I created the demo of the script-based idea using Adobe After Effects.
Animated Demo1 👉
Animated Demo2 👉
EVALUATION
I interviewed 4 experts in narrative visualization on their opinions about the two concepts.
Domain experts were asked to rate the two concepts on 6 dimensions (1-5 likert):
1. How engaging the storytelling process is?
2. How flexible the authoring process is?
3. How efficient the authoring process is?
4. How supportive the system is when exploring alternative designs?
5. How creative the concept is?
6. How feasible the technical solution is?
Concept Comparisons
The canvas-based concept won higher overall scores. According to experts, “Interacting with the canvas is a more directly engaged and a more natural form of storytelling compared with filling the forms”.
Although six qualities of the canvas-based concept had a positive evaluation, the initial version still has many limitations. It should be more supportive, flexible and efficient.
To collect user feedback on this concept, I conducted an interview on 12 participants from the workshop, which helps me gain insights on later iterations.
Takeaways
Iterations
Based on the key takeaways from user feedback, I simplified the tool panels and refined the pen+touch interactions.
MVP User Feedback
After we iterated and developed the app. I conducted a usability test, which was meant to learn more about how we can help users express their ideas and explore possible designs (which is the last takeaway from the previous user interview).
Users were asked to recreate existing storyline visualizations of well-known stories using interaction tools we provided. They are encouraged to create their desired layouts to express their own design ideas.
Key Findings
Optimizations impede users in fulling expressing design ideas
it is labor-intensive for users to iteratively use interactions to adjust the layouts and the optimized. results might not be predictable or satisfying. Non-expert users are usually limited in fully expressing their design ideas.
Non-experts have difficulties meeting the aesthetic goals
For people without good aesthetics, it's hard to adjust the lines to meet the aesthetic goals. P1 said, “I have no idea what a good layout is. Showing me others’ nice work might be helpful.”
So, how to reduce human labor and inspire Diverse & Expressive Designs?
ITERATION
New Direction: Design for Inspiration
We kept collecting useful information online and brainstormed possible solutions, e.g., incorporating a library of beautiful pieces of work for users to refer. We did a very simple experiment where we showed users possible design layouts when they were creating their own storyline visualizations.
The result was not satisfying. Users mentioned that they could not learn much from others' work since
1. Their stories are different from others and can be hard to compare
2. Different designs are needed for different plots, character relationships, etc
3. Cannot get suggestions in time and find it interruptive to refer to the library from time to time while creating his/her own storyline visualizations
This trial, however, helped us come up with a brand new direction to explore:
What if AI could help optimize the layout while providing instant suggestions based on user intentions?
Instead of jumping into AI development, I spent a long time reviewing literature related to mixed-initiative system design to understand its pros & cons, and guidelines, to make sure that we are user-orienting instead of technique-orienting.
Human-AI Co-design Principles
To seamlessly incorporate AI to assist storyline creation, I conducted literature review on human-AI co-creativity. Based on previous literature and user research, I proposed three design principles for our human-AI co-design system.
Co-design Workflow
I then designed the human co-design workflow for our mixed-initiated design system.
To support a smooth and iterative co-design process between users and the AI agent, users would start the design process by customizing an initial storyline, while the AI agent provides a set of suggestive alternative designs according to the user-specified layout.
Then, users could quickly switch between different layouts for inspiration, or simply move forward for further refinements.
Design Iterations
AI IMPLEMENTATION
I collected data for AI training and refined the model. Learning the knowledge of reinforcement learning also helped me design interactions that are technically feasible.
To obtain considerable and diverse layouts, we use story scripts from the published gallery and generated 1000 storyline layouts for each story. In total, we generated 20,000 storylines to train the AI agent.
Model Architecture
To support the collaborative design, we define the action space of the AI agent as the high-level interactions: 1. The agent can share the same action space with users so that they can work concurrently to design storyline visualizations; 2. the high-level interactions are implemented on the basis of the constrained optimization model so that the agent can produce well-optimized layouts in terms of the aesthetic goals.
Learning algorithm for the AI agent: (a) use the optimization model to produce storylines; (b) measure the similarity between user-specified and “current” layouts to obtain the reward; (c) calculate the critic value to predict the “next” action.
Qualitative Experiments
To validate the effectiveness of the human-AI co-design approach, we conducted qualitative experiments on 4 stories. To simulate the real authoring process, we create 4 typical user layouts. The results indicate that the AI agent can successfully capture the visual features the user desires and produce more expressive layouts than the initially-optimized ones.
User Feedback
I conducted semi-structured interviews on 3 experts (artist, UX/UI, VIS researcher, respectively) to evaluate the effectiveness and usability of PlotThread.
CO-DESIGN OUTCOME
Users’ work created using PlotThread
We conducted a workshop where participants were able to use PlotThread to create their own storyline visualizations on Ipads. They had great fun exploring alternative designs with the help of AI. Below is a gallery of their deign outcome. Very impressive!!!
REFLECTION
Thanks for arriving here... This research was really long and had many iterations. But I felt so happy that I learned a lot from this iterative process.
🏋️♀️ Balance between exploration & exploitation
Instead of jumping into designing a human-AI co-design system, we started with preliminary user research and targeted the most severe problem: the difficulty of preparing datasets of stories, and brainstormed ways to help users create storyline visualizations naturally from scratch. After MVP development and quick iterations, we identified the limitations of existing optimization-based methods, which led us to further explore the possibilities of human-AI co-design. I found progressive and quick evaluations and iterations help inspire new directions both in design and research.
🤔 Be critical when using automation technologies
Reflecting on our initial optimization-based semi-auto system, I realize that though it incorporates human intentions into the optimization models, the automatic generation process of storyline visualizations often limits non-expert users in fully expressing their design ideas. I think digital authoring tools should not only boost the efficiency and reduce human labor in realizing ideas, but they should craft beautiful, natural, and inspiring design experiences. Should we automate the process? And how? We should ask these questions before using automation technologies in our products.
⏳ Spend enough time on empathy establishment
Based on user behaviors patterns and visual patterns concluded from the hand-drawn storyline, I gained a deeper understanding of their pain points, needs and expectations and could wear their hats when designing pen interactions. Also, we should not just focus on the outcome users make and superficial user behaviors, instead, we should also dig deeper into their cognitive process.
🥂 Mixing technical and design knowledge drives innovation
After learning the technical knowledge of the -state-of-art optimization algorithms of storyline visualizations, I learned that the optimized layout could not satisfy users’ design ideas since users might want to sacrifice aesthetical quality for a more expressive layout. Therefore, I proposed high-level interactions for users to describe their desired layout quickly, and AI would improve the design by generating well-optimized layouts. Fine-grained interactions for them to specify their intentions in a detailed level, flexible enough to support more personalized design requirements.
By embodying both the technical and design world in one person, we could accelerate the ideation process.