{Task based Evaluation}
We invited eight data analysts that include five males and three females to participate our evaluation. They are identified from P1 to P8. Both of them have Twitter accounts. They all have experience of using data analysis tools to explore and analyse social media data.
FIFA dataset was used in this study. We replayed the tweets according to time-stamp to simulate real world scnario. Before the study, we demonstrated every detail of our system and gave each participant a tutorial document. Then we asked the participants to freely use our system and to finish some tasks. The tasks we chose exercise all the elements of our system and cover all the design goals mentioned before. Some example tasks include:
A. Hierarchy Navigation
- When did the first goal score?
- Switch the system from default time granularity to hour-level, what's the main difference?
B. Exploration
- What's the 4W in the last event during the match?
- What did people say in the second event during the match?
C. Comparison
- Which half of the match contains the most number of events?
- What's the change of topics between the second event and the third event.
D. Real-time Monitor
- What happen now?
- Who lead the match now?
After the study, we conducted a post-study interview to collect their feedback. Here comes the summary of their feedback.
System usability.
Overall, our system was well received by the participants. When asked why, most of them answered: clean and clear, understandable, quite real-time, functional, funcy but powerful lens. Their comment upheld our design rational. According to P3, \emph{``This system can help me catch the overview quickly. The multi-stage strategy facilitate tracking crucial events. It really release me from tedious works."} His comment touches many of the design considerations of our framework.
All the participants appreciated the two key features of subevent reviewing and real-time monitoring. Seven out of the eight participants were impressed by the capability of the system to naturally support the investigation of the four Ws for event exploration. The comparative analysis feature, which involves small multiples of interactive lenses and topic maps, was also greatly appreciated by the participants. P2 mentioned, \emph{``It's extremely useful for presentation that the layout of topic map can be adjusted freely. Also the interations are smooth, cool and novel. In one word, it's awesome!"}
Visualization design.The participants confirmed the intuitiveness of our visual design.
Both of them considered it is proper to use DICON to profile subevent. With the help of DICON, they can target distinctive subevent efficiently and effectively. Moreover, our design reveals the relationship between different properties of subevent.
Six participants appreciated our design, especially the hierarchical event tree and topic map, as it allowed them to easily navigate from high-level information, such as subevents and topics, to low-level information, such as keywords and users.
Five out of the eight participants particularly liked the interactive lenses. According to one postgraduate, ``The lens tool is cool. It's just like real-world lenses, but with much more powerful abilities.".
Suggestion.
The participants makes some valuable suggestions on how to improve the system. Although P3 spoke highly of the system, he suggested that the system should also allow him to track tweets according to other conditions, not just tweets detected by algorithm. Both P1 and P4 recommended that size of cells could be used instead of color opacity to encode property's value in DICON. Also P1, P2 and P3 proposed that the properties with little distinction between different subevents should be removed from DICON. P8 wanted additional functions on lens, such as sorting and re-layout.
