Overview
The Opportunity
Kaskada, a Seattle start-up in the machine learning space, sought to create and test a product idea that would empower data scientists to own the entire feature engineering lifecycle for machine learning model training.
The Challenge
Kaskada’s product concept was a paradigm shift for the industry. Up until this point, data scientists of all skill levels were processing their data by building discrete tool sets to support their workflows. In terms of product value, our client needed to understand if the end user sentiment of combining those features into one toolset, along with additional net new features, would be more valuable to data scientists than their bespoke tools.
Details
Project Length: 5 Weeks
Team: Chris Hannon (design), Paul Townsend (strategy)
Role: UX, wires flows, Prototyping / Testing
My Contributions: This was a small team. My colleague Paul Townsend was well versed in data manipulation (key function of the concept) and led the lion’s share of translating the requirements into user stories, from which I generated or refined the UX (divide and conquer).
Business Impact: Enabled client to secure next round of funding and refine the product
“How can we make the product handle enough complexity that very accomplished data scientists would see value in the tool, but simple enough that data scientists who needed more guidance could easily use it too?”
Discovery
The Approach
Smashing embedded themselves in Kaskada’s office and became part of the team. From there the teams ideated, prioritized, and created baseline screens for key scenarios that would take place in the application. Then, to ensure the prototype met the needs of the target users, we conducted two rounds of testing and refined the designs based on the feedback.
Given the short timeline, we needed to move quickly through concepting, design, and testing phases in order to meet the upcoming funding timeline:
In order to facilitate this we generated a high level overview of the cadence to align with the client on what we would need from them to get started. The client then in turn generated primary research in the form of:
Trends and insights:
Transcripts of user interviews
High level summary of insights
End user goals, needs, pain points
Business requirements
UX Patterns for data managment
List of common data science tools
Highlighted certain patterns for ML
Key Example & Use Case
Rough draft of a real data problem
Step by step through key phases
Both first-user and nth time user scenarios
In scope and out of scope items
This pre-work from the client was critical to our velocity and gave us a firm understanding of the desired scope and value that the product would need to provide to properly design and test their proof of concept.
Design
We began with rapid white boarding sessions to outline key functionality and information architecture:
A concrete example
Real world data from tax cab drivers provided ample complexity from which to create scenarios
This helped us align with the team’s vision, but to refine the vision and make it less abstract and more grounded, we needed a real world example of complex data to manipulate that would help us understand how to best architect the work flows and processing of data within the user experience.
The client then provided a complex data set that we were then able to generate distinct user scenarios from. (Only time I have seen an excel book exceed 100MBs of data. Wild.)
Complex Data from Yellow Cab
Key Scenarios
These user scenarios allowed to create sets of features and arrange them in logical work flows that we could apply the sample data to and then refine the interactions based on the pass-through results.
Micro Vs Macro - What I would have done differently
At the time, our design practice was not in the habit of performing Story Mapping activities for smaller projects on short timelines such as this. As a result this made the team’s focus bounce back and forth between macro architecture decisions that impacted overall user value, and simultaneously micro interactions within discrete workflows. From the perspective of a domain knowledge outsider this made the dependencies and impacts to user value very difficult to track. Lesson learned: New products need maps. Everyone benefits.
Emerging Prototypes
In parallel to the white-boarding efforts, we would generate and refine view states within Sketch to begin formalizing the concept for testing,
After multiple rounds of iteration and capturing in Sketch, a refined information architecture emerged that could be used to tie all the features together at a macro level:
Prototypes view states in InVision
Emerging Information Architecture
Testing
Refined Feature Set
With these refined features in place, we then generated a tightly integrated user testing script and test kit to walk testers through the prototype to understand product sentiment. Prior to the test, we reviewed the approach with the product team to align on objectives and incorporate any feedback they had.
During the test, we also took notes and video recorded the sessions. As the testing audience was highly skilled, the client agreed on remote viewing of the test through Zoom as to not bias the testers.
Recording of User Test
Test, integrate feedback, test again!
The test was extremely effective at helping us understand which areas of the product concept were of value to data scientist and what needed refining. Because of the careful test planning, the team was able to discuss the results of each section with identified insights. This led to us refining the proof of concept to make it more desirable and viable which we tested with a subsequent round of testing.
Results
Site analysis
With the final round of user feedback in hand, Smashing and Kaskada were able to refine the product concept for further market validation and succeeded in getting additional investor approval.
“There are a lot of companies out there that can help you improve an existing product. But defining a new product … that is harder work, and you guys were able to help us do it. Four weeks ago we had nothing but ideas. We have come a long way in that time. I view this as a huge success, thank you.”