In part 1 of our series on AI in the Making, we explored the importance of early-stage prototyping in validating how AI interacts with users. We discussed adopting a fail-fast approach, the necessity of user validation, and methods like Wizard of Oz to simulate AI interactions. We also addressed challenges such as limited fidelity and the need for realistic data inputs.
As we continue this journey, we will dig into mid-stage and late-stage prototyping, examining how to refine AI models, integrate them into user interfaces, and optimize the user experience. Let’s dive into these advanced stages to ensure we’re building robust and user-centered AI solutions.
Mid-Stage Prototyping
Valid Scenarios
Mid-stage prototyping scenarios are identified by the following:
- The ML team is actively refining the AI model, improving its accuracy and performance based on initial feedback and data analysis.
- There are still some undefined aspects regarding how the AI’s predictions and outputs will be presented to users. This stage focuses on refining these interactions to ensure seamless integration into user workflows.
Prototyping Goals & Guiding Questions for This Stage
Assuming you’ve gone through an initial validation, your goal now is to test a new iteration, refining both the UI and the ML output. With higher-fidelity prototypes, you can address more specific questions:
- How do users perceive the refined interactions with the AI?
- Is the AI’s output being presented in the most effective way for end-users?
- How do we design for different types of model errors?
- Can we achieve the required model accuracy?
- What technical challenges remain before full integration?
- Are there any biases present in the model, and how can they be mitigated?
Methods & Tools for This Stage
Some typical methods and tools for this stage are:
1. High-Fidelity Prototyping: Use tools like Framer or actual coding environments to build detailed, functional prototypes. These tools allow for more precise testing of the AI’s functionalities and interactions.
2. Increased Performance Expectations: Meeting user expectations for AI performance and reliability as fidelity increases.
3. Data Subsets: Selecting a smaller, representative portion of your full dataset can help you with refining crucial aspects of your ML model without over-investing on it. Generating the subset needs attention, so make sure to have a clear purpose like testing specific model functionalities, handling particular user scenarios, or identifying biases.
4. Hybrid Approaches: Combine various tools and methods to create prototypes that are both realistic and adaptable. For example, one of our preferred prototyping methods for this stage is to use a Hi-Fi UI prototype and stitch to it a working ML model that isn’t refined yet, such as a Proof of Concept ML model. This allow us to make users tinker with the model, detect where we need to add explainability and what concepts from the model need refinement.
The Challenges of Mid-Stage Prototyping
Data Quality and Relevance: the data used in prototypes needs to be accurate and representative of real-world scenarios. While this is an issue that you will start facing when mocking up data for early-stage phases, it will increase with a more refined prototype, as users will expect it to have the proper data.
Integration Complexity: using hybrid approaches is time-consuming. Whenever we can, we focus on simple interaction patterns, like the user opening a sidebar/modal to interact with the working ML Model. This might not reflect the UI ideas we have for the final interaction, but it can be important not to waste our effort in validating well-known interaction patterns
Ethical Considerations: Identifying and mitigating biases in the AI model is crucial to ensure fair and ethical outcomes. During mid-stage prototyping, we not only need to analyze datasets for imbalances. We also need to use diverse user groups in our validation sessions to help monitor and address emerging biases.
Late-Stage Prototyping
Valid Scenarios
Late-stage prototyping scenarios are identified by the following:
- The AI model is refined and integrated into the user interface. The focus is now on ensuring that the integration works smoothly and adding new functionalities to it.
- From a user perspective, our goal will be closer to optimizing the user experience by fine-tuning AI responses and interactions based on extensive user feedback.
Prototyping Goals & Guiding Questions for This Stage
Use fully functional prototypes to refine and optimize the user experience and the model output:
- Does the integrated AI perform as expected in real-world scenarios?
- How can the user experience be further enhanced?
- Are there any remaining technical issues or bugs that need to be resolved?
- What are other edge cases we need to care about?
- How scalable is the AI solution?
Methods & Tools for This Stage
1. Full-Scale Prototyping: To test new features and ideas over the working UI / ML Model, you will now use fully functional prototypes with live data to test the AI in real-world scenarios. Tools like native coding environments are essential at this stage.
2. User Testing Platforms: Conduct extensive user testing using platforms that allow for detailed feedback and analytics. Tools like UserTesting or Lookback can be beneficial.
3. Performance Testing Tools: Use tools that can simulate high loads and stress-test the AI system to ensure scalability and reliability.
The Challenges of Late-Stage Prototyping
User Feedback Integration: Continuously gathering and integrating user feedback is essential for refining the AI solution. At this stage, the feedback should focus on fine-tuning AI responses and interactions to enhance the user experience and meet high standards of usability and satisfaction.
Performance and Scalability Testing: Ensuring that the AI solution performs well under various conditions and is scalable to handle increased loads and user interactions is crucial. This involves rigorous performance testing and stress testing to identify and resolve any bottlenecks or issues that could impact scalability.
Enhancement Opportunities: Identifying areas for further enhancement is critical as refining an ML model is not always straightforward. Collaborate with ML experts to validate improvements and ensure the model continues to perform optimally. This iterative process involves testing new features, refining model parameters, and incorporating advanced techniques to enhance the AI’s capabilities and reliability.
Lack of Integrated Tools: There is a significant challenge due to the lack of tools that easily allow for simultaneous changes affecting both the UI and the ML model. This gap necessitates a more manual and time-consuming approach to integrating and testing changes, which can slow down the prototyping process and affect the efficiency of iterations.
Best Practices
No matter the prototyping stage you currently are, recalling and following these prototyping principles is always essential:
- Collaborate Closely: Involve designers, engineers, ML experts, researchers, and product managers throughout the prototyping process. Diverse perspectives ensure that the prototype addresses various aspects of user needs and technical feasibility.
- Focus on the User: Prioritize user needs and pain points, ensuring the prototype addresses them effectively. This user-centered approach increases the likelihood of creating a product that resonates with the target audience.
- Fail Fast, Learn Faster: Embrace iteration and don’t be afraid to discard prototypes that aren’t working. Each failure provides valuable insights that can be used to refine the next iteration.
- Don’t Over-Invest: Match the prototype’s fidelity to the specific questions being answered at each stage. This strategy ensures efficient use of resources and maintains focus on the most critical aspects of the design and development process.
Notice these best practices don’t have to do with AI at all, they can guide any validation process you have implemented.
Prototyping AI experiences at different stages —early, mid, and late— presents unique challenges and requires tailored approaches. One common topic is the lack of better tools to prototype with AI. This is an evolving field, and new tools are starting to emerge (look at this ProtoAI concept). As tools improve, so will our ability to create more accurate and effective AI prototypes.
Despite the prototyping and testing process not being as efficient as it could be, teams can still use Prototypes to navigate the complexities of AI development and create more user-centered AI products.
If you’re planning to develop AI functionalities and you’re not sure how to test it, remember you can always drop us a line.