Blog / AI, AI Strategy & Implementation, Digital Products, Podcasts, Product Design & Development

Chain of Thoughts, Ep. 4: The AI Design Sprint

By Damian Calderon, Maria Victoria De Santiago and José Ignacio Orlando

May 3, 2024

Introduction

Nacho: Hello, everyone. Welcome to the next episode of Chain of Thoughts. Here we are, Nacho, Vico and Calde, as in the last ones. I hope that they can smile a little bit, right? Because they are kind of frightened. Okay, now they are smiling. Now we have smiles. Okay. Today we will start a little bit differently because I think that in the last episodes, you already saw me reading the introduction. So the plan is to start right from scratch and go straight to the point. So today we are going to talk about design sprints, but not design sprints in general, but carefully tailored for AI initiatives. And why is that? Because in the last show, we reflected a lot about how chatGPT and all these large language models are being used right now are introducing a whole new generation of products, all of them based on the same UI, always with a textbox or a prompting interface. And today we want to get a little bit more practical. We want to understand how can we use tools like the design sprints to format and design AI-driven products. And of course in the process to find also better user experiences before these mere prompting interfaces. So make sure that you listen all the way through the episode if you want to learn more about this.

So maybe, as a starting point, maybe we can start discussing about the challenges that we have when it comes to designing AI products. Just to bring something to the table and then you guys can jump on, I think that one of the fundamental problems is related with the mindset that we, the data scientists, always bring to the table, which is why we tend to think only in terms of automation. It’s like you give me your data and I tell you if I can automate the tasks that you want us to do. And we know that this is one of the fundamental reasons behind the poor adoption that we are seeing about AI right now. We see that many initiatives are discarded, basically because there is not enough data without, you know, having a trade-off between automation and maybe augmentation. This idea of using an AI-augmented pipeline, maybe for start collecting the data that might be used later on in the process for automation. And also a problem that I experience very frequently because I’m biased by my academic background, is that I never tend to think of the user. And this is something that was really challenging the first day that I joined Arionkoder. I started to change my mind into that direction because the user is at the center of these solutions. And at the same time, I also experienced this chicken or egg situation in which I always push to prototype the AI first so that we can understand if this is feasible or not. But if I jump directly to prototype an AI that the user will never need, then I’m busted. And on the other hand, if I focus too much on the future user interface and how the users are expected to interact with the tool, then it might be that I’m doing a huge investment on something that really depends on how feasible the AI components are. So this is an open question for you guys. How can we challenge the challenge and remove this issue?

Vico: Just a quick comment maybe, Calde, before we jump right into the AI design sprint, one of the key things we were discussing last time was why are design sprints in general good tools on solving some of the intricacies, let’s call it, about team collaboration and just to remind everyone before we jump in on the benefits of using these types of methodologies and frameworks against, like traditional collaboration between teams that requires long meetings. There’s a lot of, I wouldn’t say disagreement, but yes, back and forth regarding how to prioritize and when to go from inspiration to ideation and really getting into prototypes in a timeframe that we often discuss even internally. When you give yourself the time, design and inspiration can take as much time as you want, and this type of engagements allows us to really box it at the time, frame it in a way that you’re moving forward with the team aligned on a specific challenge. So it has that specific structure that makes it more efficient way to achieve results. And as everyone is, was involved in listening and active note taking, questions, sessions and discussions in general tend to be more productive and you get more than just words of note. You get sketches, note questions, and even things to validate with your users, when you finish with a prototype, and also a fun part always of this type of methodologies against regular team curation are the boards that we use to collaborate that have this kind of, how to say it? It’s like a physical kind of interaction that you get in and if you got a good facilitator it can change even the mood of a whole team when you got everyone in the right vibe to collaborate, to create great things. This might seem like silly thing, but it’s not small when you’re looking to create really different solutions actually, to have the people in the right mindset and aligned. There’s a lot of careful thinking about okay, influences, and how does this new AI design sprint that is like flashing out everywhere is getting a lot of attention. There’s a lot of different sources I have been gathering to craft this type of a specific framework. And once again, not just say it in the last episode too, I think, is it’s not like any software solution. That’s also a key thing. So it has its capabilities and okay, what are those and what are the challenges that it actually brings in? It’s one of the things we’d love to explore with you today.

Understanding Phase

Calde: Yeah, right. I think something that you mentioned there Vico, the timeframe, is key also to understand the cycles of definition. And it has to do with exploring the unknown. And the same that happens at the level of designing, thinking, and training a machine learning model because it’s actually like combining these designs sprints for AI with the model definition and designing things the best that we can do, because it’s about cycles. And you timeframe these, you timeframe a proof of concept, right? Or you timeframe a design sprint because you want to navigate uncertainty. So you’re navigating uncertainty with these tools. And for doing that you need to first allocate that time to explore the problem and the solution in a holistic way. Then assign a team to explore it in a technical way and then continue iterating over that. It’s not that the design sprint will get you to a final solution. It will leave you with something to experiment with and some learnings, some basic learnings about what are the possibilities in terms of design and the technical side, and more things to explore. And the same will happen with the first iteration over a model, but at least with what we know from the software industry is that it’s better when you make progress into cycles in fast iterations. Fail fast, this fail fast mentality is what is behind this. But getting to the design sprint itself, also as you mentioned, Vico, we are looking at that it’s not a crazy idea from us, right? There’s like a whole lot of people working in these and understanding how they can adapt design processes for the AI field, and for defining AI-driven products. You have academic examples like Carnegie Mellon and the great work that they’re doing, they have been doing about ideating for AI products. That’s an inspiration for us. And we went through the papers that they published and we are trying to incorporate that into our design process.

And also you have in the other side, you have the People + AI Research (PAIR) at Google which is basically an entire guide, chapters. There’s an article. It’s an entire site dedicated to thinking about how the relationship between AI systems and people needs for interaction and needs for interactions to be designed. So those are two main influences here. Also, The Shape of the AI, it’s a site that is like registering emerging UI patterns. Something that happens with them is that while we are getting inspired by them, there are just separate resources. And I think if you just try to go there and use them, you don’t have the time frame that the design sprint has, right? So that’s the missing part. And what we are trying to do with our AI designs sprint process is to package all of that into a timeframed version. And also we are using of course the inspiration from the design sprint itself, which is pretty great.

Nacho: Yeah, that’s really interesting. And to your point, when it comes to managing everything into the same AI design sprint, I guess that the one important component is to develop a common language with the users and with the stakeholders of the project, because we can talk about AI in general terms. But, we need to at some point bring it to the table and figure out what we are talking about when we talk about AI, we’re talking about using LLMs, we are talking about training our own custom model. We are talking about doing computer vision or training an object detector or things like that. So there is a very interesting resource from Carnegie Mellon that is a really nice paper, and maybe we can share the link jointly with this episode, in which the researchers shared a series of snapshots about different problems that can be tackled with AI but in general terms, so that you understand them from an input-output perspective rather than delving into all the technical aspects of that, I think that’s very important and should be always an element in any AI design sprint. It’s an element in ours, in particular in the one that we use in Arionkoder. But it’s also important for any other initiative. Maybe some of the listeners of this episode are trying to look up for resources like that. And I think it’s a very, very valuable one so that you can develop this common background. Or at least I find that interesting because I’m also a professor in the university. So it’s an interesting resource to communicate with people.

Calde: Yeah. Yeah, 100%. It’s a great resource. And I think it’s doing an excellent job into creating a shared language which, as you were saying, the Design Sprint is also about this. About creating shared understanding. So in our process of the AI design sprint, we combine the first part which is understanding about the project and the problem space with understanding more about how to design for AI. Because we have a new problem now, right? They’re like designers and they’re like AI experts and they don’t know each other. One of them works in the definition of the product, so they need to create a common language, so this ideation tool from Carnegie Mellon based on the capabilities of AI kind of helps in creating shared knowledge. We introduce it, we present it in our design sprint, the people who are participating learn about capabilities and how to use them through this guide from Carnegie Mellon and some other examples and knowledge that we have. And then from creating that common ground, we start working on the the part of the project that has to do with ideation. But it starts just in the understanding phase. It’s like something that helps everyone understand what can be done with AI. What are the exact capabilities? And we kind of know, like for example, we know I can create a chat tool with chatGPT or that kind of examples that talks to the end user and helps him create something. But the thing is that this lets you understand what is happening in a more deep level, right? There is that capability for the system to collect information, to process it, to make sense of it and to provide also an output. So it’s a way of understanding the models and species that you can put together. Getting back to our process… sorry, you wanted to say something, Nacho?

Nacho: Yeah, actually, I was going to say that I guess, once you develop that common language, the next step is to start doing some interviews and understanding like in the, in the classical Design Sprint, right?

Calde: Yes, exactly. Yes. So we have to level the field about the new tool and new material that AI means for designers. And we also need to level the field in the process of designing a product, which will be the interesting part for ML experts, both experts and designers, and stakeholders participate in the AI design sprint. So they’re all on the same page, right? So what we do is we start by exploring the problem space. Exploring the problem space means talking to users and subject matter experts and understanding more of the field. And understanding more of the feel of the market and the tools being used. And essentially like the workflow, which is the most important part: understanding the customer journey. How does a user go from an initial moment to the outcome that he’s looking for, right? So with simple questions and just talking to users and subject matter experts, we help everybody navigate that in a fast, efficient way. We take notes into sticky notes, sticky post-its, and we try to group them later together to make sense of the different interviews that we have been doing. That’s essentially like the understanding part. But there is a new thing with AI, and the things that you can start thinking, and everybody could be thinking about the ML opportunities there, right? And there will be opportunities for automating things, and there will be opportunities for augmenting processes that, as we talked in our previous episodes, could be to create a relationship between the model and the user that favors the production. How efficiently a task is performed. So already at this level, we start to map those opportunities because it’s in the in the head of everybody right there thinking about it. So just like, dump it. Remove it from your head and put it into the shared space. That’s something that we do all the time with design thinking, so we can talk about it, and once we have done that, we have started mapping the different opportunities, we can go in into the next phase that is the ideation.

Vico: I have a question there before you jump there, Calde, because I think it’s interesting. If you allow me, you talked about the opportunities. And also a little bit about challenges of leveling the field in terms of understanding, okay, we are in a session and we’re specifically working with people that are still trying to grasp actually what AI can do. What they can ask AI to do for them, or on the contrary, they still struggle, with imagining because you have always different types of level of knowledge. You may have people that are asking to solve everything, people that may still struggle imagining AI doing anything of what they do today. And it happens to every one of us. Like we continuously discover new things that could be done or on the contrary, things that maybe are not so well solved yet, actually. So in that discovering of opportunities, and prioritization, how much translation also needs to be done. And I think we’ve been talking a lot about the, let’s call it educational aspect also of this processes, as AI has. And I think it’s worth commenting a little bit to also, and I know you’re going to jump to the ideation space, and I think there’s a lot of interesting things that are also inside of it in that space. So we call also for leveling the with educational sharing knowledge about…

Calde: Yes. During the whole process we’d be like, sharing some small details and knowledge, in particular knowledge that it has to do with task or the activity that we have to perform. And so for example, depending on the team and depending on the level of the team, in some cases we’ll share a small training about what automation means versus what augmentation means. We have some slides for that. And they help with understanding. Something that we can do also is to create moments for processing that. We can have like an initial training, we just need to review the slides and the examples, and then a review before we jump into the activity itself. So it kind of depends on the expertise of the team that we are working with. Because in some cases, some of them already know about AI and are already aware of this promise of interacting, users interacting with AI models, and how also they can create like a rejection with users, right? If presented badly or if presented in a bad way, or if they are creating expectations that they don’t get to comply. That’s also a huge issue. So I think in that sense, the materials that we need to provide have to do with the level of knowledge of the team participating in the AI design sprint, but it’s also always better to plan ahead for that. We need to have initial training with this team before jumping into the sprint itself, or it’s better if we do it and we just review the materials over the sprint. I think that answers your question, Vico?

Vico: Yes, without a doubt.

Ideation Phase

Nacho: Yeah, I think so. I would love to start talking about the ideation phase, because I think it’s at least for me, as an AI practitioner, it’s always the funniest part. I mean, not that understanding the context of the problem is not. I really love that part. And I think that it’s educational also for us as well. Every new project, it’s like a new opportunity to learn about a new domain and how AI can be leveraged in that particular domain. So it’s, it’s really nice. But yes, it comes with it. And I love to do the solution part. I think it’s like one of the activities that I enjoy the most to do, you know, behind the scenes. So, I would like for you to explain our approach because I think it’s very interesting.

Calde: Yeah. So essentially what we do in the ideation is, we have integrated a part of the Ideation from Carnegie Mellon in our process right now. And we start reviewing the opportunities that are in the customer journey again. And we try to kind of decompose those opportunities and what they really mean for AI. And having ML experts in that part is crucial because they can explain where the different parts of maybe, an opportunity, and also they have a clear idea of how feasible they are. But we are not yet looking into feasibility. We are starting to understand what’s behind the opportunities with these, capabilities material from Carnegie Mellon. And what we do with that is we decompose them together and then we select some of them based on how usable, how useful we find them for the customer journey, right? For the customer itself. Will this be really useful for customers? Is it worth it in terms of how the usability is? Does it automate long processes that take a lot of time and are error-prone for humans? Can this augment the experience of the user in significant meaningful ways? So that will be like the kind of questions in which we do an initial voting and selection. After that, we start with visual sketching, which is my preferred part of the Design Sprints because I have a background in design, but I think it’s also preferred for a lot of people because it’s also starting to materialize the ideas into interfaces. What’s your idea, Nacho? I mean, you’re not a designer so I’d really like to know that.

Nacho: No, but I, I really love that. I really love that part as well. And I would love to be a designer, by the way, but I don’t have the time now to pursue a career like that. But, for me, it’s also one of the most beautiful stages because there is an intrinsic relationship between what you do with a UI or what you expect to do with the UI and the AI component that you will end up building. So, for example, let’s say that the person that you’re talking to wants to annotate something on a medical image. And maybe you are thinking about a tool that will complement a radiological process or something like that. And then you capture that necessity from a UI perspective, and that brings you a signal about an opportunity for automation. You can say, hey, you know, do you have data, describing this particular process that you would like us to implement in this pipeline? And if the data is there, would you like to have a tool that automates this for you so that instead of doing this manually, you can simply correct the errors of the AI and simplify the process?

So again, all this design phase is bringing something to the table that might help us as AI practitioners to figure out another scenario that was not on the table. And I really love that. And another thing is that this particular phase of the design sprint, especially in times of LLMs, force the user to bring something alternative to a prompt interface, because you can simply ask, I would like to have a button that I can touch and then the moment I press it, I get a longer version of the text I’m writing, for example. And that means that there is a prompt there, but it’s going to be hidden on that very button and that brings a lot of things to the table. First, it’s easy to implement. Second, it goes outside the prompt interface that we all love and hate at the same time. Because I hate being typing all the time and writing more and more instructions. And the third thing is that it’s also more secure because since you are hiding the prompt, you are less prone to security issues. These new security issues, like prompt injection and things like that, that might jeopardize the actual platform that you’re building. So, yeah, I really believe that this part is essential and it gives you more clues about how to bring AI closer to the users without simply relying on the prompting interface, which is my obsession. I want to get rid of prompts as soon as possible. And, at the same time, as I said, it’s a really nice opportunity for learning new AI opportunities at the same time.

Calde. Yeah. And I see a lot of people in the UI field talking about instructional interfaces and trying to understand how to work with these prompts and how to do that in the process. But in reality, there’s something that you mentioned that I think is really important to grasp here. This thing of the button converting, making an entire prompt with just a click and a single action. That’s valuable. And that’s what we have been doing. What UI designers have been doing for the entire time, right? For their whole career. What they have been doing is hiding entire pieces of logic behind single UI representations, and that provides value to any kind of product. So that’s why we do visual sketching tools for AI. And maybe there are parts or there are like different things that you need to think how to work with because the generative aspect of the AI is something that is new to us. When we are working on UIs, we are used to working with big things that in some cases they are not that static, but generative AI is a new level of dynamic content because you are not sure what will be generated. But the things that we, here, for the sake of the design sprint itself, we follow the typical process. We do notes taking, we go to simple sketching, then crazy 8s, and then we present concepts.

But there are two things that differ from the typical sketching at the same time. This first one is that the note-taking part, in which you would be like, taking notes of what you consider valuable or what ideas you have in your mind. In this case, we do it in a different way. We try to put that in sync with presenting new patterns and emerging patterns. In this moment is when we use some of the examples and details on the chapters from People + AI Research, and also some patterns presented in the Shape of AI. We use that in place of the lightning demos. That would be like something that you do in the design sprint, because there aren’t really like great amount of sources of inspiration that people are aware of right now. So it’s better to present those examples and then allow them to see how that that we are looking at can be adapted toward the specific problem that they are trying to design for. So we do that and we think that’s a better approach and that it works better, in our experience.

The other thing that changes regarding this visual sketching part is that in the conceptualizing moment, instead of going with the typical representing the concept with three images and some text, we use a more structured format that was also presented… it’s inspired by the Carnegie Mellon material. We adapted it a little bit because we also want to have the first idea of the logical workflow behind, and also about how the system should optimize for it, if it’s optimized for precision or recall. So that’s what we do in the visual sketching part. And of course, the ideation part is also part of like selecting ideas. But before just doing like a pre-input voting, we do two things that have to do at the same time with bringing again the different knowledge together. We perform a Design Critique, which is a format from designers to provide feedback in which they look at each of the concepts and they make questions to them and they highlight what they like, it’s like an unstructured feedback conversation. But we do that. And that kind of includes the design perspective into the concepts. And then we perform the technical assessment. So in the technical assessment we have an asynchronous time to go there and check what is needed, data sources, the viability of using that, who are the data owners, etcetera. And once we have all of that together, we just share it. And we do the voting and selecting activity. And that’s essentially like the ideation part. Any questions about this?

Nacho: No. Actually I think that something that I would like to bring to the table, and we are running out of time and it’s such a pity, but still, is that the output of this process is like a combination of a series of sketches of how the final product might look like. But also we have a prioritized list of AI models or AI components that are necessary to bring this product to life. So these two, these first two stages can be used even though the customer or the stakeholders, even if they don’t want to advance and progress to the product, this is useful for them. For example, we have seen that with the startups when they want to have sketches about the application, but they don’t have the funding to start developing it. And it’s also useful from an AI perspective, because now you have a list of the AI components that you need, and you can start prototyping those AI components and verifying if those are feasible or not. And that we do a lot in Arionkoder. And we expect to keep doing all the time, which is once we identify these AI opportunities and once we determine if they are feasible or not, by heart, basically analyzing the problem, the next step is to see the prototype and to evaluate that which is quite similar to what you do in classical Design Sprints. Right? You prototype the UI and you test that with the users. In this case, what we want to do is to prototype the AI models, evaluate them, evaluate the numbers in a test set, verify if the numbers that we get are the numbers that the users were expecting, and then if we have a confirmation for that, the next step is to build the other prototypes that we are talking about. The UI prototype that connects all the dots.

Prototyping and Testing

Calde: What we do here is that we can jump into the third and fourth stages of design sprints, about prototyping and testing, and we can do the same that we do in a typical design sprint, we can just prototype the UI, and that will be like going to Figma linking some screens, present it to users and validate how users feel with just the UI without having built anything. And this can be like linked to the AI, the model development, in different ways. Maybe there is already like a PoC or something that the team tried. And in some cases, we can kind of connect a prototype to that like stitching them together. It won’t look the same maybe, but there could be like things that can feel more integrated, that the users could perceive as being like a cohesive product. So your testing you did at the moment, the POC itself and the design at the same time. That would be like a way of validating, but it could also happen that you don’t have anything developed, as you mention of the AI system. And so you decide to just like, delay everything or test just the UI to learn about the UI details and how users perceive the results before you design the model. That could also provide some clues, like users prefer to have, categories like the high certainty about something instead of 88% certainty. So it can also create some useful information for designing what will be the outcomes of the ML model. But again, we are getting back to this chicken or egg situation that you are talking. So it really depends on what’s your team preference data would seem to be. The good thing to grasp here is that we can use and adapt to different situations of the ML development stage that the team is in.

Nacho: Yeah. And you have user signals which is the most important part. You’re not creating AI in the void. You’re basically creating AI from a user-centric perspective. Going back to the very opening of this episode, we are doing something alternative to the way in which other people decide to do AI, especially in academia. I mean, I see that all the time. I see people publishing papers with new tools to solve a problem that no one has. So, in this context is very important because you want to focus only on that. And these user signals that you’re getting from a UI perspective, let’s say, are useful to steer the AI components. So it’s really important. But again, it’s always a chicken and egg situation in which the users slash the stakeholders, they are the ones that have the last word. So it’s up to them to decide how to move forward.

Unfortunately I have to wear the policeman hat and say that we are running out of time. So, it was lovely to talk with you. Unfortunately, we lost Vico at some point of the recording. But, I hope she’s fine. No. She’s fine. She had to leave because she is very, very, you know. But, if you made it to this point, that means that you have listened to all our episode, which is great. Thank you very much for your company today. I hope that all the topics that we have talked about with Calde today and with Vico as well were useful for you. We have covered our own proposal for design sprints for AI and UX discovery and crafting. So I think that this might be useful for everyone that have listened to the episode. Of course, if you enjoyed the episode, please leave us a review on Spotify or comment in the box below if you’re watching us on YouTube. Remember that Chain of Thoughts seeks to build a community of product designers, developers, and AI experts to keep creating great digital tools for customers and users using this new technology that we have available. So to keep this community growing, remember to like, follow and subscribe to Chain of Thoughts both on Spotify and YouTube and share with your own followers using the hashtag #ChainofThoughtspodcasts on X and LinkedIn. You can also follow Aionkoder on these social media platforms to keep yourself updated about the show and our actions as a company. Thank you very much Calde and Vico for being here and see you soon for the next piece of the Chain.

Calde: Thank you Nacho. See you next time. Bye.