Jose Ignacio Orlando (Nacho): In today’s episode of Chain of Thoughts, we will talk about how LLMs are reshaping the world of digital products. We will cover some of the most interesting applications associated with this technology, the challenges associated with those from a product perspective, and some forecasts and considerations that arise when it comes to innovating with LLMs. Make sure you listen all the way through to not miss a detail about this.
Nacho: Welcome to today’s episode of Chain of Thoughts. We are happy to be again here with you, with all of you guys. Here we are, I’m Nacho. I’m the director of the AI Labs at Arionkoder, and we have Vico. Vico, can you introduce yourself?
Maria Victoria De Santiago (Vico): Hi everyone, I’m Vico. I’m the Head of Product at Arionkoder. Thank you for listening to our past episode. And this episode!
Damian Calderon (Calde): Hey guys. I’m Calde, I’m the Product Manager at Arionkoder, and I have a background in UX and design.
Nacho: If you want to know more about us, you can listen to our previous episode where we introduce ourselves properly. But today’s idea is to, well, first of all, be a little bit more relaxed than how we were in our previous episode because we were like super serious and we were kind of frightened, but at the same time it was a pilot, so it was okay to be like that. And as we said in the introduction, the idea today is to talk more about the LLMs from a product perspective. So probably I will speak less than in the last episode where we focus on the technical details of the LLMs, and we will listen to Vico and Calde and some of the insights from them when it comes to innovating with LLMs and all these topics. So I’d like to start with a question for you guys, which is about how LLMs found you when they started to become trendy. We know that ChatGPT has been around for like almost a year, but I think that the moment when we saw it for the first time was kind of different, the landscape at that moment was different than the one that we have now, right? So I’d like to know more about your first reactions and how did you feel when you saw that happen for the first time?
Calde: Yeah, in my case, I think that initially it was great seeing like a huge amount of enthusiasm over it and getting into that, and I started to think around what ChatGPT means for us, for our world, for people close to us. Everything, right? It was like everybody was playing with it. I remember that initial moment of everybody playing with it and trying to do things with it and trying to find quickly the limits and also trying to understand how it works, and what were also like the errors. I remember this, for example, I think it was one month after it was released, but I remember this guy trying to create like, putting ChatGPT to create a business starting from 10 bucks, and I find that it is like everybody was trying new things.
Nacho: It was quite funny that time. Vico?
Vico: I was trying to, I was actually trying to remember because it’s like a year now. Feels like a whole life, especially after this tool particularly, and I read a little bit, I agree a little bit, not completely with what Calde said. I actually would tell us that it was like, okay, this is a new tool. I remember I just popped into it and start talking, chatting with it. In the beginning, I was still struggling. That felt like, okay, this is a little parrot there, just providing answers. It was like a little bit skeptical about actually how I would be able to incorporate it to use more fluidly. And then I started experimenting a little bit, also considering what the different people were saying about how you can use it and how to improve the usage of it. I started finding a few applications. I left completely the Google search basically as my first way of sourcing information and was like that immediate thought of, oh! This is interesting. Can we keep playing around it? I remember one of the experiments that at the end of last year was more excited about was when I was finally able to really, through a conversation with it, get it to understand my tone of voice, and then actually incorporate my use of emojis for content. And then I was like, okay, this could actually be even more interesting to me, more workloads like this as I was expecting. And it was really just enough to feel a little emotional. The incredible myriad of products, right? And it started exploding like we’re going a little bit out of control, also maybe.
Calde: We were like, I used also to do to these AI experiments to be just experiments, not being adopted by AI people. And I think ChatGPT brought that up, like a massive adoption so quick, so fast. And so on day one, we were just playing with it. A week after we were thinking, okay, now this will write all of our emails, right? Then maybe like a month after or two, it was like this is rewriting everything that we know. And then that’s how it works. And then a huge wave of enthusiasm started at that moment.
Vico: What about you, Nacho? How do you feel?
Nacho: You know, I was going to mention that both your experiences are quite similar to the one that I had myself. So at the very beginning, I remember that I had to write an article about ChatGPT for for our blog in Arionkoder. So it was right after the release, and I said to myself, alright, I will go through the paper or the technical report. It was not a paper. So I tried to focus on the technical details, as I always do when it comes to AI. And then I found out, I said to myself, alright, this is a good tool for chatting and for playing around and to ask questions, random questions about facts and that stuff. So I remember that well, I published that article, and a few days after that I was lying in my bed and I was super sad because I was building my house and the construction place was robbed and I was like asking myself, what can I do to make it safer? And then I said, well, maybe I can ask ChatGPT about that. And I started to ask questions, and it started to answer things that were kind of obvious but at the same time that I didn’t think about at the very beginning. So I was like, maybe there are some other use cases that I’m not seeing. And then I started to use it on a daily basis.
I realized I learned a lot of use cases by just trying to play around with it and trying to find those opportunities. And I guess that this is what is going on from a product perspective, right? Like, products are developed around these ideas that are first tried with ChatGPT and then wrapped around the user interface and becoming a digital tool. So I guess that what you mentioned, I don’t recall who of you guys mentioned this, but this, this idea of AI being always something for nerds and experimental died with ChatGPT and became something completely different. And I guess that this has changed the way in which products are developed. Which brings another question that I would like to ask you guys, which is how the product management and design community faced that hype wave around LLMs and is facing it, how they are doing it. Do you have any comments about that?
Calde: Yes. In a way, what I see is that it also started with the same enthusiasm, and now just seeing like some kind of disregard for it as a tool. LLMs are being incorporated. But the way they are seen is affecting other’s work instead of Product Management work, right? So I have been reading for example in some communities about how LLMs can be used for product discovery, and the conclusion is always that oh, it will just do like the dumb things and we’ll be doing the smart things, right? That’s like the current conclusion. And I saw that in other fields, like the same reaction, right? It’s like it’s regarded as a tool that would be integrated into our daily life much deeper. And I think that’s because the technology is in its early stages, right? So we just are playing with it over ChatGPT which is like an isolated place that yeah, isn’t aware of our field or our particular tooling. Right? If there was like, if we could integrate LLMs into the context that we have and the tools that we use, I think that would change a lot the panorama, how we integrate.
Nacho: Yes. And I have a small comment about that. I think that you’re true in the sense that people are not taking AI as seriously as they should take it. At the same time, I personally believe that LLMs will affect the way in which we work. They are actually affecting the way in which we work because now we have a conference call and we crave for these tools that extract the transcripts because now we want to analyze the transcripts afterwards. We don’t want to take notes during the conference calls. And that’s just a tiny example. But at the same time, I believe that it will change our jobs. It will eliminate some jobs for sure, some of these tedious automated jobs, let’s say. But at the same time, I think that these tools will probably unleash a new way of creativity for us, or at least I hope that we will have more time to devote to all that creativity part that cannot be automated with an AI, yet. At least with an autoregressive model.
Vico: I love that you bring this because it’s actually one of the things that I was a little bit amazed by, how the community was approaching this discussion. And bear with me a little bit. I’m going to back on background. There’s a concept that has been for a long while ago used, and was actually established by I think it was a work on it was to talk about the 4.0 industry, an industrial relation new wave at that point. And this was quite a few years ago now. I was still teaching economic history at this point and in the university and it was really interesting to see and I was actually talking with my students at that point. I think it was 2018. I think there has always been this establishment of the discussion of how the advancements of technology and the new waves of optimization were going to be affecting the different types of jobs. It was established already at that point of what is the future of work, what is the future of our lives with relation to work and how would affect these, let’s call, trajectories in some way. And there was like a whole group of people talking about this. This is going to be the reawakening of the human arts. For example, there was an expectation that we’ll go back to the enlightenment unless there was some discussion of how our jobs will be solved. We will have more time to work in the more creative things. I was also a bit worried about how this actually will create also a bigger gap. But all of these are the same because I remember, this was 2018, the first wave of not massive but huge layoffs in Japan with the release of Watson as an accounting agent at that time. And that was already happening and this conversation was establishing some spheres of work and I was amazed by not seeing that as much discussed in the technology space because I was like, hey, this is not so new and this is not such a different trend. It’s part of an already-established trend for a few years. So I was a little bit shocked by that. And with regard to what you were saying, Calde, I do agree I haven’t seen so much, but I think I was discussing with you earlier, I’ve started to see a few works there trying to actually explore the way, specifically these new trends, could be involved. But still there isn’t really much let’s call it research environment yet, and that’s maybe the most complicated thing. How do we get the design, anthropology and design. And people are specifically in design process research to actually permeate through the practice and not only ask being like, okay… But what I was trying to say is effectively there’s research that is being done mindfully on how it could affect both discovery and design of products on these new technologies. So I would like to say there’s some people that might be looking at that that we may not know about. And if you’re listening to this, do please reach us. I would love to hear and discuss with people looking at this too. But yeah, definitely.
Nacho: Definitely. Yeah. It’s very interesting. And you brought something that I was thinking about for a while, which is creativity, right? Creativity is just one of these use cases scenarios of AI, and we use it regularly for writing emails or to proofread our articles, or for things like that. And I remember that one of the first points that I raised against the massive usage of ChatGPT was this idea of having all of us a uniform language, a language that that always passed through ChatGPT before reaching the final destination. And that’s kind of sad because I prefer to get, I don’t know, best wishes for my birthday that are customized for myself, not customized by an algorithm, but this is just one of the use cases. And I’d like to know a little bit more about these use cases, in particular from a product perspective, because then we can connect the dots from the products with the solution, let’s say. So I’d like to know what are the most common use cases that you guys are seeing right now from a product perspective?
Calde: Yeah. I think there’s like a no-brainer that is like, automating processes that are tedious and also where a human in the middle can be like a liability. That’s the thing that makes more sense to adopt LLMs to do. So and I think there always has a little bit of that that I didn’t even… like transcriptions, if you think about it, it’s like a tedious work, right? We are not relying, we are trying not to rely anymore for transcriptions from audio to text. Of course you need a human in the middle to review that, but that’s one of the greatest, one of the most common cases I know. And also I am seeing a trend of AI used to summarize and in that sense to present summaries as insights, right? So the LLM is doing the thinking, right? They can present a summary. And there are tools that we use every day like Miro, which is a board, a virtual board to organize thinking into stickies and collaborate over it. But it’s basically like a virtual whiteboard. But the thing is that the tool itself tries to incorporate like… they propose, they have this promise that they will help you, like, organize the stickies that you put there and see that the output is not that good. I think they have like a long way to go, right? To get there to be something useful. But those are cases in which I’m seeing they come, LLMs incorporated, in generating insight and also processing information in terms of them.
Nacho: Well, it’s very interesting because this idea of using it for ideation is this kind of the way in which we work with other humans, right? We interact with one another and we chat with one another and we ask questions like, what are your insights about something? And with these chatbots powered by LLMs we are able to do that. I always raise the same awareness, let’s say, which is to be careful because these models are just stochastic parrots. They just know the probability distributions of our language. And they know, so to say, they don’t know anything, they are just mathematical models, but they kind of produce the words that are coherent with this probability distribution. They are not being creative, but at the same time, if they are good enough for helping us to come up with new ideas, that’s amazing.
Calde: I mean, they are stochastic parrots and they’re helping us think, right? So I have started wondering myself how I am not a stochastic parrot because I’m feeling like yeah, I mean, like I also process a lot of language and also like helping others to have conversations and helping them think, right? Like a lot of discovery that we do in product, we talk to others, we try to help them think, to organize their thinking.
Nacho: I will introduce just a small comment because you have triggered me.
Vico: Time for existential problems now, being triggered?
Nacho: It’s more from a technical perspective because I think that when it comes to language, we have like different layers here. So with language we have syntax and we also have the semantics of the language. And those are the two things that we are able right now to capture with LLMs. So that part of the equation is already solved. But the part that we are not ready to solve yet with artificial intelligence is the conceptualization of the word. These LLMs pretend to conceptualize the word, but they are just able to produce words in a way that lets you believe that. But they are unable to understand how touching a human being feels like. How seeing our children coming to the world, those kind of things, they are unable to feel and they can pretend that they know and they can put that in words, but they don’t know how to do that.
Vico: Yeah. I actually like that, and you know we have these strong different views of how this could be. And it’s really interesting to actually even discuss about the conceptualization of knowledge and intelligence broadly. But what you brought in there and actually made me think lately of experiences that I have recently had. Because you’ve been given this understanding that they are, let’s call it, pretending or making us believe that they conceptualize, right? It’s more than in the LLMs, the LLMs are clear on their limitations, it’s how we actually experience it. I actually used even for, and I’m going to stray a little bit from product and go back, just a second so bear with me, actually for creative processes even more broad. I actually did an experiment around my poetry writing actually, and that is like, okay, I did a piece, put it up to it, said review it, give me your understanding. How do you interpret this? What are you seeing from what I’m writing? And give me a critique. And I was doing, I was like, hey, it got to understand a few things that I wasn’t really expecting, like how nostalgic it was, or even growth. The images had nothing to do, it was not really straightforward there. It made me understand, incredibly, opportunities that I hadn’t even myself considered exploiting in that poem through different imagery. I was like, hey, I didn’t thought this image and that image were actually connected. What would happen if I were to introduce this other one, to re-reinforce it and add it to the plot. So in that sense, it was really a process of enhancing creativity and the creative process, even though of course, it’s simply trying to understand language and this goes, this takes me back completely to use cases around product. And I think enhancing brainstorming and creativity is without a doubt one of the tools that it’s incredibly useful for, if we understand limitations and we’re mindful of verifying also that information, specifically if you’re using it for accelerated research. I think the latest versions that allow you to actually see the citations and the sources are a huge advancement on that and help us like really understands where it comes and I’m really looking forward, as I mentioned, on seeing more how it can create more digital twins or discovery and design throughout this process and getting us a little bit into what happens when we work with synthetic data. I’m like a little bit excited on that. I know it’s not like current use case I’m seeing now, but there’s a few things are moving that way and we really like to see more of that.
Nacho: I really love that that the use cases that you guys have brought are most of them related with this idea of chatting with the computer and that’s because you have this interaction background. What I’d also like to bring here to the table because I would like the audience to also, to trigger the audience with some additional ideas. I think that this idea of chatting with a bot is amazing, but at the same time we can use the bot in the background, you know, to solve other tasks. And this is all possible through prompt engineering, which in the end is instructing the computer to do something in a very structured manner so that they do exactly what we want it to do. But I think that figuring out that by prompting and instructing the computer to do something, we can unleash like different use cases. So for example, this summarization use case that you brought, Calde, it’s just a prompt saying, I provide you with this text and please extract a summary out of that. When we talk about, for example, this other use cases that you brought before about analyzing massive amounts of data, then then we have a prompt that is essentially doing that. Or we have a retrieval augmented generation system that combines two different types of LLMs all together in the same structure. Also the few-shot learning algorithms that it’s something that we have been doing research on for lots of years. Now we are experiencing that LLMs can be used for tasks like that, kind of I explain to you how to do a task with only two or three examples, and then I have something that works as accurate as a classifier that otherwise I should train with thousands of samples. And maybe it’s not perfect, but can be used already, which is like the important part, right? But I’m bringing all these technical details because one thing that I was thinking about that that is kind of difficult in this scenario, in this landscape is to run these discovery sessions, right? Trying to connect the dots between the use cases and the solutions that are available. Now it feels like everything is possible with AI, so finding that sweet spot in which the solution connects with the problem or the user requirements. It’s kind of challenging, right?
Calde: Yes. And I have something to add here, because what I think is that it would be interesting if we focus more on how this technology can help augment user needs rather than automate them away, right? Because I’m seeing also trending that these can be automated through AI, like a dream of that, right? When AI is used to augment the experience that a human being has with his work, I find it much more interesting, right? And there’s like a human side that is interesting to keep in the loop, and it’s something that we should aim to as product creators, we start to create more and more products that incorporate AI features and the way we incorporate that feature is our decisions, so we should continue getting into the user journeys, understanding the task they have to do, try to augment that, ask an automated way what makes sense to automate, but also provide tools to help that user in those contexts to do their jobs better, right? The future is something really interesting to get into there, it also has to do with what makes humans productive, which is I think some nice conversation here, or what makes the human valuable to produce new things, because there is also like a lot of authenticity, and things that can’t and won’t be covered by LLMs, because as you say, right? They are stochastic parrots. They are repeating, and they’ve also been instructed to, like having… this has to do with how humans are productive and produce things, right? What’s the value about humans. That’s what I think is a really interesting and existential conversation.
Vico: I love that you bring this in actually on there because I’ve been thinking a lot about the whole innovation process and the way we, as you mention, we build value throughout the process and discussing a lot of how actually this source can outman our creativity, can actually provide an acceleration on even our way to get into insights. And if you think about innovation, this streaming from finding and understanding and framing problems, and it comes to me that for new problems and new solutions that then you try to actually either work forward to the solution or backwards finding the problem is that your solution actually it’s able to solve, there’s a huge amount of work around synthesizing knowledge, so that main role right now is mainly focused on people. You actually require a lot of experts. We work a lot actually throughout discovery, having research that we do from our side, working around having interviews, having a lot of conversations, to gather all that information, to walk through the process, challenging assumptions and leading the conversation or even the thought of a group to uncover the value and create new things. And there’s a possibility there that I see that it’s, I think I was commenting with you guys, about something still not completely solidified or not in the practice today, and some of that I’m becoming such like a long shot, I’m going always on about the future. No, that’s why I say please bear with me, it’s Friday. I shouldn’t be saying the day actually, but well, we’re recording this a Friday, guys, in January. It’s interesting because I think this is opening like a really interesting wave of ok, we’ve relied a lot on people with specific soft and both soft and hard skills for being like these incredible synthetizers, to manage the conversation for people from different backgrounds. Now we have a tool that can help us actually with a little bit of that translation. I agree with my colleagues that it’s very difficult, and I think we’re far away yet of having it, during the synthetization process and being able to lead a thought process like this without a doubt. But I think it enables us to reach a broader or easier to access specific knowledge that has a great power for the process. And I know I will save a few things for a much broader discussion about the future because I know we are running out of time there.
Nacho: Yes, we are. I was going to say that we are running out of time. But actually there were so many things that you said that triggered other lines in my head because on the one side, we can use them to help us during the process for ideation and for discovery. But sometimes we are trying to do a discovery for an AI application. So it’s kind of an Inception movie. You know, we have an AI component inside an AI component. So yeah, it’s it’s amazing. Unfortunately, we are running out of time, but I think that we have already enough topics to to start talking about in the next episode. So I would encourage you guys, if you are listening to us, to get ready for the next episode, because we will cover some of the points that we missed in this one. We will talk a little bit more about this creation process and how AI can be embedded in these processes. And we will also talk about our forecasts about the future of AI and the future of product when it comes to AI. But I would like to thank you, Calde and Vico, I think that it was a really insightful conversation that we had today. I really appreciate that we smiled more than in the last episode, so that’s good. We also had some interruptions as well, but it doesn’t matter. I think that it’s a more relaxed episode.
Vico: Completely, yeah.
Nacho: Yeah. And to the audience, if you guys made it to this point, that means that you have listened to the entire episode, which is always great. So thank you for your company.
Vico: Much appreciated.
Calde: like, subscribe, share.
Nacho: Yes, we have to make all that promotion. So yeah, if you enjoyed the episode, please leave us a review on Apple Podcasts and Spotify or comment in the box below if you’re watching us on YouTube and remember that Chain of Thoughts, this podcast, seeks to build a community of product designers, developers and AI experts to keep creating great digital tools for customers and users. So to keep this community growing, remember, as Calde said, like follow and subscribe to Chain of Thoughts in Spotify, YouTube, Apple Podcasts and share with your own followers using the hashtag #ChainofThoughtsPodcast on X and LinkedIn. You can also follow Arionkoder on these social media platforms to keep yourself updated about the show and our actions as a company and see you soon, on the next piece of the chain.
Vico: See you soon! Thank you Nacho.
Calde: See you soon!
—
In case you missed it, find episode 1 – And here come the LLMs – here! And stay tuned for upcoming releases every month.