Blog / AI, AI Strategy & Implementation, Digital Products, Future, Product Design & Development

Knowledge Base Management through AI

By Juan Pablo Vargas

August 27, 2024

What is a Knowledge Base?

When we talk about a knowledge base nowadays, we may assume that it’s just a library of information and data related to a company, a product or a specific project. This vague definition leaves out some key aspects to consider when discussing managing knowledge inside an organization or a team.

Let’s start by defining what knowledge is in the context of an organization. Knowledge can be anything: from a defined set of rules of engagement, to a product or service detailed description, to a repository of key information of a project and even to processes flows and maps. The main reason knowledge is needed inside an organization is to align together as one team into what constitutes value, how value is produced and how this value can be harnessed and communicated inside and outside the organization.

A knowledge base can be seen as the place (online or offline) where this knowledge lives. It can be sourced from anywhere, any data format (video, image, diagram or plain text) and even any software. One key aspect that we must understand from this knowledge base is its structure and organization, because an important reason why organizations build it is to onboard new members of the team to understand how things work and get up to speed fast and easily. FAQs, process manuals, troubleshooting step-by-step guides and even an organization chart are usually found in this knowledge base in order to explain to any newcomer what’s essential to start delivering value in their role.

Another key aspect is access to the actual knowledge base. Depending on the organization, this base can be open to anyone, or in some cases some sensitive information can be left for leadership or HR teams. However, any access management protocols should be in place and be very transparent to anyone inside the organization. Clear communication and alignment amongst the organization can basically make or break knowledge base management as expectations, roles and responsibilities are made visible for everyone.

Knowledge base management 101

To begin the creation of the knowledge base you must understand as an organization the following:

Who is in charge of the knowledge base organization and update of information?
Who is responsible for the onboarding of new employees?
Who is responsible for defining the sources of information and software needed to support them?
Which teams inside the organization are meant to create new knowledge and how to spread it around?
Who should handle knowledge base access management?
Are there any best practices in documentation inside the knowledge base?

With the aforementioned questions answered, as an organization you would be ready to build and maintain a knowledge base updated and valuable for all people inside the organization. However, with ever-changing information, managing a knowledge base can become a true hassle and difficult process. Therefore, organizations are now tending to use more software tools to decrease manual processes to handle knowledge and make it accessible easily and quickly for anyone who needs it.

Managing a Knowledge Base can become a nightmare

Contributors of new knowledge and information can come from anywhere and it could be stored in a centralized database for anyone to access it. Nevertheless, trying to navigate inside that pile of data could be a somewhat tricky process. For example, if you want to understand the organizational chart you can go to tool A, then if you need to know the holiday’s policy a tool B may be needed and even a tool C when you need to request some sick days off as an employee. So in just 3 simple questions as the ones just mentioned, you can get lost in 3 different tools or platforms and don’t get me started with other simple questions, say about an overview of a certain project inside the organization.

Anxiety and frustration might kick in as an employee, since trying to navigate through knowledge or even asking a simple question can become difficult. Finally, what usually happens is that people just ask another person where to look for something inside the myriad of places or platforms and then roam around them. Is that it? Should we just accept our fate and ask endless questions to our colleagues through Slack, Teams or <insert here any chat platform your company uses>?

I believe here’s where AI comes in. Yes, I know AI is not the answer to all questions, and we are starting to get bored when we hear those two letters together. Believe me, I have rolled my eyes in disbelief or exasperation when people come at me with the silver bullet solution of AI for everything. However, in this specific use case, I think there is an actual value to draw out of artificial intelligence. First of all, let’s get something straight out of the way: AI will not replace entirely roles to manage a knowledge base. “Entirely” is key here as some automation and augmentation of manual processes will definitely happen, and this would mean eventually that new processes will be created and some old ones will be deprecated.

How can AI help manage our knowledge base?

So now that the cat is out of the bag let’s begin to understand how AI will actually help manage our organization’s knowledge base. Let’s suppose we have already built our knowledge base from scratch. As explained before its purpose is to make knowledge accessible to anyone inside the organization and accurately navigate through all sorts of spiderwebs of data. AI can organize, store and retrieve information from this knowledge base easily, streamlining the process of search thus enhancing the information discovery process. Instead of manually organizing and storing content, through machine learning algorithms now is possible to do it automatically.

Furthermore, AI is even more important in another use case of knowledge base management and that is in the search and navigation of heaps of data. Using natural language processing (NLP) it can basically take all content from the centralized database, then understand and comprehend meaning behind plain text or even images using optical character recognition (OCR). It all sounds a bit too wordy and full of abbreviations, so let’s break it down.

First, the data retrieval process is done through ingestion from all data sources or from a centralized data source (let’s assume the organization uses Confluence, Notion or other documentation software), making this data available and accessible to an AI agent. This “agent” is basically an assistant that could act as a skilled coworker, seamlessly working side by side with humans in the knowledge base management processes.

The agent will be the one that uses NLP as explained above to interpret the data and then be ready to answer general questions about the knowledge base (the same ones we usually make to other coworkers continuously). Human interaction with this agent can be done through different user interfaces (UI), usually chats like ChatGPT are the standard nowadays. In more advanced use cases the agent will not only answer simple questions but generate actual relevant content, therefore updating the knowledge base and keeping it valuable continuously.

In the first use case, the agent will try to understand all questions and have a normal conversation with a human, using correctly the ingested data in order to offer a quick and simple answer. I know what you’re thinking, “not another chatbot please!”, and I totally sympathize with that feeling of frustration that stems from interacting with annoying bots that don’t even understand the question in the first place. Yet, finally it seems we might be able to have an actual conversation with an agent (not a chatbot), through smooth back and forth interaction as if you were actually talking to someone through Slack or right next to the water cooler at the office.

So, are agents the new search engine?

Moreover, I can totally comprehend the skeptic’s smirking in the backdrop, thus I will point out that there’s a disclaimer: not all conversations or questions will be smoothly answered in 1 second. Using large language models (LLMs), an evolution from NLP, AI agents will have to analyze and learn from data inputs to continuously improve their performance and accuracy. As in any learning process for machine learning, the more data we use the more it will learn, however it will need some fine tuning along the way. This seems reasonable when you come to think that AI is mimicking a human being, and we as humans can have conversations in which we can be misinterpreted or misunderstood, remember: humans are not perfect either, so don’t expect AI agents to be perfect from the start.

The actual question we should ask ourselves now is: are these agents the new way to search information in any knowledge base? Are we replacing simple search bars with autocomplete functions? Or are we just trying to get our tech to be more human after all? (remember when I said AI will not replace humans entirely?). It’s time to accept that AI will shape the way we search, navigate and use an organization’s knowledge base, basically any internal process for that matter.

Is AI’s purpose clear to everyone?

AI is here to stay and automate or enhance what we do in our daily tasks through our usual tools, maybe embedding it inside a current user journey will come across as “imperceptible”, but what is certain is it will completely change how we do things in general. It will be a slow change because we are humans and as behavioral animals we will need to adapt, we will begin by trying ChatGPT (I guess everyone has tried it now, right?) and then seeing how those seemingly godly powers can actually help us become more efficient and save time for tasks that are not boring (like searching for simple info from a knowledge base).

I might be getting carried away at the end here but I find ethics in AI is key to fully understanding the repercussions of using tech and main reasons behind it: should we use AI to make us happier by giving us more time back? Or should we take a step back and understand if saving time in the first place is our ultimate goal? One thing is for certain, and that is that organizations need to adapt as quickly as possible in order to create a competitive advantage by using AI, nevertheless reflecting on why they use AI is as important as the use itself, it needs to have a clear purpose behind (don’t rely on using AI because everyone else is doing it!).

One last thing I want to say is that even though AI has been our new buzzword for the past 2 years, and apparently everybody loves it, we should never forget that it is not a one-size-fits-all type of solution. AI will not fix our knowledge base if the data we have inside is garbage and content is not clear and understandable for a human being, and also AI is not magic, it’s not like a PM will use its genie powers to snap its fingers to make AI appear from nowhere to make everything great in 1 day. Maybe it’s hard to grasp that AI is just another tool in our toolkit and I think there’s a quote by Abraham Maslow that summarizes it perfectly: “If the only tool you have is a hammer, it is tempting to treat everything as if it were a nail.”