Introduction
Norwegian Refugee Council (NRC) has developed a prototype AI-powered chatbot to facilitate precise and rapid access to policies and other relevant information from NRC's extensive internal databases. It utilises a large language model (LLM) that has been trained on NRC’s documents, and a chatbot interface through which users can ask questions and receive responses for this purpose. The tool aims to provide easy access to a wealth of organisational knowledge that might take a lot of time to search through otherwise.
Background
Effectively preserving and accessing institutional knowledge is a constant challenge for nonprofits, with the restricted resources, frequent organisational changes and high turnover. Zineb Bhaby, AI lead at NRC, says that users working in programs often ask about ways to easily retrieve information they need for their work. For example, they might be looking for a specific policy or a document with approaches on provision of cash assistance. In response, Zineb and her team started searching for a way to speed up the process of retrieving this kind of information. They believed that AI, specifically large language models (LLMs) in combination with a chatbot user interface, could address this issue.
AI development
Interest in utilising AI based solutions is high at NRC, and several potential use cases have been discussed. The team conducted a thorough assessment of its technical capabilities and data infrastructure to ensure readiness for AI implementation. This evaluation included mapping potential use cases identified by both internal stakeholders and other humanitarian organisations. Among the various applications considered, knowledge management and retrieval emerged as a priority due to its potential to support programoperations and save users’ time.
Retrieving knowledge from a large variety of resources quickly is something LLMs can do. A chatbot can provide a natural language interface to interact with the LLM. The NRC team believed that the combination of both would give their users what they needed.
The team also wanted to make sure the future solution would be cost efficient and keep NRC’s data safe. They therefore opted for an architecture where they would use a small LLM and host it in their own cloud environment. This served also to avoid vendor lock-in and to have a solution that would be small enough to potentially run on a local server in low-connectivity environments. Given the focus of the use case was to accurately retrieve information and not to generate completely new content, a small LLM was powerful enough to do what NRC needed.
NRC chose one of Llama’s models and the team built a working prototype within three months, which will now be opened to a select group of users for testing. The model was solely trained on NRC’s internal information, to avoid introducing any inaccurate information from other sources.
Operationalising AI
NRC’s next step will be to introduce a core group of users from across the organisation to the tool, and train them in how to use it and, importantly, how to provide feedback when the tool does not give the correct response. One thing the project team learned during their own testing and demonstrating of the tool was that it really needed knowledge owners who are very familiar with specific policies and processes to validate the output – something that takes time and needs to be factored in when building such an application. Otherwise, the risk is that the tool may combine information from different documents that is not strictly relevant to the question asked by the user or reference out of date documents.
The team has implemented strict guardrails and system prompting to ensure the model does not hallucinate or generate content outside the scope of the knowledge base, and they will work on refining the model continuously in that way as users test and provide feedback. Accuracy and quality of the result are the key measures that will be applied.
Support for this initial group of users will be provided by the project team with the existing IT team supporting the infrastructure; however, to scale out more broadly through the organisation, additional funding for infrastructure and resources will need to be found.
Learnings
The initial experience of the project team is that the tool can provide accurate information efficiently and can do so in a variety of languages as well – besides English, French and Spanish have been tested so far. At the same time, they know further improvements and tweaks will be needed, which will be done based on the feedback of the users testing the tool. As Zineb Bhaby says, the team developed a data curation strategy as part of the project which defined which documents would be ingested by the LLM and how it would, for example, handle different versions of the same policy document, but still expert review is needed to avoid “surprises” such as the tool merging information from two documents on different subject matters into one response that is then not quite accurate. For that reason, the responses from the chatbot always provide links to the sources of the information, so that users can double check and validate the response they have been given.
In terms of the architecture chosen, the team’s experience is so far that while there has been some up front investment to build the tool, the cost of running it is very predictable and comparatively low, as it runs in the organisation’s own cloud environment. As the use of the tool increases and other use cases are added, the infrastructure and with that also the cost of running it will of course increase.
Plans for the future
The NRC team will continue to monitor the ongoing technological developments in the AI space and continue to assess their solutions’ approach and architecture based on that. Other, newer LLMs may be more suitable for this or other use cases in the future.
On top of evaluating the accuracy and quality of the chatbot’s responses, the team also wants to look at ways to assess the actual “usefulness” of the tool, through measures such as time saved by using the tool versus manually searching for relevant information. This could help in securing funding to keep the solution going and to extend it to more users and more use cases, such as supporting the writing of grant applications and donor reports.
The team has also been experimenting with adding information from external sources. So far, they have tested interfacing with ReliefWeb which is run by the United Nations Office for the Coordination of Humanitarian Affairs (OCHA). The content is carefully curated and therefore considered reliable and accurate so that it can be ingested without further validation needed on NRC’s side.
A future vision is also to share the use of the chatbot with other organisations that NRC collaborates with. A joint use across a consortium would deliver efficient information retrieval to all organisations involved and may attract funding for donors who are keen to support joint efforts between nonprofit organisations.
Where to learn more
Zineb Bhaby, NRC AI Lead Zineb.bhaby@nrc.no