QA RAG with Self Evaluation II

For this variation, we make a change to the evaluation procedure. Along with the question-answer pair, we also pass the retrieved context to the evaluator LLM.

To perform this, we add a further itemgetter function within the second RunnableParallel to gather the context string and pass it to the brand new qa_eval_prompt_with_context prompt template.

rag_chain = ( 
RunnableParallel(context = retriever | format_docs, query = RunnablePassthrough() ) |
RunnableParallel(answer= qa_prompt | llm | retrieve_answer, query = itemgetter("query"), context = itemgetter("context") ) |
qa_eval_prompt_with_context | 
llm_selfeval |
json_parser
)

Implementation Flowchart :

Considered one of the common pain points with using a sequence implementation like LCEL is the issue in accessing the intermediate variables, which is significant for debugging pipelines. We have a look at few options where we are able to still access any intermediate variables we have an interest using manipulations of the LCEL

Using RunnableParallel to hold forward intermediate outputs

As we saw earlier, RunnableParallel allows us to hold multiple arguments forward to the following step within the chain. So we use this ability of RunnableParallel to hold forward the required intermediate values all the best way till the tip.

Within the below example, we modify the unique self eval RAG chain to output the retrieved context text together with the ultimate self evaluation output. The first change is that we add a RunnableParallel object to each step of the method to hold forward the context variable.

Moreover, we also use the itemgetter function to obviously specify the inputs for the next steps. For instance, for the last two RunnableParallel objects, we use itemgetter(‘input’) to be sure that only the input argument from the previous step is passed on to the LLM/ Json parser objects.

rag_chain = ( 
RunnableParallel(context = retriever | format_docs, query = RunnablePassthrough() ) |
RunnableParallel(answer= qa_prompt | llm | retrieve_answer, query = itemgetter("query"), context = itemgetter("context") ) |
RunnableParallel(input =  qa_eval_prompt, context = itemgetter("context")) |
RunnableParallel(input = itemgetter("input") | llm_selfeval , context = itemgetter("context") ) | 
RunnableParallel(input = itemgetter("input") | json_parser,  context = itemgetter("context") )
)

The output from this chain looks like the next :

A more concise variation:

rag_chain = ( 
RunnableParallel(context = retriever | format_docs, query = RunnablePassthrough() ) |
RunnableParallel(answer= qa_prompt | llm | retrieve_answer, query = itemgetter("query"), context = itemgetter("context") ) |
RunnableParallel(input =  qa_eval_prompt | llm_selfeval | json_parser, context = itemgetter("context"))
)

Using Global variables to avoid wasting intermediate steps

This method essentially uses the principle of a logger. We introduce a brand new function that saves its input to a world variable, thus allowing us access to the intermediate variable through the worldwide variable

global context
def save_context(x):
global context
context = x
return x
rag_chain = ( 
RunnableParallel(context = retriever | format_docs | save_context, query = RunnablePassthrough() ) |
RunnableParallel(answer= qa_prompt | llm | retrieve_answer, query = itemgetter("query") ) |
qa_eval_prompt | 
llm_selfeval |
json_parser
)

Here we define a world variable called context and a function called save_context that saves its input value to the worldwide context variable before returning the identical input. Within the chain, we add the save_context function because the last step of the context retrieval step.

This feature lets you access any intermediate steps without making major changes to the chain.

Using callbacks

Attaching callbacks to your chain is one other common method used for logging intermediate variable values. Theres so much to cover on the subject of callbacks in LangChain, so I might be covering this intimately in a special post.