Using Fallbacks in LangChain (JavaScript Version)

DanielAugust 21, 2024August 21, 2024

When building chains with LangChain.js, one or more elements in the chain may fail, potentially breaking the entire solution. This is where LangChain fallbacks come into play, providing an alternate path if something goes wrong.

Let’s look at a simple chain where we use an Anthropic model to handle a request:

import { ChatGroq } from "@langchain/groq"
import { ChatAnthropic } from "@langchain/anthropic"

// ⛔️ Force a failed response
const llmAnthropic = new ChatAnthropic({
    anthropicApiKey: 'invalid_api_key'
})

const response = await llmAnthropic.invoke('Tell me about yourself')

console.log(response)

The above code will fail, as the anthropicApiKey is invalid:

return new AuthenticationError(status, error, message, headers);
AuthenticationError: 401 {"type":"error","error":{"type":"authentication_error","message":"invalid x-api-key"}}

This is a forced error, but in production environments, LLMs can fail for various reasons, such as downtimes or rate limits.

The withFallbacks() Method

To improve the stability of our apps, we can use the withFallbacks() method to provide backups for essential nodes.

Here’s an example of how the withFallbacks() method works:

import { ChatGroq } from "@langchain/groq"
import { ChatAnthropic } from "@langchain/anthropic"

// Force an error by passing an invalid API key
const llmAnthropic = new ChatAnthropic({
    anthropicApiKey: 'invalid_api_key'
})

// Backup LLM
const llmGroq = new ChatGroq({
    apiKey: 'gsk_mqGpIuTckJ1u5nt3pW7GWGdyzbEVv9BHJv1p3ZPCisdjka'
})

const llmWithFallback = llmAnthropic.withFallbacks({
    // If llmAnthropic fails, use llmGroq
    fallbacks: [llmGroq]
})

const response = await llmWithFallback.invoke(
    'Tell me about yourself'
)

console.log(response)

The above code will return:

AIMessage {
  "content": `I am a Large Language Model trained by Mistral AI. 
  I am designed to generate human-like text based on the input 
  I receive. I do not have personal experiences or emotions, but 
  I can mimic a conversational style and provide information on 
  a wide range of topics. I am meant to be a helpful and 
  respectful tool, and I am constantly learning and improving to 
  better assist you.`,
  // rest of the message here
}

Since the call to the Anthropic model failed, LangChain.js used the backup model provided by ChatGroq.

By the way, this is just a demo. You need to replace the Groq key with your own. You can get a free API Groq key from here.

One nice thing about the withFallbacks() method is that it can be used for any runnable element, so it applies to any part of a LCEL chain.

The maxRetries Parameter

A parameter often linked to fallbacks is maxRetries.

As the name implies, this parameter defines the number of attempts a model will make to fulfill a user request before triggering the fallback:

const llmAnthropic = new ChatAnthropic({
    anthropicApiKey: 'key_here',
    maxRetries: 0
})

Using LangChain Fallbacks to Reduce Costs

The LangChain documentation provides an interesting use case for fallbacks.

We can try to answer a user request with a cheaper and/or faster model first, and only if that model is unable to fulfill the request, we fallback to a more advanced model:

const cheapModel = new OpenAI({
  maxRetries: 0,
  model: "gpt-3.5-turbo",
})

const expensiveModel = new ChatOpenAI({
  model: "gpt-4"
})

const model = cheapModel.withFallbacks({
    fallbacks: [expensiveModel]
})

model.invoke('💸 💸 💸 expensive user prompt here')

You can see the full code used in this example on my GitHub. Happy coding!