LangChain JavaScript – execute generated code

DanielJune 21, 2024July 6, 2024

Let's see how we can use LangChain.js and LLMs to generate JavaScript code and execute it to improve the accuracy of the results.

Large Language Models are not great at solving math problems. They work by estimating, so we can't rely on their capacity to solve math expressions with reliable precision.

Take the following math expression:

(5^2*40+sqrt(9)+350) / (3^2*20+102/1000+80*72)

The result of this operation is 0.227773866509363.

Let's use LangChain.js and ask GPT-4o to solve this:

import { ChatOpenAI } from "@langchain/openai"
import { StringOutputParser } from "@langchain/core/output_parsers"
import { PromptTemplate } from "@langchain/core/prompts"
import * as dotenv from "dotenv"

dotenv.config()

const model = new ChatOpenAI({
    modelName: "gpt-4o",
    temperature: 0.75
})

const prompt = PromptTemplate.fromTemplate(
`What is (5^2*40+sqrt(9)+350) / (3^2*20+102/1000+80*72)? 
Just tell me the result.`
)

const chain = prompt
                .pipe(model)
                .pipe(new StringOutputParser())

const result = await chain.invoke()
console.log(result)

I ran this code a few times and each time got different results:

The result is approximately 0.067.
The result is approximately 0.014.
The result is approximately 0.006944.

The spread is quite big, and worse, it's not close to the actual 0.227773866509363 value. Updating the temperature does not significantly improve the behaviour.

Use LangChain and ChatGPT to generate and execute JavaScript code

One way to improve the determinism of AI models is to ask them to generate code to solve problems that need an accurate result.

Based on this idea, we can improve accuracy by asking ChatGPT to generate JavaScript code to solve the math problem:

const cleanCode = (code) => {
    return code.replace('javascript', '').replaceAll('```', '') 
}

const prompt = PromptTemplate.fromTemplate(
`You are an expert at writing correct executable JavaScript code.
Generate the JavaScript code to solve the the below question. 
Return the JavaScript function code only and call it with no other explanation or log statments.
\n\n
What is (5 ^ 2 * 40 + sqrt(9) + 350) /  (3 ^ 2 * 20 + 102 / 1000 + 80 * 72)?`
)

const chain = prompt
                .pipe(model)
                .pipe(new StringOutputParser())

const generatedCode = await chain.invoke()

const codeClean = cleanCode(generatedCode)

try {
    const result = eval(codeClean)
    console.log('Math operation result = ' + result)
}
catch {
    console.error('Got an exception while running the code')
}

// Math operation result = 0.227773866509363

The generated JavaScript code from the LangChain invoke() call will look like this:

'''javascript
function calculateExpression() {
    return (Math.pow(5, 2) * 40 + Math.sqrt(9) + 350) / (Math.pow(3, 2) * 20 + 102 / 1000 + 80 * 72);
}

calculateExpression();
'''

Therefore the need for the cleanCode() function:

const cleanCode = (code) => {
    return code.replace('javascript', '').replaceAll('```', '') 
} 

Also, please note that the eval() method has some serious security implications.

You can find the full code for this example on my GitHub. Happy coding!