Handling LLM JSON Output Failures And Infinite Loops

Jul 27, 2025 by James Vasile 53 views

Once LLM Fails to Output JSON, Insert Infinite Loops

Hey guys! Today, we're diving into a tricky issue we've encountered with Large Language Models (LLMs) and how they can sometimes get stuck in infinite loops when they fail to output valid JSON. This is a critical problem, especially when dealing with applications that rely on structured data from these models. Let's break down the problem, understand why it happens, and explore potential solutions.

Understanding the Issue

When an LLM fails to output valid JSON, our system, go-light-rag, falls into an infinite loop. This happens because the application continuously retries to parse the output, hoping for a valid JSON format. The log snippets clearly show this: the Insert function in golightrag keeps retrying the parsing process, as indicated by the incrementing retry count and the consistent error message: failed to parse llm result: unexpected end of JSON input. This loop not only consumes resources but also prevents the application from moving forward, effectively halting the process.

The Technical Breakdown

At the heart of the problem is the parsing mechanism within go-light-rag. The system expects the LLM to return data in JSON format, which is then parsed and processed. However, LLMs, being probabilistic models, don't always guarantee a perfect output. Sometimes, due to various factors like the complexity of the query, the model's training data, or even random chance, the output might be malformed or incomplete JSON. When the parser encounters an invalid JSON, it throws an error. In our current implementation, this error triggers a retry mechanism, leading to the loop.

Why Does This Happen?

Several factors can contribute to an LLM's failure to produce valid JSON:

Incomplete Output: The model might cut off the JSON string prematurely, leading to an "unexpected end of JSON input" error.
Syntax Errors: The generated JSON might contain syntax errors like missing commas, brackets, or incorrect key-value pairs.
Unexpected Characters: The model might include extra characters or text outside the JSON structure, making it unparseable.
Model Limitations: The LLM's training data might not have adequately prepared it for generating complex JSON structures, or the model itself might have inherent limitations.

Understanding these factors is crucial in designing robust error-handling strategies.

Diving Deep into the Code: The `Insert` Function

To get a clearer picture, let's look at the relevant code snippet from go-light-rag, specifically the Insert function mentioned in the issue:

// Assuming this is a simplified representation
func Insert(llm LLM, data interface{}) error {
  for retries := 0; retries < maxRetries; retries++ {
    result, err := llm.GenerateJSON(data)
    if err != nil {
      // Log the error and retry
      log.Warn("LLM generation failed", "error", err)
      continue
    }

    parsedResult, err := parseJSON(result)
    if err != nil {
      log.Warn("Failed to parse LLM result", "retry", retries, "error", err)
      // This is where the infinite loop occurs
      continue
    }

    // Process the parsed result
    return processResult(parsedResult)
  }
  return fmt.Errorf("failed to insert data after %d retries", maxRetries)
}

In this simplified example, you can see the core of the problem. The Insert function calls an LLM to generate JSON, then attempts to parse it. If the parseJSON function fails (which is where the unexpected end of JSON input error occurs), the loop continues. The key issue here is that there's no mechanism to break out of the loop if the LLM consistently fails to produce valid JSON.

Proposed Solutions: Breaking the Infinite Loop

To address this infinite loop issue, we need to implement a strategy that allows us to skip problematic records and log them for further analysis. Here’s a breakdown of potential solutions:

1. Implement a Retry Limit with a Skip Mechanism

The most straightforward approach is to introduce a retry limit and a mechanism to skip the record if the limit is reached. This prevents the system from getting stuck indefinitely. Here's how we can modify the Insert function:

func Insert(llm LLM, data interface{}) error {
  maxRetries := 5 // Set a reasonable retry limit
  for retries := 0; retries < maxRetries; retries++ {
    result, err := llm.GenerateJSON(data)
    if err != nil {
      log.Warn("LLM generation failed", "error", err)
      continue
    }

    parsedResult, err := parseJSON(result)
    if err != nil {
      log.Warn("Failed to parse LLM result", "retry", retries, "error", err)
      if retries == maxRetries-1 {
        log.Error("Skipping record after max retries", "data", data)
        return fmt.Errorf("failed to parse after max retries, skipping record")
      }
      continue
    }

    // Process the parsed result
    return processResult(parsedResult)
  }
  return fmt.Errorf("failed to insert data after %d retries", maxRetries)
}

In this modified version, we set a maxRetries limit. If the parsing fails repeatedly and reaches this limit, we log an error, skip the record, and return an error. This prevents the infinite loop and allows the system to continue processing other records.

2. Implement Circuit Breaker Pattern

The Circuit Breaker pattern is a more robust approach for handling failures in distributed systems. It works by monitoring the success and failure rates of an operation (in this case, parsing JSON). If the failure rate exceeds a certain threshold, the circuit breaker “opens,” preventing further attempts to execute the operation for a certain period. This gives the system a chance to recover. After the timeout, the circuit breaker enters a “half-open” state, allowing a limited number of test calls to pass through. If these calls succeed, the circuit breaker “closes,” and normal operations resume. If they fail, the circuit breaker reopens.

Implementing a circuit breaker adds complexity but can significantly improve the resilience of the system. Libraries like github.com/sony/gobreaker can help implement this pattern in Go.

3. Improve LLM Prompting and Validation

Another approach is to improve the prompts sent to the LLM to guide it towards generating valid JSON. This involves:

Clear Instructions: Provide very clear and specific instructions in the prompt about the expected JSON format.
Examples: Include examples of valid JSON output in the prompt.
Schema Definition: Explicitly define the JSON schema the LLM should adhere to.

Additionally, we can add a pre-parsing validation step to catch and correct common errors before attempting to parse the JSON. This could involve simple regex checks or more sophisticated validation techniques.

4. Logging and Monitoring

Robust logging and monitoring are essential for identifying and addressing issues like this. We need to:

Log Input Data: Log the input data that caused the parsing failure. This helps in debugging and identifying patterns in problematic inputs.
Log LLM Output: Log the raw LLM output, even if it’s invalid JSON. This allows us to analyze the output and identify potential issues with the model or prompting.
Monitor Retry Counts: Monitor the number of retries and the frequency of parsing failures. This helps us detect and respond to issues proactively.

Practical Steps and Code Examples

Let's look at some practical steps and code examples for implementing these solutions.

Implementing Retry Limit with Skip

We already saw the code modification for the retry limit. Here's a more detailed example:

import (
  "encoding/json"
  "fmt"
  "log"
)

type LLM interface {
  GenerateJSON(data interface{}) (string, error)
}

func parseJSON(result string) (interface{}, error) {
  var parsedResult interface{}
  err := json.Unmarshal([]byte(result), &parsedResult)
  if err != nil {
    return nil, err
  }
  return parsedResult, nil
}

func processResult(result interface{}) error {
  // Placeholder for processing logic
  fmt.Println("Processed result:", result)
  return nil
}

func Insert(llm LLM, data interface{}) error {
  maxRetries := 5 // Set a reasonable retry limit
  for retries := 0; retries < maxRetries; retries++ {
    result, err := llm.GenerateJSON(data)
    if err != nil {
      log.Printf("LLM generation failed: %v", err)
      continue
    }

    parsedResult, err := parseJSON(result)
    if err != nil {
      log.Printf("Failed to parse LLM result (retry %d): %v", retries, err)
      if retries == maxRetries-1 {
        log.Printf("Skipping record after max retries: %v", data)
        return fmt.Errorf("failed to parse after max retries, skipping record")
      }
      continue
    }

    // Process the parsed result
    return processResult(parsedResult)
  }
  return fmt.Errorf("failed to insert data after %d retries", maxRetries)
}

// Example LLM implementation (for testing)
type MockLLM struct{}

func (m *MockLLM) GenerateJSON(data interface{}) (string, error) {
  // Simulate an LLM that sometimes fails to produce valid JSON
  if data == "bad_data" {
    return "{\"key\": \"value\"", fmt.Errorf("simulated JSON error") // Incomplete JSON
  }
  jsonBytes, _ := json.Marshal(map[string]interface{}{