Structured Outputs

Learn how to use JSON mode to get reliable, structured outputs from Large Language Models on the Gravix Layer platform.

Overview

Large Language Models (LLMs) typically return responses in plain text. While useful for chatbots, this format can be difficult to parse when you need to extract specific information programmatically for use in your applications.

Certain models on the Gravix Layer platform support structured JSON outputs. By enabling this feature, you can ensure that model responses are predictable data objects that are easy to work with. To enable structured responses, you specify the desired format in the response_format parameter of the Chat Completions API.

Gravix Layer supports two primary modes for this:

JSON Mode: Forces the model to output a syntactically correct JSON object without a predefined structure.
JSON with Schema: Forces the model to output a JSON object that strictly conforms to a provided JSON Schema definition.

JSON Mode

This mode guarantees the model will produce a valid, well-formed JSON object. It does not enforce a specific schema, giving the model flexibility on the structure while ensuring the output is machine-readable. Set response_format to {"type": "json_object"} to enable it.

Request

Here are the requests for generating a product catalog by enabling JSON mode in Python, cURL, and JavaScript.

cURL
Python
JavaScript

# Ensure your GRAVIXLAYER_API_KEY is set as an environment variable
# export GRAVIXLAYER_API_KEY='your-api-key'

curl https://api.gravixlayer.com/v1/inference/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -d '{
    "model": "llama3.1:8b-instruct-fp16",
    "messages": [
      {
        "role": "system",
        "content": "You are a helpful assistant that generates product catalog data in JSON format."
      },
      {
        "role": "user",
        "content": "Generate a small product catalog for a tech store with 3 products. For each product include: name, price, description, in_stock (boolean), features (list of strings), and ratings (object with average and count). Make it realistic and detailed."
      }
    ],
    "response_format": {
      "type": "json_object"
    },
    "temperature": 0.7
  }'

import json
import os
from openai import OpenAI

client = OpenAI(
    base_url="https://api.gravixlayer.com/v1/inference",
    api_key=os.environ.get("GRAVIXLAYER_API_KEY"),
)

prompt = """Generate a small product catalog for a tech store with 3 products.
For each product include:
- name
- price  
- description
- in_stock (boolean)
- features (list of strings)
- ratings (object with average and count)
Make it realistic and detailed."""

response = client.chat.completions.create(
    model="llama3.1:8b-instruct-fp16",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates product catalog data in JSON format."
        },
        {"role": "user", "content": prompt}
    ],
    response_format={"type": "json_object"},
    temperature=0.7
)

print(response.choices[0].message.content)

import { OpenAI } from 'openai';

const client = new OpenAI({
    baseURL: "https://api.gravixlayer.com/v1/inference",
    apiKey: process.env.GRAVIXLAYER_API_KEY,
});

const prompt = `Generate a small product catalog for a tech store with 3 products.
For each product include:
- name
- price
- description
- in_stock (boolean)
- features (list of strings)
- ratings (object with average and count)
Make it realistic and detailed.`;

const response = await client.chat.completions.create({
    model: "llama3.1:8b-instruct-fp16",
    messages: [
        {
            role: "system",
            content: "You are a helpful assistant that generates product catalog data in JSON format."
        },
        { role: "user", content: prompt }
    ],
    response_format: { type: "json_object" },
    temperature: 0.7
});

console.log(response.choices[0].message.content);

Response

The API returns a message containing a string of valid JSON.

{
  "products": [
    {
      "name": "Apple MacBook Air",
      "price": 1299,
      "description": "The Apple MacBook Air is a powerful yet portable laptop, designed for professionals and students alike.",
      "in_stock": true,
      "features": [
        "10th Gen Intel Core i5 processor",
        "8GB RAM",
        "256GB SSD",
        "13.3-inch Retina display"
      ],
      "ratings": {
        "average": 4.7,
        "count": 123
      }
    },
    {
      "name": "Dell XPS 15",
      "price": 1499,
      "description": "The Dell XPS 15 is a high-performance laptop, ideal for gamers and creatives.",
      "in_stock": false,
      "features": [
        "10th Gen Intel Core i7 processor",
        "16GB RAM",
        "512GB SSD",
        "15.6-inch OLED display"
      ],
      "ratings": {
        "average": 4.9,
        "count": 25
      }
    },
    {
      "name": "HP Envy x360",
      "price": 899,
      "description": "The HP Envy x360 is a versatile laptop, perfect for everyday use.",
      "in_stock": true,
      "features": [
        "AMD Ryzen 5 processor",
        "8GB RAM",
        "256GB SSD",
        "15.6-inch Full HD display"
      ],
      "ratings": {
        "average": 4.2,
        "count": 76
      }
    }
  ]
}

JSON with Schema

For maximum reliability, you can provide a specific JSON Schema to constrain the model's output. The model is forced to generate a JSON object that strictly adheres to the types, formats, and properties you define.

Request

Here are the requests for generating structured data for a Person object by providing a JSON schema.

cURL
Python
JavaScript

# Using a 'here document' (<<-EOF) is recommended for complex JSON payloads
curl https://api.gravixlayer.com/v1/inference/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $GRAVIXLAYER_API_KEY" \
  -d @- <<-EOF
{
    "model": "llama3.1:8b-instruct-fp16",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant that generates JSON data according to a specific schema."
        },
        {
            "role": "user",
            "content": "Generate a person's information for John Doe, age 30, with email [email protected]. Provide two addresses: one in New York, NY and one in Los Angeles, CA. Include some metadata about their preferences. Make sure to include ALL required fields."
        }
    ],
    "response_format": {
        "type": "json_object",
        "schema": {
            "type": "object",
            "properties": {
                "name": {"type": "string", "description": "Full name of the person"},
                "age": {"type": "integer", "description": "Age in years", "minimum": 0, "maximum": 120},
                "email": {"type": "string", "description": "Email address"},
                "addresses": {
                    "type": "array",
                    "description": "List of addresses",
                    "items": {
                        "type": "object",
                        "properties": {
                            "street": {"type": "string", "description": "Street address"},
                            "city": {"type": "string", "description": "City name"},
                            "state": {"type": "string", "description": "State or province"},
                            "zip_code": {"type": "string", "description": "ZIP or postal code"},
                            "country": {"type": "string", "description": "Country name", "default": "USA"}
                        },
                        "required": ["street", "city", "state", "zip_code"]
                    }
                },
                "metadata": {"type": "object", "description": "Additional metadata"}
            },
            "required": ["name", "age", "email", "addresses"]
        }
    },
    "temperature": 0.7
}
EOF

from openai import OpenAI
from pydantic import BaseModel, Field
from typing import List, Optional
import json
import os

# Define the data structure using Pydantic models
class Address(BaseModel):
    street: str = Field(..., description="Street address")
    city: str = Field(..., description="City name")
    state: str = Field(..., description="State or province")
    zip_code: str = Field(..., description="ZIP or postal code")
    country: Optional[str] = Field(default="USA", description="Country name")

class Person(BaseModel):
    name: str = Field(..., description="Full name of the person")
    age: int = Field(..., description="Age in years", ge=0, le=120)
    email: str = Field(..., description="Email address")
    addresses: List[Address] = Field(..., description="List of addresses")
    metadata: Optional[dict] = Field(default=None, description="Additional metadata")

# Convert the Pydantic model to a JSON Schema dictionary
json_schema = Person.model_json_schema()

client = OpenAI(
    base_url="https://api.gravixlayer.com/v1/inference",
    api_key=os.environ.get("GRAVIXLAYER_API_KEY"),
)

prompt = """Generate a person's information for John Doe, age 30, with email [email protected].
Provide two addresses: one in New York, NY and one in Los Angeles, CA.
Include some metadata about their preferences. Make sure to include ALL required fields."""

response = client.chat.completions.create(
    model="llama3.1:8b-instruct-fp16",
    messages=[
        {
            "role": "system",
            "content": "You are a helpful assistant that generates JSON data according to a specific schema."
        },
        {"role": "user", "content": prompt}
    ],
    response_format={"type": "json_object", "schema": json_schema},
    temperature=0.7
)

print(response.choices[0].message.content)

// This example uses Node.js with the native fetch API
async function getPersonData() {
    const apiKey = process.env.GRAVIXLAYER_API_KEY;
    const url = "https://api.gravixlayer.com/v1/inference/chat/completions";

    const jsonSchema = {
        type: "object",
        properties: {
            name: { type: "string", description: "Full name" },
            age: { type: "integer", description: "Age in years" },
            email: { type: "string", description: "Email address" },
            addresses: {
                type: "array",
                items: {
                    type: "object",
                    properties: {
                        street: { type: "string" },
                        city: { type: "string" },
                        state: { type: "string" },
                        zip_code: { type: "string" },
                        country: { type: "string", default: "USA" }
                    },
                    required: ["street", "city", "state", "zip_code"]
                }
            },
            metadata: { type: "object" }
        },
        required: ["name", "age", "email", "addresses"]
    };

    const payload = {
        model: "llama3.1:8b-instruct-fp16",
        messages: [
            {
                role: "system",
                content: "You are a helpful assistant that generates JSON data according to a specific schema."
            },
            {
                role: "user",
                content: "Generate info for John Doe, age 30, email [email protected], with addresses in NY and LA."
            }
        ],
        response_format: { type: "json_object", schema: jsonSchema },
        temperature: 0.7,
    };

    try {
        const response = await fetch(url, {
            method: 'POST',
            headers: {
                'Content-Type': 'application/json',
                'Authorization': `Bearer ${apiKey}`,
            },
            body: JSON.stringify(payload),
        });

        const data = await response.json();
        console.log(data.choices[0].message.content);
    } catch (error) {
        console.error("Error fetching data:", error);
    }
}

getPersonData();

Response

The model's output is guaranteed to conform to the provided schema.

{
    "name": "John Doe",
    "age": 30,
    "email": "[email protected]",
    "addresses": [
        {
            "street": "123 Main St",
            "city": "New York City",
            "state": "NY",
            "zip_code": "10001"
        },
        {
            "street": "456 Elm St",
            "city": "Los Angeles",
            "state": "CA",
            "country": "USA",
            "zip_code": "90001"
        }
    ],
    "metadata": {
        "preference": {
            "language": [
                "English"
            ],
            "music_genre": [
                "Rock",
                "Pop"
            ]
        },
        "version": 1
    }
}

Overview​

JSON Mode​

Request​

Response​

JSON with Schema​

Request​

Response​

Overview

JSON Mode

Request

Response

JSON with Schema

Request

Response