Inspect Model Backends and Endpoints

List the LLM backends configured in a workspace and inspect the endpoint addresses and online status for a selected backend with the SDK.

Task flow

List workspace LLM backends ──▶ Select target backend ──▶ Inspect backend endpoints and status

Prerequisites

  1. Configure the client, personal access token, and workspace ID (PRODUCT_API_BASE_URL, PRODUCT_API_KEY, WORKSPACE_ID) by following Use the SDK.

1. List LLM backends

Initialize the client and workspace model service, then list the workspace’s LLM backends to obtain the total count, backend IDs, names, and declared model names.

import os
import moi_product_sdk as sdk

client = sdk.new_with_personal_access_token(
    os.environ["PRODUCT_API_BASE_URL"], os.environ["PRODUCT_API_KEY"]
)
workspace = client.workspace(os.environ["WORKSPACE_ID"])
models = workspace.models()

page = models.list_workspace_llm_backends()
print("Total backends:", page.total)
for backend in page.backends:
    print(backend.id, backend.name, list(backend.models))
package main

import (
	"context"
	"fmt"
	"os"

	sdk "github.com/matrixorigin/matrixflow/sdk/go-sdk"
)

func main() {
	ctx := context.Background()
	client, err := sdk.NewWithPersonalAccessToken(
		os.Getenv("PRODUCT_API_BASE_URL"), os.Getenv("PRODUCT_API_KEY"),
	)
	if err != nil {
		panic(err)
	}
	workspace, err := client.Workspace(os.Getenv("WORKSPACE_ID"))
	if err != nil {
		panic(err)
	}
	models := workspace.Models()

	page, err := models.ListWorkspaceLLMBackends(ctx)
	if err != nil {
		panic(err)
	}
	fmt.Println("Total backends:", page.GetTotal())
	for _, backend := range page.GetBackends() {
		fmt.Println(backend.GetId(), backend.GetName(), backend.GetModels())
	}
}

2. Read backend endpoints

Select a target backend ID from the returned list and query its endpoint addresses and status values.

if not page.backends:
    raise SystemExit("no LLM backends configured in this workspace")

backend_id = page.backends[0].id
for endpoint in models.list_workspace_llm_endpoints(backend_id):
    print("Endpoint:", endpoint.id, endpoint.address, "Status:", endpoint.status)
if len(page.GetBackends()) == 0 {
	fmt.Println("no LLM backends configured in this workspace")
	return
}

backendID := page.GetBackends()[0].GetId()
endpoints, err := models.ListWorkspaceLLMEndpoints(ctx, backendID)
if err != nil {
	panic(err)
}
for _, endpoint := range endpoints {
	fmt.Println("Endpoint:", endpoint.GetId(), endpoint.GetAddress(), "Status:", endpoint.GetStatus())
}

Limitations and notes

  • Endpoint status values: Endpoint status is encoded as an integer where 0 means online (the SDK also defaults to 0 when the server omits the field) and 1 means offline. Having a backend record or a 0 status indicates only that the endpoint is enabled in configuration; actual reachability and vision or reasoning capabilities still require live invocation checks.

  • Retaining offline endpoint IDs: LLM backend details and LLM endpoint queries return online endpoints only, so an empty list can also mean that all endpoints are currently offline. Save the endpoint ID returned at creation before setting an LLM endpoint offline so you can bring it back online later. Embedding model endpoint queries return both online and offline endpoints.

  • Scope differences across model lists: The workspace model service provides both a list of model names declared on workspace backends and a list of routable models permitted by the current access scope (which can include system models). Because the two lists serve different purposes, their model sets do not necessarily match.

Next steps

Last updated on