Xray API Node Health Checks And Automated Failover

Automated failover in Xray is not the same as randomly switching to another server when a request fails. A reliable workflow needs three separate capabilities: Xray must expose a restricted API, the core must measure or report outbound health, and an external controller must decide when a node should be removed or restored. Without those boundaries, a script may mistake one temporary timeout for a dead node, remove every available outbound, or create duplicate entries after a restart.

This guide uses Xray-core’s gRPC API together with the observatory service. The observatory performs periodic probes against selected outbounds, while the HandlerService API can add or remove outbound instances at runtime. The examples target a headless Linux server, use a local API listener on port 10085, and keep the original node definitions in a controlled JSON file so that recovered nodes can be restored without reconstructing their credentials by hand.

Separate observation, selection, and failover decisions

A node can be slow without being unusable, and a probe can fail even when normal user traffic would succeed. Packet loss, a temporary DNS problem, a congested route, or an overloaded probe URL can all produce misleading results. For that reason, the controller should not remove an outbound after one failed request. A practical starting policy is three consecutive failed probes before quarantine, followed by five consecutive successful probes before restoration.

The observatory reports measurements for outbounds selected by subjectSelector. It is useful for comparing response latency and detecting repeated failures, but it does not automatically make application traffic use the fastest outbound. Routing, a balancer, or a separate controller still determines which node carries traffic. Treat the observatory as the measurement layer and the HandlerService as the configuration layer.

Use stable, unique tags such as node-sg-01 and node-de-01. Do not use display names that change during subscription updates. The tag is the identifier used by routing rules, observatory selection, logs, and API commands. If a subscription provider replaces a node while retaining a similar label, the controller should still treat the new configuration as a distinct instance unless you deliberately map it to the same tag.

Configure the Xray API and observatory services

The API requires an api inbound whose protocol is dokodemo-door, with the address set to 127.0.0.1 and the port set to 10085. The inbound is not intended to carry user traffic. Its purpose is to provide a local gRPC control endpoint. The api section lists the services that the core will expose; for this workflow, HandlerService, LoggerService, and ObservatoryService are the important entries.

{
  "api": {
    "tag": "api",
    "services": [
      "HandlerService",
      "LoggerService",
      "ObservatoryService"
    ]
  },
  "inbounds": [
    {
      "tag": "api",
      "listen": "127.0.0.1",
      "port": 10085,
      "protocol": "dokodemo-door",
      "settings": {
        "address": "127.0.0.1"
      }
    }
  ],
  "observatory": {
    "subjectSelector": [
      "node-"
    ],
    "probeURL": "https://www.gstatic.com/generate_204",
    "probeInterval": "10s",
    "enableConcurrency": true
  }
}

The exact observatory fields available depend on the installed Xray-core version, so validate the configuration with the same binary that systemd will launch. A configuration accepted by a newer local test binary may not be accepted by an older production service. Run the core’s configuration check before restarting, and inspect the first error line rather than assuming that a silent command means the service is healthy.

The selector value is a regular-expression-style prefix match in common Xray configurations. If every managed outbound begins with node-, the observatory can monitor them without listing every tag individually. Keep direct, blocked, API, DNS, and balancer helper outbounds outside this naming convention. Otherwise, the controller may interpret a direct route or an internal helper as a server node.

1. Back up the file

   Copy /etc/xray/config.json to a root-readable backup and save the exact Xray version used by the service.

2. Add the API inbound

   Insert the local dokodemo-door inbound on 127.0.0.1:10085 with the api tag.

3. Enable services

   Add HandlerService, LoggerService, and ObservatoryService to the api.services array.

4. Define probe scope

   Use a dedicated tag prefix such as node-, set a 10-second interval, and choose a stable HTTPS endpoint.

5. Validate and restart

   Run the core’s configuration test, restart the systemd unit, then check journalctl -u xray for API and observatory startup messages.

After the restart, verify that the API is listening only on the loopback interface:

sudo ss -lntp | grep 10085
sudo systemctl restart xray
sudo systemctl --no-pager --full status xray
sudo journalctl -u xray -n 80 --no-pager

The expected listening address is 127.0.0.1:10085, not a wildcard address. If another process already owns the port, choose a different local port and update every controller command consistently. A port conflict is a startup problem, not a node-health problem.

Design node state before writing the controller

The controller needs durable state. At minimum, store the node tag, its original outbound definition, current status, consecutive failures, consecutive successes, the last observed latency, and the time of the last transition. Keep this file separate from the live Xray configuration. The live configuration is the core’s startup source; the state file is the controller’s record of what it has temporarily quarantined.

Use separate counters for failure and recovery. When a probe fails, increment the failure counter and reset the success counter. When it succeeds, do the opposite. A node should move from healthy to suspect after the first failure, but it should not be removed until the failure threshold is reached. After removal, continue probing the saved definition if your test method supports it, or use an independent periodic re-add-and-test cycle.

Also add a cooldown period. For example, after restoring a node, do not quarantine it again for 60 seconds unless the API reports a definitive configuration error. Cooldown prevents oscillation when a node alternates between just-under-timeout and just-over-timeout latency. A healthy failover system values stability over the fastest possible reaction.

Implement API-driven removal and recovery safely

Xray provides command-line wrappers for several API operations. On installations that include the standard Xray binary, commands commonly used for runtime outbound management are ado for adding an outbound and rmo for removing one. The exact command syntax should be checked with xray api --help and the installed binary’s documentation. Do not copy a command designed for a different core build without testing its request format.

sudo xray api --help
sudo xray api ado --help
sudo xray api rmo --help

# Example shape; verify arguments for your installed build
sudo xray api rmo --server=127.0.0.1:10085 node-sg-01
sudo xray api ado --server=127.0.0.1:10085 /etc/xray/nodes/node-sg-01.json

The saved outbound file should contain one complete outbound instance, including its protocol, settings, stream settings, and tag. For a VLESS node, that normally means the server address and port, UUID, encryption value, transport, TLS or REALITY settings, SNI, fingerprint, and flow where applicable. For a VMess node, preserve the alter ID only if the server configuration actually requires it; do not add obsolete values simply because an old example contains them.

Conceptually, the controller loop is simple. It reads the latest observatory result, maps each result to a stable tag, updates counters, and performs an API action only on a state transition. The following pseudocode shows the important guardrails without pretending that every Xray build exposes identical JSON output:

for node in managed_nodes:
    result = observatory_result(node.tag)

    if result.failed:
        node.failures += 1
        node.successes = 0

        if node.status == "healthy":
            node.status = "suspect"

        if node.failures >= 3 and node.status != "quarantined":
            if count_live_nodes() > 2:
                save_state_atomically(node)
                api_remove_outbound(node.tag)
                node.status = "quarantined"
                write_event("removed", node.tag, result.error)

    else:
        node.successes += 1
        node.failures = 0

        if node.status == "quarantined" and node.successes >= 5:
            if cooldown_expired(node):
                api_add_outbound(node.saved_definition)
                node.status = "healthy"
                write_event("restored", node.tag, result.latency)

Make state writes atomic. Write a temporary file in the same directory, flush it, apply restrictive permissions, and rename it over the old state file. A process termination during a direct overwrite can leave an empty JSON file, causing the next controller run to forget which nodes were quarantined. Keep credentials readable only by root or by a dedicated service account that is not allowed to edit the controller itself.

If your build does not provide convenient observatory output, use an external probe through a dedicated local SOCKS inbound, but make sure the test traffic is routed through the node being tested. A request sent through the default balancer only proves that some node works; it does not identify which node carried the request. The probe must select a specific outbound by tag, or the result cannot safely trigger removal.

Route around quarantined nodes without breaking traffic

Removing an outbound is only half of failover. Existing routing rules must have a valid replacement. A common design uses a balancer whose selector includes the managed node tags, while direct, block, DNS, and API traffic use explicit outbounds. If the controller removes one node from the handler set, the balancer can continue using the remaining eligible nodes. This is safer than changing every routing rule whenever one server becomes unhealthy.

Do not put the API inbound or the API outbound into the same balancer used for ordinary user traffic. The API path must remain deterministic and local. Likewise, do not use a broad selector such as .* if it can match a direct outbound, because the balancer may select a route that bypasses the intended proxy policy.

Test rollback and operate the service headlessly

Test the workflow with a disposable node or a temporary firewall rule, not during a production outage. First confirm that the node appears in the observatory result and that normal traffic uses the balancer. Next simulate a failure long enough to cross the three-probe threshold. Verify that the controller logs the transition, removes only the intended tag, keeps the minimum live-node count, and leaves direct traffic and API access working. Finally restore connectivity and confirm that the node is added only after the recovery threshold and cooldown have been satisfied.

Run the controller as a dedicated systemd service. Give it read access to the saved node definitions, write access only to its state and log directories, and permission to invoke the narrowly required Xray API command. Avoid granting unrestricted root shell access if a more limited service arrangement is possible. The controller should exit when it cannot parse state, cannot authenticate its local execution environment, or receives an unexpected API response; silently continuing with an empty node list is dangerous.

[Unit]
Description=Xray node failover controller
After=xray.service
Requires=xray.service

[Service]
Type=simple
ExecStart=/usr/local/sbin/xray-node-controller
Restart=on-failure
RestartSec=15s
User=xray-controller
Group=xray-controller
NoNewPrivileges=true
PrivateTmp=true

[Install]
WantedBy=multi-user.target

Use journal fields that make incident review possible: timestamp, node tag, previous state, new state, failure count, latency or error, API operation, and result. A useful event looks like node=node-sg-01 transition=healthy->quarantined failures=3 reason=probe_timeout api=success. Avoid logging UUIDs, private keys, full subscription URLs, or complete outbound settings. Health automation should provide evidence without creating a second credential leak.

Before enabling automatic restoration, rehearse a manual rollback. Keep the original configuration backup, a copy of every saved outbound definition, and a command that stops the controller. If the controller behaves unexpectedly, stop it first, restore the known-good startup configuration, restart Xray, and inspect the logs before re-enabling automation. The most dependable failover system is one that can be disabled without losing the node inventory or the route configuration.