Update and removal problems

This page covers the update script and the removal of Guardioz from a server. Entries that follow a change on your server say what changed, so you know where you stand before you run anything again. The procedures themselves are on Update the agent and Remove Guardioz from a server.

The examples use the port 5432. Replace 5432 with the port of your PostgreSQL instance. Messages that the agent writes to its log are quoted without the date, the time, the level and [guardioz-agent] that stand in front of every one of them.

No snapshot arrives in the hour after an update

What you see

The update ended with Update complete., and Last seen in Servers, the time since the last snapshot arrived, keeps growing. The agent’s log carries these two messages and nothing after them:

Guardioz Agent starting (interval=3600s, dsn=user=guardioz_monitor dbname=postgres host=/var/run/postgresql port=5432 )
Waiting 3600s before first collection (registration already sent a snapshot).

Why it happens

After every start the agent waits one full interval before it collects, one hour as the installer sets it. An update restarts the services, so the next snapshot arrives about one hour after the restart. The message names the registration because the agent writes the same line at every start.

Check

Read the last lines of the agent’s log. See Check that the agent works.

Fix

Nothing is broken. To get a snapshot now instead of waiting, run one collection by hand. See Manage the agent service.

After the fix

The log reports Snapshot sent (HTTP 202).

Error: the URL given does not match the agent’s configured api_url

What you see

Error: the URL given (ADDRESS_YOU_PASSED) does not match the agent's configured api_url (CONFIGURED_ADDRESS).
Re-run with that URL, or pass --allow-url-change to repoint the agent on purpose.
The agent on this server was not changed and no service was restarted.

Why it happens

The updater compares the address you passed with the address in the agent’s configuration file, and stops when the two differ. It makes the same comparison for a rollback, before it moves anything. A trailing slash is ignored on both sides, so the difference is somewhere else: http instead of https, another host name, or an extra path. The message shows the address you passed without its trailing slash.

Check

On that server, with administrator rights, read the configured address:

sudo grep '^api_url' /etc/guardioz-agent/config-5432.yaml

Fix

Run the update again with exactly that address. --allow-url-change permits the download from another address and nothing more. It does not write the configuration file, and the agent keeps sending its snapshots to the address in that file. With the flag, the updater prints a line beginning with Note: the URL, and then runs the update. After an update that ran with the flag, the rollback command on the last line of the output carries the flag too.

After the fix

The updater prints Downloading the agent from ADDRESS... and goes on.

A service did not start again after the update

What you see

A line that begins with an exclamation mark, and one more line after the restarts:

  ! guardioz-agent-5432 failed to restart - continuing with the others.
  (some instances need attention - see above)

A second kind of line reports that a service restarted and does not run under the account that it has to run under. The script prints Update complete. and ends with the same exit code whether a service started again or not, so read those lines and not the exit code.

Why it happens

The updater restarts the service of every PostgreSQL instance on the host. When one does not start again, the updater reports that instance and goes on to the next, so one failure does not leave the others on the old code.

Check

On that server, with administrator rights, look at the state of that service:

sudo systemctl status guardioz-agent-5432 --no-pager

Then read its last log lines:

sudo journalctl -u guardioz-agent-5432 -n 50 --no-pager

Fix

The log names the reason. For the messages in the log, see Agent and data problems.

When the new agent code is the reason, go back to the version that the update kept. The last line of the update output is the command that does it. When that output is no longer on your screen, list the kept versions in a shell on that server:

ls -d /opt/guardioz-agent/guardioz_agent.*

Update the agent explains what a rollback keeps.

After the fix

systemctl status reports active (running), and the log carries the starting message of the agent.

Usage: update-agent.sh

What you see

Usage: update-agent.sh <cockpit-url> [--rollback <timestamp>] [--allow-url-change]

When the script found an argument that it cannot read, a line that starts with Error: comes before the usage line, and one more line follows it:

Error: unexpected argument: ARGUMENT
Usage: update-agent.sh <cockpit-url> [--rollback <timestamp>] [--allow-url-change]
The agent on this server was not changed and no service was restarted.

Why it happens

The script needs the Guardioz address and stops without it. The message calls that address cockpit-url; cockpit is the Guardioz web app. In the piped form, the arguments of the script come after -- , and a command that ends before them leaves the script with none.

The script reads its arguments in any order. It stops at an argument that it cannot read instead of guessing what the argument means, and the Error: line names the reason:

• Error: unexpected argument: ARGUMENT: the script has no option of that name, or the command has a second argument that is not an option, for example a timestamp without --rollback in front of it. The script takes one address, and ARGUMENT is the later of the two.
• Error: --rollback needs the timestamp of a kept version after it.: --rollback is the last argument, or the argument directly after it starts with -.
• Error: --rollback was given more than once.

Check

Compare your command with the one on Update the agent.

Fix

Run the command again with the address after --, and with the timestamp directly after --rollback. Nothing on the server was changed.

Error: no kept version

What you see

Error: no kept version at /opt/guardioz-agent/guardioz_agent.TIMESTAMP
Available:
  20260918T141206Z

Under Available: the script lists the timestamps of the versions that are kept, oldest first, or prints (none).

Why it happens

The timestamp after --rollback names a directory that is not on the server. A rollback consumes the version that it restores, so a timestamp works once. A server that was never updated keeps no version at all.

Check

The list under Available: is the list of kept versions. To see the same versions as directories, list them on that server:

ls -d /opt/guardioz-agent/guardioz_agent.*

Fix

Run the command again with one of the listed timestamps after --rollback. A timestamp is the moment that version was set aside, so the newest one, at the end of the list, is the version that ran before the last update.

After the fix

The script ends with Rolled back to TIMESTAMP. and starts the service again. As after every start, the agent waits one full interval before it collects, so the next snapshot arrives about one hour after the restart. To get one now, run one collection by hand. See Manage the agent service.

Error: the downloaded agent does not import

What you see

Error: the downloaded agent does not import - nothing was changed.

Why it happens

The updater confirms that the downloaded code loads before it replaces anything on disk. The download arrived incomplete or damaged.

Check

The agent keeps running on the code it had, and the download went into a temporary directory that the script removes when it ends. On Linux, with administrator rights, confirm that the service still runs:

sudo systemctl status guardioz-agent-5432 --no-pager

Fix

Run the update again. When the same message comes a second time, write to contact@guardioz.com and include the whole output of the run. For what to send, see Get help.

The download stops with a message from curl, fetch or tar

What you see

After Downloading the agent from ADDRESS... the run stops with a message from your download tool or from tar, and Update complete. never appears. The updater has no message of its own for this.

Why it happens

The updater downloads the agent code from Guardioz. The download fails when the server has no outbound HTTPS connection to that address, when a name cannot be resolved, or when too many requests came from this network address in a short time. The download goes into a temporary directory, so nothing on the server was replaced.

Guardioz answers a refused request with this body:

{"detail":"Too many requests from this network address. Wait a minute and try again."}

Your terminal shows the error of the download tool, for example curl: (22) The requested URL returned error: 429, and then the message from tar. The check shows the answer code.

Check

On that server, ask for the same file and show the answer code:

curl -fsS -o /dev/null -w '%{http_code}\n' https://app.guardioz.com/api/v1/setup/agent-source

200 means the server can reach Guardioz. On a server without curl, for example a FreeBSD host, ask for the same file with fetch:

fetch -qo /dev/null https://app.guardioz.com/api/v1/setup/agent-source && echo reachable

Fix

Allow outbound HTTPS from that server to https://app.guardioz.com/, or wait a minute when several servers were updated from one network address at the same time. Then run the update again.

The updater stops and reports that it needs administrator rights

What you see

One of these messages:

Error: this run needs administrator rights, this account is not root, and sudo
was not found on this system.
Run this command as root, or ask an administrator of this server to run it.

Error: this run needs administrator rights, and sudo did not grant them.
Run this command as root, or ask an administrator of this server to run it.

Error: sudo refused a check that runs without asking for a password.
This run has no terminal for sudo to ask on, so Guardioz stops here instead of
failing part-way through.
Run this command as root, or from a terminal where sudo can ask.

Each of them ends with:

The agent on this server was not changed and no service was restarted.
If you need help, write to contact@guardioz.com and tell us which system this is.

Why it happens

The updater replaces files under /opt/guardioz-agent and restarts services, so it needs administrator rights, and it settles them before it changes anything. The first message means that your account is not root and the server has no sudo. The second means that sudo refused. The third means that the run has no terminal for sudo to ask for a password on. That is what happens in automation, and in a command started as ssh HOST 'COMMAND'.

When sudo needs a password for your account, and the session has a terminal, the updater first prints the same three lines as the installer:

This run needs administrator rights to set up monitoring on this server.
sudo may ask you to authenticate. Nothing has been installed or changed by
Guardioz yet.

The installer and the updater share those words. Read them as a statement about this run. The agent that is already on this server keeps its files, its configuration and its collector key, and no service was restarted.

Check

In a shell on that server, show the name of your account:

id -un

Then check whether sudo runs a command for this account without asking for a password. This is the first check the updater makes:

sudo -n true

The command prints nothing when sudo runs commands for this account without a password. When it prints sudo: a password is required, confirm that the account can use sudo with its password:

sudo -v

The command asks for your password and prints nothing more when the account can use sudo. It prints a refusal when the account cannot.

Fix

Run the update as root, or in an interactive session from an account that can use sudo.

Error: no Python interpreter found to verify the agent with.

What you see

Error: no Python interpreter found to verify the agent with.
The agent's own interpreter is named in /usr/local/etc/rc.d/guardioz_agent_* on
FreeBSD and in /opt/guardioz-agent/venv on Linux; neither was readable here.

Why it happens

Before it replaces anything, the updater loads the downloaded code once, with the Python that the agent runs on. On Linux that is the Python in /opt/guardioz-agent/venv. On FreeBSD the updater reads it from the service script of each instance, /usr/local/etc/rc.d/guardioz_agent_5432 for the port 5432, which names the Python that the service starts the agent with, and it takes the first one that is installed. When it finds none of these, it looks for python3.

The message means that the updater found none of them. On a FreeBSD server with the agent, that happens when no service script names a Python that is still installed and the server has no python3, for example after the Python version that the agent was installed with was removed. The service starts the agent with that same Python, so a restart of the service cannot start the agent either. The message comes before any change, so the agent on this server was not touched.

Check

On that FreeBSD server, read the line of the service script that starts the agent:

grep command_args /usr/local/etc/rc.d/guardioz_agent_5432

The path directly before -m guardioz_agent.cli is the Python that the service uses. Check that the file is there, with that path in place of PYTHON:

ls -l PYTHON

No such file or directory means that this Python is no longer installed.

Fix

Write to contact@guardioz.com. Put the output of both commands above in your message: it names the Python that this server’s agent runs on, and shows whether it is installed. What else to include: Get help.

The agent reports the transaction-log disk as not measured

What you see

The agent’s log carries this message, and an update does not change it:

pg_wal could not be inspected through PATH (REASON); its path and disk figures are recorded as not measured

The installer reports the same state while it runs. On Linux:

    Could not set the ACL on DIRECTORY - stopping the data-directory traversal grant here.
    The WAL-directory disk figures will be recorded as not measured.

or, when setfacl was missing and could not be installed:

  Could not install setfacl (the acl package) - skipping the data-directory
  traversal grant. The WAL-directory disk figures will be recorded as not measured.

On FreeBSD:

    could not set an ACL on DIRECTORY; the WAL disk figures read as not measured.

PATH is the transaction-log directory the agent looked for, REASON is what the operating system answered, and DIRECTORY is the directory that refused the change.

Why it happens

The agent needs permission to pass through the PostgreSQL data directory to measure the disk that holds the transaction log (WAL). The installer grants that permission; an update neither grants nor repairs it. When the permission is gone, for example because the data directory was created again or PostgreSQL was moved to a new one, the figures stay not measured.

Not measured is a result, not an error. Guardioz records it as such and does not put an estimate in its place. The parts of a check that need those figures do not run. The rest of the collection, and the other checks, are not affected.

Check

The installer records each permission it granted. On that server, with administrator rights:

sudo grep -A1 'traverse ACL' /etc/guardioz-agent/install-manifest.txt

Each entry names one directory and the command that removes the permission again. No entry means that this installation granted none, for example because you installed without the grant.

Fix

A run of the installer gives the monitoring role a new password, and the agent a new collector key. It keeps the server’s history only under one condition: the configuration file on that host still points to the same Guardioz address, and still holds a working key for a server of your organization. A command that is expired or already used still changes that password before Guardioz refuses it. The agent then stops collecting until you run a fresh command to the end.

To have the permission granted again, take a fresh install command in Guardioz and run it to the end on that server. What that run changes: What the installer changes on your server. Why it is not an update: Update the agent.

Something is still on the server after the removal

What you see

After the removal steps, a service is still listed, files are still in place, or the monitoring role still exists in PostgreSQL.

Why it happens

There is no uninstall command. Each step undoes one change, and the record of those changes is the install manifest. Three things decide what is left on the server:

• The manifest lives in /etc/guardioz-agent, and the removal deletes that directory. Copy the file somewhere else before you delete the directory. When your copy is gone, Remove Guardioz from a server still covers the services, the agent files, the pg_hba.conf lines and the monitoring role. Two steps need the lines from the manifest: the access entries on the data directory, and the system account. Write to contact@guardioz.com for those.
• The recorded undo line for pg_hba.conf removes the rule line and leaves the comment line # Guardioz monitoring (read-only, local socket only) in place. Open the pg_hba.conf file as root or as the account that owns the cluster, and delete that comment line. PostgreSQL ignores comment lines, so it does not have to read the file again.
• The manifest does not record everything. On FreeBSD the packages that the installer added with pkg are not in it; on both systems a new password given to a monitoring role that already existed is not in it. What the installer changes on your server lists what each manifest holds.

Check

On that server, list the agent services that are still registered. On Linux:

systemctl list-unit-files 'guardioz-agent-*'

On FreeBSD, list the service scripts, also those that are no longer enabled:

ls /usr/local/etc/rc.d/guardioz_agent_*

Then look at /etc/guardioz-agent and /opt/guardioz-agent.

Fix

Work through Remove Guardioz from a server again with your copy of the manifest. Two details decide whether the lines from the manifest work:

• Run the psql lines from the manifest as the operating-system account that owns the cluster. The header of the manifest says so, and the lines themselves do not: pasted from another account they fail.
• Remove the access entries on directories before the system account. Each entry names that account, so after the account is deleted the entries no longer show a name you can match, and you have to identify them by their numeric user id.

A role, an account or a pg_hba.conf rule that was on the server before the installation has no entry in the manifest, because the installer did not create it. A new password that the installer gave to a monitoring role of that name cannot be reversed.

The agent keeps running after you deleted the server in Guardioz

What you see

The agent’s log reports this once per interval, and the agent keeps running:

Authentication failed (HTTP 401). Check collector_key.

Why it happens

Deleting a server in Guardioz deletes its collector key with it. The agent on the host is not part of that deletion: it keeps running, keeps reading PostgreSQL every interval and keeps sending, and Guardioz refuses every snapshot. The agent does not stop by itself and does not register again. Deleting your account deletes every server of your organization, so the same happens on each of their hosts.

Check

Read the last lines of the agent’s log. See Check that the agent works.

Fix

Remove the agent from that server. See Remove Guardioz from a server.

When you did not want to delete the server, take a fresh install command in Guardioz and run it to the end on that host. It registers a new server, and the history of the deleted one is gone.

The server is still in Guardioz after you removed the agent

What you see

The server is still listed under Servers with its findings. Last seen grows, and the dot in the Status column turns red. On Findings, with that server picked in the server filter, Latest snapshot: reports the age of the last snapshot that arrived from it.

Why it happens

Removing the agent from a server changes nothing in Guardioz. The server, its snapshots and its findings stay until you delete them.

Check

In Guardioz, open Servers and look at Last seen in the row of that server.

Fix

Deleting a server removes it together with its snapshots, its findings and its collector key, and Guardioz has no undo for that. The full list of what a deletion removes: Manage your servers.

When you want that, open Servers and click Delete in the row of that server.

If your problem is not on this page

• The update finished and the agent reports errors, or no data arrives: Agent and data problems.
• The install command stopped, or the setup page never reported success: Installation problems.
• Nothing matches what you see: write to contact@guardioz.com. For what to include, see Get help.