Agent and data problems

This page covers what can go wrong after the installation finished. A server sends no data, a status looks wrong, the agent log reports an error, or a value reads not measured. Each entry names what you see, why it happens, what to look at, and what to do.

Most answers are in the agent log. To read the log, and to run a collection now, see Manage the agent service. The examples use port 5432. Replace 5432 with the port of your PostgreSQL instance.

Guardioz has no data after the installation

What you see: the Findings page shows the heading All clear together with the line No snapshots received yet. The Servers list shows Never in the Last seen column and a grey dot.

Why it happens: All clear here does not mean that your server is healthy. It means that no snapshot has arrived, so nothing has been checked yet. The installation sends the first snapshot itself. When that send fails, the installation still reports the server as registered. The service then waits one full interval, about an hour, before it collects on its own.

Check: the result of that first snapshot is in the output of the installation, not in the service log. When that output is still on your screen, look for this line:

The initial snapshot did not reach the cockpit. The server is registered; the agent will collect on its schedule, or run it once to retry now.

The word cockpit in that message means the Guardioz web app. The service log carries Snapshot sent (HTTP 202) only after its own first collection, about an hour after the installation.

Fix: run a collection now, as described in Manage the agent service, and look in its output for this line:

Snapshot sent (HTTP 202)

When that line is there, the data is on its way. The run closes with Guardioz Agent stopped. whether the snapshot was sent or not, so that last line proves nothing. When another message is there instead, continue with the entry for that message on this page. Guardioz takes at most one snapshot of a server every 10 minutes, so a run that shows Cockpit refused the snapshot (HTTP 429): came too soon after the last snapshot: run it again 10 minutes later.

After the fix: the Findings page shows Latest snapshot: with an age, and the first findings appear within a few minutes.

No new snapshot after a restart, a reboot or an update

What you see: the service runs, the log reports no error, and Last seen in Guardioz does not change for about an hour. The last message in the log is the following:

Waiting 3600s before first collection (registration already sent a snapshot).

Why it happens: the agent waits one full interval before its first collection after every start of the service. As the installer sets it up, the interval is one hour. A restart, a reboot of the server and an update of the agent each start the service again, so each postpones the next snapshot.

Check: the message in the log. As long as it is the last message, the service is waiting and nothing is wrong.

Fix: wait, or run a collection now to send one snapshot at once, as described in Manage the agent service. For what each collection carries, see When the agent collects.

The status dot is yellow or red

What you see: the dot in the Status column of the Servers list is yellow or red.

Why it happens: the dot shows how recent the newest snapshot is, and nothing else. The agent sends one snapshot per hour. The dot turns yellow when no snapshot has arrived for 90 minutes, and red after 150 minutes, when two hourly snapshots in a row did not arrive. A restart, a reboot or an update postpones the next snapshot by one interval, so the dot can turn yellow for a while after one of them. Guardioz sends no email when the dot changes.

Check: read Last seen in the same row, then open the server page. Its notice says when the last snapshot arrived, and Data collection, last 30 days shows the gaps and the PostgreSQL starts around them: see When snapshots stop arriving. Then read the agent log on that server. See Check that the agent works.

Fix: continue with the entry for the message you find in the agent log. When the service runs and the log shows Snapshot sent (HTTP 202) at the last collection, the next snapshot turns the dot green again. For what else the dot does not report, see Manage your servers.

The agent log reports Authentication failed (HTTP 401). Check collector_key.

What you see: this message once per collection, and no snapshot arrives. The agent does not retry after it.

Authentication failed (HTTP 401). Check collector_key.

Why it happens: Guardioz does not accept the collector key that this agent sends. One cause is a server that was deleted in Guardioz, because the key of a server is deleted with it. The other is a collector key that Guardioz replaced while this agent kept the previous one. One way that happens: a configuration file was copied to a second machine, and the installation was run again there.

Check: in Guardioz, open Servers and look for the server. When the server is not in the list, it was deleted, and its collector key was deleted with it. When the server is in the list, this agent holds a key that Guardioz no longer accepts for it.

Caution

Guardioz no longer accepts this key, so an install command that is run again cannot connect this host to the server it sent data to before. It registers a new server, and a server that was not deleted stays in the list with its old data and no new snapshots. Such a run also gives the monitoring role a new password and writes a new collector key. Read what else it changes in Update the agent.

Fix: when the server was deleted on purpose, remove the agent from the host as well. See Remove Guardioz from a server.

Otherwise get a fresh install command in Guardioz - Install the agent shows where Guardioz gives you one - and run it to the end. Guardioz has no field for entering a key by hand. The run registers a new server, so an old entry that is still in the list receives no more data. Delete it when you no longer need it, as described in Manage your servers.

After the fix: the next collection reports Snapshot sent (HTTP 202).

The agent log reports Cockpit refused the snapshot (HTTP 422)

What you see: this line once per collection, followed by the reason that Guardioz gave. The agent does not try again, and no snapshot arrives.

Cockpit refused the snapshot (HTTP 422): REASON

Why it happens: Guardioz checks the whole snapshot before it stores any of it, and the reason names the part that it refused. When the reason contains check the clock of the monitored server, the clock of the server is wrong: Guardioz refuses a snapshot whose collection time is more than 15 minutes ahead of its own clock, or more than one day old. Otherwise the agent is older than the version that Guardioz expects. An older agent writes Server returned HTTP 422 (attempt 1/3) three times instead, and then Failed to send snapshot after 3 attempts.

Check: the reason in the line. For the clock, run date -u on the server and compare it with the time in UTC.

Fix: when the reason names the clock, set the clock of the server right, for example with NTP. Otherwise update the agent in place: see Update the agent. When the message is still there after the update, write to contact@guardioz.com and send the line.

After the fix: the next collection reports Snapshot sent (HTTP 202). The service waits one interval after the restart, so run a collection now when you do not want to wait for it.

The agent log reports Connection error

What you see: a message that begins with Connection error (attempt 1/3): and ends with the error text of the network library. Two more attempts follow, and then this line:

Failed to send snapshot after 3 attempts

Why it happens: the agent could not reach Guardioz. The agent opens the connection itself, outward, over HTTPS. A name that does not resolve, a firewall that blocks the connection, and a device that inspects TLS traffic all end here. The agent drops that snapshot and collects fresh data at the next collection. Nothing is queued on the server.

Check: on the server that runs PostgreSQL, open a connection to Guardioz. The command needs no special rights.

Linux

curl -sS -o /dev/null https://app.guardioz.com/

FreeBSD

fetch -qo /dev/null https://app.guardioz.com/

The command prints nothing when the connection works. When it does not, it prints its own error message, which names the cause.

On Linux the agent connects with its own Python libraries, which carry their own list of trusted certificate authorities. A device that inspects TLS traffic can therefore let curl through and still stop the agent. To test the connection the way the agent makes it, run this with administrator rights:

sudo /opt/guardioz-agent/venv/bin/python -c "import requests; print(requests.get('https://app.guardioz.com/', timeout=10).status_code)"

It prints 200 when the agent itself can connect.

Fix: allow this server to open outbound connections to app.guardioz.com on TCP port 443 (HTTPS), and make sure the name resolves from this server. An outbound proxy is not supported: the agent’s configuration has no proxy setting.

After the fix: run a collection now. It reports Snapshot sent (HTTP 202).

The agent log reports Cockpit refused the snapshot (HTTP 429)

What you see: this line once, and no snapshot for that collection. The agent does not try again.

Cockpit refused the snapshot (HTTP 429): A snapshot of this server arrived less than 600 seconds ago. The agent sends one snapshot per collection interval.

Why it happens: Guardioz takes at most one snapshot of a server every 10 minutes. The scheduled collections are an hour apart, so this answer follows a collection that came less than 10 minutes after another one: a collection that you started by hand, or the scheduled collection that came shortly after yours. A line that ends with Too many requests from this network address. Wait a minute and try again. means something else: too many requests came from the network address of this server, for example from several servers behind one address.

Check: whether you or a script started a collection by hand shortly before the line.

Fix: nothing needs repair. For a collection by hand, wait until 10 minutes have passed since the last snapshot, and run it again. The next scheduled collection sends again, and the hour in between stays a gap in the history. When the line about the network address comes back at every collection, write to contact@guardioz.com and tell us how many monitored servers share that address.

After the fix: the next collection reports Snapshot sent (HTTP 202).

The agent log reports Server returned HTTP 503

What you see: up to three attempts, and then the snapshot is dropped.

Server returned HTTP 503 (attempt 1/3)
Retrying in 60s...
...
Failed to send snapshot after 3 attempts

Why it happens: Guardioz did not take the snapshot at that moment, for example because it was busy with other snapshots or still analysing earlier ones. The agent waits as long as Guardioz asks, at most 60 seconds, and tries 3 times. Another code from 500 to 599 is written the same way.

Check: whether the line comes back at the next collection.

Fix: nothing needs repair on your server. The next collection sends again, and the hour in between stays a gap in the history. When the line comes back at several collections in a row, write to contact@guardioz.com.

After the fix: the next collection reports Snapshot sent (HTTP 202).

The agent log reports Error during collection cycle

What you see: this message, followed by several lines of Python output. No snapshot was sent for that hour.

Error during collection cycle

In Guardioz, the server page says what the agent reported about that collection, for example The agent reported that it could not connect to PostgreSQL (PostgreSQL refused the password of the monitoring role). The words in the brackets match the causes below: nothing accepted the connection at the address the agent uses when PostgreSQL was not running, PostgreSQL reported that the monitoring role or its database does not exist when the role was removed, PostgreSQL refused the password of the monitoring role when the password no longer matches, and no rule in pg_hba.conf lets the monitoring role connect for the pg_hba.conf file. Without brackets, the agent did not recognise the message; the log line names it.

Why it happens: the collection failed as a whole, before anything could be sent. The agent connects to PostgreSQL on the same machine, over the local socket, as the monitoring role guardioz_monitor. The usual causes are these four:

• PostgreSQL was not running at that minute.
• The monitoring role was removed.
• Its password no longer matches.
• A rule in the pg_hba.conf file no longer accepts the role. That file is where PostgreSQL decides which roles can connect, and from where.

The service keeps running and tries again at the next collection.

Check: on the server that runs PostgreSQL, ask PostgreSQL about the monitoring role. The command runs as the system account that owns the PostgreSQL cluster:

sudo -u postgres psql -p 5432 -c "SELECT rolname, rolcanlogin, pg_has_role(rolname, 'pg_monitor', 'member') AS has_pg_monitor FROM pg_roles WHERE rolname = 'guardioz_monitor';"

The output is similar to the following:

     rolname      | rolcanlogin | has_pg_monitor
------------------+-------------+----------------
 guardioz_monitor | t           | t
(1 row)

t in both columns means that the role can sign in and that it holds the read-only pg_monitor membership that it needs. No row at all means that the role does not exist.

Fix: the output decides what to do.

With no row, the role was removed. Get a fresh install command in Guardioz - Install the agent shows where Guardioz gives you one - and run it to the end. The installer creates the role, gives it a new password and writes the agent’s configuration again. A run that stops before the end still changes the password of the monitoring role, so generate the command right before you use it. Read what else such a run changes in Update the agent.

With f in the has_pg_monitor column, grant the membership in an SQL session as a superuser:

GRANT pg_monitor TO guardioz_monitor;

With t in both columns, the role itself is in order, so the cause is the password or the pg_hba.conf file. Compare the time of the log line with the times PostgreSQL was running. When PostgreSQL was stopped at that minute, there is nothing to repair, and the next collection sends again. Otherwise get a fresh install command and run it to the end: it sets a new password on both sides, and it adds the rule the role needs when the role still cannot connect over the local socket. A run that stops before the end leaves the role with a password the agent does not have.

After the fix: run a collection now. It reports Snapshot sent (HTTP 202).

The service stops right after it starts

What you see: on Linux the service is not running. On FreeBSD the service keeps running, starts the agent again about once a second, and the log repeats the message. The last line in the log is one of these three, or several lines of Python output:

api_url is not configured

collector_key is not configured

interval is 60 s; the agent runs with at least 900 s. Set interval in /etc/guardioz-agent/config-5432.yaml (or GUARDIOZ_INTERVAL) to 900 or more.

The third one carries the interval from your file in place of 60.

Why it happens: the agent reads its configuration file once, at start. It stops when the file that the service names does not exist, cannot be read, is not valid YAML, carries no value for one of the two required keys, or sets an interval below 900 seconds. On Linux, systemd starts the agent again a few times and then stops trying. On FreeBSD, daemon(8) keeps starting it again.

Check: on the server that runs PostgreSQL, as root, open the configuration file of the instance on port 5432, /etc/guardioz-agent/config-5432.yaml. Only root and the account that runs the service can read it, so an ordinary account gets a permission error. Confirm that the file is valid YAML, that api_url and collector_key each carry a value, and that interval is 900 or more (the installer writes 3600). The file also holds the collector key and the password of the monitoring role, so do not paste it into a message. For every key in it, see Agent configuration.

Fix: repair the configuration file first. Then start the service again, on the server that runs PostgreSQL.

Linux

As an administrator, clear the failed state that systemd set when it stopped trying:

sudo systemctl reset-failed guardioz-agent-5432

Then start the service:

sudo systemctl start guardioz-agent-5432

FreeBSD

As root, restart the service, so that the agent reads the repaired file:

service guardioz_agent_5432 restart

After the fix: the log opens with a line that begins with Guardioz Agent starting, and then reports the wait before the first collection.

The agent log reports a schedule file that cannot be read

What you see: once every hour, a message that begins with Cadence state file. Cadence is the agent’s word for its collection schedule. The message names a file under /var/lib/guardioz-agent, names the error, and ends with this sentence:

The agent account has to own its state directory; until it does, the daily and weekly collections are suspended so they do not repeat on every cycle.

A message that begins with Failed to write state file says the same thing about writing.

Why it happens: the agent keeps a small file with the time of its last daily and its last weekly collection. When the running service cannot read or write that file, it suspends those two collections instead of repeating them every hour on your database. The hourly statistics keep arriving, so the server looks healthy while the results that need the catalog and the size measurements grow old.

Check: on the Findings page, the Age of the groups about indexes keeps growing while other groups stay recent. For what Age means, see Read your findings. The message in the log names the file it cannot use.

Fix: update the agent in place, and then run one collection as root. A collection that runs as root writes the schedule file again and hands it to the owner of its directory, so the service can read it. The file is written only when that collection reaches Guardioz, so confirm that the run reported Snapshot sent (HTTP 202) before you look for the message again. See Update the agent and Manage the agent service. When the message comes back at the next hour, write to contact@guardioz.com.

After the fix: the message is gone, and the log reports a line that begins with Collecting relation catalog for when the daily part runs again.

A value on a finding page reads Not measured

What you see: Not measured yet, Not measured, Not recorded, Not reported, Size not recorded or - in place of a number.

Why it happens: Guardioz shows a measurement it does not have as missing and never fills the gap with an estimate. The cause differs by value.

• Table and index sizes are measured once a week, so an object that appeared since the last weekly collection has no size yet.
• A measurement can fail, for example when the agent cannot reach the transaction-log directory inside the PostgreSQL data directory.
• PostgreSQL’s own statistics or catalog hold no value for that object, for example after the statistics were reset. That reads as Not recorded or Not reported, and nothing on your side has to be repaired.

The finding itself still stands; the missing number can change how severe it is once it arrives.

Check: for the transaction-log directory, look in the agent log. The line begins with pg_wal could not be inspected through, with pg_wal is not present under, or with PGDATA, and it names the directory and the error. The record of what the installation granted is in /etc/guardioz-agent/install-manifest.txt. See What the installer changes on your server.

Fix: for a size that is not measured yet, wait for the weekly collection. To get it at once, run a collection now, as described in Manage the agent service.

For the transaction-log directory, the installation grants the access that the measurement needs. It does not grant it when the install command carried --no-pgdata-access or GUARDIOZ_NO_PGDATA_ACCESS, or when the grant did not go through on this server.

Caution

Running the install command again grants that access, and it changes more on this server. The monitoring role gets a new password, and the collector key is replaced. The server keeps its history only when the configuration file for that port still points at the same Guardioz address and still holds a key that Guardioz accepts for it. Otherwise the run adds a second server. A run that stops before the end still changes the password, and the agent cannot connect until a run finishes. Read Update the agent first.

When that is acceptable, get a fresh install command in Guardioz - Install the agent shows where Guardioz gives you one - and run it to the end. When you cannot run the installer again, write to contact@guardioz.com.

Everything else in the results stays valid, and only the parts that need the missing number stay silent. A finding whose size is not measured yet is still shown, with a note next to the object that says so. For how Guardioz shows a missing value, see Read your findings.

One dataset or one database is missing from the results

What you see: the snapshot arrives, and the log carries one of these messages:

• a warning that begins with Query and then names a dataset, or one that begins with Disk usage collection failed or Archive status listing failed;
• an error that begins with Database-local statistics could not connect to database, Relation catalog collection could not connect to database or Relation size collection could not connect to database, followed by Python output;
• an error such as 23 user databases exceed the limit of 20; monitoring the first 20 by name, 3 left out, with your numbers.

Why it happens: one dataset or one database could not be read. The agent sends the rest of the snapshot and records the missing part as not measured. Three causes are common. The monitoring role is not a member of pg_monitor. Or it cannot connect to one database, because the right to connect to that database was taken away from PUBLIC. PUBLIC is the group that stands for every role in PostgreSQL. Or the instance has more than 20 user databases: the agent collects the per-database data of the first 20 by name and leaves the others out of every snapshot. The index checks and Tables behind on cleanup then do not cover those databases; the checks that read views of the whole instance still include them. See Limits.

Check: confirm the pg_monitor membership with the command in The agent log reports Error during collection cycle on this page. Confirm the connect right per database with the following command, which runs as the system account that owns the PostgreSQL cluster:

sudo -u postgres psql -p 5432 -c "SELECT datname, has_database_privilege('guardioz_monitor', datname, 'CONNECT') AS can_connect FROM pg_database WHERE datallowconn;"

The output is one row per database. f in the can_connect column names a database the agent cannot read.

Fix: in an SQL session as a superuser, run the statement that matches your output. When has_pg_monitor was f, run:

GRANT pg_monitor TO guardioz_monitor;

For every database that showed f in the can_connect column, run:

GRANT CONNECT ON DATABASE DATABASE_NAME TO guardioz_monitor;

Replace DATABASE_NAME with the name of that database.

The limit of 20 databases has no setting. When it leaves out a database that you want checked, write to contact@guardioz.com.

After the fix: run a collection now. The warning is gone from its output, and the checks for that database report again.

If none of this matches

• The installation itself stopped, or the setup page never reported success: Installation problems.
• An update or a removal went wrong: Update and removal problems.
• Nothing here matches: write to contact@guardioz.com. For what to send, see Get help.

Next steps

Check that the agent worksConfirm that the service runs and that a snapshot reached Guardioz.

When the agent collectsThe hourly, daily and weekly parts, and when a finding follows a change.

Manage the agent serviceRead the log, restart the service, and collect one snapshot now.