Troubleshooting of Reporting

Most frequently seen Zenarmor reporting issues and their solutions are given below.

MongoDB Index Problems

Sometimes, because of improper shutdown of MongoDB services you may encounter MongoDB Index problems.

If Zenarmor health check detects a problem in MongoDB, the message below automatically appears on the Zenarmor screen.

Figure 1. MongoDB Index Problem

When you click Dismiss:

• This notification will be deleted from alerts until the next system health check (in one hour)

• If the same error occurs in the next system health check ,the alert appears on the screen again.

• No operation can be done on MongoDB if you dismiss the message again.

Elasticsearch seems to be a better alternative as the backend database. If you're using the Mongodb backend and experiencing problems, it might be wise to switch to the Elasticsearch backend. You'll need at least 8GB of RAM to be able to run ES along with Zenarmor.

When you receive a warning notification about Reporting DB Index Problems(whether MongoDB or ElasticSearch) you need to reset the database by running 'Reset Reporting'

In some cases, you need to reinstall reporting database as explained below.

How do I reinstall the Reporting Database?

In some cases, you might need to reinstall the Reporting Database. For this, you don't need to uninstall and reinstall Zenarmor; but do a configuration resetTo reinstall reporting DB:

1. Log in to the console as root
2. Delete .configdone file by running rm -f /usr/local/zenarmor/etc/.configdone command.
3. Open any Zenarmor menu on the GUI
4. The wizard will run and DB selection step will reinstall reporting DB.

Some charts are broken

This is because of broken Elasticsearch/Mongodb indices. Two reasons that we're aware of:

• Reason 1: There has been an unexpected power loss on the firewall .e.g. an electricity outage, abnormal shutdown of the firewall etc. these databases do a lot of buffering, writing the buffers to the indices from some time to time. If a partial write is in place than chances are high that your indices might get corrupt.

  Solution 1: Go to Zenarmor → Settings → Data Management → Reporting Database Settings pane . Click Perform Index Check. It'll take care of the rest for you.

• Reason 2: You have enabled Use memory file system for /var from System → Settings → Miscellaneous on OPNsense configuration:

  

  Figure 2: Disk/Memory Settings

  Solution 2: Make sure you have this setting disabled. After that, go to Zenarmor → Settings → Data Management → Reporting Database Settings pane . Click Perform Index Check. You're done.

tip

According to the reports we receive from Zenarmor users; Elasticsearch seems to be a better alternative as the backend database. If you're using Mongodb backend and experiencing problems, it might be wise to switch to the Elasticsearch backend. Please check the HW requirements to be able run ES backend along with your Zenarmor deployment.

How can I remove the DB index files?

You can set the store duration time for reporting data or just erase the reporting data manually.

I do not see DNS hostnames for some IP addresses

If the engine cannot do real-time DNS enrichment, this is generally because you're running a DNS server somewhere outside your firewall (like Pi-hole or Active Directory) so that Zenarmor is missing some/all of your DNS transactions.

If this is the case, we advise your disable "caching" on the external DNS server and set your Firewall's DNS server as a forwarder to the external DNS server. In this way, Zenarmor will have a chance to witness your DNS transactions.

For a little bit of background: Zenarmor does DNS enrichment in two ways:

• Engine doing the mapping realtime: Engine keeps track of all dns transactions that it can see flowing over itself. When it detects an IP address resolution (either an A/AAAA/CNAME or PTR), packet engine caches the IP addresses and the corresponding fully qualified domain name.

  All charts/tabular reports and live session reports display this cached hostname when you view the reports.

  Note (July 2020)

  Beginning with Sensei 1.6, engine does an active real-time reverse PTR query in case it cannot detect an immediate dns enrichment data from previous attempts(available in all subscription tiers)

• UI doing mapping during reports viewing: This applies to live session reports only: When you view a live session report, while you're browsing over records, UI runs a background job to see if a particular record has its hostname resolved. If it detects an unresolved IP address, it runs a background query to resolve the IP address via the name server you've configured on Zenarmor → Settings → DNS Enrichment.

  So, if you do not see a hostname corresponding to the IP address, this means that Zenarmor was not able to see a DNS request/response which can map this IP address to a hostname. But while you're browsing over the hostname section in Live Session Explorer screens, Zenarmor will try it once more by querying the IP address from your configured DNS server.

Why are my daily reports sent from an external server?

Zenarmor uses SVN backend servers to create your PDF reports and then send them to you. The generation of "PDF-based" scheduled reports is facilitated through an API hosted on the Zenarmor Datacenter. However, it should be noted that no data processed by this API, which primarily functions to generate PDF files, is stored on our end.

On the other hand, HTML reports are created on your local system and send via your email settings. But if you didn't define a sender address in your scheduled report settings, a sunnyvalley.io account is used as a sender. You should configure the scheduled reports properly and add your own email address to the Send Mail(s) From: field.

Why do I see an unexpected number of devices in reports?

Sometimes Zenarmor users report that they are viewing a more number of local devices than they really have or unknown IP addresses that do not belong to their networks. These issues may be seen in the following reports:

• Zenarmor widget on OPNsense Dashboard
• Conns & Facts Chart on Zenarmor Dashboard
• Zenarmor License Upgrade Premium Page
• Zenarmor Live Sessions Explorers

These issues may be caused by the Zenarmor Privacy settings or firewall network architecture design. To solve these issues you must ensure that:

• Anonymize Local IP address option is disabled. While OPNsense users may navigate Zenarmor → Settings → Privacy on OPNsense UI, other platform users may go to Firewall → Settings → Privacy → Reporting and Data pane on Zenconsole to check Anonymize Local IP address option.

• WAN traffic is not passing through the Zenarmor-protected interface. If you have WAN VLAN on your protected interface, you may see the external WAN IP addresses that are communicating with your LAN devices on the Top Local Hosts report and the remote IP Addresses will be counted as your local device. You may solve this issue by excluding the WAN interface's VLAN ID in the Zenarmor → Settings → Exempted VLANs & Networks option.

No Reporting Data

Some users can not view any reporting data on their Reports and Live Sessions due to database connection problems. They may get one of the following error messages on their reports:

• There is no data to display on Elasticsearch
• Error (200): No suitable servers found (serverSelectionTryOnce set): [socket timeout calling hello on 'localhost:27017']

Enabling DDoS syncookies protection on OPNsense is one of the factors that contribute to reporting data issues. This Anti-DDoS configuration prevents database connections. For example, when these users attempt to use the subsequent command on the CLI to reconnect their Elasticsearch DB databases, they encounter the error message "connection reset."

root@OPNsense:~ # curl http://localhost:9200/_cat/indices
curl: (56) Recv failure: Connection reset by peer

To address the database connection problem caused by DDoS syncookies protection, you need to alter the following settings on your OPNsense firewall.

1. Navigate to the Firewall → Settings → Advanced on OPNsense web UI.
2. Scroll down to the Anti DDOS pane at the bottom of the page.
3. Ensure the Enable syncookies is set to never (default).

Figure 3. Anti-DDoS syncookies option

After disabling DDoS syncookies protection on OPNsense, try to view your Zenarmor reports. If the issue still exists, you may need to reinstall reporting database.

rm -f /usr/local/zenarmor/etc/.configdone

Another reason for not being able to view the reports could be malformed index file(s). To address this issue, you can run the Reset Reporting feature. This action will remove all index files and re-create them.