Create troubleshooting guide for users (#37774)

@ddribeiro After talking with Noah about troubleshooting and logs, I thought it would be good if we created a user-facing guide for troubleshooting. Our CSEs frequently send the same troubleshooting information to customers when they report issues. We can empower customers with one resource that they can work through, reducing the significant amount of time in back-and-forth steps sent over Slack. Also, we've talked about pulling the "Finding fleetd logs" section out of the [Enroll hosts](https://fleetdm.com/guides/enroll-hosts) guide. Perhaps when this is complete, we can link to this document from that guide. This certainly doesn't contain everything! Please bring the CSEs in to flesh this out since they're in the day-to-day and have much more knowledge regarding this than I do. --------- Co-authored-by: Brock Walters <153771548+nonpunctual@users.noreply.github.com>
2026-02-12 14:34:54 -06:00
parent 2787b5ddc6
commit 573bf877fe
3 changed files with 143 additions and 17 deletions
@@ -153,24 +153,8 @@ In the Google Admin console:

 ## Debugging

-If you're running into issues when enrolling hosts, the best practice is to look for errors in the fleetd logs. Fleetd will send stdout/stderr logs to the following directories:
+If you're running into issues when enrolling hosts, the best practice is to look for errors in the fleetd logs. See our [troubleshooting guide](https://fleetdm.com/guides/fleet-troubleshooting-for-it-admins) for more info.

-  - macOS: `/private/var/log/orbit/orbit.std{out|err}.log`.
-  - Windows: `C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log` (the log file is rotated).
-  - Linux: Orbit and osqueryd stdout/stderr output is sent to syslog (`/var/log/syslog` on Debian systems, `/var/log/messages` on CentOS, and `journalctl -u orbit` on Fedora).
-
-If the `logger_path` agent configuration is set to `filesystem`, fleetd will send osquery's "result" and "status" logs to the following directories:
-  - Windows: `C:\Program Files\Orbit\osquery_log`
-  - macOS: `/opt/orbit/osquery_log`
-  - Linux: `/opt/orbit/osquery_log`
-
-The Fleet Desktop log files can be found in the following directories depending on the platform:
-
-  - Linux: `$XDG_STATE_HOME/Fleet or $HOME/.local/state/Fleet`
-  - macOS: `$HOME/Library/Logs/Fleet`
-  - Windows: `%LocalAppData%/Fleet`
-
-The log file name is `fleet-desktop.log`.

 ## Advanced

@@ -0,0 +1,74 @@
+# Fleet troubleshooting for IT admins
+
+It would be great if computers always performed flawlessly, but sometimes errors occur. When things go wrong, digging into logs is one of the first steps to resolving an issue.
+
+
+## Finding fleetd logs
+
+Fleetd will send stdout/stderr logs to the following directories:
+
+- macOS: `/var/log/orbit/orbit.std{out|err}.log`.
+- Windows: `C:\Windows\system32\config\systemprofile\AppData\Local\FleetDM\Orbit\Logs\orbit-osquery.log` (the log file is rotated).
+- Linux: Orbit and osqueryd stdout/stderr output is sent to syslog (`/var/log/syslog` on Debian systems, `/var/log/messages` on CentOS, and `journalctl -u orbit` on Fedora).
+
+If the `logger_path` agent configuration is set to `filesystem`, fleetd will send osquery's "result" and "status" logs to the following directories:
+- macOS: `/opt/orbit/osquery_log`
+- Windows: `C:\Program Files\Orbit\osquery_log`
+- Linux: `/opt/orbit/osquery_log`
+
+The Fleet Desktop log files can be found in the following directories depending on the platform:
+
+- macOS: `$HOME/Library/Logs/Fleet`
+- Windows: `%LocalAppData%/Fleet`
+- Linux: `$XDG_STATE_HOME/Fleet` or `$HOME/.local/state/Fleet`
+
+The log file name is `fleet-desktop.log`.
+
+
+## Enabling debug mode for fleetd
+
+Debug mode can be helpful by providing more information in the logs.
+
+When [generating an installer package](https://fleetdm.com/guides/enroll-hosts#cli) with `fleetctl package`, add the `--debug` argument to enable debug mode for the agent installer.
+
+If you're trying to troubleshoot macOS hosts, you can [run a script](../docs/solutions/macos/scripts/manage-orbit-debug.sh) on the host to turn on debug mode. After you're done, you can run the script again to disable debug mode on the host.
+
+1. Run the script on the affected host.
+2. Wait ~10 min.
+3. Refetch the host.
+4. Wait another ~10 min.
+5. Run the script again to disable debug logging.
+6. Grab the logs from `/var/log/orbit/orbit.stderr.log`.
+
+
+## Checking MDM commands
+
+If you suspect something went wrong with an MDM command for a device (such as locking, wiping, installing an app, etc.), you can use the UI or API to view the MDM command results.
+
+For the UI, open the host details page and under **Activity** toggle the switch for **Show MDM commands**.
+
+<img width="717" height="365" alt="Show MDM commands toggle" src="https://github.com/user-attachments/assets/41e7297c-efb4-4355-841e-d46296b99505" />
+
+Hover over the command you'd like to view, and select the **"i"** button.
+
+For the API, use the [List MDM commands](https://fleetdm.com/docs/rest-api/rest-api#list-mdm-commands) endpoint to find the `command_uuid` for the command. Use this UUID with the [Get MDM command results](https://fleetdm.com/docs/rest-api/rest-api#get-mdm-command-results) endpoint. The result of this looks like a random string of characters, but this is because it's base64 encoded. A quick way to decode this is on a Mac is to copy the long string, then decode it in the Terminal:
+
+```bash
+pbpaste | base64 -d
+```
+
+
+## Server-side logs
+
+Use [fleetctl](https://fleetdm.com/guides/fleetctl) to see server logs.
+
+```bash
+fleetctl debug errors
+```
+
+
+<meta name="category" value="guides">
+<meta name="authorFullName" value="Steven Palmesano">
+<meta name="authorGitHubUsername" value="spalmesano0">
+<meta name="publishedOn" value="2026-02-09">
+<meta name="articleTitle" value="Fleet troubleshooting for IT admins">
@@ -0,0 +1,68 @@
+#!/bin/sh
+
+# This script only works on macOS hosts
+# Use it to enable (and then disable) debug mode for troubleshooting
+
+# 1. Run the script on the affected host.
+# 2. Wait ~10 min.
+# 3. Refetch the host.
+# 4. Wait another ~10 min.
+# 5. Run the script again to disable debug logging.
+# 6. Grab the logs from `/var/log/orbit/orbit.stderr.log`.
+
+function change_state {
+  echo "------[ $(date) ]--------"
+  set -x
+
+  # Wait 15 seconds to allow the script response to be sent. 
+  # This ensures that Fleet registers the script as complete.
+  sleep 15
+
+  # Update the Orbit debug setting in Orbit plist. 
+  /usr/libexec/PlistBuddy -c "set EnvironmentVariables: ORBIT_DEBUG ${target_state}" "$plist_path"
+
+  # Stop Orbit, wait for stop to complete, and then restart.
+  launchctl bootout system/com.fleetdm.orbit
+  while pgrep orbit > /dev/null; do sleep 1 ; done
+  launchctl bootstrap system $plist_path
+
+  exit
+}
+
+default_action="toggle"
+action=${1:-$default_action}
+
+plist_path=/Library/LaunchDaemons/com.fleetdm.orbit.plist
+
+debug_enabled=$(/usr/libexec/PlistBuddy -c 'print EnvironmentVariables:ORBIT_DEBUG' "$plist_path")
+
+current_state=$([ "$debug_enabled" == "false" ] && echo "dis" || echo "en")
+flip_state=$([ "$debug_enabled" == "false" ] && echo "en" || echo "dis")
+
+case $action in
+  enable)
+    target_state="true"
+    ;;
+  disable)
+    target_state="false"
+    ;;
+  toggle)
+    target_state=$([ "$debug_enabled" == "true" ] && echo "false" || echo "true" )
+    ;;  
+esac  
+
+if [ "$debug_enabled" == "$target_state" ]
+  then  
+    echo "Debug logging is already ${current_state}abled"
+    exit
+  else
+    echo "Debug logging is currently ${current_state}abled"
+    echo "Starting a new process to ${flip_state}able debug logging..."
+    set -ma
+    change_state >> /tmp/orbit_debug_script_logs.txt 2>&1 &
+    set +ma
+    echo "New process started, Orbit will restart in 15 seconds."
+    echo "If debug logging is not ${flip_state}abled, check the logs at"
+    echo "/tmp/orbit_debug_script_logs.txt"
+    exit
+fi