Resolve the matching Fleet user from the persisted SCIM record rather
than the incoming request state when handling deactivation, so
deprovisioning still works when identifiers change in the same request.
**Related issue:** Resolves#49007
# Checklist for submitter
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* SCIM user create/reactivation, replace, patch, and delete flows now
automatically record certificate resend activities when applicable.
* Certificate resend activities are generated alongside SCIM
persistence, tied to the resulting “resent certificates”.
* **Bug Fixes**
* Improved reliability and synchronization of certificate resend
activity recording during SCIM and Google Workspace reconciliation.
* Failures to record individual resend activities no longer block the
underlying SCIM operation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The SCIM endpoints are served by the elimity-com/scim library mounted as
a single prefix handler on the root ServeMux, so they are never
registered as individual gorilla/mux routes.
Since the routes can't be discovered, supply them to the validator
instead: add scim.RegisterValidationRoutes, a FeatureRouteFunc that
registers stub routes for the SCIM endpoints (handlers are never
invoked, only their path templates and methods are inspected). Wire it
into the three Validate call sites (production serve, test helper,
svctest).
### 🥞 Stack (review/merge bottom-up)
1. #48164 — Activity types (FE+BE)
2. **#48165 — Backend (cron + directory sync) ⬅ this PR**
3. #48166 — Usage statistics
4. #48167 — fleetctl generate-gitops
5. #48168 — Settings UI
📄 Documentation is tracked separately in #48169 (targets
`docs-v4.89.0`).
---
## Summary
**PR 2 of 6.** Core **backend** for the Google Workspace IdP
integration:
- Directory sync client (`ee/server/googleworkspace/`) and cron job
(`server/cron/google_workspace_cron.go`) reusing the `scim_*` tables
(Google Workspace and SCIM are mutually exclusive).
- Config types + validation (`server/fleet/google_workspace.go`,
`app.go`, `integrations.go`), appconfig handling + activity emission
(`server/service/appconfig.go`), cron registration and schedule.
- SCIM is ignored while Google Workspace is configured
(`ee/server/scim/scim.go`).
> 🥞 **Stacked PR.** Base: `42915-gw-idp-vitals-1-activities` (PR 1) —
review/merge that first.
**Related issue:** Resolves#42915
# Checklist for submitter
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements).
- [ ] Timeouts are implemented and retries are limited to avoid infinite
loops.
## Testing
- [ ] Added/updated automated tests
- [ ] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Google Workspace integration support for syncing users, groups,
and host-related identity data.
* Added a scheduled sync that keeps directory data up to date
automatically.
* Added support for configuring Google Workspace in app settings, with
validation and masking of sensitive credentials.
* **Bug Fixes**
* Prevented SCIM provisioning from overwriting data when Google
Workspace sync is configured.
* Preserved existing Google Workspace credentials when an update omits
masked API key values.
* Added handling for deleted users and group membership changes during
sync.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Resolves#41996
Some IdPs push SCIM PATCH operations that bundle `department` with other
RFC 7643 §4.3 enterprise attributes that Fleet does not store. Before
this change the SCIM library rejected those payloads with a 400 because
the schema only declared 'department'.
An environment variable was added (FLEET_DEBUG_SCIM_PAYLOADS) for aiding
in further debugging this if the issue persist.
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements), JS
inline code is prevented especially for url redirects, and untrusted
data interpolated into shell scripts/commands is validated against shell
metacharacters.
## Testing
- [X] QA'd all new/changed functionality manually
Server log when SCIM ingestion receives a deactivation event for the
last admin:
```log
ts=2026-04-01T15:23:01Z level=error msg="failed to delete fleet user on deactivation" component=SCIM err="cannot delete last global admin"
```
Server response when attempting to demote last admin:
```json
{
"message": "Validation Failed",
"errors": [
{
"name": "global_role",
"reason": "cannot demote the last global admin"
}
],
"uuid": "1d110f56-25ac-47b8-bc96-982354474a87"
}
```
Server response when attempting to delete last admin:
```json
{
"message": "Validation Failed",
"errors": [
{
"name": "id",
"reason": "cannot delete the last global admin"
}
],
"uuid": "1448c2da-30e2-4652-a9a8-a01fc4f9b9c1"
}
```
---
## Original AI Summary
- Fixes a TOCTOU race condition where two concurrent admin operations
could bypass the last-global-admin guard, leaving zero admins and
permanently locking out the Fleet instance (fleetdm/confidential#14827)
- Introduces two new atomic datastore methods
(`DeleteUserIfNotLastAdmin`, `SaveUserIfNotLastAdmin`) that wrap the
admin count check and the write in a single MySQL transaction with
`SELECT ... FOR UPDATE`
- Fixes all four vulnerable code paths: service `DeleteUser`, service
`ModifyUser` (two demotion paths), and SCIM user deletion
## Test plan
- [X] Manual verification: single admin cannot be deleted or demoted
🤖 Generated with [Claude Code](https://claude.com/claude-code)
---------
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Lucas Manuel Rodriguez <lucas@fleetdm.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38536
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Changes file present in previous PR.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Reorganized internal activity tracking infrastructure across services
to improve code maintainability and reduce complexity.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#40054
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Already added in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Updated internal logging infrastructure across multiple server
components to use standardized logging methods and improved context
propagation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38889
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Updated internal logging infrastructure to improve standardization and
maintainability. Logging functionality remains unchanged from an
end-user perspective.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38889
PLEASE READ BELOW before looking at file changes
Before converting individual files/packages to slog, we generally need
to make these 2 changes to make the conversion easier:
- Replace uses of `kitlog.With` since they are not fully compatible with
our kitlog adapter
- Directly use the kitlog adapter logger type instead of the kitlog
interface, which will let us have direct access to the underlying slog
logger: `*logging.Logger`
Note: that I did not replace absolutely all uses of `kitlog.Logger`, but
I did remove all uses of `kitlog.With` except for these due to
complexity:
- server/logging/filesystem.go and the other log writers (webhook,
firehose, kinesis, lambda, pubsub, nats)
- server/datastore/mysql/nanomdm_storage.go (adapter pattern)
- server/vulnerabilities/nvd/* (cascades to CLI tools)
- server/service/osquery_utils/queries.go (callback type signatures
cascade broadly)
- cmd/maintained-apps/ (standalone, so can be transitioned later all at
once)
Most of the changes in this PR follow these patterns:
- `kitlog.Logger` type → `*logging.Logger`
- `kitlog.With(logger, ...)` → `logger.With(...)`
- `kitlog.NewNopLogger() → logging.NewNopLogger()`, including similar
variations such as `logging.NewLogfmtLogger(w)` and
`logging.NewJSONLogger(w)`
- removed many now-unused kitlog imports
Unique changes that the PR review should focus on:
- server/platform/logging/kitlog_adapter.go: Core adapter changes
- server/platform/logging/logging.go: New convenience functions
- server/service/integration_logger_test.go: Test changes for slog
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- Was added in previous PR
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Migrated the codebase to a unified internal structured logging system
for more consistent, reliable logs and observability.
* No user-facing functionality changed; runtime behavior and APIs remain
compatible.
* **Tests**
* Updated tests to use the new logging helpers to ensure consistent test
logging and validation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#38087
More context:
https://fleetdm.slack.com/archives/C084F4MKYSJ/p1768336339026999 and
https://fleetdm.slack.com/archives/C084F4MKYSJ/p1768512354275959.
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
### Okta
NOTE: Okta does not send `DELETE` requests when deleting a user.
Therefore, we decided to perform Fleet users deletion when the
deactivation happens (`PUT` request).
There's an edge case where a `deactivated` user in Okta is `activated`
back again: Okta sends a `POST` request as if a new user was created --
due to this I added an extra check on the `UserHandler Create` function
so that we don't attempt to create a duplicate SCIM user and instead
replace the existing record (basically, the only change should be
`active=0` -> `active=1`).
What I tested:
- [x] Deactivating user in Okta switches `scim_users` record to `active
= 0` and deletes matching `users` records.
- [x] Activating a deactivated user in Okta switches `scim_users` record
to `active = 1`. Note that a `users` record is not created
automatically. For this, there are two alternatives that we'll mention
in the documentation:
1. Manually create a user from the **Users page**.
2. Log in to Fleet using SSO (must have SSO and the **Create user and
sync permissions on login** setting enabled). Note that an activation
email is sent and the user must provide a new password, or an admin in
the Okta dashboard can set up a one-time password and share it with the
user.
https://github.com/user-attachments/assets/5262a581-41f0-4a88-aa73-40768064f8f5
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Followup fix for #30888
See
https://github.com/fleetdm/fleet/issues/30888#issuecomment-3321700108
Needs to be cherry-picked into 4.74
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
Fixes#32331
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- New Features
- Expanded OpenTelemetry coverage to include scheduled jobs for improved
tracing of cron executions.
- Documentation
- Updated changelog to reflect expanded OpenTelemetry coverage.
- Refactor
- Updated scheduled job processing to use contextual tracing across
operations with improved span lifecycle handling.
- Tests
- Adjusted tests to pass context into scheduled job runs.
- Chores
- Removed obsolete commented debug logs in SCIM middleware.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Fixes#32331
Manually tested all paths. `/test` path removed in
https://github.com/fleetdm/fleet/pull/32962
Also added support for sending errors to OpenTelemetry, like we do for
APM/Sentry.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added OpenTelemetry tracing across core HTTP endpoints (health,
version, assets, metrics, enroll/root, debug, Apple MDM, SCEP, SCIM)
with dynamic per-request route instrumentation.
* Enhanced error reporting to include OpenTelemetry spans/events with
contextual user/host attributes.
* **Tests**
* Added unit tests validating SCIM and error-handling telemetry, span
naming, and sensitive-data redaction.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
#29609
Verified the changes with [Entra ID's
validator](https://scimvalidator.microsoft.com/) and adding the
department attribute to the tester:
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 32 AM"
src="https://github.com/user-attachments/assets/45a5deb8-7c65-49df-b3e8-eb05bea11f6b"
/>
<img width="1312" alt="Screenshot 2025-06-27 at 8 54 21 AM"
src="https://github.com/user-attachments/assets/91b554b5-b0b9-4bb6-a0cf-4e3b40e6ce21"
/>
- Tested with Okta
- TODO: Test with Entra ID and Google Workspace.
- I decided to not fail profile deployment if a user has no department
because it's not a required attribute, instead the
`FLEET_VAR_HOST_END_USER_IDP_DEPARTMENT` will be replaced with the empty
string.
---
- [X] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [X] If database migrations are included, checked table schema to
confirm autoupdate
(https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- For database migrations:
- [X] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [X] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [X] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [X] Added/updated automated tests
- [X] Manual QA for all new/changed functionality
For #28196
This PR depends on PR https://github.com/fleetdm/fleet/pull/28832
This PR adds support for excludeAttributes=members, which is being used
by Microsoft Entra ID.
This PR modifies the primary key of host_scim_user table to be host_id.
This should have been done initially and has added accidental complexity
and maintainability challenges, so we are doing it now. This means a
host can have a maximum of 1 SCIM user associated with it. A SCIM user,
on the other hand, can be associated with multiple hosts.
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #28196
This PR adds full patching for SCIM Users and Groups, and adds the
ability to filter Groups by displayName.
The changes have been tested with [Entra ID SCIM
Validator](https://github.com/fleetdm/fleet/blob/67dfd91c0cfb1546177ad533d02ee94eb199c3eb/docs/Contributing/SCIM-integration.md#entra-id-integration)
and Okta SCIM 2.0 SPEC Test (to make sure we didn't break Okta).
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #28196
This PR adds the capability to SCIM API to PATCH a User using
op:replace. This brings us closer to fully supporting Entra ID SCIM.
In future PR:
- Add support op:add
- Add support for op:remove
- Add support for PATCH to Group
- Add filter support
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #23236
Bug fix, per Figma
# Checklist for submitter
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #27281
This PR adds `/api/{version}/fleet/scim/details` endpoint, along with
some frontend fixes.
# Checklist for submitter
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #27284
This PR:
- Adds SCIM as a fallback for username during macOS end user
authentication during setup experience
- Adds SCIM/endUsers details to host details
# Checklist for submitter
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/Committing-Changes.md#changes-files)
for more information.
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
For #27287
This PR adds integration tests for SCIM API endpoints as well as some
bug fixes found by these tests.
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #27287
This PR adds SCIM Groups to Fleet's SCIM endpoint as a follow on to SCIM
Users. The logic has been manually tested with Okta, and integration
tests will be in the next PR.
# Checklist for submitter
- [x] Added/updated automated tests
- [x] Manual QA for all new/changed functionality
For #27287
Video explaining the PR: https://www.youtube.com/watch?v=ZHgFUAvrPEI
This PR adds SCIM Users support for Okta. The goal is to first add
Users/Groups support so that the remaining backend SCIM work can be done
in parallel.
This PR does not include the following, which will be added in later PRs
- Changes file
- Groups support for Okta
- Full support for Entra ID
- Integration tests
# Checklist for submitter
- [x] If database migrations are included, checked table schema to
confirm autoupdate
- For database migrations:
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
- [x] Added/updated automated tests
- [x] A detailed QA plan exists on the associated ticket (if it isn't
there, work with the product group's QA engineer to add it)
- [x] Manual QA for all new/changed functionality
Juan Fernandez