<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34847
# Details
In testing the new end-user auth flow for enrolling hosts on Windows, I
noticed that the `&host_uuid=<uuid>` part of the query string was
missing when opening the browser window, causing the SSO login process
to ultimately fail. I discovered that this is because the command we're
using to open the browser interprets `&` as a command separator, so it
needs to be escaped. This PR updated the `open` package for Windows, by
adding escaping to all `&` characters in a URL that are not already
escaped.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [X] QA'd all new/changed functionality manually
Added a new tool for testing:
```
go run ./tools/open -url "http://google.com?x=y&owl=hoot^&foo=bar"
```
to test that it escapes ampersands correctly (and doesn't
double-escape).
## fleetd/orbit/Fleet Desktop
- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#33776
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34899
Cherry pick for #35127
Co-authored-by: Martin Angers <martin.n.angers@gmail.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34528
# Details
This PR implements the agent changes for allowing Fleet admins to
require that users authenticate with an IdP prior to having their
devices set up. I'll comment on changes inline but the high-level is:
1. Orbit calls the enroll endpoint as usual. This is triggered lazily by
any one of a number of subsystems like device token rotation or
requesting Fleet config
2. If the enroll endpoint returns the new `ErrEndUserAuthRequired`
response, then it opens a window to the `/mdm/sso` Fleet page and
retries the enroll endpoint every 30 seconds indefinitely.
3. Any other non-200 response to the enroll request is treated as before
(limited # of retries, with backoff)
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-
changes.md#changes-files) for more information.
Will add changelog when story is one.
## Testing
- [X] Added/updated automated tests
Added test for new retry logic
- [X] QA'd all new/changed functionality manually
This is kinda hard to test without the associated backend PR:
https://github.com/fleetdm/fleet/pull/34835
## fleetd/orbit/Fleet Desktop
- [X] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
This is compatible with all Fleet versions, since older ones won't send
the new error.
- [X] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
This is compatible with all platforms, although it currently should only
ever run on Windows and Linux since macOS devices will have end-user
auth taken care of before they even download Orbit.
- [ ] Verified that fleetd runs on macOS, Linux and Windows
Testing this now.
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added SSO (Single Sign-On) enrollment support for end-user
authentication
* Enhanced error messaging for authentication-required scenarios
* **Bug Fixes**
* Improved error handling and retry logic for enrollment failures
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
**Related issue:** Resolves#32084
This PR modifies `isValidAppFilePath` to allow subdirectors in
`Applications/`, like in this case `Applications/Cisco/Cisco Secure
Client.app`.
This also changes the metadata extraction from packageinfo to trim
`.app` from the name in all cases.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [x] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
### Test plan:
---
I ran this on my local environment and it seemed fine
- Have environment with the bug recreated, it has two software titles
for "Cisco Secure Client", and the one with the bundle id
`com.cisco.pkg.anyconnect.vpn` is used by the installer.
- URL to pkg:
https://fndtnfleetmsp.blob.core.windows.net/fndtnpkgs/cisco-secure-client-macos-5.1.3.62-core-vpn-webdeploy-k9.pkg
- Cisco Secure Client doesn't show as installed in UI even after
installing.
- Run the new migration.
- Cisco Secure Client shows as installed now in ui, software title with
bundle id `com.cisco.pkg.anyconnect.vpn` is gone from the database, and
the software installer references the correct title
(`com.cisco.secureclient.gui`).
- Check that deleting and reuploading the installer doesn't recreate the
bad software title.
### QA Note:
---
There are some problems with the install script, but that is probably a
different scope than this ticket.
`Reinstall` wont work, it says Cisco Secure Client is already installed.
Uninstalling through Fleet then Installing again works fine though.
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34500 and Resolves#33758
Video demo: https://www.youtube.com/watch?v=4HZlKG0G1B0
- Added a new aggregation table
`operating_system_version_vulnerabilities` for faster queries. The table
is currently used only for Linux vulnerabilities, but could be used for
other OS vulnerabilities.
- Added `max_vulnerabilities` parameter per [API
doc](https://github.com/fleetdm/fleet/pull/33533)
- Also added `max_vulnerabilities` parameter to `os_versions/{id}`
endpoint, but not making it public since that endpoint is still slow and
needs other API changes. bug #34974
- Removed `"kernels": []` from `os_versions` endpoint result
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
- [x] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [x] Added/updated automated tests
- [x] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [x] QA'd all new/changed functionality manually
## Database migrations
- [x] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [x] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [x] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added ability to limit the number of vulnerabilities displayed for
operating system versions via an optional parameter.
* Introduced vulnerability count tracking for operating system versions,
now visible in API responses and UI displays.
* Enhanced operating system vulnerability visualization with improved
count-based rendering.
* **Tests**
* Added comprehensive test coverage for vulnerability limiting behavior
across multiple operating system versions and architectures.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34500
osquery-perf changes only
- Updated Ubuntu template to add a patch version. This increases the
number of OS versions in line with customer environments.
- Updated Ubuntu template to send from a set of kernels per agent. This
closer replicates a customer environment with a lot of Linux hosts.
- Updated software versions to be deterministic. Once an agent selects a
software version, then it will use the same version for future checkins.
# Checklist for submitter
## Testing
- [x] QA'd all new/changed functionality manually
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added embedded Ubuntu 22.04 kernel dataset for agents
* Implemented consistent software version selection across queries
* Enhanced kernel data assignment for Ubuntu systems
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
- Use global form and "modal-cta-wrap" styling for certificate authority
forms
- Confirmed all forms maintain desired style. See example below:
L: this branch | R: Dogfood (main)
<img width="2562" height="1228" alt="Screenshot 2025-10-31 at 1 45
00 PM"
src="https://github.com/user-attachments/assets/68766ef4-9abe-47dc-bd2a-458bd04a5073"
/>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#35005
# Checklist for submitter
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [x] Confirmed that the fix is not expected to adversely impact load
test results
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Software API responses now properly include installed paths and path
signature information when populated.
* **Tests**
* Added test coverage for software data serialization and installed
paths functionality in API responses.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Victor Lyuboslavsky <2685025+getvictor@users.noreply.github.com>
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34500
Terraform changes after my latest loadtest.
VPC consolidation: updated (and deployed) shared VPC so that Signoz
backend can now use it
- Removed eks-vpc/ directory
- Moved VPC management to shared/vpc.tf
- Updated shared/init.tf to reflect VPC changes
Infra improvements
- infra/internal_alb.tf - changed suffix from -internal to -int since I
hit max 32 characters issue
OTEL
- OTEL Collector configuration overrides for production stability
Correcting description of action when no team is applied to fleet
premium when modifying OS settings.
https://fleetdm.slack.com/archives/C019WG4GH0A/p1761829754255319
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves #
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
- [ ] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
- [ ] If paths of existing endpoints are modified without backwards
compatibility, checked the frontend/CLI for any necessary changes
## Testing
- [ ] Added/updated automated tests
- [ ] Where appropriate, [automated tests simulate multiple hosts and
test for host
isolation](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/reference/patterns-backend.md#unit-testing)
(updates to one hosts's records do not affect another)
- [ ] QA'd all new/changed functionality manually
For unreleased bug fixes in a release candidate, one of:
- [ ] Confirmed that the fix is not expected to adversely impact load
test results
- [ ] Alerted the release DRI if additional load testing is needed
## Database migrations
- [ ] Checked schema for all modified table for columns that will
auto-update timestamps during migration.
- [ ] Confirmed that updating the timestamps is acceptable, and will not
cause unwanted side effects.
- [ ] Ensured the correct collation is explicitly set for character
columns (`COLLATE utf8mb4_unicode_ci`).
## New Fleet configuration settings
- [ ] Setting(s) is/are explicitly excluded from GitOps
If you didn't check the box above, follow this checklist for
GitOps-enabled settings:
- [ ] Verified that the setting is exported via `fleetctl
generate-gitops`
- [ ] Verified the setting is documented in a separate PR to [the GitOps
documentation](https://github.com/fleetdm/fleet/blob/main/docs/Configuration/yaml-files.md#L485)
- [ ] Verified that the setting is cleared on the server if it is not
supplied in a YAML file (or that it is documented as being optional)
- [ ] Verified that any relevant UI is disabled when GitOps mode is
enabled
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [ ] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [ ] Verified that fleetd runs on macOS, Linux and Windows
- [ ] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))
- @noahtalerman: We decided to cut the `santa_rules` table and tweak
descriptions: https://github.com/fleetdm/fleet/pull/33825
- Looks like these doc changes got added back to `main` by accident
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34529
# Details
This PR implements the backend (and some related front-end screens) for
allowing Fleet admins to require that users authenticate with an IdP
prior to having their devices set up. I'll comment on changes inline but
the high-level for the device enrollment flow is:
1. The handler for the `/orbit/enroll` endpoint now checks whether the
end-user authentication is required for the team (or globally, if using
the global enroll secret).
2. If so, it checks whether a `host_mdm_idp_accounts` row exists with a
`host_uuid` matching the identifier sent with the request
3. If a row exists, enroll. If not, return back a new flavor of
`OrbitError` with a `401` status code and a message
(`END_USER_AUTH_REQUIRED`) that Orbit can interpret and act accordingly.
Additionally some changes were made to the MDM SSO flow. Namely, adding
more data to the session we store for correlating requests we make to
the IdP to initiate SSO to responses aimed at our callback. We now store
a `RequestData` struct which contains the UUID of the device making the
request, as well as the "initiator" (in this case, "setup_experience").
When our SSO callback detects that the initiator was the setup
experience, it attempts to add all of the relevant records to our
database to associate the host with an IdP account. This removes the
enrollment gate in the `/orbit/enroll` endpoint.
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
- [ ] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.
Will put the changelog in the last ticket for the story
- [X] Input data is properly validated, `SELECT *` is avoided, SQL
injection is prevented (using placeholders for values in statements)
## Testing
- [X] Added/updated automated tests
will see if there's any more to update
- [X] QA'd all new/changed functionality manually
To test w/ SimpleSAML
1. Log in to your local Fleet dashboard with MDM and IdP set up for
SimpleSAML
1. Go to Settings -> Integrations -> Identity provider
2. Use "SimpleSAML" for the provider name
3. Use `mdm.test.com` for the entity ID
4. Use `http://127.0.0.1:9080/simplesaml/saml2/idp/metadata.php` for the
metadata URL
1. Set up a team (or "no team") to have End User Authentication required
(Controls -> Setup experience)
1. Get the enroll secret of that team
1. In the browser console, do:
```
fetch("https://localhost:8080/api/fleet/orbit/enroll", {
"headers": {
"accept": "application/json, text/plain, */*",
"cache-control": "no-cache",
"content-type": "application/json",
"pragma": "no-cache",
},
"body": "{\"enroll_secret\":\"<enroll secret>", \"hardware_uuid\":\"abc123\" }",
"method": "POST",
});
```
replacing `<enroll secret>` with your team's enroll secret.
8. Verify in the network tab that you get a 401 error with message
`END_USER_AUTH_REQUIRED`
1. Go to
https://localhost:8080/mdm/sso?initiator=setup_experience&host_uuid=abc123
1. Verify that a new screen appears asking you to log in to your IdP
1. Log in to SimpleSAML with `sso_user / user123#`
1. Verify that you're taken to a success screen
1. In your database, verify that records exist in the `mdm_idp_accounts`
and `host_mdm_idp_accounts` tables with uuid `abc123`
1. Try the `fetch` command in the browser console again, verify that it
succeeds.
## fleetd/orbit/Fleet Desktop
- [ ] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
This is _not_ compatible with the current version of fleetd or the
soon-to-be-released 1.49.x. Until #34847 changes are released in fleetd,
this will need to be put behind a feature flag or withheld from Fleet
releases.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added support for device UUID linkage during MDM enrollment to enable
host-initiated enrollment tracking
* Introduced setup experience flow for device authentication during
enrollment
* Added end-user authentication requirement configuration for macOS MDM
enrollment
* **Improvements**
* Enhanced MDM enrollment process to maintain device context through
authentication
* Updated authentication UI to display completion status for device
setup flows
* Refined form layout styling for improved visual consistency
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Add the related story/sub-task/bug number, like Resolves#123, or
remove if NA -->
**Related issue:** Resolves#34251
# Checklist for submitter
If some of the following don't apply, delete the relevant line.
## Testing
- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually
This PR adds a small log warn statement to the fleetdm proxy for a 400
error on creating enterprise that is not an invalid token.
See this slack thread about the issue, and missing error log to further
debug the issue, unless running it locally, and then it is sometimes not
reproducable.
Rachael Shaw