always send webhook while device is unmanaged for MDM migration (#39416)

<!-- Add the related story/sub-task/bug number, like Resolves #123, or
remove if NA -->
**Related issue:** Resolves #38322 

This PR utilizes the ping/status ticker that sees if the device is
Unmanaged (aka. not enrolled from a Fleet server perspective), if the
Migrate to Fleet flow before had set the `mdm_migration.txt` file, but
somehow not successfully unenrolled the device, we now keep sending it
if you trigger the modal again.

We wait 90seconds after start, so at most the user can go through the
flow every 90s, but the server has a hard limit on at most one webhook
every 3m, but still it means the user can wait a bit and retry and still
see the webhook gets sent now.

_PS: Updated the old migration test to go from 1,5m to ~2s execution
time with parallel and configurable waitForUnenrollment time (to allow
test to set lower values)

# Checklist for submitter

If some of the following don't apply, delete the relevant line.

- [x] Changes file added for user-visible changes in `changes/`,
`orbit/changes/` or `ee/fleetd-chrome/changes`.
See [Changes
files](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/guides/committing-changes.md#changes-files)
for more information.


## Testing

- [x] Added/updated automated tests
- [x] QA'd all new/changed functionality manually

## fleetd/orbit/Fleet Desktop

- [x] Verified compatibility with the latest released version of Fleet
(see [Must
rule](https://github.com/fleetdm/fleet/blob/main/docs/Contributing/workflows/fleetd-development-and-release-strategy.md))
- [x] If the change applies to only one platform, confirmed that
`runtime.GOOS` is used as needed to isolate changes
- [x] Verified that fleetd runs on macOS, Linux and Windows
- [x] Verified auto-update works from the released version of component
to the new version (see [tools/tuf/test](../tools/tuf/test/README.md))

---------

Co-authored-by: Jordan Montgomery <elijah.jordan.montgomery@gmail.com>

docs/Contributing/product-groups/mdm/README.md

@@ -16,6 +16,7 @@ Fleet's MDM functionality allows organizations to manage and secure devices acro
 - [Custom Configuration Web URL](custom-configuration-web-url.md)
 - [Custom SCEP Integration](custom-scep-integration.md)
 - [Provide end user email address w/o relying on end user](set-up-custom-end-user-email.md)
+- [Migrate to Fleet via Orbit/Fleet Desktop](migrate-to-fleet-orbit.md)
 
 ## Related Resources
 

docs/Contributing/product-groups/mdm/migrate-to-fleet-orbit.md

@@ -0,0 +1,26 @@
+# Migrate to Fleet via Orbit/Fleet Desktop
+
+We provide a way to MDM migrate devices enrolled via other MDM solutions to Fleet using Fleet Desktop (Orbit). 
+
+## Relevant code pieces
+- [mdm_migration_darwin.go](../../../../orbit/pkg/useraction/mdm_migration_darwin.go) handles the Orbit side showing the migration dialogs and unenrollment checking logic.
+- [migrate_mdm endpoint](https://github.com/fleetdm/fleet/blob/main/ee/server/service/devices.go#L22-L98) handles the Fleet server side of the migration request and triggering the unenrollment webhook.
+
+## Prerequisites for the option to show
+- The device must have Fleet Desktop (Orbit) installed, and be Orbit enrolled into Fleet.
+- The device must be enrolled in an MDM solution that is not Fleet.
+- Fleet server needs to see and recognize the device being MDM enrolled elsewhere.
+
+> If the device is ADE enrolled, it requires the device to be assigned to Fleet in ABM.
+
+## Migration flow
+1. User initiates migration on their device via Fleet Desktop (Orbit)
+2. Fleet Desktop (Orbit) checks the locally placed file [`mdm_migration.txt`](https://github.com/fleetdm/fleet/blob/main/orbit/cmd/desktop/desktop.go#L710-L715) value, to determine what kind of previous MDM enrollment was done.
+3. Fleet Desktop (Orbit) hits the endpoint `POST /api/_version_/fleet/device/{token}/migrate_mdm` on the Fleet server, to notify Fleet of the migration request, which will in turn notify the [configured Webhook URL](https://github.com/fleetdm/fleet/blob/main/ee/server/service/devices.go#L84).
+4. Fleet Desktop (Orbit) then waits for unenrollment locally to be completed. See [how long it waits](https://github.com/fleetdm/fleet/blob/main/orbit/pkg/useraction/mdm_migration_darwin.go#L54).
+    1. If it successfully unenrolled while waiting, the loading dialog will disappear.
+        a. If manual enrollment, then the My Device page will pop up and instruct the user to manually enroll.
+        b. If ADE enrollment, then it will close and Orbit will periodically call the `profiles renew -type enrollment` command to trigger the native ADE enrollment flow from Apple.
+    2. If it failed to unenroll, the user can trigger the Migrate to Fleet flow again, which will keep sending the unenroll webhook until we sucessfully unenroll. _Currently the Fleet server limits webhook requests to [every 3 minutes](https://github.com/fleetdm/fleet/blob/main/server/fleet/mdm.go#L29)_ 
+
+Once the new enrollment steps have been followed, the device should now be MDM enrolled into Fleet.

orbit/changes/38322-send-webhook-while-unmanaged

@@ -0,0 +1 @@
+- Updated Migrate to Fleet webhook to always send when device is seen as unmanaged.

orbit/pkg/useraction/mdm_migration_darwin.go

@@ -48,10 +48,10 @@ const (
 // people that build integrations on top of the migration flow.
 var mdmEnrollmentFile = "/private/var/db/ConfigurationProfiles/Settings/.cloudConfigProfileInstalled"
 
-// mdmUnenrollmentTotalWaitTime defines how long the dialog is going to wait
+// defaultMDMUnenrollmentTotalWaitTime defines how long the dialog is going to wait
 // for the device to be unenrolled before bailing out and showing an error
 // message.
-const mdmUnenrollmentTotalWaitTime = 90 * time.Second
+const defaultMDMUnenrollmentTotalWaitTime = 90 * time.Second
 
 // defaultUnenrollmentRetryInterval defines how long we're going to wait
 // between unenrollment checks.
@@ -97,12 +97,20 @@ var mdmMigrationTemplateOffline = template.Must(template.New("").Parse(`
 
 // baseDialog implements the basic building blocks to render dialogs using
 // swiftDialog.
+// it should fulfil the dialog interface.
 type baseDialog struct {
 	path        string
 	interruptCh chan struct{}
 }
 
-func newBaseDialog(path string) *baseDialog {
+// dialog is an interface that MDMMigrator needs to act on dialog windows
+type dialog interface {
+	CanRun() bool
+	Exit()
+	render(flags ...string) (chan swiftDialogExitCode, chan error)
+}
+
+func newBaseDialog(path string) dialog {
 	return &baseDialog{path: path, interruptCh: make(chan struct{}, 1)}
 }
 
@@ -115,6 +123,10 @@ func (b *baseDialog) CanRun() bool {
 	return true
 }
 
+func (m *swiftDialogMDMMigrator) CanRun() bool {
+	return m.baseDialog.CanRun()
+}
+
 // Exit sends the interrupt signal to try and stop the current swiftDialog
 // instance.
 func (b *baseDialog) Exit() {
@@ -122,6 +134,10 @@ func (b *baseDialog) Exit() {
 	log.Info().Msg("dialog exit message sent")
 }
 
+func (m *swiftDialogMDMMigrator) Exit() {
+	m.baseDialog.Exit()
+}
+
 // render is a general-purpose render method that receives the flags used to
 // display swiftDialog, and starts an asyncronous routine to display the dialog
 // without blocking.
@@ -186,7 +202,7 @@ func (b *baseDialog) render(flags ...string) (chan swiftDialogExitCode, chan err
 }
 
 // NewMDMMigrator creates a new  swiftDialogMDMMigrator with the right internal state.
-func NewMDMMigrator(path string, frequency time.Duration, handler MDMMigratorHandler, mrw *migration.ReadWriter, fleetURL string, showCh chan struct{}) MDMMigrator {
+func NewMDMMigrator(path string, frequency time.Duration, handler MDMMigratorHandler, mrw readWriter, fleetURL string, showCh chan struct{}) MDMMigrator {
 	if cap(showCh) != 1 {
 		log.Fatal().Msg("swift dialog channel must have a buffer size of 1")
 	}
@@ -195,19 +211,27 @@ func NewMDMMigrator(path string, frequency time.Duration, handler MDMMigratorHan
 		baseDialog:                newBaseDialog(path),
 		frequency:                 frequency,
 		unenrollmentRetryInterval: defaultUnenrollmentRetryInterval,
+		maxUnenrollmentWaitTime:   defaultMDMUnenrollmentTotalWaitTime,
 		mrw:                       mrw,
 		fleetURL:                  fleetURL,
 		showCh:                    showCh,
 	}
 }
 
+// readWriter is an interface that abstracts the reading and writing of the migration file
+type readWriter interface {
+	GetMigrationType() (string, error)
+	SetMigrationFile(typ string) error
+	RemoveFile() error
+}
+
 // swiftDialogMDMMigrator implements MDMMigrator for macOS using swiftDialog as
 // the underlying mechanism for user action.
 type swiftDialogMDMMigrator struct {
-	*baseDialog
-	props     MDMMigratorProps
-	frequency time.Duration
-	handler   MDMMigratorHandler
+	baseDialog dialog
+	props      MDMMigratorProps
+	frequency  time.Duration
+	handler    MDMMigratorHandler
 
 	// ensures only one dialog is open at a time, protects access to
 	// lastShown
@@ -223,7 +247,8 @@ type swiftDialogMDMMigrator struct {
 	// the enrollment status of the host
 	testEnrollmentCheckStatusFn func() (bool, string, error)
 	unenrollmentRetryInterval   time.Duration
-	mrw                         *migration.ReadWriter
+	maxUnenrollmentWaitTime     time.Duration
+	mrw                         readWriter
 	fleetURL                    string
 }
 
@@ -329,7 +354,7 @@ func (m *swiftDialogMDMMigrator) renderError() (chan swiftDialogExitCode, chan e
 // device to unenroll from the current MDM solution. If the device doesn't
 // unenroll, an error is returned.
 func (m *swiftDialogMDMMigrator) waitForUnenrollment(isADEMigration bool) error {
-	maxRetries := int(mdmUnenrollmentTotalWaitTime.Seconds() / m.unenrollmentRetryInterval.Seconds())
+	maxRetries := int(m.maxUnenrollmentWaitTime.Seconds() / m.unenrollmentRetryInterval.Seconds())
 	checkFileFn := m.testEnrollmentCheckFileFn
 	if checkFileFn == nil {
 		checkFileFn = func() (bool, error) {
@@ -422,13 +447,14 @@ func (m *swiftDialogMDMMigrator) renderMigration() error {
 			return nil
 		}
 
-		if previousMigrationType == constant.MDMMigrationTypeADE {
+		if previousMigrationType == constant.MDMMigrationTypeADE && m.props.IsUnmanaged {
+			// Only skip if we know the device is unamanged, but then
 			// Do nothing; the Remote Management modal will be launched by Orbit every minute.
 			return nil
 		}
 
-		if previousMigrationType == constant.MDMMigrationTypeManual || previousMigrationType == constant.MDMMigrationTypePreSonoma {
-			// Launch the "My device" page.
+		if (previousMigrationType == constant.MDMMigrationTypeManual || previousMigrationType == constant.MDMMigrationTypePreSonoma) && m.props.IsUnmanaged {
+			// Launch the "My device" page, only if the device is marked as unmanaged else keep trying to unenroll
 			log.Info().Msg("showing instructions")
 
 			if err := m.handler.ShowInstructions(); err != nil {
@@ -818,8 +844,8 @@ func (o *offlineWatcher) showSwiftDialogMDMMigrationOffline(ctx context.Context)
 }
 
 type swiftDialogMDMMigrationOffline struct {
-	*baseDialog
-	props MDMMigratorProps
+	baseDialog dialog
+	props      MDMMigratorProps
 }
 
 func (m *swiftDialogMDMMigrationOffline) render(flags ...string) (chan swiftDialogExitCode, chan error) {

orbit/pkg/useraction/mdm_migration_darwin_test.go

@@ -5,24 +5,86 @@ import (
 	"testing"
 	"time"
 
+	"github.com/fleetdm/fleet/v4/orbit/pkg/constant"
 	"github.com/stretchr/testify/require"
 )
 
-type dummyHandler struct{}
+// mockDialog is a mock implementation of the dialog interface for testing.
+type mockDialog struct {
+	exitCh chan int // exit code
+}
 
-func (d dummyHandler) NotifyRemote() error {
+func (d *mockDialog) CanRun() bool {
+	return true
+}
+
+func (d *mockDialog) Exit() {
+	select {
+	case d.exitCh <- unknownExitCode:
+	default:
+	}
+}
+
+func (d *mockDialog) exitWithCode(code int) {
+	select {
+	case d.exitCh <- code:
+	default:
+	}
+}
+
+func (d *mockDialog) render(flags ...string) (chan swiftDialogExitCode, chan error) {
+	exitCodeCh := make(chan swiftDialogExitCode, 1)
+	errCh := make(chan error, 1)
+	go func() {
+		select {
+		case code := <-d.exitCh:
+			exitCodeCh <- swiftDialogExitCode(code)
+		case <-time.After(15 * time.Second):
+			errCh <- errors.New("timeout waiting for mock dialog to exit")
+		}
+	}()
+	return exitCodeCh, errCh
+}
+
+// mockReadWriter is a mock implementation of the readWriter interface for testing.
+type mockReadWriter struct {
+	migrationType string
+}
+
+func (rw *mockReadWriter) GetMigrationType() (string, error) {
+	return rw.migrationType, nil
+}
+
+func (rw *mockReadWriter) SetMigrationFile(typ string) error {
+	rw.migrationType = typ
+	return nil
+}
+
+func (rw *mockReadWriter) RemoveFile() error {
+	rw.migrationType = ""
+	return nil
+}
+
+type dummyHandler struct {
+	TimeCalled int
+}
+
+func (d *dummyHandler) NotifyRemote() error {
+	d.TimeCalled++
 	return nil
 }
 
 func (d dummyHandler) ShowInstructions() error { return nil }
 
 func TestWaitForUnenrollment(t *testing.T) {
-	t.Parallel()
-	m := &swiftDialogMDMMigrator{
-		handler:                   dummyHandler{},
-		baseDialog:                newBaseDialog("foo/bar"),
-		frequency:                 15 * time.Minute,
-		unenrollmentRetryInterval: 300 * time.Millisecond,
+	getMigratorInstance := func() *swiftDialogMDMMigrator {
+		return &swiftDialogMDMMigrator{
+			handler:                   &dummyHandler{},
+			baseDialog:                newBaseDialog("foo/bar"),
+			frequency:                 15 * time.Minute,
+			unenrollmentRetryInterval: 1 * time.Millisecond,
+			maxUnenrollmentWaitTime:   1 * time.Second,
+		}
 	}
 
 	cases := []struct {
@@ -39,6 +101,8 @@ func TestWaitForUnenrollment(t *testing.T) {
 
 	for _, c := range cases {
 		t.Run(c.name, func(t *testing.T) {
+			t.Parallel()
+			m := getMigratorInstance()
 			tries := 0
 			m.testEnrollmentCheckFileFn = func() (bool, error) {
 				if tries >= c.unenrollAfterNTries {
@@ -63,6 +127,8 @@ func TestWaitForUnenrollment(t *testing.T) {
 	}
 
 	t.Run("fallback to enrollment check file", func(t *testing.T) {
+		t.Parallel()
+		m := getMigratorInstance()
 		m.testEnrollmentCheckFileFn = func() (bool, error) {
 			return true, nil
 		}
@@ -76,6 +142,8 @@ func TestWaitForUnenrollment(t *testing.T) {
 	})
 
 	t.Run("only check file during ADE enrollment", func(t *testing.T) {
+		t.Parallel()
+		m := getMigratorInstance()
 		var fileWasChecked bool
 		m.testEnrollmentCheckFileFn = func() (bool, error) {
 			fileWasChecked = true
@@ -95,3 +163,102 @@ func TestWaitForUnenrollment(t *testing.T) {
 		require.True(t, fileWasChecked)
 	})
 }
+
+func TestShouldSendWebhookUntilUnmanaged(t *testing.T) {
+	for _, typ := range []string{constant.MDMMigrationTypeADE, constant.MDMMigrationTypeManual, constant.MDMMigrationTypePreSonoma} {
+		t.Run(typ, func(t *testing.T) {
+			t.Parallel()
+			handler := &dummyHandler{}
+			mockDialog := &mockDialog{exitCh: make(chan int, 10)}
+			m := &swiftDialogMDMMigrator{
+				handler:                   handler,
+				mrw:                       &mockReadWriter{},
+				baseDialog:                mockDialog,
+				frequency:                 15 * time.Minute,
+				unenrollmentRetryInterval: 50 * time.Millisecond,
+				maxUnenrollmentWaitTime:   100 * time.Millisecond,
+				props: MDMMigratorProps{
+					IsUnmanaged: false,
+				},
+			}
+
+			// Set up enrollment check functions - device stays enrolled throughout
+			m.testEnrollmentCheckFileFn = func() (bool, error) {
+				return true, nil // Always enrolled (file exists)
+			}
+
+			m.testEnrollmentCheckStatusFn = func() (bool, string, error) {
+				return true, "example.com", nil // Always enrolled
+			}
+
+			// First migration attempt - should call webhook and see device never unenrolls for unenrollment
+			mockDialog.exitWithCode(0) // Start button clicked
+			mockDialog.exitWithCode(0) // Error ok? clicked
+			err := m.renderMigration()
+
+			// Should get host is still enrolled error
+			require.Error(t, err)
+			require.Contains(t, err.Error(), "host didn't unenroll from MDM") // This is okay
+			require.Equal(t, 1, handler.TimeCalled)
+
+			// We fake the migration file being set even though an error returned to simulate this weird state
+			err = m.mrw.SetMigrationFile(typ)
+			require.NoError(t, err)
+
+			// Second migration attempt - device is still managed, should call webhook again
+			mockDialog.exitWithCode(0)
+			mockDialog.exitWithCode(0)
+			err = m.renderMigration()
+
+			// Should still get not unenrolled error
+			require.Error(t, err)
+			require.Contains(t, err.Error(), "host didn't unenroll from MDM")
+			require.Equal(t, 2, handler.TimeCalled) // webhook was still called
+
+			// Now we let it unenroll the device, and then simulate the ping for IsUnmanaged
+			fileTries := 0
+			statusTries := 0
+			m.testEnrollmentCheckFileFn = func() (bool, error) {
+				fileTries++
+				if fileTries > 1 { // Unenroll after 2nd try
+					return false, nil
+				}
+				return true, nil
+			}
+
+			m.testEnrollmentCheckStatusFn = func() (bool, string, error) {
+				statusTries++
+				if statusTries > 1 {
+					return false, "", nil // Not enrolled
+				}
+				return true, "example.com", nil
+			}
+
+			go func() {
+				// start button click
+				time.Sleep(10 * time.Millisecond)
+				mockDialog.exitWithCode(0)
+
+				// There is a loading spinner that takes over the exit call, so we need to call it ourselves again.
+				time.Sleep(100 * time.Millisecond)
+				mockDialog.Exit()
+			}()
+			err = m.renderMigration()
+
+			// Now it successfully unenrolls
+			require.NoError(t, err)
+			require.Equal(t, 3, handler.TimeCalled) // webhook was called again.
+
+			// Device is now seen as unmanaged by Fleet server
+			m.props.IsUnmanaged = true
+
+			// This simulates our runner that periodically shows the window - should NOT call webhook since device is unmanaged, it will hit the early exit
+			mockDialog.exitWithCode(0)
+			err = m.renderMigration()
+
+			// Should succeed without error
+			require.NoError(t, err)
+			require.Equal(t, 3, handler.TimeCalled) // webhook was not called
+		})
+	}
+}