Moving a data-science team from Azure VMs to Kubernetes and MLflow on two clouds

The team had 18 data scientists working on hand-built virtual machines in Azure. Environments, data paths, experiment history and deployment steps were tied to specific machines and to the people who'd set them up. A new environment took 2 days. Getting a model released took 10.

We moved them to a platform built on Kubernetes and MLflow, running in both Azure and Google Cloud, with one supported way to request an environment, run an experiment and release a model. What mattered was that any scientist could rerun any workload without the person who'd built its VM.

Pulling a contract out of the VMs

We started by inventorying what produced each model: dependencies, local datasets, scheduled jobs and the feature code. Each workload became an immutable image, a dataset manifest, a source revision and a declared resource request. Notebooks stayed, but as consumers of shared code rather than a second production implementation that slowly drifted.

The first move was one representative workload on AKS, with the original VM path left running. We compared outputs within a numerical tolerance the workload owner agreed to, and retired the VM only when that owner accepted the replacement. Nobody promised faster training. The win was not depending on one particular machine.

The platform as the ordinary path

Scientists asked the platform for an environment or a run, picked an approved cloud and watched its status. Behind that, the platform checked ownership, budget and identity, then submitted a standard Kubernetes Job with resource requests, a quota, a deadline and bounded retries.

Notebooks, CPU data preparation, GPU training and serving each had their own resource policy. The status view counted completed work, not accepted requests or pending pods.

Two clouds that don't share storage

MLflow holds run metadata, evaluation results, lineage and model versions. We kept tracking data and artifacts in the cloud where each run executed, and the platform's catalog recorded where every run lived and which digest had been approved.

Moving a model across clouds was an explicit, authorized copy: verify the bytes, check access on the destination, account for the egress. A model version was identified by its registry location and digest, never by a bare version number or a mutable alias.

That's the decision I'd defend hardest. A single shared artifact store spanning both clouds would have looked simpler on a diagram. It would also have hidden transfer costs, residency questions and a cross-cloud dependency inside every training run.

Identity without long-lived keys

AKS workloads used Microsoft Entra Workload ID and GKE workloads used Workload Identity Federation, so nobody handed out durable cloud keys. Permissions were scoped to the project and storage each job needed. A person's login, the cluster-management identity and a job's data-access identity were kept apart.

A registry entry didn't authorize a release on its own. Promotion pinned an exact model version, image, feature schema and evaluation report, and the previous compatible release stayed ready to restore. We tested revoked identities and wrong-project access before trusting any of it.

Results

The cost drop is $75K a month, or $900K a year, at comparable workload. The transition had a $180K budget, which pays back in 2.4 months on simple arithmetic, before commitments and finance adjustments.

The 18 scientists got back 194.85 hours a month that had gone into maintaining their own machines. At $110 an hour that's $21,433.50 of capacity. At 12 promotions a month, cutting manual release work from 8 hours to 45 minutes freed another 87 hours. Those two overlap somewhat, so I wouldn't add them together.

Environment lead time is waiting time, not labor. Going from 2 days to 20 minutes doesn't mean 2 days of work saved per request. It means a scientist who asks after lunch is running experiments the same afternoon.

What it cost

Two execution domains need two sets of support, validation and runbooks, and we paid for that. Cross-cloud promotion became an explicit copy-and-verify step, which is slower than repointing an alias. Evaluation and an accountable approval also stayed on the release path, because a training run that finished isn't proof a model is fit to ship.

The team got one familiar way to work, and each cloud kept its own identity, data and quota boundaries instead of having them papered over.

Client name and identifying details are left out.