Migration playbook · v2.0Target: Google Managed Service for Apache Kafka

Cloudera Kafka to Google Managed Kafka

An 11-week, rolling cutover from Cloudera-managed Kafka (CDP/HDP) to a Google-managed cluster, replicating with MirrorMaker 2.0 and moving consumers before producers. This version replaces the self-managed Apache Kafka target in v1.0.

Against v1.0

What changes when Google runs the brokers

The v1.0 plan spent most of its effort building and operating a KRaft cluster. Here that work disappears, but several things you could do yourself are no longer possible.

Handled by the service

  • ZooKeeper to KRaft migration The service runs KRaft only; ZooKeeper mode isn't supported.
  • Provisioning brokers and controllers You set vCPUs and memory; Google sizes brokers.
  • Cruise Control Rebalancing is built into the service.
  • Broker tuning Most broker configs are service-managed.
  • JMX Exporter sidecars JMX isn't supported; metrics go to Cloud Monitoring.

Replaced

  • SCRAM-SHA-512 SASL/OAUTHBEARER (recommended) or mTLS.
  • Kafka ACLs + OPA IAM roles plus Kafka ACLs.
  • Self-run MirrorMaker 2 MirrorMaker 2.0 connectors on a managed Connect cluster.
  • Confluent / Apicurio Schema Registry Google's Schema Registry (Preview).
  • Prometheus, ELK / Loki Cloud Monitoring and Cloud Logging.

New constraints

  • Schemas: Avro and Protobuf only. No JSON schemas.
  • Authentication: Kerberos (GSSAPI) and SCRAM aren't supported.
  • Three zones, equal resources, no single-zone option.
  • Local disk is fixed at 100 GiB or more per vCPU, not configurable.
  • Connect ships nine built-in connector types.
  • ksqlDB isn't provided; you run it yourself or replace it.
Architecture

Source, bridge and target

Replication runs inside Google Cloud: a Connect cluster in the target region pulls from Cloudera over Cloud VPN or Cloud Interconnect.

Source · Cloudera CDP / HDP

  • Brokers managed by Cloudera Manager
  • ZooKeeper
  • Cloudera Schema Registry
  • Kafka Connect and KSQL
  • Ranger / Sentry authorization
  • Atlas lineage

Bridge · temporary

  • Cloud VPN or Cloud InterconnectInterconnect for high throughput
  • Connect cluster, same region as the targetbilled per DCU while it runs
  • MirrorMaker 2.0 Sourcetopics, partitions, topic ACLs
  • MirrorMaker 2.0 Checkpointconsumer group offsets
  • MirrorMaker 2.0 Heartbeat
  • Consumer lag monitor

Target · Google Managed Kafka

  • Cluster across 3 zonesKRaft · Apache Kafka 3.7.1
  • Private Service Connect endpointscreated when you pick a subnet
  • Schema Registry (Preview)Avro, Protobuf
  • IAM + Kafka ACLsOAUTHBEARER or mTLS
  • Cloud Monitoring, Cloud Logging
  • Built-in Connect sinksBigQuery, Cloud Storage, Pub/Sub

Around itFlink or Dataflow for stream processing, BigQuery and Cloud Storage sinks, OpenMetadata or DataHub for lineage, Kafka UI or AKHQ as clients, Terraform for clusters, topics, Connect and ACLs.

Timeline

Eleven weeks, five phases

Order carries meaning here: nothing moves until replication has been shadowed, and Cloudera stays alive until the stabilization window closes.

Hatched: both clusters live and replicating. Solid: traffic moves. Cloudera remains running until day 14 after the last cutover.

Phases

What happens in each phase

Weeks 1–2Discovery & baseline
  • Inventory topics (partitions, replication factor, configs), consumer groups, schemas, Ranger policies, connectors and KSQL queries.
  • Measure BytesInPerSec and BytesOutPerSec from broker JMX on the source and sum across brokers. Google's sizing rule is write-equivalent rate = write bandwidth + (read bandwidth / 4).
  • Confirm the source runs Kafka 2.4.0 or later, which MirrorMaker 2.0 requires.
  • Flag blockers: JSON schemas, clients that only speak Kerberos or SCRAM, plaintext listeners, connectors outside the nine built-in types, topics with fewer than three replicas.
  • Pick the network path: Cloud VPN for trials or modest bandwidth, Cloud Interconnect for sustained throughput.

GateEvery blocker has an owner and a replacement decision.

Weeks 3–4Target build
  • Create the cluster in the region closest to your workloads: at least 3 vCPUs, 1 to 8 GiB of memory per vCPU. Selecting a subnet makes the service create the Private Service Connect endpoints for you.
  • Create a Connect cluster in the same region with the new cluster as its primary.
  • Create one service account per workload and grant IAM roles. Write Kafka ACLs before any client connects: with no ACLs, every authenticated principal has full cluster access.
  • Stand up Schema Registry and register Avro and Protobuf schemas. Check compatibility against your existing subject naming.
  • Build Cloud Monitoring alerts and Cloud Logging queries, and define the cluster, topics, Connect and ACLs in Terraform.
  • Deploy OpenMetadata or DataHub and start the Atlas metadata export.

GateA test client authenticates with its own identity, and is denied what its ACLs don't grant.

Weeks 5–6Replication & shadowing
  • Create the MirrorMaker 2.0 Source connector (it creates missing topics and partitions and syncs topic ACLs), the Checkpoint connector with sync.group.offsets.enabled=true, and the Heartbeat connector.
  • Tune offset.lag.max, emit.checkpoints.interval.seconds and sync.group.offsets.interval.seconds. Offset translation is only as precise as these intervals and your consumers' auto.commit.interval.ms.
  • Run shadow consumers for at least 72 hours and compare message parity topic by topic.
  • Rehearse failover for each consumer group in a development environment first.
  • Rewrite KSQL queries in Flink or Dataflow, or run ksqlDB yourself as a client of the new cluster.
  • Review the synced ACLs: they come from source principals, which won't match IAM principals.

Gate72 hours of parity with no unexplained offset drift.

Weeks 7–8Cutover
  • Move consumers first, non-critical before critical, then producers.
  • Switch client authentication and bootstrap servers through a config map or environment variable. Connections must be encrypted and authenticated; plaintext and unauthenticated access isn't allowed.
  • While both clusters are live, disable side effects that must happen once per message, to avoid double processing.
  • Optionally put a dual-write interceptor on critical producers during their switch.
  • After validation, delete the MirrorMaker connectors and update DNS and client configs.

GateAll producers write to the new cluster and lag is flat for 48 hours.

Weeks 9–11Stabilize & decommission
  • Monitor for two weeks: consumer lag, under-replicated partitions, CPU and memory, and Cloud Logging errors.
  • Load test at 120% of peak production throughput. Resize vCPUs and memory if needed; the service checks that each broker can carry its share before resizing.
  • Train the ops team on the console, gcloud managed-kafka, Cloud Monitoring and IAM.
  • Decommission Cloudera at day 14, then export the final Atlas metadata.

GateCloudera is switched off and no client references it.

Component mapping

Where each Cloudera piece goes

Effort is my estimate for a typical estate, from none (▯▯▯) to high (▮▮▮). Rows marked with a dotted underline depend on a decision in the open items list.

ClouderaOn Google Managed KafkaEffort
ZooKeeperKRaft, run by the service. Nothing to migrate.▯▯▯ None
Cloudera ManagerGoogle Cloud console, gcloud managed-kafka, Terraform and Cloud Monitoring.▮▯▯ Low
Cruise ControlBuilt-in rebalancing (an --auto-rebalance option on the cluster).▯▯▯ None
Cloudera Replication ManagerMirrorMaker 2.0 Source, Checkpoint and Heartbeat connectors on a managed Connect cluster.▮▯▯ Low
Kerberos authenticationSASL/OAUTHBEARER with service accounts, Workload Identity Federation for clients outside Google Cloud, or mTLS. GSSAPI and SCRAM are unsupported.▮▮▯ Medium to high
Ranger / SentryIAM role bindings plus Kafka ACLs. Tag-based policies driven by Atlas classifications have no direct equivalent.▮▮▯ Medium to high
Cloudera Schema RegistryGoogle Schema Registry (Preview) with the Confluent REST API. Avro and Protobuf only.▮▮▯ Medium, high with JSON schemas
Kafka ConnectManaged Connect with nine built-in connector types. Custom plugins and SMTs aren't confirmed.▮▮▯ Medium
KSQLNot provided. Rewrite in Flink or Dataflow, or self-host ksqlDB as a client.▮▮▯ Medium
Metrics and alertsCloud Monitoring and Cloud Logging. JMX isn't available, so exporters can't scrape brokers.▮▯▯ Low
Atlas lineageOpenMetadata or DataHub. The Google documentation reviewed describes no native Kafka topic lineage.▮▮▮ High
Risk register

What can go wrong

High

Consumer offset desync at cutover

Run the Checkpoint connector with sync.group.offsets.enabled=true, shadow for 72 hours or more, and rehearse each group's failover.

Clients can't authenticate

Kerberos and SCRAM clients have no equivalent mechanism.

Inventory every client's mechanism in week 1. Move to OAUTHBEARER, or to mTLS (port 9192, certificates issued through CA Service) for clients outside Google Cloud.

Authorization gaps

Ranger policies don't map one to one, and a cluster with no ACLs gives every authenticated principal full access.

Write ACLs before clients connect, and test deny cases. Keep any tag-based rules as an explicit list to rebuild.

Schema incompatibility

JSON schemas are unsupported, the registry is in Preview, and subject naming strategies may differ.

Find JSON-schema subjects in discovery. Decide early whether to convert them or keep a self-hosted registry for them.

Medium

Connector and SMT dependencies

Only nine connector types are built in.

List every connector class and custom SMT. Confirm custom plugin support with Google, or run Connect yourself against the cluster.

Source topics below three replicas

Default topic settings require at least three replicas, with a minimum of two in sync.

Review source replication factors before MirrorMaker creates the topics.

Less control over the brokers

Local disk size is fixed, and read-only broker configs only change at Google's maintenance restarts.

Compare your tuned configs with the changeable list. Size by vCPU and memory, not by disk.

No protection from regional failure

The service doesn't cover regional or dual-zone failures.

If your SLA needs it, plan a second cluster in another region, and replicate with the same MirrorMaker approach.

Authentication request quota

The default is 1,200 connection authentication requests per minute, per project and region.

Check for fleets of short-lived clients, and request a higher quota before cutover if needed.

Low

Operational unfamiliarity

Smaller than in v1.0: there are no KRaft controllers or broker tuning to learn.

Train on IAM, ACLs and Cloud Monitoring instead.

Commands & configs

Starting points

The cluster command and PLAIN settings are as Google documents them; replace the placeholders. Check the full OAUTHBEARER class name in Google's client auth readme.

Create the cluster
gcloud managed-kafka clusters create CLUSTER_ID \
  --location=LOCATION \
  --cpu=CPU_COUNT \
  --memory=MEMORY \
  --subnets=projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET_ID \
  --auto-rebalance \
  --async

At least 3 vCPUs and 1 to 8 GiB per vCPU. --encryption-key is optional.

Client: OAUTHBEARER (Java)
security.protocol=SASL_SSL
sasl.mechanism=OAUTHBEARER
sasl.login.callback.handler.class=\
  GcpLoginCallbackHandler
sasl.jaas.config=org.apache.kafka.common.\
  security.oauthbearer.OAuthBearerLoginModule \
  required;

Uses Application Default Credentials. Non-Java clients get tokens from a local authentication server.

Client: PLAIN (development only)
security.protocol=SASL_SSL
sasl.mechanism=PLAIN
sasl.jaas.config=org.apache.kafka.common.\
  security.plain.PlainLoginModule required \
  username="SERVICE_ACCOUNT_EMAIL" \
  password="BASE64_ENCODED_KEY_JSON";

Google discourages long-lived keys for production.

MirrorMaker 2.0 properties
# Source connector
offset.lag.max

# Checkpoint connector
sync.group.offsets.enabled=true
emit.checkpoints.interval.seconds
sync.group.offsets.interval.seconds

# Consumers
auto.commit.interval.ms

Shorter intervals give more precise offset translation, at the cost of more writes.

Cost & limits

What you pay during and after

Prices are for us-central1. During the overlap you pay for both clusters, plus the Connect cluster that carries the replication.

$0.09
per DCU-hour
1 DCU = 1 vCPU + 4 GiB
$0.12
per DCU-hour for Connect
no committed use discount
$0.01
per GiB inter-zone replication
20% / 40%
committed use discount
1 year / 3 years
5
clusters per project and region by default
750
broker vCPUs per project and region by default
10 MiB
default max message size
100 GiB
minimum local disk per vCPU, billed regardless of use
Rollback

Getting back to Cloudera

Keep the Cloudera cluster running until day 14 and keep every client's bootstrap servers and credentials behind a config map or environment variable, so a switch back is a configuration change rather than a deploy.

Open items

Settle these before week 1

Things I couldn't confirm from Google's documentation and which change the plan if the answer is no.

  1. Reverse replication. Can a managed Connect cluster write back to an external Kafka cluster for rollback? If not, run MirrorMaker 2 yourself for that direction.
  2. Custom connectors and SMTs. Does managed Connect accept your own plugins? The documentation lists nine built-in types.
  3. Schema Registry status. It was still Preview at the last documentation update. Decide whether Preview terms are acceptable for production.
  4. ACL principal mapping. How the Source connector's synced topic ACLs behave when source principals don't exist in IAM.
  5. Ranger tag policies. Which of your rules depend on Atlas classifications, and how you'll express them with IAM and ACLs.
  6. Kafka version. Clusters run 3.7.1 and minor upgrades aren't automatic. Confirm every client library works against it.
Sources

Where this comes from