What changes when Google runs the brokers
The v1.0 plan spent most of its effort building and operating a KRaft cluster. Here that work disappears, but several things you could do yourself are no longer possible.
Handled by the service
ZooKeeper to KRaft migrationThe service runs KRaft only; ZooKeeper mode isn't supported.Provisioning brokers and controllersYou set vCPUs and memory; Google sizes brokers.Cruise ControlRebalancing is built into the service.Broker tuningMost broker configs are service-managed.JMX Exporter sidecarsJMX isn't supported; metrics go to Cloud Monitoring.
Replaced
SCRAM-SHA-512SASL/OAUTHBEARER (recommended) or mTLS.Kafka ACLs + OPAIAM roles plus Kafka ACLs.Self-run MirrorMaker 2MirrorMaker 2.0 connectors on a managed Connect cluster.Confluent / Apicurio Schema RegistryGoogle's Schema Registry (Preview).Prometheus, ELK / LokiCloud Monitoring and Cloud Logging.
New constraints
- Schemas: Avro and Protobuf only. No JSON schemas.
- Authentication: Kerberos (GSSAPI) and SCRAM aren't supported.
- Three zones, equal resources, no single-zone option.
- Local disk is fixed at 100 GiB or more per vCPU, not configurable.
- Connect ships nine built-in connector types.
- ksqlDB isn't provided; you run it yourself or replace it.
Source, bridge and target
Replication runs inside Google Cloud: a Connect cluster in the target region pulls from Cloudera over Cloud VPN or Cloud Interconnect.
Source · Cloudera CDP / HDP
- Brokers managed by Cloudera Manager
- ZooKeeper
- Cloudera Schema Registry
- Kafka Connect and KSQL
- Ranger / Sentry authorization
- Atlas lineage
Bridge · temporary
- Cloud VPN or Cloud InterconnectInterconnect for high throughput
- Connect cluster, same region as the targetbilled per DCU while it runs
- MirrorMaker 2.0 Sourcetopics, partitions, topic ACLs
- MirrorMaker 2.0 Checkpointconsumer group offsets
- MirrorMaker 2.0 Heartbeat
- Consumer lag monitor
Target · Google Managed Kafka
- Cluster across 3 zonesKRaft · Apache Kafka 3.7.1
- Private Service Connect endpointscreated when you pick a subnet
- Schema Registry (Preview)Avro, Protobuf
- IAM + Kafka ACLsOAUTHBEARER or mTLS
- Cloud Monitoring, Cloud Logging
- Built-in Connect sinksBigQuery, Cloud Storage, Pub/Sub
Around itFlink or Dataflow for stream processing, BigQuery and Cloud Storage sinks, OpenMetadata or DataHub for lineage, Kafka UI or AKHQ as clients, Terraform for clusters, topics, Connect and ACLs.
Eleven weeks, five phases
Order carries meaning here: nothing moves until replication has been shadowed, and Cloudera stays alive until the stabilization window closes.
Hatched: both clusters live and replicating. Solid: traffic moves. Cloudera remains running until day 14 after the last cutover.
What happens in each phase
- Inventory topics (partitions, replication factor, configs), consumer groups, schemas, Ranger policies, connectors and KSQL queries.
- Measure
BytesInPerSecandBytesOutPerSecfrom broker JMX on the source and sum across brokers. Google's sizing rule is write-equivalent rate = write bandwidth + (read bandwidth / 4). - Confirm the source runs Kafka 2.4.0 or later, which MirrorMaker 2.0 requires.
- Flag blockers: JSON schemas, clients that only speak Kerberos or SCRAM, plaintext listeners, connectors outside the nine built-in types, topics with fewer than three replicas.
- Pick the network path: Cloud VPN for trials or modest bandwidth, Cloud Interconnect for sustained throughput.
GateEvery blocker has an owner and a replacement decision.
- Create the cluster in the region closest to your workloads: at least 3 vCPUs, 1 to 8 GiB of memory per vCPU. Selecting a subnet makes the service create the Private Service Connect endpoints for you.
- Create a Connect cluster in the same region with the new cluster as its primary.
- Create one service account per workload and grant IAM roles. Write Kafka ACLs before any client connects: with no ACLs, every authenticated principal has full cluster access.
- Stand up Schema Registry and register Avro and Protobuf schemas. Check compatibility against your existing subject naming.
- Build Cloud Monitoring alerts and Cloud Logging queries, and define the cluster, topics, Connect and ACLs in Terraform.
- Deploy OpenMetadata or DataHub and start the Atlas metadata export.
GateA test client authenticates with its own identity, and is denied what its ACLs don't grant.
- Create the MirrorMaker 2.0 Source connector (it creates missing topics and partitions and syncs topic ACLs), the Checkpoint connector with
sync.group.offsets.enabled=true, and the Heartbeat connector. - Tune
offset.lag.max,emit.checkpoints.interval.secondsandsync.group.offsets.interval.seconds. Offset translation is only as precise as these intervals and your consumers'auto.commit.interval.ms. - Run shadow consumers for at least 72 hours and compare message parity topic by topic.
- Rehearse failover for each consumer group in a development environment first.
- Rewrite KSQL queries in Flink or Dataflow, or run ksqlDB yourself as a client of the new cluster.
- Review the synced ACLs: they come from source principals, which won't match IAM principals.
Gate72 hours of parity with no unexplained offset drift.
- Move consumers first, non-critical before critical, then producers.
- Switch client authentication and bootstrap servers through a config map or environment variable. Connections must be encrypted and authenticated; plaintext and unauthenticated access isn't allowed.
- While both clusters are live, disable side effects that must happen once per message, to avoid double processing.
- Optionally put a dual-write interceptor on critical producers during their switch.
- After validation, delete the MirrorMaker connectors and update DNS and client configs.
GateAll producers write to the new cluster and lag is flat for 48 hours.
- Monitor for two weeks: consumer lag, under-replicated partitions, CPU and memory, and Cloud Logging errors.
- Load test at 120% of peak production throughput. Resize vCPUs and memory if needed; the service checks that each broker can carry its share before resizing.
- Train the ops team on the console,
gcloud managed-kafka, Cloud Monitoring and IAM. - Decommission Cloudera at day 14, then export the final Atlas metadata.
GateCloudera is switched off and no client references it.
Where each Cloudera piece goes
Effort is my estimate for a typical estate, from none (▯▯▯) to high (▮▮▮). Rows marked with a dotted underline depend on a decision in the open items list.
| Cloudera | On Google Managed Kafka | Effort |
|---|---|---|
| ZooKeeper | KRaft, run by the service. Nothing to migrate. | ▯▯▯ None |
| Cloudera Manager | Google Cloud console, gcloud managed-kafka, Terraform and Cloud Monitoring. | ▮▯▯ Low |
| Cruise Control | Built-in rebalancing (an --auto-rebalance option on the cluster). | ▯▯▯ None |
| Cloudera Replication Manager | MirrorMaker 2.0 Source, Checkpoint and Heartbeat connectors on a managed Connect cluster. | ▮▯▯ Low |
| Kerberos authentication | SASL/OAUTHBEARER with service accounts, Workload Identity Federation for clients outside Google Cloud, or mTLS. GSSAPI and SCRAM are unsupported. | ▮▮▯ Medium to high |
| Ranger / Sentry | IAM role bindings plus Kafka ACLs. Tag-based policies driven by Atlas classifications have no direct equivalent. | ▮▮▯ Medium to high |
| Cloudera Schema Registry | Google Schema Registry (Preview) with the Confluent REST API. Avro and Protobuf only. | ▮▮▯ Medium, high with JSON schemas |
| Kafka Connect | Managed Connect with nine built-in connector types. Custom plugins and SMTs aren't confirmed. | ▮▮▯ Medium |
| KSQL | Not provided. Rewrite in Flink or Dataflow, or self-host ksqlDB as a client. | ▮▮▯ Medium |
| Metrics and alerts | Cloud Monitoring and Cloud Logging. JMX isn't available, so exporters can't scrape brokers. | ▮▯▯ Low |
| Atlas lineage | OpenMetadata or DataHub. The Google documentation reviewed describes no native Kafka topic lineage. | ▮▮▮ High |
What can go wrong
High
Run the Checkpoint connector with sync.group.offsets.enabled=true, shadow for 72 hours or more, and rehearse each group's failover.
Kerberos and SCRAM clients have no equivalent mechanism.
Inventory every client's mechanism in week 1. Move to OAUTHBEARER, or to mTLS (port 9192, certificates issued through CA Service) for clients outside Google Cloud.
Ranger policies don't map one to one, and a cluster with no ACLs gives every authenticated principal full access.
Write ACLs before clients connect, and test deny cases. Keep any tag-based rules as an explicit list to rebuild.
JSON schemas are unsupported, the registry is in Preview, and subject naming strategies may differ.
Find JSON-schema subjects in discovery. Decide early whether to convert them or keep a self-hosted registry for them.
Medium
Only nine connector types are built in.
List every connector class and custom SMT. Confirm custom plugin support with Google, or run Connect yourself against the cluster.
Default topic settings require at least three replicas, with a minimum of two in sync.
Review source replication factors before MirrorMaker creates the topics.
Local disk size is fixed, and read-only broker configs only change at Google's maintenance restarts.
Compare your tuned configs with the changeable list. Size by vCPU and memory, not by disk.
The service doesn't cover regional or dual-zone failures.
If your SLA needs it, plan a second cluster in another region, and replicate with the same MirrorMaker approach.
The default is 1,200 connection authentication requests per minute, per project and region.
Check for fleets of short-lived clients, and request a higher quota before cutover if needed.
Low
Smaller than in v1.0: there are no KRaft controllers or broker tuning to learn.
Train on IAM, ACLs and Cloud Monitoring instead.
Starting points
The cluster command and PLAIN settings are as Google documents them; replace the placeholders. Check the full OAUTHBEARER class name in Google's client auth readme.
gcloud managed-kafka clusters create CLUSTER_ID \ --location=LOCATION \ --cpu=CPU_COUNT \ --memory=MEMORY \ --subnets=projects/PROJECT_ID/regions/LOCATION/subnetworks/SUBNET_ID \ --auto-rebalance \ --async
At least 3 vCPUs and 1 to 8 GiB per vCPU. --encryption-key is optional.
security.protocol=SASL_SSL sasl.mechanism=OAUTHBEARER sasl.login.callback.handler.class=\ GcpLoginCallbackHandler sasl.jaas.config=org.apache.kafka.common.\ security.oauthbearer.OAuthBearerLoginModule \ required;
Uses Application Default Credentials. Non-Java clients get tokens from a local authentication server.
security.protocol=SASL_SSL sasl.mechanism=PLAIN sasl.jaas.config=org.apache.kafka.common.\ security.plain.PlainLoginModule required \ username="SERVICE_ACCOUNT_EMAIL" \ password="BASE64_ENCODED_KEY_JSON";
Google discourages long-lived keys for production.
# Source connector offset.lag.max # Checkpoint connector sync.group.offsets.enabled=true emit.checkpoints.interval.seconds sync.group.offsets.interval.seconds # Consumers auto.commit.interval.ms
Shorter intervals give more precise offset translation, at the cost of more writes.
What you pay during and after
Prices are for us-central1. During the overlap you pay for both clusters, plus the Connect cluster that carries the replication.
1 DCU = 1 vCPU + 4 GiB
no committed use discount
1 year / 3 years
Getting back to Cloudera
Keep the Cloudera cluster running until day 14 and keep every client's bootstrap servers and credentials behind a config map or environment variable, so a switch back is a configuration change rather than a deploy.
- Target: reverse the config switch for any client within 15 minutes.
- Data written after cutover: v1.0 relied on reverse MirrorMaker 2 replication. Google's guide documents only the inbound direction, so prove a reverse path in the dry run before you depend on it (see open items).
- Alternative: for critical producers, keep dual writes through the stabilization window so the old cluster never falls behind.
Settle these before week 1
Things I couldn't confirm from Google's documentation and which change the plan if the answer is no.
- Reverse replication. Can a managed Connect cluster write back to an external Kafka cluster for rollback? If not, run MirrorMaker 2 yourself for that direction.
- Custom connectors and SMTs. Does managed Connect accept your own plugins? The documentation lists nine built-in types.
- Schema Registry status. It was still Preview at the last documentation update. Decide whether Preview terms are acceptable for production.
- ACL principal mapping. How the Source connector's synced topic ACLs behave when source principals don't exist in IAM.
- Ranger tag policies. Which of your rules depend on Atlas classifications, and how you'll express them with IAM and ACLs.
- Kafka version. Clusters run 3.7.1 and minor upgrades aren't automatic. Confirm every client library works against it.