diff --git a/assets/images/cfk/create-Kafka-Advance-configuration.png b/assets/images/cfk/create-Kafka-Advance-configuration.png new file mode 100644 index 0000000..759fdbd Binary files /dev/null and b/assets/images/cfk/create-Kafka-Advance-configuration.png differ diff --git a/assets/images/cfk/create-Kafka-Basic-configuratio.png b/assets/images/cfk/create-Kafka-Basic-configuratio.png new file mode 100644 index 0000000..53c8060 Binary files /dev/null and b/assets/images/cfk/create-Kafka-Basic-configuratio.png differ diff --git a/assets/images/cfk/create-Kafka-ClusterTypeSelection.png b/assets/images/cfk/create-Kafka-ClusterTypeSelection.png new file mode 100644 index 0000000..024afc5 Binary files /dev/null and b/assets/images/cfk/create-Kafka-ClusterTypeSelection.png differ diff --git a/assets/images/cfk/create-Kafka-Fleet-Selection.png b/assets/images/cfk/create-Kafka-Fleet-Selection.png new file mode 100644 index 0000000..89cb74c Binary files /dev/null and b/assets/images/cfk/create-Kafka-Fleet-Selection.png differ diff --git a/assets/images/oci/basic-configuration-oci.png b/assets/images/oci/basic-configuration-oci.png new file mode 100644 index 0000000..8c36166 Binary files /dev/null and b/assets/images/oci/basic-configuration-oci.png differ diff --git a/assets/images/oci/bootstrap-provider-selection.png b/assets/images/oci/bootstrap-provider-selection.png new file mode 100644 index 0000000..3dd4e6f Binary files /dev/null and b/assets/images/oci/bootstrap-provider-selection.png differ diff --git a/assets/images/oci/create-bootstrap-provider-basic-info.png b/assets/images/oci/create-bootstrap-provider-basic-info.png new file mode 100644 index 0000000..7ce2a2f Binary files /dev/null and b/assets/images/oci/create-bootstrap-provider-basic-info.png differ diff --git a/assets/images/oci/create-bootstrap-provider-oci-credentials.png b/assets/images/oci/create-bootstrap-provider-oci-credentials.png new file mode 100644 index 0000000..a777364 Binary files /dev/null and b/assets/images/oci/create-bootstrap-provider-oci-credentials.png differ diff --git a/assets/images/oci/oci-advance-configuration.png b/assets/images/oci/oci-advance-configuration.png new file mode 100644 index 0000000..8af1337 Binary files /dev/null and b/assets/images/oci/oci-advance-configuration.png differ diff --git a/assets/images/oci/oci-placement-configuration.png b/assets/images/oci/oci-placement-configuration.png new file mode 100644 index 0000000..8e86c56 Binary files /dev/null and b/assets/images/oci/oci-placement-configuration.png differ diff --git a/bootstrapping-fleet/oracle.md b/bootstrapping-fleet/oracle.md index c031b95..090d59f 100644 --- a/bootstrapping-fleet/oracle.md +++ b/bootstrapping-fleet/oracle.md @@ -8,6 +8,55 @@ nav_order: 2 Oracle Cloud Infrastructure (OCI) is supported as a bootstrap provider for creating Kubernetes fleets in Streamtime, enabling you to deploy and manage Kafka clusters on Oracle Kubernetes Engine (OKE) with integrated automation. + + +### Prerequisites: + + **1. Before you can create a Kubernetes fleet in **StreamTime**, you need a user that belongs to a group with the right level of permissions in OCI.** + * **Create a Group** in OCI for StreamTime users (e.g., streamtime-admins). + * **Add your user** to this group. + * **Attach a policy** to the group that grants the required permissions. The policy should include the following statement: + + Allow group to manage all-resources in compartment id + + This ensures that StreamTime can provision, manage, and monitor all OCI resources (compute, networking, storage, and OKE clusters) within the specified compartment. + + **2. Generate an API key for the user. Refer to the official documentation-** [Managing API Keys](https://docs.oracle.com/en-us/iaas/Content/API/Concepts/apisigningkey.htm#two) +
+ +#### Steps to Create a Bootstrap Provider on Oracle in StreamTime + + +**Step-1: Create a Bootstrap Provider** +Navigate to **Settings → Bootstrap Providers → Select OCI OKE → Next** + +![Creating a Bootstrap Provider]({{ site.baseurl }}/assets/images/oci/create-bootstrap-provider-basic-info.png) + + + + +
+ +**Step-2: Fill in the Configuration Form** + +![bootstrap provider oci credentials]({{ site.baseurl }}/assets/images/oci/create-bootstrap-provider-oci-credentials.png) + +- **Tenancy OCID** – OCID of the OCI Tenancy in which the resources should be created. +- **User OCID** – OCID of the IAM user StreamTime will use to create and manage resources. +- **Key Fingerprint** – Fingerprint of the above user’s API key. +- **Private Key** – Paste the PEM private key that matches the API key. +- **Defined Tags** – Tags that StreamTime will automatically apply to every OCI resource it provisions (compute, networking, storage, OKE). Enter tags as `namespace.key=value`. Using defined tags ensures consistent ownership, cost allocation, governance, and auditing across all StreamTime-created resources. When you create the tag key definition, you choose its value type, which determines how users assign values to resources. StreamTime automatically applies these defined tags to all resources it creates in OCI. +**You can add multiple tags,** ensuring that every compute, network, and storage resource provisioned through StreamTime is consistently tagged under your chosen namespace, making it easier to: + + * Track resource ownership and usage + * Align deployments with cost centers or projects + * Enforce governance policies + * Simplify reporting and auditing across your environment +- **Identity Domain OCID** – OCID of the OCI Identity Domain for creating an user for object storage bucket access. If not provided, the default domain in the tenancy is used. This is required in scenarios where the user does not have access to the default Identity Domain and needs to use a different Identity Domain. + +By defining and applying these tags at the StreamTime level, you get end-to-end visibility and control over your OCI resources without needing to manually tag them later. +[Learn more about defined tags in Oracle Cloud Infrastructure](https://docs.oracle.com/en-us/iaas/Content/Tagging/Tasks/managingtagsandtagnamespaces.htm#overviewtags) + --- ## When to Use Oracle (OKE) @@ -32,20 +81,155 @@ Oracle Cloud Infrastructure (OCI) is supported as a bootstrap provider for creat ## How to Deploy on Oracle (OKE) -1. **Start Fleet Creation in Streamtime** - - In the Streamtime UI, click "Create Kubernetes Fleet". - - Select **Oracle** as your bootstrap provider. -2. **Configure Tenancy & Sizing** - - Choose tenancy mode (shared, isolated, or dedicated). - - Set base domain, max tenancy, and max Kafka units. -3. **Placement Configuration** - - Select the OCI region for your fleet. +### Prerequisites: + +* An OCI account with appropriate permissions. +* API keys configured in Streamtime for OCI access. +* A defined compartment in OCI where the fleet will live. +* A Base Domain is required (you can add it in the Settings Panel). +* (Optional) SSH key for accessing worker nodes. + +
+ +#### Steps to Create a OKE on StreamTime + +**Step-1:Bootstrap provider selection** +Navigate to **Bootstrap Providers → Add Kubernetes Fleet → Select OCI OKE** + +![bootstrap-provider]({{ site.baseurl }}/assets/images/oci/bootstrap-provider-selection.png) + +
+ +**Step-2: Basic Configuration** + Basic Configuration step with fields for Identifier, Tenancy, Base Domain, Alert Channels, and sliders for Max Tenants and Max Kafka Units + +![ocibasic-configuration]({{ site.baseurl }}/assets/images/oci/basic-configuration-oci.png) + +* **Identifier** + + * A unique name you assign to your Kubernetes fleet (Use the auto-generated name or provide your own unique name. Here it is `scared-finch`). + + * Must be unique within your account. + +* **Tenancy** + + * Defines how resources are shared, refer the docs for more information [Tenancy in Streamtime]({{ site.baseurl }}/concept-architecture/tenancy.html#tenancy) + +* **Base Domain** + + * The root domain used for accessing services in the fleet. + + * For example: if you set example.com, your workloads will be exposed as service.example.com. + + * The Base Domain must be configured prior to fleet creation if it’s not already set in the Settings panel., Refer to the docs on [Domains]({{ site.baseurl }}/setup/domains-certificates.html) + + +* **Alert Channels** + + * Where Streamtime will send alerts for fleet events (scaling issues, failures, upgrades, etc.). + + * Can be things like Slack channels, Email etc + +* **Max Tenants** + + * The maximum number of tenants (kafka clusters) that can be hosted in this fleet. + + * Example: If you set 5, you can host up to 5 different tenants (clusters) on this fleet. + +* **Max Kafka Units** -4. **Advanced Configuration** - - Provide VCN ID or create a new one. - - Set the node shape, optionally specify a KMS Key ID, and set `cluster_public` (default: false). + * The maximum Kafka capacity units (a resource abstraction Streamtime uses for sizing Kafka clusters). + * One “Kafka Unit” typically maps to a certain amount of broker resources (CPU, memory, storage, throughput). Refer, [Scaling Kafka Clusters in Streamtime]({{ site.baseurl }}/concept-architecture/tenancy.html#tenancy) + + * Setting this defines how much Kafka workload this fleet can handle. + + * Example: If you set 10, tenants can request Kafka resources up to 10 units in total. + +
+ +**Step-3: Placement Configuration** + +![oci-placement-configuration]({{ site.baseurl }}/assets/images/oci/oci-placement-configuration.png) + +* **Account** + * This is the OCI account you already onboarded into Streamtime (via API keys). + + * Example: data-platform-oci (your configured account). + +* **Region** + * The OCI region where your Kubernetes fleet (and Kafka Cluster cluster) will be deployed. + + * Example: ap-hyderabad-1. + +* **Compartment OCID** + * You must paste the OCID of the OCI compartment where your Kubernetes resources should live. + + * This decides which compartment Streamtime will use to spin up the fleet/cluster. + + * Navigate to: Identity & Security → Compartments → [Your Compartment] → OCID + + * Looks like: ocid1.compartment.oc1..aaaaaaaexampleuniqueID12345 + +
+ + **In the OCI Console, you can locate your Compartment OCID by navigating to:** + + OCI Console → Identity & Security → Compartments → [Your Compartment] → OCID + +
+ +**Step-4: Advanced Configuration** + +![oci-advance-configuration]({{ site.baseurl }}/assets/images/oci/oci-advance-configuration.png) + +* **VCN (Virtual Cloud Network)** + + * The VCN provides the networking backbone for your Kubernetes fleet, including subnets for both the control plane and worker nodes. + + * You can **create a new VCN** or **select an existing VCN** from your OCI account. + + * Streamtime will use this VCN to allocate IP addresses, route traffic, and manage network security for the cluster. + + **Tip:** Ensure the VCN has enough IP address space for all nodes and services in your fleet. + + +* **Node Shape** + * This defines the compute instance shape for worker nodes + + +* **KMS Key OCID** (Optional) + + * KMS Key OCID (Optional): If you have a customer-managed encryption key stored in OCI Vault, you can specify its OCID (Oracle Cloud Identifier) here. This key will be used to securely encrypt your Kubernetes cluster data and node volumes, providing an additional layer of data protection and control over encryption keys. If you choose to leave this field blank, Oracle-managed encryption keys will be used by default, meaning Oracle handles all encryption management without requiring you to specify a custom key. Using a customer-managed key gives you greater control over key lifecycle, rotation, and access, enhancing your overall security posture. + +* **Public Cluster**(Checkbox) + + * If checked, the Kubernetes API server endpoint will be publicly accessible. + + * If unchecked, it will be private and accessible only within the VCN. + +* **API Server Allowed CIDRs** + + * These are CIDR ranges allowed to access the Kubernetes API server (control plane) and for SSH into worker nodes. + + * Example shown: 0.0.0.0/0 → This means the API server is open to all IPs (security risk). + + * Best practice: Restrict to your office IP or VPN CIDR (e.g., 203.x.x.x/32). + +* **SSH Public Key** + + * Here you can provide your SSH public key. + * This allows you to SSH into the OKE worker nodes. + + * If left blank, you won’t be able to SSH into nodes directly (still manageable via Kubernetes API). --- + + + + + + + diff --git a/setup-kafka/confluent.md b/setup-kafka/confluent.md index c9bf434..e67d851 100644 --- a/setup-kafka/confluent.md +++ b/setup-kafka/confluent.md @@ -42,29 +42,203 @@ Confluent for Kubernetes (CFK) is a fully managed Kafka distribution optimized f - **Multi-Cloud Support**: Can be deployed across multiple cloud providers, ensuring high availability and disaster recovery. ## How to Deploy CFK -Deploying Confluent for Kubernetes (CFK) using Streamtime is straightforward and can be done through the Streamtime UI. Here’s a step-by-step guide: - -1. **Create a Kafka Cluster**: - - Use the Streamtime UI to create a new Kafka cluster. - - Select "Confluent for Kubernetes" as the distribution type. - ![Create Kafka Cluster]({{ site.baseurl }}/assets/images/cfk/cluster-type.png) - -2. **Configure Cluster Settings**: - - Choose a cloud provider (AWS, GCP, OCI, Azure) or use BYOK (Bring Your Own Kubernetes). - - Specify the region. - - Specify the number of Kafka units (1 unit = 20 MBps throughput). - - Set the tenancy model (`shared`, `isolated`, or `dedicated`). - ![Cluster Basic Configuration]({{ site.baseurl }}/assets/images/cfk/cluster-basic-config.png) - -3. **Fleet Selection**: - - Choose the Kubernetes fleet where the Kafka cluster will be deployed. - - Ensure the fleet is healthy and ready for deployment. - ![Cluster Fleet Selection]({{ site.baseurl }}/assets/images/cfk/cluster-fleet-selection.png) - -4. **Advanced Configuration**: - - Optionally configure advanced settings such as replication factor, partition count, and resource limits. - - ![Cluster Advanced Configuration]({{ site.baseurl }}/assets/images/cfk/cluster-advanced-config.png) +Deploying Confluent for Kubernetes (CFK) using Streamtime is straightforward and can be done through the Streamtime UI. + +#### **What StreamTime Creates for You** + +* **CFK Components**: Kafka brokers with KRaft, Schema Registry (if enabled), Control Center. +* **Kubernetes Objects**: Namespaces, Deployments/StatefulSets, Services (LB/ClusterIP), PersistentVolumeClaims. +* **Networking**: LoadBalancers/ingress according to your **Outbound Access** and **Private Access** choices. +* **Storage**: Block Volumes‑backed Persistent Volumes sized per your config. + +### **Prerequisites** + +* **A Kubernetes Fleet** +Before you can create a CFK cluster, ensure that you have a **Kubernetes Fleet** already bootstrapped on any supported cloud platform using StreamTime. +[How to deploy A Kubernetes fleet.]({{ site.baseurl }}/bootstrapping-fleet/) + + +* **Bootstrap Provider Permissions for Tiered Storage Setup** +A Bootstrap Provider configured with an user that has permission to create Object Storage buckets for Tiered Storage and Users for granting access to the created bucket. If the bucket is already created, the user will need permissions to create a credentials (Ex - Access Key in AWS, Customer Secret Key in OCI) and have access to the bucket. + + +#### **Steps to Create a CFK Cluster in StreamTime** + +**Step-1: Select Cluster Type** + +Navigate to **Clusters → Create Cluster →** select **Confluent Platform** (Commercial) **→** Click **Begin Configuration**. + +![Kafka-ClusterTypeSelection]({{ site.baseurl }}/assets/images/cfk/create-Kafka-ClusterTypeSelection.png) + +**Step-2: Basic Configuration** + +Configure the sizing and placement of your CFK cluster. + +![Kafka-Basic_configuration]({{ site.baseurl }}/assets/images/cfk/create-Kafka-Basic-configuratio.png) + +* **Identifier** + * Human-readable cluster ID + * You can use the auto-generated identifier or provide your own, which must be unique per organization. (Auto-generated names are provided only for UI convenience and not intended for real-world cluster names, hence use meaningful identifiers) + * *Example*: `clickstream-pipe` + +* **Tags** + * Key/value labels applied to StreamTime objects + * Useful for environment, owner, cost-center, etc. + * *Example*: `environment=non-prod` + +* **Cloud Provider** + * Target cloud for deployment + * *Guidance*: Choose any cloud provider of your choice (Example shown for **OCI**) +* **Region** + * OCI region where the Fleet is created + * Must match an existing Fleet region + * *Example*: `ap-hyderabad-1` +* **Tenancy** + * Defines the resource sharing model + * Options: **Shared**, **Dedicated** or **Isolated,** Refer to the docs on [Tenancy]({{ site.baseurl }}/concept-architecture/tenancy.html#tenancy) +* **Kafka Units (KU)** + * Baseline throughput capacity + * Each KU ≈ 20 MB/s aggregate +* **Alert Channels** + * Health and incident notifications + * Optional; choose Slack/Email channels configured in Settings + +
+ +**Sizing Tip-** +1 KU (\~20 MB/s) is good for dev/test or light prod. For production with replication factor \= 3 and bursty writes, consider higher suitable KU to avoid broker IO saturation. + +
+ +**Step‑3: Fleet Selection** (This step applies only to Administrator users.) +* Select the Kubernetes Fleet that will host this CFK cluster. +* Choose preferred Kubernetes cluster for deploying Confluent Platform. +* This ensures that StreamTime deploys the Confluent Platform inside your existing Kubernetes fleet. + + +![Kafka-Fleet-Selection]({{ site.baseurl }}/assets/images/cfk/create-Kafka-Fleet-Selection.png) + +
+ +**Step-4: Advanced Configuration** (This step applies only to Administrator users.) + +Fine‑tune networking, security, components, and storage. + +![Kafka-Advance-Configuration]({{ site.baseurl }}/assets/images/cfk/create-Kafka-Advance-configuration.png) + + +* **Optimization Goal** +You have three choices for controlling how the cluster manages egress networking (traffic going out of the cluster): + + - **Cost** → Optimized for lowest cost. + - Limits or routes outbound traffic through shared, cheaper paths. + - Best for dev/test clusters where traffic is light. + + - **Balanced (default)** → Mix between performance and cost. + - Suitable for most production environments. + - Provides decent throughput with moderate costs. + + - **Performance** → Optimized for maximum throughput and lowest latency. + - Uses premium networking resources. + - Best for high-throughput, latency-sensitive production workloads. + + +
+ +* **Cluster Access** +Controls how clients connect to Kafka brokers: + + - **Internal** → Only accessible within the Kubernetes cluster (ideal for private, secure setups where client workloads run in the same cluster). + - **External** → Accessible from outside the Kubernetes cluster (needed when clients run outside the cluster). + - **Internal & External** → Both internal and external clients can connect. + +
+ +* **Authentication Mechanism** +Defines how clients authenticate to Kafka: + + - **SASL/OAUTH** → OAuth-based authentication. + - More secure, integrates with identity providers. + - Recommended by StreamTime. + + - **SASL/PLAIN** → Username/password-based authentication. + - Simpler but less secure. + + +
+ +* **Additional System Admin** +You can assign extra system administrators who will have full access to manage the CFK cluster. + +
+ +* **Additional System Admin Type** +Specifies the type of principal mentioned in Additional System Admin. If the admin name provided is a group in the identity provider, set the Additional System Admin Type as Group, else, set it as user +
+ +* **Storage Tier Configuration** +Defines how Kafka tiered storage manages log segment offloading to tiered storage: + + - **Aggressive** → Offloads data to tiered storage quickly, keeping local disk usage minimal. + (Suitable when you want to optimize for cost and use cloud storage heavily) + + - **Balanced** → Default option. Keeps a healthy balance between local disk and tiered storage usage. + + - **Conservative** → Keeps data locally for longer before offloading to tiered storage. + - (Useful if you want faster local access and rely less on cloud storage) + - (Requires larger local disk size, which increases infrastructure cost.) + +
+ +* **Tiered Storage Bucket Name** +The name of the Object Storage bucket which will be used for tiered storage. Ensure that this is globally unique for a provider. +Tip: Use a naming convention that is unique for your organization and use numbers to ensure global uniqueness. + +
+ +* **Tiered Storage Provider** +All Confluent Platform clusters created by StreamTime have Tiered Storage enabled for lowering storage costs. Learn more [here](https://docs.confluent.io/platform/current/clusters/tiered-storage.html) + + + Choose where offloaded Kafka data will be stored: + + - **AWS** + - **AWS Account** → Your AWS account identifier configured as a Bootstrap Provider in the Organization Settings. Ensure the user configured for this Provider has the following policy - + ```json + { + "Version" : "2012-10-17", + "Statement" : [ + { + "Effect" : "Allow", + "Action" : [ + "s3:*" + ], + "Resource" : "" + } + ] + } + ``` + + + - **Region** → The AWS region where your S3 bucket exists. + + - **OCI** + - **OCI Account** → Your Oracle Cloud Infrastructure account ID (Tenancy OCID). + Your OCI account identifier configured as a Bootstrap Provider in the Organization Settings. Ensure the user configured for this Provider has the following policy - + ```Allow group to manage buckets in compartment id ``` + + ```Allow group to manage objects in compartment id ``` + - **Region** → The OCI region where the bucket exists. + - **Compartment ID** → OCID of the compartment where the storage bucket resides. + +
+ + +**Once all steps are complete-** +- Review the configuration shown on the right side of the screen. +- Click **Create Cluster**. + +--- ## Monitoring and Management - **Metrics Dashboards**: Visualize key metrics such as throughput, latency, and error rates.