|
| 1 | +# ClusterSettings CRD Examples |
| 2 | + |
| 3 | +This directory contains examples of how to use the `ClusterSettings` Custom Resource Definition (CRD) to configure PerfectScale profiles via GitOps. |
| 4 | + |
| 5 | +## Recommended Workflow |
| 6 | + |
| 7 | +**We prefer users to use refByID rather than inline profiles.** |
| 8 | + |
| 9 | +1. **Create and test profiles in UI first** |
| 10 | + - Create profiles through the PerfectScale UI |
| 11 | + - Test and validate the configuration |
| 12 | + - Once satisfied, note the profile ID shown in UI (format: {type|class}-N, e.g., jira-1, slack-2) |
| 13 | + |
| 14 | +2. **Reference tested profiles via refByID in CR (preferred method)** |
| 15 | + - Use `refByID` to reference the UI-created profile by its ID |
| 16 | + - No `assigned` field needed - refByID profiles are always assigned |
| 17 | + - See examples in `clustersettings-assign-existing.yaml` |
| 18 | + |
| 19 | +3. **Alternative: Create inline profiles for testing** |
| 20 | + - Create inline profiles with `assigned: false` (default) |
| 21 | + - Profiles will be visible in UI and can be tested |
| 22 | + - Once tested, update CR to set `assigned: true` to activate the profile |
| 23 | + |
| 24 | +## Validation Rules |
| 25 | + |
| 26 | +- **Only ONE profile of each type can be assigned to a cluster** |
| 27 | +- **For inline profiles**: Only ONE can have `assigned: true` per type (validation fails if multiple) |
| 28 | +- **For refByID profiles**: Only ONE can be specified per type (validation fails if multiple) |
| 29 | +- **IMPORTANT**: You CANNOT have BOTH a refByID AND an inline profile with `assigned: true` for the same type |
| 30 | + - If refByID exists for a type, ALL inline profiles of that type MUST have `assigned: false` (or unset) |
| 31 | + - Having both will fail validation |
| 32 | + - Valid: refByID + inline profiles with `assigned: false` |
| 33 | + - Invalid: refByID + any inline profile with `assigned: true` |
| 34 | +- **refByID profiles**: Always considered assigned (no `assigned` field needed or required) |
| 35 | +- **Non-existent refByID**: If referenced profile doesn't exist, validation passes but nothing happens (noop) |
| 36 | +- **CRD Override**: CRD profiles (both refByID and inline) ALWAYS WIN over UI-assigned profiles |
| 37 | + |
| 38 | +## Important Notes |
| 39 | + |
| 40 | +- **All ClusterSettings resources MUST be applied in the `perfectscale` namespace** |
| 41 | +- Only **one** ClusterSettings resource is allowed per cluster |
| 42 | +- Secrets referenced in the CRD must exist in the `perfectscale` namespace |
| 43 | +- **Profiles assigned via CRD cannot be deleted via UI** - deletion must be done via CRD |
| 44 | + |
| 45 | +## UI-only Features |
| 46 | + |
| 47 | +Some PerfectScale features are currently available only through the UI and cannot be configured via CRD: |
| 48 | +- **Custom Grouping** (Ephemeral Pods Grouping) - See: https://docs.perfectscale.io/customize-workflow/ephemeral-pods-grouping |
| 49 | +- **Automation Policies** - Automation rules and maintenance windows |
| 50 | + |
| 51 | +All other profiles are fully supported via CRD. |
| 52 | + |
| 53 | +## Files Overview |
| 54 | + |
| 55 | +### Complete Examples |
| 56 | +- **`clustersettings-all.yaml`** - Comprehensive example showing all profile types and configuration options |
| 57 | +- **`clustersettings-mixed.yaml`** - Example combining inline profiles and profile references |
| 58 | + |
| 59 | +### Simple Examples |
| 60 | +- **`clustersettings-assign-existing.yaml`** - Assign existing profiles to cluster |
| 61 | +- **`clustersettings-custom-pricing.yaml`** - Create custom node pricing inline |
| 62 | +- **`clustersettings-podfit-labels.yaml`** - Configure PodFit label visibility |
| 63 | +- **`clustersettings-customization.yaml`** - Configure workload labels |
| 64 | +- **`clustersettings-resiliency-alerts.yaml`** - Resiliency alerts configuration |
| 65 | + |
| 66 | +### Pricing Examples |
| 67 | +- **`clustersettings-aws-cur.yaml`** - AWS Cost and Usage Report integration |
| 68 | +- **`clustersettings-gcp-billing.yaml`** - GCP billing export integration |
| 69 | +- **`clustersettings-azure-billing.yaml`** - Azure billing integration |
| 70 | +- **`clustersettings-cloud-billing.yaml`** - Multi-cloud billing (GCP + Azure) |
| 71 | + |
| 72 | +### Integration Examples |
| 73 | +- **`clustersettings-slack.yaml`** - Slack integration for alerts |
| 74 | +- **`clustersettings-teams.yaml`** - Microsoft Teams integration |
| 75 | +- **`clustersettings-jira.yaml`** - Jira ticketing integration |
| 76 | +- **`clustersettings-datadog.yaml`** - Datadog monitoring integration |
| 77 | +- **`clustersettings-observability.yaml`** - Custom observability platform |
| 78 | +- **`clustersettings-integrations.yaml`** - Multiple integrations (Teams + Jira + Datadog) |
| 79 | + |
| 80 | +## Profile Types |
| 81 | + |
| 82 | +### Pricing Profiles |
| 83 | +Control how costs are calculated for your cluster. |
| 84 | + |
| 85 | +Optional field for any pricing profile: |
| 86 | +- `global_discount` - Applies a single discount percent to on-demand resource prices for any pricing profile type (not applied to DoiT pricing profiles). Discount applies only when `start_date` is provided and valid. |
| 87 | + - `percentage` - Discount percent (0-100) applied to on-demand resource prices only. |
| 88 | + - `start_date` - ISO date (yyyy-mm-dd) when the discount starts; if missing or invalid, the discount is not applied. Changing this date does not automatically update past reports; contact support if historical data needs recalculation. |
| 89 | + |
| 90 | +**Types:** |
| 91 | +- `custom` - Define custom pricing per instance type |
| 92 | + - `instanceType`: The name of the instance that is given by the provider (e.g., c5.large, m5.xlarge) |
| 93 | + - `memGBHourPrice`: Indicates the price ($) of 1 GB of memory per hour |
| 94 | + - `cpuCoreHourPrice`: Indicates the price ($) of 1 core per hour |
| 95 | + - `nodeHourPrice`: Indicates the price ($) of a node per hour |
| 96 | +- `aws_cur` - AWS Cost and Usage Report integration |
| 97 | + - Authentication Method 1 (Recommended): IAM Role |
| 98 | + - `role_arn`: The Amazon Resource Name (ARN) associated with the role possessing the necessary credentials to execute calls on your behalf |
| 99 | + - `aws_external_id`: The ID for cross-account access in AWS Identity and Access Management (IAM). A unique, user-defined string that ensures only trusted third-party entities can assume a specific role |
| 100 | + - Authentication Method 2 (Alternative): IAM User |
| 101 | + - `access_key_id`: AWS access key ID (via `access_key_id` in CRD) |
| 102 | + - `secret_access_key`: AWS secret access key (via `secret_access_key_from` secret reference in CRD) |
| 103 | + - Configuration fields: |
| 104 | + - `athena_database`: Name of the database that was created during the Athena setup |
| 105 | + - `athena_region`: AWS region where Athena is running |
| 106 | + - `athena_result_bucket`: S3 bucket where Athena stores query results |
| 107 | + - `athena_table`: Name of the table that was created on the Athena setup |
| 108 | + - `aws_account_id`: AWS account where cluster is running (NOT the AWS billing account ID) |
| 109 | + - Full guide: https://docs.perfectscale.io/cloud-billing-integration/connecting-aws-cur |
| 110 | +- `gcp_billing` - GCP BigQuery billing export integration |
| 111 | +- `azure_billing` - Azure billing integration |
| 112 | + |
| 113 | +### Integration Profiles |
| 114 | +Configure external systems for notifications and monitoring. |
| 115 | + |
| 116 | +**Types:** |
| 117 | +- `slack` - Slack notifications |
| 118 | + - `slack_token`: A mandatory field used to permit PerfectScale to interact with your Slack (via `slack_token_from` secret reference in CRD) |
| 119 | + - Setup: Create an app at https://api.slack.com/apps |
| 120 | + - Required scopes: chat:write.public, chat:write, channels:read, conversations.list |
| 121 | + - `channel`: Slack channel name that indicates where to receive PerfectScale alerts |
| 122 | + - `routings`: Optional field allows alerts to be sent to different Slack channels for different workloads in a cluster |
| 123 | +- `teams` - Microsoft Teams notifications |
| 124 | + - `webhook_url`: A mandatory field used to permit PerfectScale to interact with your Teams (via `webhook_url_from` secret reference in CRD) |
| 125 | + - Setup: Go to Apps at https://teams.microsoft.com/, add Incoming Webhook, and connect to a specific channel |
| 126 | + - `routings`: Optional field allows alerts to be sent for specific alert types |
| 127 | +- `jira` - Jira ticketing |
| 128 | +- `datadog` - Datadog monitoring |
| 129 | +- `observability` - Custom observability platform |
| 130 | + - Setup: Go to https://github.com/perfectscale-io/observability, download either the Grafana or DataDog dashboard, import into your instances, and copy the dashboard link |
| 131 | + |
| 132 | +### Resiliency Alerts Profiles |
| 133 | +Configure workload health and resiliency alerts. **No type field required.** |
| 134 | + |
| 135 | +### Podfit Labels Profiles |
| 136 | +Configure which labels are displayed in the PodFit view. **No type field required.** |
| 137 | + |
| 138 | +### Customization Profiles |
| 139 | +Configure custom workload labels. **No type field required.** |
| 140 | + |
| 141 | +**Fields:** |
| 142 | +- `workload_labels`: Defines labels that can be applied to Kubernetes resources (pods, deployments, or stateful sets) to identify and manage workloads within a cluster |
| 143 | +- `node.spot_labels`: Identifies spot nodes (key-value pairs) |
| 144 | +- `node.node_group_labels`: Identifies the nodes group in a cluster |
| 145 | +- `node.architecture_labels`: Specifies the architecture of the node (x86_64, arm64, etc.) |
| 146 | +- `node.os_labels`: Specifies the operating system running on a node within the Kubernetes cluster |
| 147 | +- `node.instance_type_labels`: Specifies the type or configuration of VMs (instances) within a cloud infrastructure |
| 148 | +- `node.region_labels`: Specifies the geographical region where a particular resource is located |
| 149 | +- `node.availability_zone_labels`: Specify the availability zone where a particular resource is located |
| 150 | +- `ignored_pod_labels`: Defines labels that Kubernetes components should disregard when making pod decisions |
| 151 | +- `ignored_node_labels`: Defines labels that Kubernetes components should disregard when making node decisions |
| 152 | + |
| 153 | +**NOTE:** Labels should follow Prometheus conventions. Learn more at: https://prometheus.io/docs/concepts/data_model/#metric-names-and-labels |
| 154 | + |
| 155 | +## Profile Assignment Methods |
| 156 | + |
| 157 | +### Method 1: Reference Existing Profile (refByID) |
| 158 | +Reference a profile that was created in the UI: |
| 159 | + |
| 160 | +```yaml |
| 161 | +profiles: |
| 162 | + pricing: |
| 163 | + - type: custom |
| 164 | + refByID: "custom-1" # ID format: {type|class}-N |
| 165 | +``` |
| 166 | +
|
| 167 | +**Note:** Profiles created via CRD are cluster-specific and cannot be referenced by other clusters. |
| 168 | +
|
| 169 | +**Important Rules:** |
| 170 | +- `refByID` can only reference UI-created profiles (not inline CRD profiles) |
| 171 | +- Profile IDs are shown in UI in format {type|class}-N (e.g., jira-1, slack-2, custom-3) |
| 172 | +- **IMPORTANT**: Underscores in type/class names are converted to hyphens in IDs: |
| 173 | + - Class `resiliency_alerts` → ID `resiliency-alerts-1` |
| 174 | + - Class `podfit_labels` → ID `podfit-labels-1` |
| 175 | + - Type `aws_cur` → ID `aws-cur-1` |
| 176 | + - Type `gcp_billing` → ID `gcp-billing-1` |
| 177 | + - Type `azure_billing` → ID `azure-billing-1` |
| 178 | +- The type field must match the referenced profile's type |
| 179 | +- refByID profiles are always assigned (no `assigned` field needed) |
| 180 | +- If the referenced profile doesn't exist, the assignment won't happen |
| 181 | +- If the referenced profile is updated in UI, changes propagate to referencing clusters |
| 182 | +- Only ONE profile per class can have `assigned: true` (applies to inline profiles only) |
| 183 | + |
| 184 | +### Method 2: Create Inline Profile |
| 185 | +Create a new profile directly in the CRD: |
| 186 | + |
| 187 | +```yaml |
| 188 | +profiles: |
| 189 | + pricing: |
| 190 | + - type: custom |
| 191 | + name: "my-inline-profile" |
| 192 | + assigned: true |
| 193 | + value: |
| 194 | + nodeTypes: |
| 195 | + - instanceType: c5.large |
| 196 | + pricing: |
| 197 | + cpuCoreHourPrice: 0.08 |
| 198 | + memGBHourPrice: 0.02 |
| 199 | +``` |
| 200 | + |
| 201 | +**Note:** Inline profiles are cluster-specific and only available to the cluster where they are created. |
| 202 | + |
| 203 | +## Using Secrets |
| 204 | + |
| 205 | +You can't specify sensitive data directly in the CRD. Use secret references instead. |
| 206 | + |
| 207 | +```yaml |
| 208 | +secret_access_key_from: |
| 209 | + secretKeyRef: |
| 210 | + name: aws-credentials # Secret name in perfectscale namespace |
| 211 | + key: secret-access-key # Key within the secret |
| 212 | +``` |
| 213 | + |
| 214 | +## Validation Rules |
| 215 | + |
| 216 | +1. **Singleton:** Only one ClusterSettings resource per cluster |
| 217 | +2. **Valid Types:** Profile types must be valid for their class |
| 218 | +3. **Name Requirements:** |
| 219 | + - `refByID` cannot be empty if provided |
| 220 | + - Inline profiles must have a `name` |
| 221 | +4. **Assignment:** Only one inline profile per class can have `assigned: true` |
| 222 | +5. **Type Requirements:** |
| 223 | + - `pricing` and `integrations` MUST have a `type` |
| 224 | + - `resiliency_alerts`, `podfit_labels`, `customization` MUST NOT have a `type` |
| 225 | + |
| 226 | +## Applying Examples |
| 227 | + |
| 228 | +```bash |
| 229 | +# Apply all resources (including secrets) in the example file |
| 230 | +kubectl apply -f examples/clustersettings-aws-cur.yaml |
| 231 | +
|
| 232 | +# View the ClusterSettings |
| 233 | +kubectl get clustersettings -n perfectscale |
| 234 | +
|
| 235 | +# Describe to see details |
| 236 | +kubectl describe clustersettings cluster-settings-main -n perfectscale |
| 237 | +
|
| 238 | +# Check logs for processing status |
| 239 | +kubectl logs -n perfectscale -l app=perfectscale-exporter --tail=100 |
| 240 | +``` |
| 241 | + |
| 242 | +## Deleting Configuration |
| 243 | + |
| 244 | +When you delete the ClusterSettings resource, the cluster returns to having no profiles (or default profiles): |
| 245 | + |
| 246 | +```bash |
| 247 | +kubectl delete clustersettings cluster-settings-main -n perfectscale |
| 248 | +``` |
| 249 | + |
| 250 | +## Documentation Links |
| 251 | + |
| 252 | +All profile types are documented in detail: |
| 253 | + |
| 254 | +### Pricing |
| 255 | +- **Custom Pricing**: https://docs.perfectscale.io/customize-workflow/pricing/custom-pricing-configuration |
| 256 | +- **AWS CUR**: https://docs.perfectscale.io/cloud-billing-integration/connecting-aws-cur |
| 257 | +- **GCP Billing**: https://docs.perfectscale.io/customize-workflow/pricing/gcp-cloud-billing-configuration |
| 258 | +- **Azure Billing**: https://docs.perfectscale.io/cloud-billing-integration/connecting-azure-cost-management |
| 259 | + |
| 260 | +### Integrations |
| 261 | +- **Slack**: https://docs.perfectscale.io/customize-workflow/communication-and-messaging/slack-integration |
| 262 | +- **Teams**: https://docs.perfectscale.io/customize-workflow/communication-and-messaging/ms-teams-integration |
| 263 | +- **Jira**: https://docs.perfectscale.io/customizations/ticketing-and-bug-tracking |
| 264 | +- **Datadog**: https://docs.perfectscale.io/customize-workflow/communication-and-messaging/datadog-alerts-integration |
| 265 | +- **Observability**: https://docs.perfectscale.io/customize-workflow/observability |
| 266 | + |
| 267 | +### Customizations |
| 268 | +- **Label Customizations**: https://docs.perfectscale.io/customizations/label-customizations |
| 269 | +- **PodFit Labels**: https://docs.perfectscale.io/customize-workflow/podfit-labels |
| 270 | +- **Resiliency Alerts**: https://docs.perfectscale.io/customize-workflow/alerting/resiliency-alerts |
| 271 | + |
| 272 | +## Troubleshooting |
| 273 | + |
| 274 | +### CRD Creation Failed |
| 275 | +- Check if ClusterSettings already exists: `kubectl get clustersettings -n perfectscale` |
| 276 | +- Review logs: `kubectl logs -n perfectscale -l app=perfectscale-exporter` |
| 277 | + |
| 278 | +### Profile Not Applied |
| 279 | +- Check logs for sync errors |
| 280 | +- Verify secrets exist: `kubectl get secrets -n perfectscale` |
| 281 | +- Check profile format matches the type |
| 282 | + |
| 283 | +### Secret Not Found |
| 284 | +- Verify secret exists: `kubectl get secret <secret-name> -n perfectscale` |
| 285 | +- Ensure secret is in `perfectscale` namespace |
0 commit comments