AWS CDK and GitHub Actions: Infrastructure automation trap where “destruction” is more scary than construction

Have you ever accidentally deleted RDS in your production environment and had to restore it from a snapshot at 3 a.m. with a blank face? I am. That was the price of underestimating CDK's stack deletion privileges. I want to punch myself in the face for believing that if I implement IaC, I will be able to sleep soundly at night.

By combining AWS CDK and GitHub Actions, you can certainly create an ideal zero-downtime environment. However, there are deeply buried land mines that shatter the illusion that ``automation = safety''. In this article, I will share the ``Survival Strategy for CDK Operations'' that I learned through bloodshed.

---

1. Abandon the illusion that “automation = safety”

The biggest appeal of CDK is the flexibility of writing infrastructure in TypeScript. However, this flexibility is a disaster, and every time you hit `cdk deploy`, the infrastructure is rebuilt, resulting in many accidents where services are instantly blown out.

In particular, hitting `npx cdk deploy --require-approval never` in GitHub Actions without thinking is equivalent to driving blindfolded at 200 km/h the wrong way down the highway. **

“diff” strategy to avoid dying on site

Be sure to run `cdk diff` before deploying, and include a process in your pipeline that allows humans to visually check the differences.

`

# Recommended GitHub Actions flow
jobs:
  deploy:
    steps:
      -run: npm install
      - name: CDK Diff
        run: npx cdk diff --fail # Stop deployment if there are differences
      - name: CDK Deploy
        run: npx cdk deploy --require-approval before-upgrade

`
`--require-approval never` is ``prohibited'' in the production pipeline except during debugging.

---

2. Destruction is scarier than construction: The darkness of RemovalPolicy

The biggest mistake was setting `RemovalPolicy.DESTROY` too easily. If resource replacement is performed by changing the stack configuration with this in place, RDS and DynamoDB will disappear without any questions asked.

// [Warning] Writing this in a production environment is suicidal
const table = new dynamodb.Table(this, 'MyTable', {
  removalPolicy: cdk.RemovalPolicy.DESTROY, // Change = Erase all data
});

`

In a production environment, `RemovalPolicy.RETAIN` is the only option. This alone can dramatically reduce the risk that mistakes in code will lead directly to physical data loss.

---

3. Why “synthesis” is the key to debugging

CDK is not a "magic wand" to generate CloudFormation. It's just a "template generator". Running `cdk synth` without reading the YAML output is the same as being lost without a map.

Once, when I separated VPC and security groups into separate stacks and the deployment was completely locked due to cross references, it was the templates output by `cdk synth` that saved me. Do not question whether the CDK code is correct, but whether the generated template complies with AWS constraints. This is the dividing line between intermediate and advanced.

---

4. Achieve “non-disruption” with CodeDeploy

CodeDeploy's blue/green deployment is essential to achieve zero downtime in an ECS environment. However, writing in CDK is laborious.

// CodeDeploy configuration example: The key points are roles and listeners
const deploymentGroup = new codedeploy.EcsDeploymentGroup(this, 'BlueGreenGroup', {
  service: myEcsService,
  blueGreenDeploymentConfig: {
    blueTargetGroup: blueTargetGroup,
    greenTargetGroup: greenTargetGroup,
listener: productionListener,
  },
  deploymentConfig: codedeploy.EcsDeploymentConfig.LINEAR_10PERCENT_EVERY_1MINUTES,
});

`
The important thing here is that the moment you define `blueGreenDeploymentConfig`, the existing service will be forcibly migrated to "under CodeDeploy management". When applying it to an existing environment, unless you thoroughly eliminate conflicts (drift) with manually created resources, you will end up in an ``infinite loop hell'' where the deployment will never end.

---

5. Checklist for experts

I'll leave you with "my golden rules" for using CDK on the battlefield.

---

Conclusion: Battlefield at the edge of automation

CDK is a powerful weapon, but you are the one who pulls the trigger. Instead of relying too much on automation, deeply understand CloudFormation behavior and always write code with a "worst case scenario" in mind for breaking changes.

There is no paradise at the end of automation. It is a battlefield that requires advanced management skills and a constant sense of tension. However, only those who control the battlefield will be able to obtain ``true peace,'' where they will not be woken up at 3 a.m.

I wish you good luck.

---

🤖 Developer blog/public content

The developer's personal blog is also available, so please take a look. 👉 muttyan.com blog

We also have a special page that summarizes Amazon's convenient and advantageous free trials and discount plans!
👉 Amazon great free trial/benefits summary