Published on

Modern DevOps: Docker, AWS, and CI/CD Pipeline Implementation

Authors

Introduction

Modern application deployment requires a robust DevOps pipeline that ensures reliability, scalability, and maintainability. In this guide, we'll explore how to build a comprehensive DevOps workflow using Docker for containerization, AWS for cloud infrastructure, and implementing CI/CD pipelines for automated deployment.

Docker Containerization

Basic Docker Concepts

Let's start with a typical Node.js application Dockerfile:

# Development stage
FROM node:18-alpine AS development

WORKDIR /usr/src/app

COPY package*.json ./
RUN npm install

COPY . .

# Build stage
FROM node:18-alpine AS builder

WORKDIR /usr/src/app

COPY --from=development /usr/src/app ./

RUN npm run build

# Production stage
FROM node:18-alpine AS production

ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}

WORKDIR /usr/src/app

COPY --from=builder /usr/src/app/dist ./dist
COPY --from=builder /usr/src/app/package*.json ./

RUN npm ci --only=production

EXPOSE 3000

CMD ["node", "dist/main"]

Docker Compose for Local Development

Create a docker-compose.yml for local development:

version: '3.8'

services:
  app:
    build:
      context: .
      target: development
    volumes:
      - .:/usr/src/app
      - /usr/src/app/node_modules
    ports:
      - "3000:3000"
    environment:
      - NODE_ENV=development
      - DATABASE_URL=postgresql://postgres:password@db:5432/myapp
    depends_on:
      - db

  db:
    image: postgres:14-alpine
    environment:
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=password
      - POSTGRES_DB=myapp
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - "5432:5432"

volumes:
  postgres_data:

AWS Infrastructure Setup

Infrastructure as Code with AWS CDK

// lib/app-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as ec2 from 'aws-cdk-lib/aws-ec2';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as ecr from 'aws-cdk-lib/aws-ecr';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as rds from 'aws-cdk-lib/aws-rds';

export class AppStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // VPC
    const vpc = new ec2.Vpc(this, 'AppVPC', {
      maxAzs: 2,
      natGateways: 1
    });

    // ECS Cluster
    const cluster = new ecs.Cluster(this, 'AppCluster', {
      vpc,
      containerInsights: true
    });

    // RDS Instance
    const database = new rds.DatabaseInstance(this, 'Database', {
      engine: rds.DatabaseInstanceEngine.postgres({
        version: rds.PostgresEngineVersion.VER_14
      }),
      vpc,
      instanceType: ec2.InstanceType.of(
        ec2.InstanceClass.T3,
        ec2.InstanceSize.MICRO
      ),
      databaseName: 'appdb',
      allocatedStorage: 20,
      maxAllocatedStorage: 100,
      autoMinorVersionUpgrade: true,
      deleteAutomatedBackups: true,
      removalPolicy: cdk.RemovalPolicy.DESTROY,
    });

    // ECR Repository
    const repository = new ecr.Repository(this, 'AppRepository', {
      repositoryName: 'app-repository',
      removalPolicy: cdk.RemovalPolicy.DESTROY,
      lifecycleRules: [{
        maxImageCount: 5,
        tagStatus: ecr.TagStatus.ANY
      }]
    });

    // ECS Task Definition
    const taskDefinition = new ecs.FargateTaskDefinition(this, 'AppTask', {
      memoryLimitMiB: 512,
      cpu: 256,
    });

    // Add container to task definition
    taskDefinition.addContainer('AppContainer', {
      image: ecs.ContainerImage.fromEcrRepository(repository),
      logging: ecs.LogDrivers.awsLogs({ streamPrefix: 'app' }),
      environment: {
        DATABASE_URL: `postgresql://postgres:${database.secret!.secretValue}@${database.instanceEndpoint.hostname}:5432/appdb`,
        NODE_ENV: 'production'
      },
      portMappings: [{ containerPort: 3000 }]
    });
  }
}

AWS Security Best Practices

// lib/security-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as iam from 'aws-cdk-lib/aws-iam';
import * as kms from 'aws-cdk-lib/aws-kms';

export class SecurityStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // KMS Key for encryption
    const key = new kms.Key(this, 'AppKey', {
      enableKeyRotation: true,
      alias: 'app/main',
      description: 'Main encryption key for the application'
    });

    // IAM Role for ECS Tasks
    const taskRole = new iam.Role(this, 'TaskRole', {
      assumedBy: new iam.ServicePrincipal('ecs-tasks.amazonaws.com'),
      description: 'Role for ECS tasks'
    });

    // Add least privilege permissions
    taskRole.addToPolicy(new iam.PolicyStatement({
      effect: iam.Effect.ALLOW,
      actions: [
        's3:GetObject',
        's3:PutObject'
      ],
      resources: ['arn:aws:s3:::app-bucket/*']
    }));
  }
}

CI/CD Pipeline Implementation

GitHub Actions Workflow

# .github/workflows/main.yml
name: CI/CD Pipeline

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Set up Node.js
      uses: actions/setup-node@v3
      with:
        node-version: '18'
        
    - name: Install dependencies
      run: npm ci
      
    - name: Run tests
      run: npm test

  build-and-push:
    needs: test
    runs-on: ubuntu-latest
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    
    steps:
    - uses: actions/checkout@v3
    
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1
        
    - name: Login to Amazon ECR
      id: login-ecr
      uses: aws-actions/amazon-ecr-login@v1
      
    - name: Build and push Docker image
      env:
        ECR_REGISTRY: ${{ steps.login-ecr.outputs.registry }}
        ECR_REPOSITORY: app-repository
        IMAGE_TAG: ${{ github.sha }}
      run: |
        docker build -t $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG .
        docker push $ECR_REGISTRY/$ECR_REPOSITORY:$IMAGE_TAG

  deploy:
    needs: build-and-push
    runs-on: ubuntu-latest
    
    steps:
    - name: Configure AWS credentials
      uses: aws-actions/configure-aws-credentials@v1
      with:
        aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
        aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
        aws-region: us-east-1
        
    - name: Update ECS service
      run: |
        aws ecs update-service --cluster app-cluster \
          --service app-service \
          --force-new-deployment

AWS CodePipeline (Alternative Approach)

// lib/pipeline-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as codepipeline from 'aws-cdk-lib/aws-codepipeline';
import * as codepipeline_actions from 'aws-cdk-lib/aws-codepipeline-actions';
import * as codebuild from 'aws-cdk-lib/aws-codebuild';

export class PipelineStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    const pipeline = new codepipeline.Pipeline(this, 'Pipeline', {
      pipelineName: 'AppPipeline',
      crossAccountKeys: false
    });

    // Source stage
    const sourceOutput = new codepipeline.Artifact();
    const sourceAction = new codepipeline_actions.CodeStarConnectionsSourceAction({
      actionName: 'GitHub',
      owner: 'your-github-username',
      repo: 'your-repo-name',
      branch: 'main',
      output: sourceOutput,
      connectionArn: 'your-connection-arn'
    });

    pipeline.addStage({
      stageName: 'Source',
      actions: [sourceAction],
    });

    // Build stage
    const buildProject = new codebuild.PipelineProject(this, 'Build', {
      environment: {
        buildImage: codebuild.LinuxBuildImage.STANDARD_5_0,
        privileged: true
      },
      environmentVariables: {
        REPOSITORY_URI: {
          value: 'your-ecr-repository-uri'
        }
      },
    });

    const buildOutput = new codepipeline.Artifact();
    const buildAction = new codepipeline_actions.CodeBuildAction({
      actionName: 'Build',
      project: buildProject,
      input: sourceOutput,
      outputs: [buildOutput],
    });

    pipeline.addStage({
      stageName: 'Build',
      actions: [buildAction],
    });
  }
}

Monitoring and Logging

CloudWatch Configuration

// lib/monitoring-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as cloudwatch from 'aws-cdk-lib/aws-cloudwatch';
import * as logs from 'aws-cdk-lib/aws-logs';

export class MonitoringStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // CloudWatch Log Group
    const logGroup = new logs.LogGroup(this, 'AppLogs', {
      retention: logs.RetentionDays.ONE_WEEK,
      removalPolicy: cdk.RemovalPolicy.DESTROY
    });

    // CloudWatch Dashboard
    const dashboard = new cloudwatch.Dashboard(this, 'AppDashboard', {
      dashboardName: 'AppMetrics'
    });

    // Add metrics
    dashboard.addWidgets(
      new cloudwatch.GraphWidget({
        title: 'CPU Utilization',
        left: [
          new cloudwatch.Metric({
            namespace: 'AWS/ECS',
            metricName: 'CPUUtilization',
            statistic: 'Average'
          })
        ]
      }),
      new cloudwatch.GraphWidget({
        title: 'Memory Utilization',
        left: [
          new cloudwatch.Metric({
            namespace: 'AWS/ECS',
            metricName: 'MemoryUtilization',
            statistic: 'Average'
          })
        ]
      })
    );

    // Create alarms
    new cloudwatch.Alarm(this, 'HighCPUAlarm', {
      metric: new cloudwatch.Metric({
        namespace: 'AWS/ECS',
        metricName: 'CPUUtilization',
        statistic: 'Average',
        period: cdk.Duration.minutes(5)
      }),
      threshold: 80,
      evaluationPeriods: 3,
      alarmDescription: 'CPU utilization is too high'
    });
  }
}

Scaling and High Availability

Auto Scaling Configuration

// lib/scaling-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as ecs from 'aws-cdk-lib/aws-ecs';
import * as applicationautoscaling from 'aws-cdk-lib/aws-applicationautoscaling';

export class ScalingStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create scaling target
    const scalableTarget = new applicationautoscaling.ScalableTarget(this, 'ScalingTarget', {
      serviceNamespace: applicationautoscaling.ServiceNamespace.ECS,
      maxCapacity: 10,
      minCapacity: 1,
      resourceId: `service/app-cluster/app-service`,
      scalableDimension: 'ecs:service:DesiredCount'
    });

    // CPU utilization scaling policy
    scalableTarget.scaleToTrackMetric('CpuScaling', {
      targetValue: 70,
      predefinedMetric: applicationautoscaling.PredefinedMetric.ECS_SERVICE_AVERAGE_CPU_UTILIZATION,
      scaleInCooldown: cdk.Duration.seconds(60),
      scaleOutCooldown: cdk.Duration.seconds(60)
    });

    // Request count scaling policy
    scalableTarget.scaleToTrackMetric('RequestCountScaling', {
      targetValue: 1000,
      predefinedMetric: applicationautoscaling.PredefinedMetric.ALB_REQUEST_COUNT_PER_TARGET,
      scaleInCooldown: cdk.Duration.seconds(60),
      scaleOutCooldown: cdk.Duration.seconds(60)
    });
  }
}

Disaster Recovery and Backup

Backup Strategy Implementation

// lib/backup-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as backup from 'aws-cdk-lib/aws-backup';
import * as iam from 'aws-cdk-lib/aws-iam';

export class BackupStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create backup vault
    const backupVault = new backup.BackupVault(this, 'AppBackupVault', {
      backupVaultName: 'app-backup-vault',
      removalPolicy: cdk.RemovalPolicy.RETAIN
    });

    // Create backup plan
    const backupPlan = new backup.BackupPlan(this, 'AppBackupPlan', {
      backupVault: backupVault
    });

    // Add backup rules
    backupPlan.addRule(new backup.BackupPlanRule({
      completionWindow: cdk.Duration.hours(2),
      startWindow: cdk.Duration.hours(1),
      scheduleExpression: backup.Schedule.cron({
        day: '*',
        hour: '3',
        minute: '0'
      }),
      deleteAfter: cdk.Duration.days(14)
    }));

    // Add resources to backup plan
    backupPlan.addSelection('AppBackupSelection', {
      resources: [
        backup.BackupResource.fromRds(databaseArn),
        backup.BackupResource.fromEfs(fileSystemArn)
      ]
    });
  }
}

Security Best Practices

AWS WAF Implementation

// lib/security/waf-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as waf from 'aws-cdk-lib/aws-wafv2';

export class WafStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create WAF ACL
    const webAcl = new waf.CfnWebACL(this, 'AppWAF', {
      defaultAction: { allow: {} },
      scope: 'REGIONAL',
      visibilityConfig: {
        cloudWatchMetricsEnabled: true,
        metricName: 'AppWAFMetrics',
        sampledRequestsEnabled: true
      },
      rules: [
        {
          name: 'RateLimit',
          priority: 1,
          statement: {
            rateBasedStatement: {
              limit: 2000,
              aggregateKeyType: 'IP'
            }
          },
          action: { block: {} },
          visibilityConfig: {
            cloudWatchMetricsEnabled: true,
            metricName: 'RateLimitMetric',
            sampledRequestsEnabled: true
          }
        },
        {
          name: 'SQLInjectionRule',
          priority: 2,
          statement: {
            sqlInjectionMatchStatement: {
              fieldToMatch: { 
                allQueryArguments: {}
              },
              textTransformations: [{
                type: 'URL_DECODE',
                priority: 1
              }]
            }
          },
          action: { block: {} },
          visibilityConfig: {
            cloudWatchMetricsEnabled: true,
            metricName: 'SQLInjectionMetric',
            sampledRequestsEnabled: true
          }
        }
      ]
    });
  }
}

Secrets Management

// lib/security/secrets-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as secretsmanager from 'aws-cdk-lib/aws-secretsmanager';

export class SecretsStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    // Create secrets
    const databaseSecret = new secretsmanager.Secret(this, 'DatabaseSecret', {
      secretName: 'app/database',
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          username: 'admin'
        }),
        generateStringKey: 'password',
        excludePunctuation: true,
        passwordLength: 16
      }
    });

    // Create API key secret
    const apiKeySecret = new secretsmanager.Secret(this, 'ApiKeySecret', {
      secretName: 'app/api-key',
      generateSecretString: {
        generateStringKey: 'api_key',
        excludePunctuation: true,
        passwordLength: 32
      }
    });
  }
}

Cost Optimization

Cost Management Strategies

  1. Resource Tagging
// lib/tagging.ts
export const addResourceTags = (resource: cdk.IConstruct) => {
  cdk.Tags.of(resource).add('Environment', process.env.ENV || 'development');
  cdk.Tags.of(resource).add('Project', 'MyApp');
  cdk.Tags.of(resource).add('CostCenter', 'CC123');
};
  1. AWS Budgets Setup
// lib/cost/budget-stack.ts
import * as cdk from 'aws-cdk-lib';
import * as budgets from 'aws-cdk-lib/aws-budgets';

export class BudgetStack extends cdk.Stack {
  constructor(scope: cdk.App, id: string, props?: cdk.StackProps) {
    super(scope, id, props);

    new budgets.CfnBudget(this, 'MonthlyBudget', {
      budget: {
        budgetName: 'Monthly-Budget',
        budgetLimit: {
          amount: 1000,
          unit: 'USD'
        },
        timeUnit: 'MONTHLY',
        budgetType: 'COST'
      },
      notificationsWithSubscribers: [
        {
          notification: {
            comparisonOperator: 'GREATER_THAN',
            notificationType: 'ACTUAL',
            threshold: 80,
            thresholdType: 'PERCENTAGE'
          },
          subscribers: [{
            address: 'team@example.com',
            subscriptionType: 'EMAIL'
          }]
        }
      ]
    });
  }
}

Conclusion

Building a modern DevOps pipeline requires careful consideration of multiple aspects:

  1. Containerization

    • Use multi-stage Docker builds
    • Implement proper layer caching
    • Follow security best practices

  2. AWS Infrastructure

    • Implement Infrastructure as Code
    • Follow the principle of least privilege
    • Use managed services when possible

  3. CI/CD Pipeline

    • Automate testing and deployment
    • Implement proper security checks
    • Maintain deployment consistency

  4. Monitoring and Security

    • Implement comprehensive logging
    • Set up proper alerts
    • Follow security best practices

  5. Cost Management

    • Implement proper resource tagging
    • Set up budgets and alerts
    • Regular cost optimization reviews

Remember to:

  • Regularly update dependencies and security patches
  • Monitor application performance and costs
  • Maintain proper documentation
  • Implement proper backup and disaster recovery procedures
  • Follow security best practices

Resources