Step 1 : Containerization

The first step is to containerize the application using Docker. We will write a multi-stage Dockerfile to ensure that our images are both secure and lightweight. The Dockerfile will include:

A base image for building the application.

A distroless image for running the application.

This approach not only reduces the image size but also enhances security by minimizing the attack surface.

First you need to understand the application. Here we're dealing with a Go-based portfolio web app. So, we will first run the application locally to understand how it works, what ports it uses, and how it’s built.

First, clone the repository from GitHub using:

git clone https://github.com/sgawali/news-demo-app.git

Now refer to readme file and follow all prerequisites & get started steps to run demo app locally

As shown in screenshot News Demo App will be launched in your browser. We can type any topic to search news related to that topic in search window and then hit enter to view all news related to that topic.

Writing the Multi-Stage DockerFile:

Using a multi-stage Dockerfile saves space and adds security layers to the Docker image. We first use Go lang base image to build the app and then copy it into a minimal distribution image, ensuring we reduce the size and potential vulnerabilities.

Here's a snapshot of what the Dockerfile looks like:

FROM golang:1.23 AS base

WORKDIR /app
# We want to populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .

RUN go mod download

COPY . .

# Build the Go app
RUN go build -o ./out/news-demo-app .

# Start fresh from a smaller distroless image
FROM gcr.io/distroless/base

COPY --from=base /app/out/news-demo-app /app/news-demo-app

COPY --from=base /app/assets ./assets

COPY --from=base /app/news ./news

COPY --from=base /app/index.html .

# This container exposes port 3000 to the outside world
EXPOSE 3000

# Run the binary program produced by `go install`
CMD ["/app/news-demo-app"]

This Dockerfile is an industry-standard approach. It builds the app, then packages it into a lightweight, distroless container which is more secure and faster to deploy.

Once your Dockerfile is ready, you can build and run your image:

docker build -t news-demo-app:v1 .

docker run -p 3000:3000 go-web-app:v1

Visit localhost:3000 and you'll see your app running in the container!

Step 2 : Creating Kubernetes Manifests

Next we will create kubernetes manifests, including deployment, service and ingress configurations. This will allow us to deploy our application on a kubernetes cluster effectively. The deployment manifests will specify the number of replicas, container image and ports.

Deployment YAML

  apiVersion: apps/v1
kind: Deployment
metadata:
  name: news-demo-app
  labels:
    app: news-demo-app
spec:
  replicas: 2
  selector:
    matchLabels:
      app: news-demo-app
  template:
    metadata:
      labels:
        app: news-demo-app
    spec:
      containers:
      - name: news-demo-app
        image: samgawali/news-demo-app:v3
        ports:
        - containerPort: 3000

Service YAML

A Service allows your app to be discoverable

apiVersion: v1
kind: Service
metadata:
  name: news-demo-app
spec:
  selector:
    app: news-demo-app
  type: ClusterIP
  ports:
  - port: 80
    targetPort: 3000
    protocol: TCP

Ingress YAML

while an Ingress helps map external traffic into your cluster’s services.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: news-demo-app
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /

spec:
  ingressClassName: nginx
  rules:
  - host: "*.samgawali.com"
    http:
      paths:
      - pathType: Prefix
        path: "/"
        backend:
          service:
            name: news-demo-app
            port:
              number: 80

At this stage, your app is ready for deployment into a Kubernetes cluster.

Step 3 : Creating a Kubernetes Cluster

For production-grade deployments, we'll set up an Civo Kubernetes Cluster

Install the necessary CLI tools including kubectl, Civo CLI

Authenticate your Civo account using api key

civo apikey add key-name “Your Civo API Key“

Create the cluster

civo kubernetes create demoapp-cluster --size "g4s.kube.medium" --nodes 2 --wait --save --merge --switch --region FRA1 -r=Traefik -a=Nginx --create-firewall --firewall-rules "80,443:0.0.0.0/0"

Once done, use kubectl to deploy the Kubernetes manifests you created earlier. A simple kubectl apply command will do the trick:

kubectl apply -f deployment.yaml
kubectl apply -f service.yaml
kubectl apply -f ingress.yaml

Be sure to expose the service using a NodePort or Load Balancer for external access!

Ingress Controller Installation & Configuration

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.8.2/deploy/static/provider/cloud/deploy.yaml

This will install nginx ingress controller in ingress-nginx namespace

A Loadbalancer will be created in Civo Kubernetes Cluster as shown in below screenshot

DNS Mapping

We will map our domain to the application using DNS. This will allow users to access the application via a friendly url instead of an ip address. I am using Civo DNS in Civo dashboard to configure my website as DSN for our application.

Step 4 : Setting Up Helm Chart

Helm simplifies managing your Kubernetes configs by allowing you to easily deploy to different environments (e.g., Dev, QA, Prod). If you need to deploy the same app across multiple environments with minimal changes, Helm is your go-to tool.

To get started, create a Helm chart:

helm create news-demo-app-chart

Now delete all sample files in templates folder and copy all our app Kubernetes manifests files into it for details you can refer to my git repository.

Then, define the values for each environment in values.yaml file

Test if generated helm chart is perfect or not with below commands

helm lint ./news-demo-app-chart    

helm template ./news-demo-app-chart 

helm install news-demo-app ./news-demo-app-chart --dry-run

Deploy the Helm chart with a command :

helm install go-web-app ./go-web-app-chart

Step 5 : Implementing Continuous Integration (CI) with GitHub Actions

GitHub Actions is a powerful tool to automate your development workflows. Here's how you set up CI for the app:

Create a GitHub Actions Workflow

Under .github/workflows/ci.yml, implement several stages:

• Build and Test: Automatically build the Go app whenever code is pushed.

• Test: Test Go app

• Docker Image Build & Push: After tests pass, push the Docker image to Docker Hub.

• Update Helm: Push the updated Helm values file with the latest Docker image.

Here’s a snippet from the workflow:

# CICD using GitHub actions

name: CI/CD

# Exclude the workflow to run on changes to the helm chart
on:
  push:
    branches:
      - main
    paths-ignore:
      - 'helm/**'
      - 'k8s/**'
      - 'README.md'

jobs:

  build:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Set up Go 1.23
      uses: actions/setup-go@v2
      with:
        go-version: 1.23

    - name: Build
      run: go build 



  code-quality:
    runs-on: ubuntu-latest

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Test
      run: go test



  push:
    runs-on: ubuntu-latest

    needs: build

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v1

    - name: Login to DockerHub
      uses: docker/login-action@v3
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Build and Push action
      uses: docker/build-push-action@v6
      with:
        context: .
        file: ./Dockerfile
        push: true
        tags: ${{ secrets.DOCKERHUB_USERNAME }}/news-demo-app:${{github.run_id}}

  update-newtag-in-helm-chart:
    runs-on: ubuntu-latest

    needs: push

    steps:
    - name: Checkout repository
      uses: actions/checkout@v4
      with:
        token: ${{ secrets.TOKEN }}

    - name: Update tag in Helm chart
      run: |
        sed -i 's/tag: .*/tag: "${{github.run_id}}"/' helm/news-demo-app-chart/values.yaml

    - name: Commit and push changes
      run: |
        git config --global user.email "sameer@sgawali.com"
        git config --global user.name "sgawali"
        git add helm/news-demo-app-chart/values.yaml
        git commit -m "Update tag in Helm chart"
        git push

Now, pushing to the repo triggers automated testing, Docker pushing, and Helm updates.

Step : 6 Continuous Delivery (CD) with Argo CD

Argo CD provides a seamless way to implement GitOps, allowing automatic updates to your Kubernetes cluster every time there’s a new version.

Install Argo CD

Run the following to install Argo CD in your Kubernetes cluster:

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

Set Up Argo CD

Install a Helm chart directly through Argo CD by creating a new app in the Argo CD dashboard:

1. Set the project to "default."

2. Connect to the GitHub repository.

3. Specify the path to your Helm chart and set the sync policy to Automatic.

Now, Argo CD will monitor the Helm chart for new changes and deploy updated versions without any manual intervention!

As shown in Screenshots our News demo application is now accessible on my websites for which i have created DNS records. A change in app’s source code will automatically trigger Ci/CD pipelines and changes will be deployed automatically by Argo CD tool.

Conclusion

When you take a non-DevOps project and transform it into a fully automated, scalable, and reliable system, the possibilities are endless. We started with a basic Go lang demo news app and implemented a full DevOps cycle: containerizing the app, deploying it in Kubernetes with Helm, and automating continuous integration and deployment using GitHub Actions and Argo CD.

Now, it’s time for you to give it a shot! Check out the project repositories, follow the steps, and "DevOpsify" your next project.

Picture this: we are creating a YouTube-like platform where users can explore videos and channels. It’s a fantastic project, but managing the development workflow can be complex. That’s where GitLab CI/CD comes in, offering a straightforward and powerful solution to automate your work.

This step-by-step tutorial is for developers and devops enthusiasts looking to make their projects more efficient with CI/CD. By the end, you’ll know how to set up a robust CI/CD pipeline, including automated testing, code quality checks, and secure containerization. Let’s embark on this journey together and see how CI/CD can transform your development experience while building a React YouTube app clone.

Ready? Let’s simplify CI/CD and make your development life easier!

Step 1: Create an API key for Youtube.

Step 2: Create a Repository and push it to GitLab.

Step 3: Launch an Ec2 instance and run Sonarqube on it.

Step 4A: Create a .gitlab-ci.yml File.

Step 4B: Add the required variables for the project.

Step 5: Install Gitlab Runner on Ec2.

Step 6: Run the Application on the Docker container.

Step 7: Access the Application on Browser.

Step 1: Create an API key from Rapid API

Open a new tab in the browser and search for rapidapi.com

It will automatically provide your mail and select a mail to create an account

You will see the page like this and click on signup

Now click on Sign Up with Google

Select your mail here

It will automatically create your account now

Now in the search bar search for YouTube and select YouTube v3

Copy API Key as shown in screenshot and save it for further use at the docker stage.

docker build –build-arg REACT_APP_RAPID_API_KEY=<API-KEY> -t ${imageName} .

Step 2: Create a Repository and push it to GitLab

Go to GitLab.com and login to your account

Click on New Project

Click on Create Blank Project

Provide a name for the Project

Keep Visibility to the public

Uncheck the Readme and create the Project

Use the below commands to push code to GitLab

cd existing_folder
git init --initial-branch=main
git remote add origin https://gitlab.com/sgawali/youtubev1.git
git add .
git commit -m "Initial commit"
git push --set-upstream origin main

Files pushed to GitLab

Step 3: Launch an Ec2 instance and run Sonarqube on it

Log into AWS Console: Sign in to your AWS account.

Launch an Instance:

Choose “EC2” from services. Click “Launch Instance.”

Choose an AMI: Select an Ubuntu image.

Choose an Instance Type: Pick “t2.medium.”

Key Pair: Choose an existing key pair or create a new one.

Configure Security Group:

Create a new security group. Add rules for HTTP, and HTTPS, and open all ports for learning purposes. Add Storage: Allocate at least 10 GB of storage.

Launch Instance: Review and launch the instance.

Access Your Instance: Use SSH to connect to your instance with the private key.

Keep in mind, that opening all ports is not recommended for production environments; it’s just for educational purposes.

Connect to Your EC2 Instance and install docker:

Run the below commands to install the docker

sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER   #my case is ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock

After the docker installation, we will create a Sonarqube container (Remember to add 9000 ports in the security group).

Run this command on your EC2 instance to create a SonarQube container:

docker run -d --name sonar -p 9000:9000 sonarqube:lts-community

Now copy the IP address of the ec2 instance

<ec2-public-ip:9000>

Enter username and password, click on login and change password

username admin
password admin

Update New password, This is Sonar Dashboard.

Step 4A: Create a .gitlab-ci.yml File.

Now go to GitLab click on ‘+’ and click on Newfile

File name .gitlab-ci.yml

Content

stages:
    - npm

Install dependecy:
    stage: npm    
    image:
        name: node:16
    script:
        - npm install

Commit the changes and it will automatically start the build

Now click on Build and Pipelines

Now click on Running.

Click on Install dependency

You will see build output

Now add the Sonarqube stage to the pipeline

Go to the Sonarqube dashboard and click on Manually.

Provide the name of the Project and click on Setup

Select the CI tool as GitLab CI

Select Other because we are using the JS App

It will provide code and we need to create a file inside our repo

Go to Gitlab and click on + and Newfile

Filename is sonar-project.properties

Paste the content that you got from Sonarqube

The file looks like this and click on commit changes

Go to Sonarqube and click on continue

Now it will provide Variables to add to our GitLab

Step 4B: Add the required variables for the project.

Variables Generated

Now go to GitLab

Click on settings and CI/CD

Click on Expand in variables

Click on Add variable

Now go back to Sonarqube and copy the Key

Click on Generate a token

Again Click on Generate

Copy the token

Now come back to GitLab and add them like the below image and click on add variable.

Sonar token is added

Now go to the Sonarqube Dashboard again

Let’s add another variable, copy them

Now go to GitLab and click on Add variable

Add the copied values like the below image

Two variables were added.

Now go back to the Sonarqube Dashboard

Click on continue

It will provide and CI configuration file copy it and use it inside our .gitlab-ci.yml file

Now go back to GitLab and edit the .gitlab-ci.yml file

Full file (update with your content)

stages:
    - npm
    - sonar
Install dependecy:
    stage: npm
    image:
        name: node:16
    script:
        - npm install
sonarqube-check:
  stage: sonar
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script:
    - sonar-scanner
  allow_failure: true
  only:
    - main

Commit changes and it will automatically start the build.

Click on Build –> Pipelines

Click on Running

Now click on Sonarqube-check

Build output

Now add the next stage of the Trivy file scan

Update the .gitlab-ci.yml file

stages:
    - npm
    - sonar
    - trivy file scan
Install dependecy:
    stage: npm
    image:
        name: node:16
    script:
        - npm install
sonarqube-check:
  stage: sonar
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script:
    - sonar-scanner
  allow_failure: true
  only:
    - main
Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs .

Commit changes and go to pipeline stages

Click on the Trivy file scan

Build output

Add the Docker build and push stage

Before that Add docker credentials to GitLab Variables as secrets.

Go to the docker hub and create a Personal Access token

Click on your profile name and Account Settings

Now click on Security –> New Access Token

Provide a name –> Generate

Now copy the token and keep it safe

Now go back to Gitlab

Click on settings and CI/CD

Click on Expand in variables

Click on Add variable

Use your DockerHub username in value and Add variable

Key DOCKER_USERNAME

Click on Add variable again

Key DOCKER_PASSWORD

Variables added

Now add the below stage to the Configuration .gitlab-ci.yml file

Added Docker and Trivy image scan stages

stages:
    - npm
    - sonar
    - trivy file scan
    - docker
    - trivy image scan
Install dependecy:
    stage: npm
    image:
        name: node:16
    script:
        - npm install
sonarqube-check:
  stage: sonar
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script:
    - sonar-scanner
  allow_failure: true
  only:
    - main
Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs .
Docker build and push:
  stage: docker
  image:
    name: docker:latest
  services:
    - docker:dind
  script:
    - docker build --build-arg REACT_APP_RAPID_API_KEY=f0ead79813mshb0aa7ddf114a7dap1adb3djsn483b017de1a9 -t youtubev1 .
    - docker tag youtubev1 sevenajay/youtubev1:latest
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker push sevenajay/youtubev1:latest
Scan image:
  stage: trivy image scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image sevenajay/youtubev1:latest

Added stages

Commit changes and it will automatically start building.

Go to Pipelines view

Now click on Docker build and push

Build view

Go to Dockerhub and see that docker image is uploaded there

Now come back to GitLab and click on Trivy image scan

Output Log

Step 5: Install Gitlab Runner on Ec2

Go to GitLab and Click on Settings and CI/CD

Click on Expand at Runners

Click on Three dots and then click on Show Runner installation

Click on Linux and amd64 and copy the commands

Now come back to Putty or Mobaxtreme

Create a new file

sudo vi gitlab-runner-installation

Paste the below commands into it

# Download the binary for your system
sudo curl -L --output /usr/local/bin/gitlab-runner https://gitlab-runner-downloads.s3.amazonaws.com/latest/binaries/gitlab-runner-linux-amd64
# Give it permission to execute
sudo chmod +x /usr/local/bin/gitlab-runner
# Create a GitLab Runner user
sudo useradd --comment 'GitLab Runner' --create-home gitlab-runner --shell /bin/bash
# Install and run as a service
sudo gitlab-runner install --user=gitlab-runner --working-directory=/home/gitlab-runner
sudo gitlab-runner start

Provide executable permissions and run the script

sudo chmod +x <file-name>
./<file-name>

Installation completed

Start the GitLab runner

sudo gitlab-runner start

Now run the below command or your command to register the runner

Update the token is enough

sudo gitlab-runner register --url https://gitlab.com/ --registration-token <token>

Provide the details for registering the runner

1. Provide Enter for GitLab.com

2. For token we already added with token, so click on Enter again

3. Description as your wish

4. Tags also and you can use multiple tags by providing a comma after each tga

5. Maintenance note is just optional

6. For executors use Shell

Runner added successfully.

Start the GitLab runner

sudo gitlab-runner start

Run the GitLab runner

sudo gitlab-runner run

Go to GitLab and refresh the page once or click on Enable for this project

Now the runner is active and waiting for jobs

Click on the Pencil mark to edit

Click on the Check box to indicate whether this runner can pick jobs without tags.

Click on save changes.

Step 6: Run the Application on the Docker container

Now edit the .gitlab-ci.yml file for the deploy stage

The complete file

stages:
    - npm
    - sonar
    - trivy file scan
    - docker
    - trivy image scan
    - run container
Install dependecy:
    stage: npm
    image:
        name: node:16
    script:
        - npm install
sonarqube-check:
  stage: sonar
  image:
    name: sonarsource/sonar-scanner-cli:latest
    entrypoint: [""]
  variables:
    SONAR_USER_HOME: "${CI_PROJECT_DIR}/.sonar"  # Defines the location of the analysis task cache
    GIT_DEPTH: "0"  # Tells git to fetch all the branches of the project, required by the analysis task
  cache:
    key: "${CI_JOB_NAME}"
    paths:
      - .sonar/cache
  script:
    - sonar-scanner
  allow_failure: true
  only:
    - main
Trivy file scan:
  stage: trivy file scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy fs .
Docker build and push:
  stage: docker
  image:
    name: docker:latest
  services:
    - docker:dind
  script:
    - docker build --build-arg REACT_APP_RAPID_API_KEY=f0ead79813mshb0aa7ddf114a7dap1adb3djsn483b017de1a9 -t youtubev1 .
    - docker tag youtubev1 sevenajay/youtubev1:latest
    - docker login -u $DOCKER_USERNAME -p $DOCKER_PASSWORD
    - docker push sevenajay/youtubev1:latest
Scan image:
  stage: trivy image scan
  image:
    name: aquasec/trivy:latest
    entrypoint: [""]
  script:
    - trivy image sevenajay/youtubev1:latest
deploy:
  stage: run container
  tags:
    - youtube        #use your own tags 
  script:
    - docker run -d --name youtube -p 3000:3000 sevenajay/youtubev1:latest

Commit changes, it will automatically start to build

Click on Build –> Pipelines

Click on Running

The last stage is added to the Pipeline

If you get an error like this

Go to GitLab and click on deploy job

Let’s see what is the error

If you get an error like this, click on that link

It will open a new tab and provide a solution for that

Now go to Mobaxtreme and stop the Runner

Go to root and use the below commands

sudo su
sudo vi /home/gitlab-runner/.bash_logout

You will see file like this

Comment all lines

Save and exit from that file and restart GitLab runner

sudo gitlab-runner restart
exit #from root

Now start and Run the GitLab runner

sudo gitlab-runner start
sudo gitlab-runner run

Now go to GitLab –> Build –> Pipelines

Click on Run Pipeline

Build completed and click on Deploy job

See the output it ran a container on ec2 instance

Now go to MobaXtreme or Putty connect to ec2 instance and Provide the below command to see running containers.

docker ps

Container running.

Step 7: Access the Application on Browser

Copy the Public IP of the ec2 instance and paste it into the Browser.

Don’t forget to open the 3000 port in the Security Group

<Public-ip:3000>

Output

Step 8: Termination

1. Delete the personal Access token of the docker.

2. Delete the containers.

    docker stop <container name>
    docker rm <container name>
   

3. Delete the Ec2 instance.

And there you have it, folks! You’re now the DevOps master of your YouTube app deployment. We’ve journeyed through code, quality, security, containers, and automation, all with GitLab. Now it’s your turn to unleash the power of DevOps and take your YouTube app to the next level!

So, what’s next on your development adventure? Whether it’s conquering new projects, exploring even cooler tech, or simply celebrating with a well-deserved coffee break, remember that DevOps is all about making your life easier and your code better. Embrace the joy of seamless deployments, keep learning, and never stop having fun in the ever-evolving world of software development. 🚀😎