Kubernetes Cloud Agent (on GCP)

1. What to use Kubernetes Cloud Agent for?
Docker Cloud Agent is a agent of Jenkins to clone source, build, test for CI.

2. How to use Kubernetes Cloud Agent?
- Install Kubernetes Plugin (to integrates Jenkins with Kubernetes)

3. What does Kubernetes Cloud Agent include? 

--------------------------
Detailed instructions

Thank you.

Jenkins nodes on GCP Compute engine with SSH

1. What to use Jenkins nodes on GCP Compute for?

- For testing, deployment application

2. How to use Jenkins nodes on GCP Compute?

- Create GCP Compute instance

- Create permanent node in Jenkins

References:
- Youtube: 17. CI/CD with Jenkins. Jenkins nodes on GCP Compute engine with SSH

Pipelines & Jenkinsfile in Jenkins

1. What to use Pipelines & Jenkinsfile for?

-  A Pipeline is a series of automated executed job steps, is used to automate CICD (build, test, deploy, report)

- A Jenkinsfile is a text file, is used to define a Jenkins Pipeline.

2. How to use Pipelines & Jenkinsfile?

- Create a Pipeline is create a job in Jenkins >> see more: Create a Pipeline

- Jenkinsfile syntax:

pipeline {

    agent none

options {
timeout(time: 1, unit: 'HOURS') // Sets a timeout limit for the pipeline
retries(2) // Retries the pipeline 2 times in case of failure
buildDiscarder(logRotator(numToKeepStr: '5')) // Keeps the logs of the last 5 builds
} 

  environment {
MY_VARIABLE = "someValue" // Define a simple environment variable
PASSWORD = credentials('my-credential-id') // Safely inject a password from Jenkins credentials
}

    stages {

stage('Advanced Script') {
steps {
script {
// Your Groovy code here

                   echo "Environment variable is $MY_VARIABLE"
}
}
} 

        stage('Build') {

            agent any 

            steps {

                // Steps to build the application

                echo 'This is Build stage.'

            }

        }

        stage('Test') {

            agent { 

                label 'linux-server-test' 

            } 

            steps {

                // Steps to Test the application

            }

            post { 

                always { 

                    echo 'Test on linux-server-test finished.'

                } 

            }

        }

        stage('Deploy') {

            agent { 

                label 'linux-server-deploy' 

            } 

            steps {

                // Steps to Deploy the application

            }

            post { 

                always { 

                    echo 'Deploy on linux-server-deploy finished.'

                } 

            }

        }

    }

    post {

        always {

            echo 'This message will always be printed.'

        }

        success {

            echo 'This message will only be printed if the pipeline succeeds.'

        }

        failure {

            echo 'This message will only be printed if the pipeline fails.'

        }

    }

}  

Explain:
+ pipeline: define main pipeline
+ agent: specifies agent stages will be executed
+ option: to change default behavior of Jenkins pipeline
+ environment: define variables for using by all stages, steps, script in pipeline.
+ stages: define stages of pipeline
+ stage: define a specific stage
+ steps: contents steps to perform work in stage
+ script: to using Groovy code in pipeline
+ post: define actions or steps that are executed at the end of pipeline
+ always: steps are executed after every run of pipeline
+ success: steps are executed only if pipeline completes success
+ failure: steps are executed only if pipeline fails 

Reference
- https://www.jenkins.io/doc/book/pipeline/jenkinsfile/

Thank you.

Permanent Agent server using VPS in Jenkins

1. What to use Permanent Agent server for?

- Permanent Agent is a Jenkins agent for deploy (CD) app.

2. How to use Permanent Agent server?

- Install Java in server agent: install java version, setup JAVA_HOME path the same with jenkin master server, example jdk11.0.11, JAVA_HOME=/usr/lib/jvm/..

- Create agent: Mange Jenkins > Manage node and cloud > New node > Node name: Agent1 > Type: checked Permanent Agent > Number of executors: 2 > Remote root directory: /home/jenkins/Agent1 > Labels: Agent1 Linux > Usage: Use this node as much as possible > Launch method: Launch agents via SSH > Disable WorkDir: not checked > Custom WorkDir path: /home/jenkins/ Availability: Keep this agent online as much as possible > Save
+ Number of executors: with Executor is represent a separate process capable of executing a job, ex = 2 => can run 2 jobs same time.
+ Launch agent has two ways:
/ By connecting it to master (also call controller): Agent connect to Jenkins, by run jar file download from jenkins (using when don't want open ssh).
/ Via SSH: Jenkins connect to Agent, by using ssh to agent (using when has ssh on agent): Host: Agent1 > Credentials: (Select existed) || Add new > Domain: Global credentials (unrestricted) > Kind: SSH Username with private key || (Username & Password) > ID: CredentailLinuxAgent > return to select existed Credential already create > Host Key Verification Strategy: None verify verification Strategy. > Click Advanced > Port: 22 (your ssh agent port) 
+ Remote root directory: the path for jenkin create, manage working file...
+ Host key verification Strategy:
/ None verify verification Strategy (just connect, ignore everything)
/ Know hosts file Verification Strategy (need config ssh agent host in known host verification on jenkins master host: go to host master > cd /var/lib/jenkins/ (|| jenkins path docker if you use docker run jenkin: ../laradock/jenkins/jenkins_home) > mkdir -p /var/jenins/.ssh > ssh-keyscan -H Agent1 >> var/lib/jenkins/.ssh/know_hosts (collect ssh key from Agent1 server, after write them to know_hosts file > chow -R jenkins:jenkins /var/lib/jenkins/.ssh).

- Run the permanent Agents:
+ Configure global security: Agent: TCP port for inbound agents: Fixed: 50000
+ Manage nodes and clouds > Click Open the agent > run from agent command line in /home/jenkins (to install java, javac)

- Using the Permanent Agent: Select job/ Configure/ checked Restrict where this project can be run?/ Label Expression: Agent1/ Save/ Build job again/

Reference: Setting up Permanent Agents

3. What does Permanent Agent server include? 

--------------------------
Detailed instructions

References:
- https://www.youtube.com/watch?v=99DddJiH7lM

Thank you.

Docker Pipeline Plugin in Jenkins

1. What to use Docker Pipeline Plugin for?
- To use Docker as a Agent

2. How to use Docker Pipeline Plugin?

- Install Docker pipeline plugin: Manage Jenkins/ manage Plugins/ search Docker Pipeline/ Download now and restart/ Restart

- Using this git repository

- Create job pipeline: New item/ name: Docker/ Select Pipeline/ OK/ select Pipeline script from SCM/ SCM: Git/ Reposiotry: url git repository/ Credentials none (because this repository is public) / branch: main/ script path: Jenkinsfile-1 | Jenkinsfile-2 | Jenkinsfile-3 / Save / Build -> show Result

3. What does Docker Pipeline Plugin include? 

-------------------------
Detailed instructions

Thank you.

Jenkins tutorial from basic to advanced

1. Outline learning jenkins
- Jenkins
- Jobs, Builds & Triggers
- Builds
- Pipelines & Jenkinsfile
- Views
- Master server
- Agents (Nodes/Slaves)
- Plugins
- Credentials
- Parameters
- SCM (Source code management)
- Ngrok
- Python Jenkins (automated for Jenkins)
- Laravel CI/CD with Jenkins, Docker, AWS
- Implementing CI/CD for a Laravel App with Jenkins Using Docker Cloud Agents

2. What to use Jenkins for?
- Jenkins is used for CI/CD (continuous integration/ continuous delivery)
+ Automated builds
+ Continuous integration (CI)
+ Automated testing
+ Continuous Deployment (CD)  
+ Monitoring and reporting

- Jenkins Infrastructure:
+ Master server: Controls pipelines, Schedules Builds
+ Agents: Perform the Build

- Workflow in Jenkins:
+ Developer: commit the code
+ Jenkins Master server: aware the commit change code, then Triggers Pipeline, and distribute to a Agents to run (The Agent was selected base on Configure Labels)
+ Agents run the build: Build, test, distribute code to Server to deploy.

3. How to use Jenkins?
- Installation
- Accessing Jenkins
- Creating Jobs
- Building Jobs
- View Build results
- Monitoring and maintenance
- Scaling and Optimization
- Continuous integration and Deployment

Reference
- https://www.jenkins.io/doc/tutorials/
- https://www.tutorialspoint.com/jenkins/index.htm

Thank you

Docker Cloud Agent in Jenkins

1. What to use Docker Cloud Agent for?
Docker Cloud Agent is a agent of Jenkins to clone source, build, test for CI.

2. How to use Docker Cloud Agent?
- Setting up Docker cloud Agents

3. What does Docker Cloud Agent include? 

--------------------------
Detailed instructions

* Setting up Docker Cloud Agents (1)
- Prerequisite: Docker installed
- Prepare to connect docker in host
+ Install Plugin cloud provider/ select plugin: Docker/ Download now and restart after install.
/ There are many plugins for Cloud providers: Docker, Kubernetes, Amazon EC2, Azure VM Agents,...
+ Build and run socat container (2) to connect Jenkins with Docker (running jenkins as container, the container can't reach docker host port).

docker run -d --restart=always -p 127.0.0.1:2376:2375 --network jenkins -v /var/run/docker.sock:/var/run/docker.sock alpine/socat tcp-listen:2375,fork,reuseaddr unix-connect:/var/run/docker.sock
docker inspect <container_id> | grep IPAddress

- Connect docker in host: Dashboard/ Manage Jenkins/ Manage nodes and clouds/ Configure Clouds/ (Go to plugin manager to install Plugin cloud provider)/Add a new cloud/ Select Docker/ Docker cloud detail/ Docker Host URI: tcp://<IPAddress_Socat>:2375/Check Enable/ Test Connection.
- Create Docker Agent templates (can create multiples): Add Docker Template/ Labels: Docker-agent-alpine/ checked Enabled/ Name: Docker-agent-alpine/ Docker Image: jenkins/agent alpine-jdk11/ Instance Capacity: 2/ Remote File System Root: /home/jenkins/ Save
+ Labels: for jobs select which agent should run the build.
+ Instance Capacity: is how many actually spawn.
+ Remote File System Root: workspace of Jenkins is going to get created.
- Using Agents in job: Select job/ Configure/ checked Restrict where this project can be run?/ Label Expression: Docker-agent-alpine/ Save/ Build job again/

Reference
- (1) (Setting up Docker Cloud Agents 00:33:40 Setting up Docker Cloud Agents)
- (2) Build and run socat container

Implementing CI/CD for a Laravel App with Jenkins Using Docker Cloud Agents

1. Architecture
- Source Repository for store source code (Github)
- Jenkins server for master server handle Pipeline CICD (using local)
- Docker host to clone source, build, test for CI (using local)
- Server to deploy Laravel app for CD (VPS || Cloud)

2. Install tools & prepare source code
- Prepare source code Laravel
- Install Docker in local
- Install Jenkins using Docker in local: Using Docker plugin | Docker pipeline plugin
- Setup Docker cloud agent for CI (to clone source, build, test)
- Setup Permanent Agent server (using VPS) for CD (to deploy Laravel app)
|| Cloud Agent server (using AWS||GCP||Azure) for CD (to deploy Laravel app)

3. Setup Pipeline CICD
- Create job DeployLaraveApp
+ Type: pipeline
+  Trigger: schedule to check source change || when source code change (case Jenkin server not using in local by using webhook in github)
+ Using jenkinFile
- Run build when trigger happen.

-------------------------
Detailed instructions

* Prerequisite
- Install Jenkins, Docker

* Source Repository (Github)
- Source Laravel
- dockerfile/ docker_compose file
- Jenkinfile

* Setup Docker cloud agent for CI (to clone source, build, test)
+ Using: Docker Pipeline Plugin in Jenkins
Or using: Docker Cloud Agent in Jenkins

* Setup Permanent Agent server (using VPS) for CD (to deploy Laravel app)

* Create job DeployLaraveApp

* Using Ngrok for Jenkins in localhost: https://ngrok.com
- run: ngrok http http://localhost:8090
- endpoint page: https://dashboard.ngrok.com/cloud-edge/endpoints

* Setting webhook github for Jenkins in localhost

- When create Pipeline:.../ Build Triggers: select Github hook trigger for GITScm polling.

- Add webhook in Github: Select Repository/ Setting/ Webhooks/ Add webhook/ Payload Url: enter url get from Ngrok (for jenkins localhost)/: {JenkinServer}/github-webhook/ / Content type: application/json / Which event would you like to trigger this webhook: Let me select individual events: Enable Pull request, Pushes.. / Add webhook.

* Set permission for jenkin handle with /var/www

- Copy source code from workspace in jenkins to VPS server

cp -r * /var/www/html

- When install nginx/ apache2, then auto create group www-data
- or using this: groupadd www-data
- Add user jenkin to this group: 

sudo usermod -aG www-data jenkins

sudo newgrp www-data // to switch to www-data group for current user (because one user can has many groups)

- Change owner directory, change permission

sudo chown jenkins:www-data -R html

sudo chmod -R 2771 html 

// 2: is the set-group-id: ensure that the created file in html directory inherit the group ownership of html directory

Thank you.

Jobs, Builds & Triggers in Jenkins

1. What to use Jobs, Builds & Triggers for? 

- Jobs is individual tasks or workflows that need to be executed. These tasks could include building, testing, deploying, or any other automation task.

- Builds is the execution of a job. When a job is triggered Jenkins initiates a build process to perform the defined tasks within that job.

- Triggers in Jenkins determine how and when a build should start:

+ Build after other projects are built: This trigger allows a job to be executed after one or more specified projects have completed their builds. It's useful for setting up a sequence of jobs where the output of one job is required before starting another. For example, you might have a deployment job that should only run after a successful build job

+ Build periodically:With this trigger, you can schedule jobs to run at regular intervals, using a cron-like syntax to specify the schedule. It's ideal for workflows that need to occur regularly regardless of code changes, such as nightly builds or weekly integration tests.
GitHub hook trigger for GITScm polling:

+ This trigger is designed for projects hosted on GitHub. It utilizes webhooks to start builds automatically when a change is pushed to the repository. Instead of polling GitHub for changes, which can consume resources and delay builds, the webhook immediately notifies Jenkins of any SCM changes, triggering the build process. 

+ Poll SCM: "Poll SCM" periodically checks the source code management (SCM) system for changes and triggers a build if any are found. You define the polling frequency using cron syntax. While effective, it's less efficient than webhook-based triggers (like the GitHub hook trigger) because it periodically checks for changes rather than reacting to them instantly.

+ Quiet period: The quiet period is not a trigger by itself but a configuration option that can be used with other triggers. When a build is triggered, Jenkins waits for the specified quiet period (in seconds) before starting the build. This delay can be useful to batch SCM changes or to reduce the load on the build system by preventing too many builds from starting in quick succession.

+ Trigger builds remotely (e.g., from scripts):This option allows builds to be triggered remotely via HTTP GET or POST requests. It's particularly useful for integrating Jenkins with other tools or scripts that aren't directly supported by Jenkins plugins. You can secure this trigger with an authentication token to ensure that only authorized sources can initiate the build.

2. How to use Jobs, Builds & Triggers?

- Jobs: Create Jobs, Configure Jobs, Run Jobs.

- Builds: View Builds, Monitor Builds, Analyze Builds.

- Triggers: Configure Triggers, Define Trigger Conditions, Monitor Trigger Events.

----------------------------
Detailed instructions

- Triggers:
+ Add triggers for job: select job/ Build Triggers: Poll SCM/ Schedule: */5**** (execute a task every 5 minutes)/ Save 

- Environment Variables in command of Build (JenkinFile)
+ Configure job/ Pipeline Syntax/ Global Variable Reference

Reference

Thank you

Agents (Nodes/Slaves) in Jenkins

1. What to use Agents for?
- Agents allow you to run jobs on different machines
- When don't has agent jobs will run on Master server (but will many jobs run same time is easy errors happen)
=> jobs can run on: 1) master server (default); 2) Permanent Agents; 3) Cloud Agents

- Agent types:
+ Permanent Agents: Use for VPS (Virtual private server: DigitalOcean, Linode, PAVietNam...): Manual manage, Jenkins connect to by SSH to distribute the job.
+ Cloud Agents: Use for Cloud (Docker, AWS EC2, Google Compute Engine (GCE), Azure Virtual Machines (VMs)...): Automated mange, Jenkins connect to by Cloud Provider, type of cloud provider want to connect to (e.g., Amazon EC2, Google Compute Engine, Azure VMs).

2. How to use Agents?
- Setting up Permanent Agents
- Setting up Cloud Agents

3. What does Agents include?
- Permanent Agent
- Permanent Agent on GCP
- Docker Cloud Agent (using Docker plugin)
- Docker Cloud Agent (using Docker Pipeline plugin)
- Kubernetes Cloud Agent (on GCP)
- Aws fleet manager (on AWS)

--------------------------
Detailed instructions

* Setting up Cloud Agents ( traditional cloud agents such as AWS, Google Cloud, or Azure)

- TO DO...

Thank you.

Jenkins automation using Python

* Read more: Key concepts in Jenkins

* Docs API: https://python-jenkins.rea=dthedocs.io/en/latest/

* Automation for Laravel CI/CD with Jenkins, Docker, AWS  

1. Install Plugin

2. Create credential connect to Github

3. Create credential connect to EC2(AWS)/ VM(GCP)/ VPS

4. Create a job CI/CD

5. Run build job above

Thank you

Key concepts in Jenkins

* Jobs object

- Jobs are the fundamental building blocks in Jenkins. They represent tasks that Jenkins will execute, such as building code, running tests, or deploying applications.

- Types of job: Freestyle project, Pipeline...

* Views object

- Display A list of jobs

* Agents (Nodes/Slaves) object

- Agents are machines configured to execute Jenkins jobs. They can be physical machines, virtual machines, or containers. Jenkins can distribute jobs across multiple agents for parallel execution. 

- Manage jenkins/ manage node and clouds/...

* Plugins

- Plugins extend Jenkins functionality by adding new features, integrations with third-party tools, and additional build steps. There are thousands of plugins available in the Jenkins ecosystem.

* Credentials

- Credentials are used to securely store sensitive information such as usernames, passwords, API tokens, SSH keys, etc., which are needed for authentication with external systems.

* Builds & Triggers

- A build is an instance of a job being executed by Jenkins. Each build produces logs, artifacts, and other metadata.

- Triggers define conditions under which a job should be executed. They can be based on SCM changes, timers, manual triggers, or other events.

* SCM

- Source code management systems (SCMs) such as Git, Subversion, Mercurial, etc. This allows Jenkins to automatically trigger builds based on code changes. 

Thank you

Laravel CI/CD with Jenkins, Docker, AWS

1. Install docker, docker compose

- add current $USER to docker group

sudo usermod -aG docker $USER

2. Download source code and build in local using docker-compose

https://github.com/thecaringdeveloper/laravel-cicd-tutorial (1)

docker-compose.yml

3. Install, start Jenkins

- install java

- install jenkins

- start jenkins

- access jenkins localhost:8090

- add user jenkins to docker group: sudo usermod -aG docker jenkins

- add plugin for jenkins

+ Pipeline Utility Steps

+ File Operations

+ SSH Agent

4. Create github repository

- push code in (1) to repository

5. Setup pipeline in source local (a job)

- create Makefile

- create Jenkinsfile

- create a job:

 Enter an item name/ Pipeline 

/ General: Testing CICD

/ Pipe script from SCM

/ Repository URL: get from repository in github

/ Credentials select credential create below

/ Branch: select branch in git repository

/ Script path: Jenkinsfile

6. create Credential for jenkins connect to github

Manage Jenkins/ manage credentials/ global /add credentials

    / Kind: SSH username with private key 

    / scope: Global (jenkins, nodes, items, all child items, ect)

    / id: github

    / username: your_github_username

    / create private, public key: private_key for jenkins, public_key  for github

    / Create

* Create a job for CICD

    / name: laravel-test

    / Pipeline: Piple script from SCM

    / SCM: git

    / Repository Url: Your_repo_url

    / Credentials:credential already create above

    / Script path: jenkinsfile 

7. Populate .env by using jenkin

+ copy .env file to jenkin: 

    / create folder in workspace jenkins include .env file: var/lib/jenkins/workspace/envs/laravel-test

    / create .env file in var/lib/jenkins/workspace/envs/laravel-test

    / add script to copy this .env file when deploy 

stage("Populate .env file") {

steps {

dir("/var/lib/jenkins/workspace/envs/LaravelTest02") {

fileOperations([fileCopyOperation(excludes: '', flattenFiles: true, includes: '.env', targetLocation: "${WORKSPACE}")])

}

}

}

- Run pipeline

6. Setup EC2

- get key-ec2.pem to ssh to ec2

- chmode  400 key-ec2.pem

- create credential for ec2 in jenkins

7. Create Artifact 

Create artifact.zip file

8. Copy Artifact artifact.zip file to ec2

update Jenkinsfile to copy Artifact artifact.zip file to ec2

9. Deploy on ec2, install server on ec2

- install server: sudo yum -y install httpd

- start server: sudo service httpd start

- edit inbound, outbound rules

- install php and extension: sudo yum install php-{...}

- set permission: sudo chown ec2-user:ec2-user /var/www/html

- sudo server httpd restart

Jenkinsfile

pipeline {

agent any

stages {

stage("Verify SSH connection to server") {

steps {

sshagent(credentials: ['aws-ec2']) {

sh '''

ssh -o StrictHostKeyChecking=no ec2-user@52.221.199.29 whoami

'''

}

}

}

stage("Verify tooling") {

steps {

sh '''

docker info

docker version

docker compose version

'''

}

}

stage("Clear all running docker containers") {

steps {

script {

try {

sh 'docker rm -f $(docker ps -a -q)'

} catch (Exception e) {

echo 'No running container to clear up...'

}

}

}

}

stage("Start Docker") {

steps {

sh 'make up'

sh 'docker compose ps'

}

}

stage("Run Composer Install") {

steps {

sh 'docker compose run --rm composer install'

}

}

stage("Populate .env file") {

steps {

dir("/var/lib/jenkins/workspace/envs/LaravelTest02") {

fileOperations([fileCopyOperation(excludes: '', flattenFiles: true, includes: '.env', targetLocation: "${WORKSPACE}")])

}

}

}

stage("Run Tests") {

steps {

sh 'docker compose run --rm artisan test'

}

}

}

post {

success {

sh 'cd "/var/lib/jenkins/workspace/LaravelTest02"'

sh 'rm -rf artifact.zip'

sh 'zip -r artifact.zip . -x "*node_modules**"'

withCredentials([sshUserPrivateKey(credentialsId: "aws-ec2", keyFileVariable: 'keyfile')]) {

sh 'scp -v -o StrictHostKeyChecking=no -i ${keyfile} /var/lib/jenkins/workspace/LaravelTest02/artifact.zip ec2-user@52.221.199.29:/home/ec2-user/artifact'

}

sshagent(credentials: ['aws-ec2']) {

sh 'ssh -o StrictHostKeyChecking=no ec2-user@52.221.199.29 unzip -o /home/ec2-user/artifact/artifact.zip -d /var/www/html'

script {

try {

sh 'ssh -o StrictHostKeyChecking=no ec2-user@52.221.199.29 sudo chmod 777 /var/www/html/storage -R'

} catch (Exception e) {

echo 'Some file permissions could not be updated.'

}

}

}

}

always {

sh 'docker compose down --remove-orphans -v'

sh 'docker compose ps'

}

}

}

Makefile

#!/usr/bin/make

SHELL = /bin/sh

UID := $(shell id -u)

GID := $(shell id -g)

USER:= $(shell whoami)

export UID

export GID

export USER

up:

docker compose up -d

Thank you