What Jenkins is—and where it came from

Jenkins began as a fork of Hudson in 2011, after a trademark dispute split the community. Hudson faded; Jenkins became the de facto open-source CI server for Java shops and spread to every stack via plugins. Today it is maintained by the Jenkins open-source community (governance under the CD Foundation) with LTS (long-term support) and weekly release lines.

Jenkins is not a single pipeline product like GitHub Actions or GitLab CI. It is a pluggable automation platform: you assemble CI/CD from jobs, steps, and hundreds of plugins. That flexibility built its install base—and its operational complexity.

For why CI servers mattered in the DevOps story, see Historical foundations of DevOps. For Git-native delivery patterns that often sit beside Jenkins, see Git as the control plane (GitOps) and Git & GitHub in depth.

CI, CD, and what Jenkins typically automates

Jenkins excels at glue work: talking to legacy SCM, mainframes, on-prem artifact repos, and bespoke deploy scripts. Greenfield teams often choose managed CI (GitHub Actions, GitLab CI, CircleCI) for lower ops burden—but Jenkins remains entrenched where plugins, compliance, or multi-vendor toolchains are non-negotiable.

Architecture: controller, agents, and remoting

At runtime Jenkins splits into two roles:

• Controller (formerly “master”) — UI, configuration, scheduling, fingerprints, credentials store, plugin host. Holds job definitions and build history (by default in $JENKINS_HOME on disk).
• Agents (formerly “slaves” / “nodes”) — machines or containers that execute build steps. The controller assigns work; agents report logs and status back.

Communication uses the remoting channel (JNLP/WebSocket depending on version and setup). Agents can be:

• Permanent — always-on VMs with labels like linux && docker.
• SSH agents — controller SSHs into a host and starts the agent process.
• Inbound agents — agent initiates connection to controller (common behind firewalls).
• Ephemeral — Kubernetes, Docker, EC2, Azure VM plugins spin agents per build and discard them after.

Each agent has executors (slots). One executor runs one build at a time on that agent. Bottlenecks usually mean too few executors, slow disks, or agents without required tools (JDK, Docker socket, cloud CLI).

# Mental model
Developer pushes commit
  → webhook / polling triggers Jenkins job
    → controller queues build on agent matching label
      → agent clones repo into workspace, runs steps
        → archives logs + artifacts back to controller
          → downstream jobs or notifications fire

Core concepts every operator should know

Plugins define most behavior. The Pipeline plugin family turns Jenkins into a code-driven orchestrator; without it, you are mostly clicking freestyle build steps in the UI—fine for 2010, painful for reviewable change management today.

Job types: freestyle, Pipeline, and Multibranch

Freestyle jobs

Sequential build steps configured in the UI: SCM checkout, shell, Ant/Maven, publishers. Still common in brownfield estates. Hard to version-control and review; prefer migrating to Pipeline.

Pipeline jobs

A Jenkinsfile in the repository defines stages. Two DSL flavors:

• Declarative Pipeline — opinionated, structured pipeline { } block; better errors and Blue Ocean visualization; recommended for new work.
• Scripted Pipeline — full Groovy (node { }); maximum flexibility, easier to write unmaintainable logic.

Multibranch Pipeline

Discovers branches and pull requests from SCM, creates a job per branch, runs the Jenkinsfile found in that revision. This is how Jenkins competes with “CI on every PR” products: one repo, many automatic pipelines.

Organization Folder plugins extend discovery to entire GitHub/GitLab organizations.

Declarative Jenkinsfile anatomy

// Jenkinsfile (declarative) — illustrative CI for a containerized service
pipeline {
  agent { label 'linux && docker' }

  options {
    buildDiscarder(logRotator(numToKeepStr: '30'))
    timeout(time: 45, unit: 'MINUTES')
    disableConcurrentBuilds()
    timestamps()
  }

  environment {
    REGISTRY = '123456789.dkr.ecr.us-east-1.amazonaws.com'
    IMAGE    = "${REGISTRY}/my-app"
  }

  parameters {
    choice(name: 'DEPLOY_ENV', choices: ['none', 'staging', 'prod'], description: 'Deploy target')
  }

  stages {
    stage('Checkout') {
      steps { checkout scm }
    }
    stage('Test') {
      steps {
        sh 'make test'
      }
      post {
        always { junit '**/target/surefire-reports/*.xml' }
      }
    }
    stage('Build image') {
      steps {
        sh "docker build -t ${IMAGE}:${BUILD_NUMBER} ."
      }
    }
    stage('Scan') {
      steps {
        sh 'trivy image --exit-code 1 --severity HIGH,CRITICAL ${IMAGE}:${BUILD_NUMBER}'
      }
    }
    stage('Push') {
      when { branch 'main' }
      steps {
        withCredentials([usernamePassword(credentialsId: 'ecr-push',
                          usernameVariable: 'U', passwordVariable: 'P')]) {
          sh 'echo $P | docker login -u $U --password-stdin $REGISTRY'
          sh "docker push ${IMAGE}:${BUILD_NUMBER}"
        }
      }
    }
    stage('Deploy staging') {
      when {
        allOf {
          branch 'main'
          expression { params.DEPLOY_ENV == 'staging' }
        }
      }
      steps {
        sh './deploy.sh staging ${IMAGE}:${BUILD_NUMBER}'
      }
    }
  }

  post {
    success { slackSend(color: 'good', message: "Build ${env.BUILD_URL} succeeded") }
    failure { slackSend(color: 'danger', message: "Build ${env.BUILD_URL} failed") }
  }
}

Key declarative building blocks:

• agent — where the pipeline runs (any, label, kubernetes { } pod template).
• stages / steps — ordered work units; steps are shell, bat, or plugin calls.
• when — conditional execution (branch, tag, parameter, environment).
• environment — env vars and credentials bindings.
• tools — auto-install JDK, Maven, Node from Jenkins tool installers.
• parallel — fan-out tests or matrix builds inside a stage.
• post — always / success / failure / unstable cleanup (archive, notify).

Store the Jenkinsfile on main (or default branch) so Multibranch discovery always has a template. For PR validation, use a lighter Jenkinsfile on feature branches or when { changeRequest() } blocks.

Shared Libraries: DRY pipelines across repos

Duplicated Groovy across fifty microservices becomes unmaintainable. Global Pipeline Libraries are Git repos Jenkins loads at runtime:

// vars/buildAndPush.groovy — callable step
def call(Map cfg) {
  pipeline {
    agent cfg.agent ?: 'linux'
    stages {
      stage('Build') { steps { sh "docker build -t ${cfg.image} ." } }
      stage('Push')  { steps { sh "docker push ${cfg.image}" } }
    }
  }
}

// Jenkinsfile in app repo
@Library('platform-pipelines@v2') _
buildAndPush(image: 'my-app:ci')

Libraries version like any dependency—pin @v2 tags, test library changes in a canary job before rolling org-wide. Treat library repos with the same review bar as production code; Groovy in libraries runs on the controller with broad powers.

Triggers: webhooks beat polling

• SCM webhook (GitHub, GitLab, Bitbucket plugins) — push/PR events enqueue builds immediately; lowest latency and load.
• Polling — H/5 * * * * cron asks SCM “anything new?”; simple but wasteful at scale.
• Timer — nightly security scans, dependency reports.
• Upstream — job B after job A succeeds (artifact promotion chains).
• Remote API — curl -X POST …/job/foo/build with token (guard with auth).

Configure webhooks with a shared secret; restrict ingress to Jenkins URL. For GitHub, use the GitHub App or OAuth plugin instead of personal tokens tied to individuals.

Agents at scale: Docker, Kubernetes, and cloud

Static agents are easy to reason about but expensive idle. Modern patterns:

• Docker plugin / Docker Pipeline — run each build in a container with pinned tool versions; mount Docker socket only when you accept the security trade-off (prefer Kaniko, BuildKit rootless, or cloud builders).
• Kubernetes plugin — define a podTemplate with JNLP inbound agent + sidecar containers (Maven, kubectl, terraform). Pods die after the build—clean isolation, cluster autoscaler friendly.
• EC2 / Azure VM agents — burst capacity for heavy jobs; watch AMI drift and boot time.

Label design matters: linux && docker && large routes big image builds; keep Windows/macOS agents separate. Pin tool versions in the agent image, not “whatever was installed years ago on the VM.”

For the cluster layer Jenkins agents often target, see Kubernetes architecture in simple terms and Docker’s hidden side.

Credentials, secrets, and supply chain

Jenkins stores secrets in its credentials store (scoped global, folder, or domain). Bind them in pipelines with withCredentials, sshagent, or withVault—never echo secrets in logs (mask known patterns; use wrap([$class: 'MaskPasswordsBuildWrapper']) where needed).

• Prefer short-lived cloud credentials (OIDC to AWS/GCP/Azure) over static keys.
• Integrate HashiCorp Vault or cloud secret managers so Jenkins is not the long-term vault of record.
• Rotate deploy keys and API tokens on a schedule; audit who can use each credential via RBAC.
• Scan Jenkinsfiles in PRs for hard-coded passwords—the same rule as application code.

The controller’s $JENKINS_HOME contains secrets on disk. Encrypt secrets at rest (credentials plugin + OS disk encryption), restrict filesystem backups, and treat controller snapshots as sensitive.

Artifacts, promotion, and immutable delivery

CI should produce one immutable artifact per build (container digest, JAR with hash, Helm chart version). Jenkins archives artifacts on the controller or uploads to Nexus, Artifactory, S3, or a container registry.

Promotion means the same artifact moves staging → prod with new configuration—not a rebuild “from the same commit” that might pick different dependencies. Use fingerprints or explicit version pins in downstream deploy jobs.

For Git-declared desired state in clusters, Jenkins often builds while GitOps controllers reconcile—see GitOps principles for how those roles split cleanly.

Security model and hardening checklist

Jenkins historically shipped open by default. Production hardening is non-optional:

• Enable authentication (LDAP, SAML/OIDC, GitHub OAuth)—disable anonymous admin.
• Use Matrix Authorization Strategy or Role-Based Strategy plugin—least privilege per folder/job.
• Keep Jenkins and plugins on LTS; run plugin compatibility checks in a staging controller.
• Disable or sandbox script approval for Groovy that runs on the controller; audit approved signatures.
• Run controller behind HTTPS with modern TLS; restrict agent port exposure.
• Separate controller and agent networks; agents should not reach admin APIs.
• Enable audit logging (who triggered deploys, who changed jobs).
• Back up JENKINS_HOME with encrypted storage; test restore.

Agent compromise is a common blast radius: a malicious PR can exfiltrate secrets from the workspace or abuse mounted cloud credentials. Use ephemeral agents, read-only SCM for PR builds where possible, and separate credentials for “build PR” vs “deploy prod.”

High availability, performance, and operations

The controller is a single logical brain. Options for resilience:

• Active/passive — standby controller, restore JENKINS_HOME from backup (minutes of downtime).
• Shared storage — NFS/EFS for JENKINS_HOME with care for file locking and plugin I/O.
• CloudBees or enterprise distributions — HA features, tested plugin sets, support contracts.

Performance tuning levers:

• Archive less junk—use artifacts {} only for what downstream needs.
• Rotate build history (buildDiscarder) and logs.
• Move heavy work to agents; keep controller CPU for scheduling and UI.
• Use webhook triggers instead of five-minute SCM polling across hundreds of repos.
• Split monolithic “god pipelines” into parallel stages and separate jobs.

Monitor queue time, executor utilization, disk on controller, and plugin update lag. Pair Jenkins metrics with your observability stack (Prometheus plugin, health check URLs).

Jenkins vs GitHub Actions, GitLab CI, and Tekton

Many organizations run more than one: Jenkins for release trains and regulated paths, Actions for library repos, Tekton for cluster-native platform teams. Standardize artifacts and signing—not necessarily a single CI engine.

Production practices that survive audits

1. Everything in Git — Jenkinsfile, shared libraries, Job DSL or Configuration as Code (JCasC) for controller settings.
2. Folder per team — isolate credentials, quotas, and naming (team-a/service-x/main).
3. Separate controllers — prod deploy credentials never on the same controller that runs untrusted PR jobs.
4. Fast feedback — target sub-ten-minute PR pipelines; push slow e2e to nightly.
5. Immutable artifacts + signed images — Cosign/Sigstore or Notary in the pipeline.
6. Policy gates — fail builds on critical CVEs, license violations, or missing tests.
7. Document escape hatches — who can replay a prod deploy job and under what change ticket.

Configuration as Code example (JCasC fragment):

# jenkins.yaml (JCasC) — illustrative
jenkins:
  systemMessage: "Production CI — changes via PR to infra-jenkins repo"
  numExecutors: 0   # controller builds nothing; agents only
securityRealm:
  saml:
    binding: "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-Redirect"
    idpMetadataConfiguration:
      url: "https://idp.example.com/metadata"
authorizationStrategy:
  roleBased:
    roles:
      global:
        - name: "admin"
          permissions: ["Overall/Administer"]
          assignments: ["platform-admins"]

Troubleshooting playbook

Use Pipeline Steps view and Console Output first; enable timestamps() and Blue Ocean or Stage View for stage-level timing. For shell debugging on agents, see Terminal & DevOps: Linux in depth.

Learning path (hands-on)

1. Run Jenkins LTS locally: docker run -p 8080:8080 -p 50000:50000 jenkins/jenkins:lts (lab only—not production defaults).
2. Create a Multibranch Pipeline from your GitHub repo; add a minimal Jenkinsfile with checkout + test.
3. Add a Docker agent or Kubernetes pod template; move builds off the controller.
4. Wire a webhook; disable SCM polling.
5. Extract a repeated stage into a Shared Library; version it.
6. Add credential binding for registry push; scan images before push.
7. Export controller config with JCasC; store in Git and review changes like application code.

Further reading on this site

• GitHub CI/CD in depth — Actions vs Jenkins trade-offs
• GitLab CI/CD in depth — integrated platform alternative
• Bitbucket Pipelines CI/CD — Atlassian cloud pipelines
• Historical foundations of DevOps — CI servers in context
• DevOps life & business value — why pipeline speed maps to outcomes
• Git & GitHub in depth — SCM and webhooks
• GitOps principles — delivery after the build
• Docker’s hidden side — images agents run
• Incidents & disaster response — when deploy pipelines fail in prod

Blog index · GitHub CI/CD · GitLab CI/CD · GitOps principles

Back to blog list

Jenkins® is a registered trademark of LF Projects, LLC. Product names and plugins mentioned are for education; always verify versions and security advisories for your deployment.