Shared libraries

One reviewed pipeline vocabulary for every repo.

A restaurant chain with forty kitchens hands every cook a photocopied recipe card. The food safety rule changes one morning, and now somebody has to chase down forty cards. Twelve get fixed. Twenty-eight quietly keep serving the old version. That is what you have when forty repositories each carry their own Jenkinsfile, the file that defines a build pipeline as code. They all do roughly the same thing: check out source, build it, run a security scan, push an image. Someone wrote the first one and copy-pasted it thirty-nine times, and the copies have drifted ever since. A shared library is the fix. It is a separate Git repository written in Groovy (the scripting language that runs on the Java Virtual Machine, and the same language Jenkins pipelines are written in) that holds your common pipeline logic in one reviewed place. Every Jenkinsfile imports it. Write a step once and every project gets it, and each Jenkinsfile shrinks from a hundred lines to a few.

Where things live: vars/ and src/

The layout is fixed, and Jenkins reads it by convention rather than from any config file. vars/ is the front counter and src/ is the back office. Every .groovy file in vars/ becomes a pipeline step named after the file, so vars/standardPipeline.groovy is called as standardPipeline(...) in a Jenkinsfile. The wiring is a method named call: when you write standardPipeline(scan: 'trivy'), Jenkins runs the call method inside that file and hands it your arguments. The src/ directory is a classpath of ordinary Groovy classes filed into package folders, so src/org/acme/BuildContext.groovy is class BuildContext in package org.acme. Put logic there once it has grown big enough to deserve real objects and methods. A third folder, resources/, holds the non-Groovy odds and ends (a report template, a policy file) that a step pulls in with the built-in libraryResource step.

$ find my-shared-lib -type f -not -path '*/.git/*' | sort

my-shared-lib/README.md
my-shared-lib/resources/org/acme/report.tpl
my-shared-lib/src/org/acme/BuildContext.groovy
my-shared-lib/vars/notify.groovy
my-shared-lib/vars/standardPipeline.groovy

A whole pipeline in one vars/ step

Here is a global step you could ship tomorrow. def call(Map config = [:]) is the front door: it takes a map of named arguments and defaults to an empty one, so a caller passes scan: 'trivy' and the step reads config.scan. A vars step can do anything a Jenkinsfile can do. It can also declare a full declarative pipeline block, which turns an entire pipeline into one reusable function. The Jenkins docs draw a hard line around that trick: it is allowed only inside a vars/ file, only in a call method, and a build may run only one such pipeline. That is the templated-pipeline pattern. The library owns the stages, the agent and the post actions, while each project's Jenkinsfile picks options. Notice how this step hands the image-tag logic off to a class from src/, which keeps the step itself short enough to read in one sitting.

// vars/standardPipeline.groovy — call it as standardPipeline(...) in any Jenkinsfile
import org.acme.BuildContext
 
def call(Map config = [:]) {
  String scanner = config.get('scan', 'trivy')
 
  pipeline {
    agent { label 'linux' }
    stages {
      stage('Build') {
        steps { sh 'make build' }
      }
      stage('Scan') {
        steps {
          script {
            def ctx = new BuildContext(this, env.BUILD_NUMBER, env.GIT_COMMIT)
            echo "Scanning ${ctx.imageTag()} with ${scanner}"
            sh "${scanner} image ${ctx.imageTag()}"
          }
        }
      }
    }
    post {
      always { echo "Build ${env.BUILD_NUMBER} complete" }
    }
  }
}

Real classes in src/

src/ is where ordinary object-oriented code lives. A class there can hold state, be imported anywhere with import org.acme.BuildContext, and, best of all, be unit-tested on your laptop with no Jenkins in sight. The file path is the package: src/org/acme/BuildContext.groovy has to declare package org.acme and class BuildContext. One piece of Jenkins plumbing shows up right here. Jenkins rewrites pipeline Groovy through a transform called CPS (Continuation-Passing Style), which is what lets a build pause halfway and carry on after the controller restarts. The build keeps a written diary instead of holding everything in its head. The catch is that anything meant to survive in that diary has to be writable down. So library classes whose objects live across steps should implement Serializable, as BuildContext does, or a long build can die with a NotSerializableException. Steps in vars/ create these classes and pass this, the pipeline context, so the class can call steps like sh or echo on its own.

package org.acme
 
// Serializable so the object survives a controller restart mid-build (CPS)
class BuildContext implements Serializable {
  private final def steps
  private final String buildNumber
  private final String commit
 
  BuildContext(steps, String buildNumber, String commit) {
    this.steps = steps          // the pipeline context ('this'), lets us call sh/echo
    this.buildNumber = buildNumber
    this.commit = commit
  }
 
  String imageTag() {
    String shortSha = commit ? commit.take(7) : 'unknown'
    return "registry.acme.internal/app:${buildNumber}-${shortSha}"
  }
}

Loading the library from a Jenkinsfile

A Jenkinsfile pulls the library in with an annotation on the very first line: @Library('my-shared-lib@main') _. The name my-shared-lib points at a library registered in Jenkins' own configuration. The part after @ is the version, and it can be any Git ref: a branch, a tag, or a commit SHA (Secure Hash Algorithm, the long hexadecimal fingerprint Git gives every commit). That lone underscore is not a typo. A Groovy annotation has to be attached to something, and _ is the throwaway target people use when the only thing they want is the side effect of loading the library. After that line runs, every file in vars/ is available as a step by name, so a hundred lines of copy-pasted pipeline collapse into one call to standardPipeline.

@Library('my-shared-lib@main') _
 
// the whole pipeline is one reviewed function from the library
standardPipeline(scan: 'trivy')

Started by user Sachin
Loading library my-shared-lib@main
Attempting to resolve main from remote references...
 > git --version # 'git version 2.39.5'
 > git ls-remote -h -- https://github.com/acme/my-shared-lib.git
Found match: refs/heads/main revision 8c1f4a2e9b3d...
Checking out Revision 8c1f4a2e9b3d (main)
[Pipeline] Start of Pipeline
[Pipeline] node
Running on linux-01 in /home/jenkins/workspace/app
[Pipeline] stage
[Pipeline] { (Build)
[Pipeline] sh
+ make build
[Pipeline] }
[Pipeline] stage
[Pipeline] { (Scan)
[Pipeline] echo
Scanning registry.acme.internal/app:42-d4f9b02 with trivy
[Pipeline] sh
+ trivy image registry.acme.internal/app:42-d4f9b02
[Pipeline] }
[Pipeline] End of Pipeline
Finished: SUCCESS

Read the top of that log before anything else. Jenkins prints Loading library my-shared-lib@main and goes off to resolve the ref against the remote: an ls-remote to find which commit main points at right now, then a checkout of the library into a hidden per-build workspace. Only after that does standardPipeline run and expand into the Build and Scan stages, with BuildContext working out the image tag 42-d4f9b02 you can see further down. Two different commits are in play here, which trips a lot of people up. 8c1f4a2 is the library revision Jenkins resolved. d4f9b02 comes from env.GIT_COMMIT, the application repo's commit that the build is actually scanning. Turning a name and a version into running code is the whole trick, so it pays to see the steps laid out.

There is a second way in: the library step, written mid-pipeline as library 'my-shared-lib@main'. The @Library annotation resolves before the script is even parsed. The library step loads at runtime instead, so you can work the version out first, for example library "my-shared-lib@${env.BRANCH_NAME}" to follow whatever branch triggered the build. You pay for that flexibility. Because it resolves after parsing, the library's steps and src/ classes are unknown when the pipeline compiles, so you have to reference classes by their fully-qualified name and you give up static type checking. Most teams default to @Library for their trusted global library and keep the runtime step for the rare build where the version genuinely has to be decided on the fly.

Register it in code, not by clicking

Jenkins does not discover libraries on its own. You have to tell it that my-shared-lib exists and where its Git repo lives. The usual route is the web interface: Manage Jenkins, then System, then Global Trusted Pipeline Libraries. A library configured by clicking is a snowflake, though, and nobody can rebuild it from memory the day the controller dies. Configuration as Code (JCasC), the plugin that writes Jenkins settings into a single YAML file, records the same registration in version control. defaultVersion is the ref used when a Jenkinsfile omits one. allowVersionOverride lets a pipeline pin its own. implicit: false means a pipeline has to ask for the library out loud rather than being handed it silently on every build.

# jenkins.yaml — registers the library so a fresh controller is reproducible
unclassified:
  globalLibraries:
    libraries:
      - name: "my-shared-lib"
        defaultVersion: "main"        # used when a Jenkinsfile omits @version
        implicit: false               # pipelines must import it explicitly
        allowVersionOverride: true    # a Jenkinsfile may pin its own tag/commit
        retriever:
          modernSCM:
            scm:
              git:
                remote: "https://github.com/acme/my-shared-lib.git"
                credentialsId: "github-app"

What "trusted" really means

A global library is trusted, and trusted has a specific meaning here. Its Groovy runs outside the script-security sandbox, straight on the controller, with full reach into Jenkins internals and every stored credential. Handy, because trusted code can do things a sandboxed Jenkinsfile is blocked from doing. Expensive, because it makes the library repository exactly as privileged as the controller itself. Anyone who can merge to the branch your pipelines track can run whatever code they like on your Jenkins. Folder-level libraries are the opposite: always untrusted, always sandboxed. The controls are ordinary engineering hygiene. Protect the library repo with required reviews and branch protection, and pin a tag or a commit instead of a branch that keeps moving. That is the same supply-chain discipline the next lesson applies to plugins.

Try this

Run find my-shared-lib -type f -not -path '*/.git/*' | sort on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a trusted library is code running on your controller. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related