r/jenkinsci•
RevRagnarok

Pipeline paradigm question (and a possible hybrid approach)

I've been using Jenkins for a while, and we have about 15-20 jobs. I use "copy artifacts from another job" a ton to link them together. They all have "GIT_BRANCH_NAME" as a parameter to help keep them together.

Many of my jobs are broken down into "launcher" and "builder." The launcher is a matrix job that decides what parameters to build for, and then they call the builder job with a single set of parameters. The branch name is the same across 2-3 repositories; the build jobs all use that to pull from their respective git repositories.

A major problem my team has with the current setup is that if a single branch is failing, the job fails and it looks like all hell's breaking loose, when really, it was just one branch. But that branch caused one builder job to fail seven times in a row, because all seven platforms it was cross-compiling on failed. But since the branch is a Job input, Jenkins itself doesn't understand that. The other problem is as things roll off and are erased, there's no way (that I am aware of) to say "keep at least one set of artifacts per branch."

I think pipelines are the answer for some of these issues.

I took a two-day course recently, and while it didn't cover Pipelines as much as I had hoped (sidebar - I'm annoyed by that, since certification is 2.0-based since Mar 1), I now "get" them, to an extent. But every example I see online and in the courseware is simple compilation / testing of Java or C/C++. I'm synthesizing FPGA designs, where a single compilation can take hours to complete. Some of the "test" simulations can take days depending on the complexity. Right now, after I build the base code and throw a handful of tests at it (< 20 minutes), the output is "done enough" to have the team start working with it. I don't want to wait a few days.

So, what I'm thinking I should do is write up a Pipeline that encompasses the outline of what needs to be done, but have each "stage" really be the jobs that I already have. Instead of the current system of "Job A triggers B triggers C" etc, the flow would be controlled by the Pipeline. But the work would be the jobs I already have. I think that would then have Jenkins on the right track concerning how things fit together, while still being able to have reasonable output of each stage (without the enterprise-only checkpointing). It would also allow my team to go in and re-run certain sections based on other parameters the way they can now, e.g. the default build does CentOS 7 for "isim," but I want the CentOS 6 version.

Has anybody attempted a hybrid approach like this before? Any lessons learned to share?

Posted Apr 26, 2017, 5:33 PM UTC

14
Loading comments…