Daily Bulletin


The Conversation

  • Written by James Healy, Technical Director, The Conversation

The Conversation launched over six years ago and our continuous integration infrastructure has evolved over that time.

For the first few years we ran happily ran Jenkins on a single physical server. Our feature set grew rapidly and by early 2013 the build time for our core CMS code base peaked at an unhelpful 28 minutes. We explored options for speeding up our feedback loop and eventually ended up with a 6 minute parallel build running on three physical build servers, coordinated by Buildkite.

It feels like no development team is ever happy with their build time, but we found 6 minutes a pragmatic average that avoided most long waits for a result.

The challenge from there was avoiding the inevitable build time increase as new tests were added over time, and that required a way to monitor the long term trends.

We use librato for monitoring and it felt like a good fit for this situation too. Buildkite provide webhooks and we have a lita-powered slack bot that is well suited to hosting glue-code like this.

The solution we settled on provides a librato dashboard displaying the four-week build time trend for each of our codebases. Each chart includes the raw data that fluctuates a bit, plus a smoothed line based on the average of the last ten builds.

image Our build time dashboard for May 2017. CC BY-NC-ND

There were three puzzle pieces:

  1. Adding lita-buildkite to our lita bot and configuring buildkite to send it webhook events
  2. Adding lita-librato to our lita bot to simplify submitting metrics
  3. Writing a custom lita handler to pipe the timing data from buildkite webhooks across to librato

The glue code isn’t in a public repo, but it’s small enough to replicate here:

require "lita"
require "lita-buildkite"

module Lita
  module Handlers
    class BuildkiteJobStats < Handler
      on :buildkite_build_finished, :build_finished

      def build_finished(payload)
        event = payload[:event]
        record_build_stats(event) if event.build_branch == "master"
      end

      private

      def record_build_stats(event)
        runtime_seconds = (event.build_finished_at - event.build_started_at).to_i
        robot.trigger(:librato_submit, name: "ci.build-runtime.#{event.pipeline_slug}", type: :gauge, source: "buildkite", value: runtime_seconds)

        total_seconds = (event.build_finished_at - event.build_created_at).to_i
        robot.trigger(:librato_submit, name: "ci.build-totaltime.#{event.pipeline_slug}", type: :gauge, source: "buildkite", value: total_seconds)
      end
    end

    Lita.register_handler(BuildkiteJobStats)
  end
end

It’s only 27 lines, but it works.

In May 2017 we upgraded our standard operating environment from Ubuntu 14.04 to Ubuntu 16.04. When the upgrade was applied to our build servers we inadvertently changed a postgresql setting that added over a minute to the build. Within a few days the increase became obvious on the trend line and we identified then resolved the issue.

image Total build time for our core CMS over a four week period. The period of slower builds was due to build server misconfiguration after an upgrade to the host operating system. CC BY-NC-ND

As an extra safety net we have librato configured to alert us via slack if the average build time passed a threshold.

image When the average build time for our project exceeds a threshold, we’re alerted via slack. CC BY-NC-ND

We’ve had this setup in place for about a year and we’ve had at least three occasions where configuration or spec changes have impacted build time significantly and been reversed once discovered.

With a bit of luck, the bad old days of 28 minute builds will remain a bad memory.

Authors: James Healy, Technical Director, The Conversation

Read more http://theconversation.com/monitoring-build-times-maintaining-our-fast-feedback-loop-75548

Writers Wanted

Physical Therapist Talks About This New Massage Gun On The Block - The HYDRAGUN

arrow_forward

Too much information: the COVID work revolution has increased digital overload

arrow_forward

Ammonite: the remarkable real science of Mary Anning and her fossils

arrow_forward

The Conversation
INTERWEBS DIGITAL AGENCY

Politics

Prime Minister's Remarks to Joint Party Room

PRIME MINISTER: Well, it is great to be back in the party room, the joint party room. It’s great to have everybody back here. It’s great to officially welcome Garth who joins us. Welcome, Garth...

Scott Morrison - avatar Scott Morrison

Prime Minister Interview with Ben Fordham, 2GB

BEN FORDHAM: Scott Morrison, good morning to you.    PRIME MINISTER: Good morning, Ben. How are you?    FORDHAM: Good. How many days have you got to go?   PRIME MINISTER: I've got another we...

Scott Morrison - avatar Scott Morrison

Prime Minister Interview with Kieran Gilbert, Sky News

KIERAN GILBERT: Kieran Gilbert here with you and the Prime Minister joins me. Prime Minister, thanks so much for your time.  PRIME MINISTER: G'day Kieran.  GILBERT: An assumption a vaccine is ...

Daily Bulletin - avatar Daily Bulletin

Business News

Getting Ready to Code? These Popular and Easy Programming Languages Can Get You Started

According to HOLP (History Encyclopedia of Programing Languages), there are more than 8,000 programming languages, some dating as far back as the 18th century. Although there might be as many pr...

News Co - avatar News Co

Avoid These Mistakes When Changing up Your Executive Career

Switching up industries is a valid move at any stage in your career, even if you’re an executive. Doing so at this stage can be a lot more intimidating, however, and it can be quite difficult know...

News Co - avatar News Co

4 Costly Mistake To Avoid When Subdividing Your Property

As a property developer or landowner, the first step in developing your land is subdividing it. You subdivide the property into several lots that you either rent, sell or award to shareholders. ...

News Co - avatar News Co



News Co Media Group

Content & Technology Connecting Global Audiences

More Information - Less Opinion