I also don’t think we’ve ever publicized the links to our buildduty resource wiki before. The curious and/or masochistic may be interested to see just how much our buildduty representative is responsible for in a given week. It’s not a small list.

Yes, we’re looking to automate everything on that list, but hopefully you’ll keep the list in the back of your mind the next time you need some buildduty help.

I can dream, can’t I?

His first post on make-makefile is up now, and there is more coming soon.

However, I *do* sometimes look at the waterfall while messing around on my own development master, and I still have my userscript installed, which I’ve updated over time to label some of our newer build types and platforms.

Here’s the current version:

• Download: ThinnerBox

It will be business as usual for John for now, but in the weeks to come we’ll be looking hard at how best to go about integrating the Mozilla Messaging release processes and continuous integration into the current Mozilla releng systems.

Note: this also upsets the delicate John-to-Chris ratio in releng. If you’re a release engineer looking for work (bonus points if your name is Chris), we are still hiring.

Welcome, John! Glad to have you aboard.

If you have scripts that traverse the list of dated nightly directories (for archival, bisection, … whatever), your scripts will need to be updated to accommodate both dated dir formats, at least until the old dated directories are eventually expired.

Here are examples of both the old and new directory formats:

Old: 2011-06-21-03-mozilla-central-l10n

New: 2011-06-22-03-08-44-mozilla-central-l10n

Essentially we’ve added the minutes and seconds to the dated dirs to avoid nightly/clobber builds that happen in quick succession from landing in the same directory. This can cause some indeterminism with updates.

If you have any questions, feel free to comment here or in the bug.

Joey Armstrong is a Makefile whiz, and he’s going to be taking a stab at improving turnaround time on incremental builds and generally simplifying our Makefile story. John rightly gave up waiting for me to blog about Joey starting, and has written a post of his own explaining in more depth why Joey’s work is important. I’ll just add that we’ve done a lot of work recently in RelEng trying to identify long poles in our current build/test process. This effort has largely focused on re-assorting jobs and using hardware in smarter ways. It is awesome to have someone on the team now who is specifically tasked with trying to shorten the build process itself.

So…welcome, Joey!

Rather than re-opening the issue for fresh debate, I’m hoping I can simply restate our intentions, and also get the attention of anyone relying on the current directory format who hasn’t chimed in yet.

John identified two issues mixed up in this bug:

1. Improve regression hunting, starting with a tool to get BuildID+changesets of nightlies from the buildbot database. This tools should also construct URL to feed to hg pushlog. Laura recently picked up this work, and it is happening in over in bug 487036.
2. Normalize our directory naming on ftp so that each build is in a unique and consistent location. Specifically, we will be changing the dated directory that each build ends up in from “YYYY-MM-DD-HH” to “YYYY-MM-DD-HH-mm-ss.” This information is already available via the build ID, and allows us to sort builds chronologically AND handle respins within the same hour easily.

If you’re interested in further details about why we’re choosing to do this and not, say, using changesets for directories, I encourage you to read both the bug and the associated Google groups discussion before commenting.

If you’re a developer who currently relies on the “YYYY-MM-DD-HH” directory format, please comment in the bug like the War on Orange people did so I can give you adequate warning to fix your scripts. Ideally, I would like to target next week (June 6th-10th) for landing this change.

We still need a source code checkout for each repack because the source contains the actual instructions for repackaging. Mercifully we’re able to share a single source checkout between multiple localized repacks because, depending on the code branch, we could have upwards of 80 locales to deal with and each source checkout could be almost 1GB .

Sharing source code is helpful in that regard…until something goes wrong.

Sometimes we clobber the source code (for nightly builds, or by request, or to reclaim space on the buildslave for other builds) and due to network or machine vagaries, the shared checkout gets corrupted. If we weren’t sharing checkouts, this would result in a single failed l10n nightly repack.

However, since we are sharing, the initial l10n repack fails, but we compound our problem by returning the buildslave that just failed the repack to the slave pool more quickly (due to the early failure) where it happily picks up more l10n jobs and fails them all in quick succession.

In late April, I switched our l10n builds to use a script called hgtool.py that takes care of most hg operations and automatically deals with the common failure cases. We hope to use hgtool.py almost everywhere soon to help make all our source code operations more robust.

Ignore for now the chicken-and-egg problem that the tool itself currently lives in hg.

If you’re a current Litmus user, you’re already more awesome than most. If you’d like to take a stab at updating the Litmus Tutorial to reflect the current state of Litmus usage, I’m more than happy to help out any way I can with proofreading, etc.

I’m still not entirely up-to-date, but we’re getting closer.