mozilla :: #releng

28 Apr 2017
00:38cmorenthomas: fyi. verdi is working on changes to the distribution.ini files, but we did notice one error that was duplicated across all of them with this line: app.partner.mozilla107="mozilla110" which should be app.partner.mozilla110="mozilla110"
00:39cmorenthomas: check it:
00:39cmorenthomas: can you sanity check with verdi put in
01:45nthomasgak, no notifications of pings
01:45nthomascmore: yes, we should fix that up. Not sure if it had any impact
01:46nthomasoremj: ping
05:28nthomas'/usr/lib64/nagios/plugins/check_ntp_time -v -H' is unhappy (from buildbot-master132). Checking against ns2.private.releng.scl3 is fine, as is the IT box ns1.private.scl3
10:48sfraserspacurar|buildduty: May I ask about the instance dev-linux64-ec2-sfraser, please? It was set up for bug 1359758 and I can ssh to it but I get prompted for a password, so it looks like my ssh key isn't there
11:19spacurar|builddutysfraser: What credentials did you use? I managed to log on using cltbld username
11:20spacurar|builddutysorry for the delay btw
11:21sfraserspacurar|buildduty: ah, I was expecting my own username to work. Sorry - yes, cltbld does work, I just didn't think to try it.
11:21spacurar|builddutysfraser: ok :)
11:31aselagea|builddutysfraser: root works as well
11:38sfrasernice, thank you! :)
12:06sfraserI think I'm drowning in email, I've only just found the one from spacurar about the loaner
12:22Pikewhat kind of error is this: ?
12:23Pikealso, while there are N3 builds (two) eventually, there's no Ns3 that succeeded?
12:47* Callek looks
12:48CallekPike: ooo wtf, thats *odd*, very very odd
12:48CallekPike: looks like an issue with taskcluster/hg's robustcheckout
12:48* Callek isn't sure how/why the NsN's succeeded/failed
12:49Pikespeculation, totally based on nothing but jetlag: another facet of S3 acting up randomly?
12:50* Pike pictures himself throwing bones at clouds
12:56CallekPike: ahhh I think I know why there are purple Ns3's, because there were green N3's -- but the N3's didn't fully match the proper dependencies from the initial taskgraph. Therefore "Chain Of Trust" failed.
12:56* bhearsum preps to enable devedition CI builds on mozilla-beta
12:56Callekthe hg repo stuff could have been s3
13:07travis-cibuild-buildbot-configs#2555 (production - 456244e : Ben Hearsum): The build passed. (
13:07travis-cibuild-buildbot-configs#2554 (master - e8d94d0 : Ben Hearsum): The build passed. (
13:14bhearsumspacurar|buildduty: do i need to do anything other than merge my buildbot-configs patch to the "production" branch to trigger a reconfig these days? i feel like may be out of date
13:17aselagea|builddutybhearsum: well, after reaching the production branch of the bb-configs repo, your change must also get on the bb masters
13:18aselagea|builddutybhearsum: if no rush, that is automatically done when the bb masters get reconfig-ed hourly
13:18bhearsumok, that's what i was hoping :)
13:18bhearsumno big rush, i can wait ~40min
13:18aselagea|builddutybhearsum: ok :-)
13:18flodbhearsum: ping (not sure if you answered and I missed the other day)
13:18bhearsumflod: hey, what's up?
13:19flodis there a bug tracking cleanup post dawn?
13:19flodmostly concerned about making sure that locales are not getting reset in the move central->beta
13:20flodwe used to do that for central->aurora, but we don't need it for central->beta anymore
13:20bhearsumwell, we won't be running the central -> aurora migration stuff in the next merge cycle
13:20bhearsumso that won't happen automatically
13:20bhearsumwe still need to look at what parts of central -> aurora we need to migration to central -> beta migration i guess, so i'll make a note to ensure we don't do the locale reset
13:21flodgreat, that works for me
13:21travis-cibuild-buildbot-configs#2556 (master - ebd43cd : Kim Moir): The build passed. (
13:21bhearsumthanks for pointing that out
13:21flod(I never understood why those files are in a /testing folder)
13:22bhearsumyeah, it's confusing
13:22bhearsumi think it's because it was mostly tests that were run through mozharness when we imported it into the tree
13:33travis-cibuild-buildbot-configs#2557 (master - 1134991 : Kim Moir): The build passed. (
13:37travis-cibuild-buildbot-configs#2558 (production - e68ccb9 : Kim Moir): The build passed. (
14:59oremjdavehunt: + causes the thread leak. I forgot to mention that json schema validator causes leaks only if that one method is called
15:00davehuntoremj: wow!
15:00oremjit doesn't really make sense. I assume it's some weird interaction between that library and some jenkins internals
15:01oremjmaybe it's causing a crash somewhere on a slave thread or something
15:01oremjthat isn't being logged anywhere
15:01davehuntoremj: was this since a recent Jenkins/plugin update or since we started using JsonSchemaFactory?
15:03oremjdavehunt: we noticed the thread stuff started happening right when fxtest 1.6 was tagged
15:04oremjaside from that there were a few plugin upgrades, but just standard stuff matrix-auth email-ext envinject role-strategy
15:04oremjwere around that time
15:04oremjpipeline-model-definition and pipeline-model-api were upgraded close to, but I think the thread thing started happening right before that
15:05RyanVMcatlee: I'm wondering if we should disallow --rebuild in Try syntax if -u all is specified
15:07catleeRyanVM: yeah, that's on the bug
15:07RyanVMoh, cool :)
15:07catleebut I don't know how hard that is
15:11oremjdavehunt: !
15:12oremjthat seems to be where the threads are starting
15:13oremjthat makes sense each forever thread has a lock that is shared by 2 other threads
15:14oremjso NTHREADS = 3 seems right
15:15oremj !
15:16bhearsumaki-away: do you happen to know if it's intentional that the platform_overrides don't override the main branch overrides in ?
15:17bhearsumeg: if i set update_channel in one of those platform overrides, it still ends up as "beta". but if i remove the "update_channel" from the "mozilla-beta" dict, i get my overridden channel
15:18davehuntoremj: hmm.. there's even a couple of pull requests
15:18aki-awayit's in order. the last config to specify a key wins
15:18bhearsumaki-away: this is all in one file though
15:19aki-awayj.lund knows more than me here, but i believe there's logic to how these are applied, which is orderrd
15:19bhearsumok, thanks for the pointer
15:20bhearsumit looks like all of the platform/variant specific overrides are done prior to the branch overrides, yeargh
15:21aki-awayconfig files are in tree, so could be munged on merge day if that helps
15:22pmooregrenade: are the config(s) used in MDT for preparing the windows images available in a source repo somewhere, or are they only accessible via a running instance of the MDT tool somewhere?
15:22bhearsumnot sure it helps in this case, since i'm looking at two different build types (devedition and beta) on the same branch
15:22bhearsumanyways, i'll poke at it some more on my own
15:22pmooregrenade: i'm wondering if i can have a play switching user profiles to be on Z: drive when we create the windows 10 base image
15:22aki-awaythere might be an env var to add more config files
15:23pmooregrenade: is this something i can hack on myself, or do i need someone else (with privileged) to do it for me?
15:46asarihHello. I am working to make unsigned Firefox binaries available for CI, and I have a question. has numeric subdirectories only, and it is very difficult to figure out which directory is the latest without scraping what directories are there. Is it possible to have
15:46asariha symlink "latest" (or any suitable name) that points to the latest and the greatest directory?
16:23Callekasarih: if you're looking for ..."available for CI" you likely want to use the TC index
16:25asarihCallek: Thanks for the info. I'll take a look.
16:25Callekasarih: that said, theres probably better ways still to do what you need, depending on how you're doing it.... and what your requirements are explicitly
16:28asarihCallek: normally, we would download from and expand the downloaded archive.
16:29asarih(do the similar for the Mac .dmg)
16:30asarihin TC, looks like public/build/firefox-53.0.2.en-US.linux-x86_64-add-on-devel.tar.bz2 is the file we are looking for.
16:31travis-cibuild-buildbot-configs#2559 (master - cf3df35 : Kim Moir): The build passed. (
16:32asarihCallek: hmmm. even here, though, the file name seems to be tied to the version number currently in development (e.g., 53.0.2).
16:32asarihThis is for Travis CI, by the way. ;-)
16:32Callekasarih: yea, this is a task using PRETTY NAMES :/
16:33Callek(buildbot runs it and uploads it)
16:33Callekthats not our desired end state but it is the case today
16:33asarihCallek: I see. so, for now, I'd have to deal with what it's got?
16:33Callektc jobs have the simple named target.dmg style :/
16:34Callekasarih: note has the version info
16:35asarihthat might work.
16:35asarihit sure beats scraping HTML.
16:36asarihas long as the URL doesn't change.
16:36Callekwell the taskID of it does change, but you can use the index'd copy of buildbot_properties.json for it
16:37Callekthat file may be missing/empty once it moves off being this format
16:37travis-cibuild-buildbot-configs#2560 (production - 31e5187 : Kim Moir): The build passed. (
16:37Callekbut it can at least help until then
16:37asarihCallek: thank you for the pointers. I really appreciate it.
16:37Callekno problem
16:47grenadepmoore: i believe it's a nasty Windows point and click GUI. No config's AFAIK. But I've never used it.
16:48pmooregrenade: ah ok - thanks
16:48pmooregrenade: who should i ask to do that for me?
16:51RyanVMCallek: do you happen to know offhand why we seem to be double-scheduling OSX & Windows buildbot builds on mozilla-beta today? BBB related?
16:51pmooremaybe Q or markco?
16:52Callekugh, could be, kmoir changed the scheduler code to try and support BB OSX Builds via BBB only (on a branch)
16:52RyanVMosx, win32, win64 opt builds are the 3 affected
16:52kmoirI didn't touch beta, only holly
16:52kmoirand only for macosx
16:53CallekRyanVM: c.f. kmoir and catlee and Bug 1353376 -- theres also Bug 1359595 which does relate to beta and DE
16:53RyanVMhrm, and I guess I still need to file that "OSX debug tests aren't running on Beta" bug, don't I
16:53travis-cibuild-buildbot-configs#2562 (production - cbe2338 : ffxbld): The build passed. (
16:53travis-cibuild-buildbot-configs#2561 (master - dc4aa99 : ffxbld): The build passed. (
16:54travis-cibuild-buildbotcustom#1033 (production-0.8 - 07f2549 : ffxbld): The build passed. (
16:54Callekbhearsum: could that be related to you ^ (beta double sched)
16:55travis-cibuild-tools#1752 (master - 4fb53d3 : ffxbld): The build passed. (
17:04RyanVMCallek: oh crap, that's exactly what it is
17:04RyanVMit's a devedition build that identifies as a regular build on TH
17:04RyanVMand triggers its own tests too
17:04CallekDE is really fun, huh
17:05RyanVMthe TC ones at least have their own line on TH
17:06gpshow can i download a non-public file from tooltool?
17:08Callekgps: get a tooltool token from
17:09Callekgps: then with
17:09Callekgps: (some of the second part may be usable with various forms of scripts around the tree)
17:09Callek(depending on how/where you need the magic)
17:09grenadepmoore: Q or markco
17:10gpsCallek: what server do i use?
17:10Callekgps: from local or automation?
17:10gpsoh, worked from normal server with new token
17:10Callekgps: and if automation TC or Buildbot
17:10gpsi figured the non-public file would be on a different server
17:10Callekheh, yea normal server with token is all you need
17:11gpssweet - thanks!
17:19bhearsumRyanVM: anything i need to do to help fix this?
17:20RyanVMbhearsum: sounds like the main issue is patching TH to recognize them as different builds and display them accordingly
17:20bhearsumi think ed merged that already
17:20RyanVMok, just not on prod then
17:21bhearsumi might need to tweak the linux stuff too, not sure yet
17:21pmooremarkco: would you be able to create a win10 vm for me, with the user profiles on Z: drive?
17:21bhearsumbut that's in-tree, not treeherder
17:21pmooremarkco: or can you give me tips how i can do that for myself?
17:21RyanVMbhearsum: yeah, IIUC, the OSX/Win ones will be 'B' on a new line, and Linux are tc(DE) on a line
17:21bhearsumi find it weird that we have "tc" groups with builds in them
17:22bhearsumi guess that's leftover from when we had buildbot and tc builds in parallel
17:22RyanVMIMO, we should drop tc- from things that don't have buildbot equivalents at this point
17:22pmooremarkco: do you know if MDT just creates an "answers" file?
17:22bhearsumok, maybe i'll do that for devedition since they've got their own row anyways
17:23RyanVMbhearsum: SGTM. Call them 'B' too
17:23markcopmoore: off hand i am not quite sure how to do that. MDT creates a few different answer files.
17:23bhearsumRyanVM: rgr
17:23bhearsumthanks! sorry for any confusion they caused
17:23markcopmoore: I could create a win 10 iso
17:23RyanVMand the purdy "Linux DevEdition" and "Linux x64 DevEdition" instead of linux32-devedition
17:24RyanVMbhearsum: no worries
17:24pmooremarkco: is MDT something i can download and install myself on a windows vm? or do i need a license?
17:24markcoa license
17:24pmooreah ok
17:24pmooredo we have a corporate one?
17:24markcoyes but you would need to ask Q how to get to it. or grenade might know.
17:26pmooremarkco: i think i'll create a bug and cc you, q, grenade - ideally i'd like to be able to create my own images without needing to interrupt other people, and this way i can document the process too etc in case other people want to hack on it
17:26pmooremarkco: the motivation is that if user profiles are on Z: drive initially, it simplifies some later steps when we want to run tasks on Z: drive but not get e.g. appdata and other folders on C:
17:27pmoorewe can work around these issues, but according to microsoft, the user profile drive assignment is an OS-install-time thing, and they don't really support it being changed later (and it isn't so trivial to change)
17:27pmooreso i thought i'd give it a stab
17:27pmoorethanks for your help - i'll CC you on the bug
17:30markcopmoore: np. I will take look at how that could be done as well. Initially i would think creating the user profiles on something other than what is consider to be OSdisk may be a bit tricky, but i will let you know if find anything .
17:34markcopmoore: it may not be that hard to address after the oS installation following the user creation. It may just be move and update HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\ProfileList for the corresponding SID
17:35markcoWe could address that in OCC
17:37pmooremarkco: i can't find the link now, but microsoft say that it should be done at installation time for some reason - let me see if i can find it
17:38pmooree.g. there is the ProfilesDirectory setting in the unattend.xml file
17:41travis-cibuild-buildbot-configs#2564 (production - 23f80ad : ffxbld): The build passed. (
17:43markcopmoore: in AWS is the Z drive there at the time installation or is attached after the OS in installed?
17:43markcois installed
17:44travis-cibuild-tools#1753 (master - 169d4f4 : ffxbld): The build passed. (
17:45pmooregood question - i'd need to check we can image it after installation, and attach it before an instance starts up
17:45pmooremarkco: ^
17:45travis-cibuild-buildbot-configs#2563 (master - e942199 : ffxbld): The build passed. (
17:45pmooremarkco: i'll have a play next week - i've seen there are some unattended installation walkthroughs on msdn
17:48bhearsumaki: just in case you were curious, i found a solution -- if i set stage_platform in the devedition variants, i can override the channel in the branch specific configs
17:49bhearsumbut AFAICT, you can't override anything in branch specifics through a later config, or command line args
17:49akithat sounds wrong
17:49akibranch specifics should apply first, then later configs
17:49akinot sure what may break if we change that though =\
17:49bhearsumyeah, i didn't want to touch that :)
17:50bhearsumi could be wrong, i still don't fully understand it
17:50akii suppose if the config files tell which branch specifics to use, then it's tricky
17:50bhearsumyeah...maybe that's it
17:50akii suppose we could also create a branch-specifics for beta-devedition
17:50bhearsumi also found it weird that the variant configs are earlier than branch and pool in
17:50bhearsumyup, maybe
17:51bhearsumit might be more work to get that file used in the right places
17:51akiiirc, i reviewed that, but j.lund wrote it. he may have reasoning
17:51bhearsumi'm sure it was a fine assumption at the time, and it may still be now
17:52akinever underestimate the need for more complexity in configs ? :)
17:52bhearsums/in configs/at mozilla/
18:05akiwhere is hg installed?
18:09bhearsumRyanVM: since we were talking about treeherder - do you have any idea why my builds aren't showing up as completed on
18:09bhearsum ? win32 and osx are long done according to
18:17catleebhearsum: are they run via BBB?
18:17bhearsumi'm not sure
18:18bhearsumi don't think so though...
18:18bhearsumi didn't have to add them to anything in tree to make them run
18:18catleebhearsum: there are dead command queue item alerts in #buildduty
18:18catleecould be related
18:18bhearsumah, sounds like it
18:19bhearsumi bet the new stage_product is causing issues
18:20RyanVMbhearsum: i do not
18:20RyanVMalso, it appears that the resulting test jobs from devedition builds aren't identified as such
18:21bhearsumthere are test jobs? i'm surprised if those are running already
18:22RyanVMbhearsum: I'm definitely seeing opt tests double-triggered :)
18:22RyanVMon the buildbot jobs
18:23bhearsumwell, i knew something would go wrong here
18:24catleesendchange something something
18:24bhearsumyeah, i probably need to tweak some other bit
18:24bhearsumRyanVM: that sounds pretty serious - i suspect that one set is running vanilla beta tests, and one is running devedition...but they're reporting the same
18:25catleebhearsum: I think that's mozharness
18:25bhearsumof course it is...
18:25bhearsumso much dead code here
18:26bhearsumanyways, i'll see if there's quick fixes for any of this, but i may end up backing out for now
18:26RyanVMi'm not necessarily *opposed* to running the tests on DevEdition (though as you know, there's not any significant differences between them and vanilla at the moment anyway), but we should at least make sure they're identified as such
18:26catleebhearsum: do they have their own mozharness configs?
18:26bhearsumthey have variant configs
18:26bhearsumnot full-on separate configs
18:26bhearsumlike debug builds
18:26catleethere's some kind of 'enable_sendchange: False' thing you can do
18:27bhearsumRyanVM: yes, definitely
18:27catleeas a quick fi
18:27bhearsumRyanVM: the requirement is that we run tests, fyi
18:27bhearsumcatlee: yeah, i may have to back out anywys because of the upload issues though
18:27bhearsumso i'm going to poke that first
18:27catleeenable_unittest_sendchange: False
18:27catleeand talos too
18:28RyanVMgood, I was expecting us to need to eventually anyway - I expect that DE will eventually pick up more config differences vs. vanilla Beta
18:28bhearsumoremj: you around? i'm trying to figure out what i need to do to allow ffxbld to upload to /pub/devedition
18:29catleebhearsum: did you find the error?
18:29akihe's not responding to . is there anyone else we can ping?
18:29catleeI'll poke travis
18:29bhearsumcatlee: yeah, it's a 403 during the upload
18:37bhearsumhuh, i'm surprised the builds themselves didn't fail during upload
18:37bhearsumthey made artifacts on the taskcluster queue just fine (no surprise), but didn't put anything on s3
18:38catleebhearsum: for the builds themselves, or the logs?
18:42bhearsumcatlee: the builds
18:42bhearsumi'm going to backout the buildbot patch - there's too much bustage here to deal with this late on a friday
18:42bhearsumi think it's fine to leave the linux builds on though
18:43akicatlee: travis is oremj's backup?
18:43akior are you talking about the ci system
18:43catleeaki: the person
18:43catleehe manages that team
18:43akicould we poke him about ?
18:44catleeI did
18:44akiah, ok
18:44catleehe said that they don't watch bugzilla closely right now
18:44catleeand to please ping if we need something acted on
18:45akidoes that mean they're acting on it?
18:46catleeno - oremj is AFK ATM
18:46catleedo we need to page him?
18:46bhearsumnot for devedition reasons
18:47akiwe need it soon. not sure if that means we should page him now
18:48bhearsumlooks like jason is the backup according to
18:48catleelike if we can wait an hour, let's wait an hour before paging
18:49akihi jason, are you able to handle ?
18:49jasongive me 5 mins
18:50travis-cibuild-buildbot-configs#2565 (master - b0b561c : Ben Hearsum): The build passed. (
18:50travis-cibuild-buildbot-configs#2566 (production - 5785a06 : Ben Hearsum): The build passed. (
19:01bhearsumjason: when you have a few min (no rush), can we chat about how to allow ffxbld to upload to ?
19:03jasonbhearsum: I think oremj would be a better person to talk to about that, /cc oremj ^
19:11bhearsumjason: sure
19:11bhearsumok, it looks like devedition mac and windows builds are disabled now
28 Apr 2017
Last message: 40 minutes and 27 seconds ago