mozilla :: #releng

19 May 2017
08:13travis-cibuild-buildbot-configs#2634 (master - d24ce87 : Mihai Tabara): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/233924645)
08:21travis-cibuild-buildbot-configs#2635 (production - 7c1471c : Mihai Tabara): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/233926480)
10:01whimboojcristau: hey Julien. do you know who is running the update tests for esr52 now?
10:01whimbooi cannot find florin online
10:01whimbooand someone triggered them
10:02whimboolooks like all of those tests failed due to download issues of the updates for esr-cdntest
10:02jcristauavaida maybe?
10:06jcristaulooks like bogdan_maris starred the failures from yesterday
10:06bogdan_mariswhimboo: Yes, I am
10:08bogdan_marisI ran the tests for esr52 early in the morning by mistake when the updates were not available for esr-cdntest, that's the cause of those failures
10:32whimboobogdan_maris: k, so for the future please really wait for the official go. thanks
10:33whimboobogdan_maris: can you please star them all_
12:11bogdan_mariswhimboo: Sure thing, I just starred them. Sorry again
12:54pascalclinux nightly builds were not generated today http://ftp.mozilla.org/pub/firefox/nightly/latest-mozilla-central-l10n/
13:01catleepascalc: looks like the nightly decision task died
13:01catleehttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=nightly&selectedJob=100402344
13:02catleeI see some nightlies / l10n repacks, but not all of them
13:04catleemtabara: do you happen to know why?
13:05catleeit died trying to submit signing-l10n-linux-nightly-3/opt
13:05* mtabara reads scrollback
13:06catleeinsufficient scopes
13:09mtabaraI don't remember us changing any scopes yesterday
13:09* mtabara double checks m-c
13:25FlorinMezeiif there's anyone handling the pushes to release and esr... do you have an estimate for having them pushed?
13:26jcristauFlorinMezei: i got everyone confused with the wnp, we're trying to figure it out :)
13:27FlorinMezeiah, ok :)
13:55catleemtabara|mtg: garndt and I got nightly sorted
13:59catleepascalc: I retriggered nightlies, let's see if they work
14:05bhearsumhm, i think i broke the balrog scriptworkers
14:05bhearsumlooks like they need a new version of the balrogclient library
14:06bhearsumin investigating that, i noticed that https://hg.mozilla.org/mozilla-central/rev/183c3537157290d7ccef475651b6bc35db914924 landed yesterday, maybe that's what broke the decision task?
14:10bhearsumsfraser: are you an appropriate reviewer for bumping the balrogclient version for balrogworker?
14:11mtabarab.hearsum: yeah, that seems so. I was gonna say I never saw those priorities before in the scopes, must be something that landed from outside of our side
14:11mtabaracatlee: thanks! sorry, got side-tracked with releaseduty + other stuff. did you fix the scopes?
14:11garndtbhearsum mtabara: yup, that's exactly what broke the decision task. There was a role that was missed when migrating the scopes
14:12garndtcatlee and I are also talking in #taskcluster about ideas of how we can detect issues with submitting the graph early on to prevent issues where only partial graphs get entered making retriggers a huge problem because of duplicate tasks
14:13bhearsummtabara: looks like sfraser may be afk, are you able to review https://bugzilla.mozilla.org/attachment.cgi?id=8869437&action=edit ?
14:13mtabaraokay, thanks
14:15sfraserSorry, was getting a drink, I am around
14:15bhearsumnp
14:16mtabarabhearsum: did you push the balrogclient 0.0.2 to puppet? I don't see it there
14:16sfraserfwiw I found the puppet docs on the wiki say that you can commit with a=versionbump for a self-review for that sort of thing
14:16bhearsumnot yet, i'm working on that
14:16bhearsumsfraser: oh, cool
14:17mtabaraah, cool. I always use "r=trivial". Maybe I should've read the docs before O_O
14:17sfraserpuppet doesn't autoland, so either works, really
14:17sfraserWe should just make sure noone joins with an irc nick of 'trivial' ;)
14:18bhearsumhehe
14:18mtabaraor `versionbump` :P
14:18bhearsumok, i think i've got the package in place on releng-puppet2
14:18bhearsumprobably need to wait for that to sync around?
14:19mtabarayeah, a few more minutes
14:19mtabarawhat I usually do is that I push to default in the meantime, since it takes a bit of time for tests to run
14:20mtabaraand usually they finish in the same time, both package sync and also build-puppet tests. so you can merge and push, followed by workers puppet sync
14:20bhearsumTIL we have tests on the puppet repo
14:21bhearsumwhere do i see the test results?
14:21mtabarawell, actuall the hg puppet is mirrors here - https://github.com/mozilla/build-puppet
14:21mtabaraand the tests you see is actually Travis being triggered per your commit there
14:21bhearsumhuh
14:21mtabaraAFAIK
14:21bhearsumthat's a bit weird, but ok!
14:21mtabara*mirrored
14:22bhearsumcertainly easier than setting up tests for the hg.m.o repo
14:23mtabarayeah, found in the history this: "travis-ci> build-puppet#1353 (master - da42a70 : Dragos Crisan): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/233238906)"
14:23sfraserI can't see the tests being run on that, do I need to be in the right github group?
14:24mtabaraso more or less, the process is similar to steps 7-10 from https://github.com/mozilla-releng/beetmoverscript/#deploy-a-new-version
14:24mtabarahm, maybe I should update that to say there's automatic tests for build-puppet
14:24bhearsumso i don't need to manually pull the manifest changes on the puppet masters then, it sounds like
14:25mtabaraI always wait for the automatic sync there
14:25bhearsumok
14:27bhearsumwhere do these puppet sync mails go to?
14:28sfraserreleng-puppet-mail@mozilla.com
14:28sfraserI have one from 4 minutes ago that says ' recv python/packages/balrogclient-0.0.2.tar.gz'
14:29bhearsumlooks like we're still waiting on at least releng-puppet1 in scl3 (i don't see it in http://releng-puppet1.srv.releng.scl3.mozilla.com/python/packages/)
14:36bhearsumit looks like releng-puppet1.srv.releng.usw2.mozilla.com is having a bit of trouble syncing...there's a lot of "disconnected by user" in papertrail
14:39bhearsumthere it goes
14:40bhearsumalright, i'm moving this version bump to prod
14:43bhearsumthanks for all the help mtabara, sfraser
14:50bhearsumok, the library is upgraded on the balrogworkers, i'm going to retrigger something to verify
14:50catleebhearsum: there are some busted updates on trunk right now
14:50catleehttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=nightly&selectedJob=100444485
14:50bhearsumyeah
14:50bhearsumi notified #sheriffs
14:50bhearsumi'll be retriggering once i know this works
14:53bhearsumok, retrigger worked, going to do all the failed stuff on m-c
14:55bhearsumboy, those retriggers are fast
15:01philoryeah, there are some taskcluster test suites that are so fast I wind up retriggering multiple times because I'm only showing failed and running jobs, and my retrigger is done and green before I even come back to look again
15:05RyanVMso, what in the world happened to make TC Windows builds so much flakier recently?
15:06bhearsumphilor: yeah, that's what just about happened to me!
15:06jcristaupascalc: just got a nightly update
15:06philorRyanVM: something TC-related, like using something as a shared cache without locking, which means that bug 1364651 is in the wrong place
15:06firebothttps://bugzil.la/1364651 NEW, nobody@mozilla.org Intermittent-infra mozmake.EXE[5]: *** [win32.obj] Error 127 OR [win64.obj] Error 127
15:07philorcatlee: OMG, no startup cache!
15:08philorI hated the precompile crashes with no way of ever seeing how it crashed more than most things
15:11pascalcjcristau,, cool :)
15:12pmooreRyanVM: how long ago did the flakiness start?
15:12pmooreRyanVM: is this win7 / win10 / win2012r2 (or all three?)
15:12RyanVMcouple days ago?
15:12RyanVMish?
15:12RyanVMphilor: you'd probably know better
15:13catleephilor: well, let's hope it sticks
15:14pmooreRyanVM: by flaky, is this worker exceptions (shutdowns), or jobs failing?
15:14RyanVMjobs failing, like bug 1364651 above
15:18philornot long before Aryx filed that, so maybe 8 days tops
15:19pmoorehmmmm
15:20philormost of them look like something being removed out from under the job, "cl: command not found" or "no such file or directory", though since Windows builds are terrible about reporting failures some are opaque
15:25pmooreyeah, that is weird, that intermittently, cl which just be gone
15:25pmoore*cl would just be gone*
15:26pmoorehmmmm
15:26pmoorei wonder if somehow a task that runs before it is messing with the user's default PATH setting
15:26philornot just cl, sometimes zip, sometimes something else I forget
15:30pmooreso on these builders, we are running each task under a different user account, so that blows the theory that it could be a task affecting the state of another task (as they run under different user accounts)
15:31pmooreare we getting any failure due to full disks? i can't imagine being out of disk would manifest itself like this
15:31pmooremore likely we'd see failures about not being able to create handles or write files etc
15:33pmooreso i guess the question is, can the files not be found, because they are not there, or because the system is looking in the wrong place, or because they were there and got deleted, or are there, but no access rights to read them
15:34pmoorewell on the bright side, we're hoping to get a loaner process of some sort out for windows in the short term, which should help empower users to dive in, which at the moment is kinda tricky
15:36FlorinMezeimtabara: throttling tests seem to indicate we are at 0%, can you double check?
15:36FlorinMezeiI need to step out now
15:36mtabaraFlorinMezei: which release?
15:36FlorinMezei53.0.3
15:37FlorinMezeileaving now
15:37mtabara25%
15:39mtabaraFlorinMezei: I confirm the updates are 25%. could it be something else from your side?
15:48bhearsumyeah, it looks fine to me too
15:48bhearsumi'd be curious which update URL you're having issues with
15:59kmoir-mtgbhearsum: I notice that buildbot opt and debug tests are disabled in project_branches.py for jamun, I was going to enable opt builds so we could land the changes for devedition tests, is that okay with you?
16:07kmoir-mtgactually I don't think that matters because it is set at a per platform level
16:58bhearsumkmoir: sorry, missed that message
16:59bhearsumit's fine with me if it's necessary though
16:59kmoirbhearsum: okay
17:29StefanG_QAHello,
17:29StefanG_QAwe've started to run the Throttling tests and we are getting 0% Throttling two times in a row
17:29StefanG_QAany ideas on how to get the better results
17:29StefanG_QAthank you
17:30bhearsumStefanG_QA: do you have an example update URL?
17:31bhearsumwhen we looked earlier, everything seemed fine in our tests
17:31bhearsumalso note that 25% of requests will be getting 53.0.3, and the other 75% will be getting 53.0.2, so if you're only looking for empty update vs. has an update, that's not going to be a good verification
17:33StefanG_QAI don't have an example update url
17:34StefanG_QAwe get the scripts from https://wiki.mozilla.org/Releases/Firefox_53/Test_Plan
17:34StefanG_QA53.0.3
17:34bhearsumi don't know what that throttling test does or have it verifies it
17:34StefanG_QAand steps to run them from here https://wiki.mozilla.org/QA/Throttling_Tests
17:34bhearsumit looks like it uses https://bug873666.bmoattachments.org/attachment.cgi?id=8532130 ?
17:35firebothttps://bugzil.la/873666 FIXED, thereallove@gmail.com please update AUS to AUS2_PRODUCTION tag (throttle Firefox 21.0 updates to 5%)
17:35bhearsumand AFAICT, all that does is see if "hashValue" is present
17:35bhearsumin this case, hashValue is present on both sides of the throttle, so that script isn't going to be a useful test
17:38travis-cibuild-buildbot-configs#2636 (master - 70b3ac6 : Kim Moir): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/234102454)
17:44RyanVMaki: thanks for pushing those version bumps to esr52 - I went ahead and canceled the builds on them, though
17:44akiRyanVM: we may need builds on the relbranch, no?
17:44RyanVMno
17:45akiok
17:45akii just pushed a tag
17:45RyanVMwe haven't landed anything for a 52.1.3 ESR yet, so if/when we need one, we'll have new uplifts w/ builds on them
17:45RyanVMthanks much, I nearly forgot to push the 52.1.2 bump last time around!
17:45RyanVM(on the relbranch)
17:45travis-cibuild-buildbot-configs#2637 (production - f8949b6 : Kim Moir): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/234104785)
17:46akiyeah, it's https://bugzilla.mozilla.org/show_bug.cgi?id=1362480
17:46firebotBug 1362480 NEW, nobody@mozilla.org we can't version bump for releases off a relbranch
17:47StefanG_QAhere is the url that we used to run the throttle tests "https://aus3.mozilla.org/update/3/Firefox/53.0/20170413192749/WINNT_x86-msvc/en-US/release/Windows_NT%206.3(nowebsense)/default/default/update.xml"
17:48bhearsumStefanG_QA: ok, did you see my comments above about the script?
17:48bhearsumit's not going to work in this scenario
17:50StefanG_QAok
17:50StefanG_QAthanks
17:51jcristauhmm
17:51jcristauthe script adds ?x=$RANDOM to the url?
17:53jcristaui guess that's to avoid caching, but when i try that i get a 400 error back with "Extra query parameter(s) x not in spec"
19:34jlundjcristau: has anything changed in your test scripts since this last worked?
19:50jlundbhearsum: given the 400 jcristau is seeing ^, makes me wonder if this new part of balrog is the reason: https://github.com/mozilla/balrog/commit/3afc89072d986e832bcd67afa58e04361ac49c08#diff-2a32a8391314d73e34898f1e9317c06cR22
19:52jlundmore specifically: https://github.com/mozilla/balrog/commit/3afc89072d986e832bcd67afa58e04361ac49c08#diff-2a32a8391314d73e34898f1e9317c06cR19
19:52bhearsumyeah, i think so
19:53bhearsumi noticed that too, i didn't realize anything actually relied on the ability to send random chars
19:53bhearsumyou can probably send force=N still
19:53bhearsumwhere N is any integer
19:53bhearsumthat throttling script is still unlikely to work though for the reasons i mentioned earlier
20:04jlundre: hashValue showing regardless of throttling, still makes me wonder why tests were 'passing' before
20:05jlundin other news, that's cool that you are using swagger and connexion
20:06bhearsumyou can thank exdev and aksareen for all of that :)
20:06bhearsumall i did was advise and review
20:07bhearsumkmoir: re: https://bugzilla.mozilla.org/show_bug.cgi?id=1362387#c18 - i don't think you want to do anything on inbound. devedition only exists on Beta
20:07firebotBug 1362387 NEW, spacurar@mozilla.com enable CI tests for Linux DevEdition on mozilla-beta
20:07akibhearsum: probably for merge day, so devedition doesn't disappear next merge day
20:07kmoirbhearsum: right, but don't the patches have to exist there so they will be uplifted? My understanding is that they won't run because they depend on the builds
20:08kmoirwhich are limited by branch
20:08bhearsumoh, i see
20:08bhearsumsorry!
20:08bhearsumi thought we were talking about buildbot patches, i got all mixed up
20:08kmoirtoo many patches :-)
20:08bhearsumyeah
20:08bhearsumthe theme of this project...
20:08bhearsumit looks like the tests are coming along really nicely now though
20:12kmoirthe linux64 ones look good the windows ones are very red
20:13kmoiroh, i see what's wrong
20:13bhearsumi count scheduling as coming along :)
20:22philorand good luck getting the OS X ones to run
20:45kmoirwell, the backlog on osx is a separate issue we're trying to address
20 May 2017
No messages
   
Last message: 5 days and 10 hours ago