mozilla :: #releng

10 Aug 2017
00:10btangyou found it for me, thanks!
01:37travis-cibuild-buildbotcustom#1071 (production-0.8 - f62e59b : ffxbld): The build passed. (
01:37travis-cibuild-buildbot-configs#2813 (master - 4776b40 : ffxbld): The build passed. (
01:38travis-cibuild-buildbot-configs#2814 (production - 620c999 : ffxbld): The build passed. (
01:41travis-cibuild-tools#1913 (master - 019d6f1 : ffxbld): The build passed. (
04:36ewongI'm looking at and was wondering what image is being used? and whether gtk3 is installed on this image
05:30jlorenzonthomas: hey! Yesterday QE had trouble to start their update tests because `mozdownload.errors.NotFoundError: Specified build has not been found:` . pointed me to bug 1381774. I don't know too much about OSX cross-compile, but it seems we don't
05:30jlorenzoupload mac artifacts there. Would have an idea about this?
05:30firebot WONTFIX, Build log files for Tinderbox builds on OS X ending up in tinderbox-builds/mozilla-beta-macosx64/
05:31jlorenzofull stacktrace at:
05:32nthomastinderbox-builds is the past, they'll have to adapt
05:32jlorenzothat's what I thought too :)
05:33jlorenzonthomas: the new builds should be fetched from taskcluster, right?
05:35nthomasYep, presumably that bridge was already crossed for Linux, but the routes in the index might be slightly different
05:37nthomasBut if version >= 56 use the index for all platforms
05:40jlorenzoalright, I'll figure the routes out. Thanks for the help ntho.mas!
07:49travis-cibuild-puppet#1754 (master - 7d7f62a : Alin Selagea): The build passed. (
07:57travis-cibuild-puppet#1755 (production - f972ae3 : Alin Selagea): The build passed. (
08:37jlorenzowhimboo: hey! I confirmed with ntho.mas, builds should noow be downloaded from Taskcluster. For instance: . What locales do you usually download? cc bogdan_maris
08:40whimboojlorenzo: i dont run the update tests
08:40whimbooondemand tests are always done by SF
08:41whimbooand we stop doing those by Monday next week
08:43jlorenzowhimboo: okay, what will happen next Monday? Will there be some kind of update tests? Or there won't be any replacement?
08:43whimbooi dont know
08:44whimboowe informed any people involved
08:44whimboowe hope there will be
08:44whimbooreleng has tests, and we have unit tests for firefox
08:45whimboomaybe spotchecks of updates? talk to rd person for releases
08:45whimbooSylvestre: ^ whoever this is
08:48Sylvestrejlorenzo, we hope that that the tests that we currently have are enough and that QE will do some spot checks
08:51Sylvestrefyi, for the 3+ years that i have been there, it found 2 actual issues. One was important (RTL issues in the update window)
08:51Sylvestrethe other wasn't
08:51Sylvestrebut we had tones of false positive
08:54jlorenzobogdan_maris: so, it sounds like we won't fix the trigger job.
08:58whimbooSylvestre: i don't concur on that. We found lots of issues with those tests
08:58whimboothankfully most on nightly and aurora channels so not affecting a release
08:58Sylvestrewhimboo, you are talking about the jenkins instance ?
08:59whimboothat's what we are talking about here, yes
08:59SylvestreI was talking about beta & release myself, nightly & aurora stuff, I wasnt informed
08:59Sylvestreseems like a bad news what you are telling me :(
09:00whimboothat was all listed in the email thread
09:00whimboodevs improved test coverage through
09:00whimboobut it's not a catch all
09:01whimbooi'm still at the point that integration tests are useful spanning all areas
09:01whimboobut yeah, if update tests are needed we need a PI request and then we can setup those in TC
09:02Sylvestreprobably then, I didn't realize the usage on nightly :(
09:04whimbooSylvestre: we are doing this since the beginning
09:04Sylvestrewhimboo, my bad
09:04whimbooeven started earlier as the ondemand tests
09:04SylvestreI am replying to the thread
09:04whimbooi'm fairly sure that I added a link to bugs we found
09:04whimbooa bug querz
09:17travis-cibuild-tools#1914 (master - a90f38d : tbirdbld): The build passed. (
14:12bhearsumrstrong: do you mind waiting until Monday to land it's the safest day for me to do the next balrog deployment
14:12firebotBug 1386176 ASSIGNED, Provide value in update url to indicate whether a Firefox 32 bit install hasn't been migrated to Fir
14:19RyanVMaki-away: the download links at are broken (at least for Windows)
14:31CallekRyanVM: hrm
14:32* Callek has no idea why the link is broken tbh
14:32Callekprobably just the `.exe`... stuff
14:40Callekgarbas: ping
14:40CallekI'm getting ssl issues on
14:42garbasCallek: i never touched, but i can look into the issue if you need help
14:42Callekgarbas: interesting things seem to be working now
14:43Callekgarbas: its served by relengweb cluster
14:44RyanVMCallek: can you please retrigger task ID FnA25Eo4TB24ZAv-JCPJ1Q ?
14:44RyanVMthanks :)
14:44CallekRyanVM: no scopes?
14:44CallekRyanVM: if so, what scopes are you missing to do so?
14:45RyanVM"You are not authorized to perform the requested action. Please sign in and try again."
14:45* Callek does the `rerun` anyway
14:45RyanVMfrom the task actions
14:45RyanVMi can't install the cli per ^
14:45CallekRyanVM: oo, `taskcluster signin`
14:45CallekRyanVM: I think I linked you to the task that had the cli, but either way its all set
14:46RyanVMsorry, been in a meeting
14:46* Callek has never tried to install the cli on windows
14:46Callekyea, no worries and completely understandable
14:58RyanVMCallek: OK, so I put taskcluster.exe in my path
14:58RyanVMtaskcluster signin launches an okta sign-in
14:58CallekRyanVM: running `taskcluster signin` should pop open a browser window to get you to signin...
14:58* RyanVM signs in and now my console is sitting on "Listening for a callback on: http://localhost:51865"
14:59Callekand then you should be able to do things like `taskcluster task name FnA25Eo4TB24ZAv-JCPJ1Q`
14:59CallekI admittedly never ran it on windows, so I would *hope* it worked there, maybe ask in #taskcluster
14:59* RyanVM doesn't have time for this right now, but this isn't encouraging
15:00Callek(on the not encouraging)
15:00CallekI'll test on my windows machine in a bit
15:16philordoes Win10 have a hot-corner for new desktop, like Win8 had a hot-corner for the Start screen?
15:16philorand do our Win10 desktops have a text file on them, that stray clicking could try to open?
15:20philorsaw a couple of "how do you want to open" screenshots yesterday in tests which had suggested bugs for "on Windows 8 where the screenshot shows the Start screen", which makes me even more suspicious that we're chasing the same Win8 thing we never really caught
15:57akiRyanVM: there's a 'grant' button
15:58akigrant access maybe
16:10RyanVMaki: i wasn't getting a grant button, just stuck in a login loop it appeared (I still had the 3 login options at the top even after supposedly logging in)
16:12akiRyanVM: are you using containers at all?
16:12akii get a url like
16:13akithen i sign in with okta
16:13akii do a duo push
16:13akithen i get a taskcluster login page with 3 buttons at the top
16:14akiand a grant access to localhost:port in the middle, and a 'grant access' button
16:14akii click that, and my cli client gets its temp creds
16:14tomprinceMaybe bug 1389135 ?
16:14firebot FIXED, is down due to internal server failure
16:14akiah, resolved 15min ago
17:28StefanG_QA_SV_LVhello Drivers,
17:28StefanG_QA_SV_LVI just ran the onDemand update tests for 55.0.1 release on release-cdntest channel.
17:28StefanG_QA_SV_LVCurrently the tasks failed to find osx and linux builds.
17:28StefanG_QA_SV_LV10:15:58 mozdownload.errors.NotFoundError: Specified build has not been found:
17:28StefanG_QA_SV_LV10:15:58 Build step 'Execute shell' marked build as failure
17:28StefanG_QA_SV_LV10:15:58 Finished: FAILURE
17:28StefanG_QA_SV_LVthis is part of the build output.
17:28StefanG_QA_SV_LVCan you take a look and advise on how to proceed ?
17:28StefanG_QA_SV_LVDo I need to rebuild ?
17:48mtabara|afkthat sounds really weird
17:48mtabara|afk looks right to me
17:48mtabara|afkyou sure there's no misconfigs in setting up the tests?
18:24erahmanyone know how to deal with this? remote has heads on branch 'default' that are not known locally: 4e1d0438c31b
18:24erahmthat's pushing to inbound
18:24RyanVMerahm: there's known issues with vcs replication
18:24RyanVMsupposed to be fixed in the next couple minutes
18:24RyanVMthen pull --rebase after
18:25erahmwas going to ask why tip is 7/28...
18:28RyanVMCallek: bad news, Windows talos runs on Beta are using opt builds, not the nightlies (I've verified by checking the URLs of the downloaded artifacts)
18:29RyanVM(why are those even being run on Beta?)
18:30RyanVMprobably the same for linux64
18:30CallekRyanVM: thats how talos has been run for linux on beta too
18:30Callekiirc catlee and jmaher had a discussion about it a weeks or two back
18:30RyanVMwe turned *off* the PGO talos runs in bug 1381520
18:30firebot FIXED, we run talos on mozilla-beta for win10 opt and win10 pgo, we should only run on pgo
18:30RyanVMbecause "we PGO nightlies anyway"
18:30RyanVMexcept we're not *testing* the nightlies
18:31RyanVMI just checked, we're definitely using the tc(B), not tc(N)
18:31Callekdo you know offhand when joel is back?
18:31sphilpjoel's back monday
18:31CallekRyanVM: are we willing to postpone caring about it until monday when catlee and joel are both back?
18:32RyanVMsphilp: ^
18:33sphilpit can wait, not critical, but it should get fixed soon. but yeah we need pgo, opt is kind of pointless
18:35RyanVMerahm: should be resolved now
18:35RyanVMsphilp: can you file it? We should turn off the Windows opt builds on Beta anyway as far as I'm concerned
18:36RyanVMhaving them there just makes things more confusing IMO
18:41CallekRyanVM: sphilp: fwiw 1381520 is what I recall of bugs from the past
18:41* Callek cc's jlorenzo on this convo since he made last comment in that bug
18:44RyanVMCallek: I don't see an actual verification in that bug about the assumption that those win talos runs were on nightlies
18:44RyanVMthe logs certainly seem to indicate otherwise
18:44CallekRyanVM: I think those win/opt builds are PGO'd (on beta) is the point
18:44Calleksince merge day scripts currently modify that stuff
18:44RyanVMeasy enough to check
18:45Callekbut yea, we *should* make the talos run off `nightly` rather than `opt` for human mental sanity, if the "test what we ship" reason isn't good enough :-)
18:46RyanVMok, yeah, win tc(B) on Beta have "17:49:11 INFO - MOZ_PGO_INSTRUMENTED=1 JARLOG_FILE=jarlog/en-US.log EXTRA_TEST_ARGS=10 z:/build/build/src/mozmake.EXE -C z:/build/build/src/obj-firefox pgo-profile-run"
18:46RyanVMso that bodes well then
18:46RyanVMbut yeah, we then promote the tc(N) builds
18:47RyanVMso we aren't actually testing what we ship
18:47RyanVMwe run automated tests on the tc(N) builds but we run talos on the tc(B) builds
18:47akifor talos? yeah, known
18:48akionce we get to shippable builds, we may have enough signal for talos to be useful. 1x/day nightlies don't have enough signal
18:48akisomeone suggested running talos n times per nightly, which might make it semi useful
18:48akibut may make nightly tests on date kill the talos pool :)
18:49RyanVMaki: on Beta, we run nightlies on every push
18:50RyanVMliterally same frequency as tc(B)
18:50akiyeah, we don't have nightly talos on central, though, so you don't know if a change is a regression
18:50akion merge day
18:50RyanVMI don't fully follow that, since tc(B) on trunk vs. beta aren't comparable
18:51akiif running talos against beta nightlies is useful, let's do it
18:52RyanVMI think it'd be a less confusing state if we turned off tc(B) on Beta and ran everything on tc(N)
18:53sphilpwe should be running talos on every beta that we *may* ship, we can talk about the frequecny/signal thing as well but at a minimum it should be running on the correct build
18:54RyanVMaki: and yes, shippable builds can't happen fast enough :P :)
18:54sphilpit's useful for us in an aggregate/trending sense at least, even if individual uplifts don't have enough signal. and reduces the confusion on what it's being run against
18:54akiyeah. likely it'll be pushed out further b/c we're going to focus on releasetasks in-tree first
18:54akibut sometime in q4/q1 maybe
18:56dividehexaki: mac-v2-signing5 is the emergency spare in toronto
19:54arrwho's looking after the signing scriptworkers these days? (see #buildduty)
19:55akiarr: the queue warning?
19:55arraki: all the the cluster alert
19:56arr(yes, the queue)
19:56akithat's me. i'm keeping an eye on it. i can ack
19:56akiwe may need to increase the numbers since we're adding way more tasks per burst
19:57arrokay, just wanted to make sure someone was aware :}
19:57akiyeah, i'm stalking scriptworker and some other keywords
20:19RyanVMaki: OK, time for round 2 of trying to get taskcluster-cli working since I have 3 jobs on Beta now blocking gtb :\
20:20RyanVMwhen I click on the "Sign in with Okta" button, I get the "Logging into Taskcluster" interstitial, then I end up on a page with the 3 login buttons at the top and a credentials section on the bottom
20:20akiRyanVM: you don't see the 'grant access' in the middle?
20:21akiRyanVM: let me know what needs rerunning, and i'll rerun; can you open a taskcluster-cli issue for this?
20:21* RyanVM curses bug 1381768 yet again
20:21firebot NEW, Some gecko-1-b-win2012 workers take forever to checkout mercurial files
20:21akii filed for osx, but that involved patching my install locally
20:21akithe problem was opening the browser tab at all
20:22RyanVMaki: WGxPsxHYR8mZlNGS3bPTDA KJmBKAXBRUSx9FOnk8JfKQ YoOsGKdKQE-w5O4DLjv6Ig all need retriggering
20:22akiwill do
20:23RyanVMaki: lol
20:23RyanVMI stole your URL from earlier and manually changed the port #
20:23RyanVMand now I was able to grant access
20:24Callekphilor: RyanVM: just because you both are more likely to see it and wonder, I triggered a new file update task a while ago now, forgetting that it takes 4-6 hours on average. And this task should appear on m-c aiui (when done)
20:24RyanVMi'll try that out if I need to retrigger any others
20:24Callekthis was triggered via a manual task creation and BBB though, so the "hey, we can see it pending on treeherder" is not to be expected for the next runs `yet`
20:24akiRyanVM: hah
20:25akiRyanVM: sounds like a taskcluster-cli bug or something between it and launching the browser
20:25akii patched it to output the url to the console
21:09travis-cibuild-tools#1915 (master - 366fe5a : ffxbld): The build passed. (
21:29RyanVMCallek: I officially have insufficient scopes to retrigger from taskcluster-cli
21:29RyanVMwhich means none of the other sheriffs probably do either
21:30* KWierso hasn't tried
21:30akiRyanVM: do you know which scope(s) you're missing?
21:30RyanVMYou do not have sufficient scopes. This request requires you to have one of the following sets of scopes:
21:31RyanVMprobably the same reason I can't do it from the Task Actions menu
21:31RyanVM(for the record, those broken Windows builds never got restarted)
21:31akido you have level 3 commit access?
21:32RyanVMyes :)
21:32akiawesome. let's ask the tc team for those ^
21:32akii mean tied to your ldap acct or w/e
21:32RyanVMoh, that could be a problem
21:32akisome people use a personal email
21:32RyanVMmy l3 is on my personal email
21:32akithat could be the problem
21:32RyanVMgee, this problem sounds familiar
21:33RyanVMsame reason I can't add jobs to Try pushes
21:33akiit's almost worth asking whether it's important enough to keep your l3 tied to your personal email :)
21:34KWiersoexciting: I no longer have the scopes to open/close trees, apparently
21:35RyanVMaki: anyway, none of those 3 jobs I pinged about earlier ever restarted - can you please try again? They're blocking 56.0b2 gtb now
21:35KWiersoweird, I see assume:project:releng:treestatus/admin, project:releng:treestatus/recent_changes/revert, project:releng:treestatus/trees/create, project:releng:treestatus/trees/delete, project:releng:treestatus/trees/update in my list of scopes
21:36KWiersoRyanVM: fwiw, I have queue:scheduler-id:gecko-level-3 in my scopes, so maybe I'm set for the cli stuff?
21:36akiRyanVM: looking like they're rerunning now
21:36akiKWierso: hopefully :)
21:36RyanVMaki: thx
21:36* RyanVM files the "give my moco ldap level 3" bug now :P\
22:16nthomasmkaply: ping
22:38dividehexaki: will/does partner-repack have any listening services that need a firewall allow rule?
22:44akidividehex: i think it's all sshing in and pulling
22:44dividehexgot it. Thanks!
22:46akidividehex: oh, it'll need access to the signing servers =\
22:47akiboth scl3 and mdc1 partner-repack1's
22:53dividehexaki: all signing servers including dep port?
22:54akidividehex: might as well, and we'll try to obsolete the partner repack boxes
22:54akiin practice we might only need the release port
22:55dividehexok. I'll add it to the speadsheet. good catch!
11 Aug 2017
No messages
Last message: 11 days and 7 hours ago