mozilla :: #releng

21 Apr 2017
02:41philorwhat's better for releng, leave aurora closed and keep building nightly after nightly after nightly after nightly after nightly on the same revision, or reopen it and let the (otherwise a waste of four hours of AWS time) periodic update pushes land?
02:41philoror, alternative three, be reminded that either shutting off the nightlies or shutting off the periodic updates or both would be a good idea
02:44philorof the first two, I know reopening is vastly better from a sheriffing standpoint, since once you get up over a few thousand jobs on a single push (currently at 1321 buildbot jobs), treeherder become unworkable
02:56ewongphilor: what about just removing aurora?
02:56ewongthere are still pushes on aurora?
02:57philorewong: that's what we'd do if we decided to stop using aurora, sat down to plan out what it would take to drop it and how long it would require to do all that, got all the patches and problems worked out, and then dropped it
02:57ewongwell... we've taken the first step. "stop using aurora"...
02:58philorinstead, we said "hey, in a couple of weeks, let's just drop aurora" so since we can't yet move aurora users to beta (or central in the case of Fennec) we have to be ready to chemspill for them
02:58ewongoh
03:04ewongdustin methinks I've unhorked the whole sea-puppet horkage.. the million dollar question is how do I find out if I've *really* fixed it?
03:05ewongpuppet agent -t produces 'green' results..
03:12Callekphilor: I say keep closed + bug to disable nightlies. (And periodical stuff), manually check in whitespace if you need a new cset....
03:13CallekImo
09:33Tomcat|sheriffdutyjlorenzo: hi!
09:33Tomcat|sheriffdutyoh
09:33Tomcat|sheriffdutynevermind :)
09:34Tomcat|sheriffdutyjlorenzo: wanted to say i retrigger nightlys for beta because of the bustage on the beta tree
09:34Tomcat|sheriffdutybut since we don't have beta nightlys i do not even need to do it :)
09:34jlorenzoTomcat|sheriffduty: hehe, wfm :)
09:35Tomcat|sheriffduty:)
12:39bakuhi, I need to use a one-click-loaner. Is this the right place to ask for it?
12:39Tomcat|sheriffdutyaselagea|buildduty: ^ hi aselagea do you know
12:42aobreja|builddutybaku: what kind of tests do you want to check?do you have an example?
12:43bakuaobreja|buildduty: https://treeherder.mozilla.org/logviewer.html#?job_id=93098089&repo=try&lineNumber=3086 I need to debug this. locally I cannot reproduce it.
12:45aobreja|builddutybaku: if you check the revision you will get this page: https://treeherder.mozilla.org/#/jobs?repo=try&revision=cdbb8bb37a467cfb0dca43c42277f4456d413a38&selectedJob=93098089
12:46aobreja|builddutygo to "Job Details" and at the buttom you will have a link to "One click loaner"
12:46Tomcat|sheriffdutyCallek: ping
12:46Tomcat|sheriffdutyCallek: could you take a look at https://treeherder.mozilla.org/logviewer.html#?job_id=93288181&repo=mozilla-central&lineNumber=23394
12:47Tomcat|sheriffdutyseems something is wrong with android trunk nightlys
12:48bakuaobreja|buildduty: thanks!
12:49aobreja|builddutybaku:you must login with your credentials and click on "One-Click Loaner",here you have a short documentation: https://developer.mozilla.org/en/docs/Running_automated_tests/TaskCluster_interactive_session
12:49bakuaobreja|buildduty: yeah done. I'm already using the shell. Thanks
12:50aobreja|builddutybaku: You're welcome
13:13travis-cibuild-puppet#1222 (master - 16fe0f3 : Dustin J. Mitchell): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/224343262)
13:17travis-cibuild-puppet#1223 (production - b0694bd : Dave House): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/224344837)
13:22travis-cibuild-puppet#1224 (master - e4ca89d : Johan Lorenzo): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/224346242)
13:26travis-cibuild-puppet#1225 (production - 74e61a7 : Johan Lorenzo): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/224347690)
13:43Tomcat|sheriffdutyjlorenzo: hi! not sure if callek is around but seems android trunk nightlys have a problem - https://treeherder.mozilla.org/logviewer.html#?job_id=93287341&repo=mozilla-central
13:43Tomcat|sheriffdutyfiling a bug now
13:44* jlorenzo takes a look
13:50jlorenzoTomcat|sheriffduty: I don't know if backing out bug 1358142 can fix the nightlies. What do you think pmoore|afk ?
13:50Tomcat|sheriffdutyhm jlund mshal ^
13:50pmoore|afkjlorenzo: what is the error?
13:50jlorenzopmoore|afk: https://treeherder.mozilla.org/logviewer.html#?job_id=93287341&repo=mozilla-central&lineNumber=23101
13:51Tomcat|sheriffdutyalso filed bug 1358493
13:51Tomcat|sheriffdutyjlorenzo: if backout helps here i can do it and retrigger nightlys
13:51pmoore|afkoh dear
13:51pmoore|afki'm sorry it looks like i indeed broke the nightlies
13:53Tomcat|sheriffdutypmoore|afk: np can do the backout
13:54Tomcat|sheriffdutypmoore|afk: working on that now
13:55pmoore|afkjlorenzo: i wonder why that is bad syntax, it works for me locally with python 2.7.13 - do you know which python version is being used on the nightlies?
13:55pmoore|afk:'-(
13:56Tomcat|sheriffdutyok backout done
13:57jlorenzopmoore|afk: per the logs, it's 2.7 as well. I don't see which minor version, though
13:57pmoore|afkstrange :(
13:57pmoore|afkcoop: do you know which version of python is used? ^
13:58pmoore|afkor Callek
13:58pmoore|afkTomcat|sheriffduty: thanks for backing out
13:58Tomcat|sheriffdutynp
13:59Tomcat|sheriffdutynightlys retriggered for android too
14:03pmoore|afkjlorenzo: here is local test:
14:03pmoore|afkhttps://irccloud.mozilla.com/pastebin/USg93c73/
14:05travis-cibuild-buildbot-configs#2550 (master - 066e305 : Andrei Obreja): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/224362862)
14:06jlorenzopmoore|afk: I wonder if we're not using a different python at this line https://treeherder.mozilla.org/logviewer.html#?job_id=93287341&repo=mozilla-central&lineNumber=23026
14:07pmoore|afkjlorenzo: yeah, could be
14:07pmoore|afki'm afraid i have to look after the kids now but i'll be back around later to help out
14:08pmoore|afksorry for the bustage :((((
14:11jlorenzopmoore|afk: np! see you later
14:51travis-cibuild-buildbot-configs#2551 (production - 1949fd4 : Andrei Obreja): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/224379165)
15:15pmoorejlorenzo: so looking at https://docs.python.org/2/reference/compound_stmts.html#the-with-statement it looks like multiple with clauses were added in python 2.7, so i'm guessing this is running against python 2.6 for some reason
15:15pmoorejlorenzo: luckily, there is a simple fix, which is just to split into two with statements :)
15:16pmoorehttps://irccloud.mozilla.com/pastebin/lu5wDEg9/
15:16pmooreso i'll just do that
15:17pmoorealthough does seem kinda weird that we're still using python 2.6, but i know upgrading is a nightmare ....
15:17jlorenzopmoore: okay! Another fix could be to call python2.7 at https://dxr.mozilla.org/mozilla-central/rev/8b854986038cf3f3f240697e27ef48ea65914c13/testing/mozharness/mozharness/mozilla/building/buildbase.py#1684
15:17jlorenzobecause this machine seems to have 2.7 installed
15:18jlorenzopmoore: but jlund may have a different view on it ^
15:19pmoorejlorenzo: i think i'll leave that hornet's nest for a braver person to tackle, as that might introduce other weirdness :)
15:20jlorenzoagreed!
15:49AryxCallek: hi, are you the contact for android l10n nightlies? they fail and complain about missing credentials
15:50Aryxhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=c8198aa6e7677e90cc7f1e2df0a14a5cc2719055&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
16:02PikeAryx: https://hg.mozilla.org/mozilla-central/rev/6a0e9a3f2673 ?
16:03PikeI think they're actually failing to generate repacks at all due to a missing thing to generate some manifest xml file
16:04Aryxthanks
16:04Aryx[task 2017-04-21T15:35:15.064757Z] 15:35:15 INFO - retry: Calling _get_revision with args: (<mozharness.base.vcs.mercurial.MercurialVCS object at 0x7f9d8a6f9e10>, &#39;zh-TW&#39;), kwargs: {}, attempt #1
16:06Pikethere&#39;s a ton of unhealthy stuff in that log
16:06Aryxas usual
16:09Aryxnalexander is looking into it
16:29jlundjlorenzo: pmoore iirc - every machine should have 2.7 available, in some cases we must explicitly &quot;python2.7&quot; it.
16:34pmoorejlund: but do you know if there is a reason we run 2.6? the rest of the internals seem to be 2.6 compliant, as this was the only failure :/
16:34pmooreat least for this particular execution path
16:42jlundpmoore: afaik, not on purpose. though there is always unknown when updating :)
16:54tedcripes, still python 2.6 somewhere?
17:01akii&#39;m guessing it&#39;s bare &#39;python&#39; with an old centos that has system py26 that can&#39;t be upgraded without breaking yum
17:08tedheh
17:08tedwe should just put pyenv on all our machines
18:22mshalCallek: sorry for the delay - what is the patch supposed to do in bug 1357867 that you wanted feedback for? (And are you still wanting feedback? I see a lot of things have already landed)
18:29kmoirdhouse: do you know why I wouldn&#39;t be able to push to holly? Is there something else we need to do to enable the branch so we can push code there? https://bugzilla.mozilla.org/show_bug.cgi?id=1354284
18:37dhousekmoir: it looks like the branch is disabled. I think we need catlee/philor/? (https://wiki.mozilla.org/ReleaseEngineering/How_To/Close_or_Open_the_Tree) to open it or suggest a different branch
18:41kmoirdhouse: I have rights there was able to add it and open. let me try gain
18:42dhousecool!
18:42KWiersokmoir: you midaired me adding it to treestatus :P
18:42dhousefrom all I see on the hg servers it is ready/set. so I hope that is the only thing needed
18:44kmoiryay, I could push!
18:45dhouseexcellent
18:47kmoirhmm, still don&#39;t see anything on treeherder https://treeherder.mozilla.org/#/jobs?repo=holly
18:47kmoirnot sure if that&#39;s because of how we configured it in buildbot, will look
19:00kmoirI think I need to add the new repo to th will open pr
19:01KWiersokmoir: need to change active_status at https://github.com/mozilla/treeherder/blob/6efa85479104b658ceb4bb3b160e0fee8fcb1373/treeherder/model/fixtures/repository.json#L358 to &quot;active&quot;
19:02KWiersomaybe a few other changes, but I think that&#39;s the big one
19:02kmoirokay
20:24marco_what&#39;s Firefox 52.0.2esr?
20:24marco_I&#39;ve seen a Firefox try to update from 52.0.2 to 52.0.2esr today
20:25marco_actually, two different installations on two different machines
20:25bhearsummarco_: that was a known issue, since fixed
20:25bhearsum52.0.2esr is a build on the esr channel
20:25bhearsumwe accidentally messed up updates for a short period of time
20:25marco_it happened just now
20:25bhearsumdid you just restart firefox?
20:25bhearsumit was probably cached, we don&#39;t show the dialog right away
20:26marco_yeah, maybe it had been already downloaded
20:26bhearsumand i assume you got an error when it tried?
20:26marco_is it possible that it was downloaded yesterday?
20:26bhearsumyup
20:26bhearsumextremely likely
20:26bhearsumi got it last night too, hours after we already fixed the issue
20:26marco_ok, yes, this machine was just booted now and it was last used yesterday night
20:27bhearsumyeah, 99.9% sure that&#39;s the same issue
20:27bhearsumi just double checked and things are still configured correctly
20:28bhearsumthanks for poking about it though
20:41marco_np!
22 Apr 2017
No messages
   
Last message: 123 days and 8 hours ago