mozilla :: #releng

12 Sep 2017
00:14iamXXXanyone there?
00:15KWiersoiamXXX: probably lots of people here. what's up?
00:17iamXXXI have a doubt
00:17iamXXXI found a good first bug
00:17iamXXXbug no 1139444
00:17iamXXXand I dont know how to go about it
00:17iamXXXI just want to contribute
00:17iamXXX I just want to solve a bug
00:17iamXXXand keep on contriuting
00:19akithat one might already be fixed, not sure. but https://www.joshmatthews.net/bugsahoy/?releng=1 has a list of mentored bugs
00:19nthomashave you seen www.joshmatthews.net/bugsahoy/ ? it turns out that you chose a bug which is non-trivial, and probably not going to ever happen
00:19nthomassnap
00:20nthomashuh, that shows bugs with assignees
00:20nthomasby default anyway - https://www.joshmatthews.net/bugsahoy/?releng=1&unowned=1 is better
00:21iamXXXshow me a way plz
00:32mkaplynthomas: Still here.
00:32nthomasmkaply: I was going to ask if you wanted just cliqz or the other chipde, then realised from the changes it would just be cliqz
00:33mkaplynthomas: Yeah. The test is on chip sites, but these were just cliqz. Thanks!
01:43* bholley is getting decision task failures on a try push: https://treeherder.mozilla.org/#/jobs?repo=try&revision=029e824a0cad8ae97ceb34208130a20aa47f60d3
01:47KWierso|afkbholley: looks like line 1 in .taskcluster.yml isn't commented out correctly
01:48KWierso|afkyou've got https://hg.mozilla.org/try/raw-file/029e824a0cad8ae97ceb34208130a20aa47f60d3/.taskcluster.yml
01:50KWierso|afkbholley: rebase on m-c and repush
01:50KWierso|afkI think you caught some mid-bustage... bustage
01:55bholleyKWierso|afk: thanks
02:35mkaplynthomas: I'm getting AccessDenied errors on the cliqz builds? AccessDeniedRequest has expired2017-09-12T01:51:52Z2017-09-12T02:34:43Z039E41A8076777BC7gQhxdXPuCRt6DPBcBIHvEpkJbcG5sdshr6e7Ku/fjKeClQctWqMH6HbYeV8iWPBO2a0BbksXM0=
02:35nthomasmkaply: not aware of anything changing our side, and we don't admin the bucket/website so hard to debug
02:36mkaplyOdd. It's happening with all of our partner builds.
02:36mkaplynthomas: I'll figure it out. Tx
02:37mkaplynthomas: Somehow my google auth had timed out. I'll have to remember that one :)
02:38nthomashuh, ok
02:38mkaplyThe partner download site uses Google auth. I had to reauth with auth0. but the errors looked like an s3 error.
08:44travis-cibuild-buildbot-configs#2871 (master - 91c914b : Andrei Obreja): The build was broken. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/274513940)
09:01travis-cibuild-buildbot-configs#2872 (master - a24c828 : Andrei Obreja): The build is still failing. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/274519194)
09:18travis-cibuild-buildbot-configs#2873 (master - 163c65e : Andrei Obreja): The build was fixed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/274523942)
09:32travis-cibuild-buildbot-configs#2874 (production - dca6069 : Andrei Obreja): The build was broken. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/274528950)
09:46travis-cibuild-buildbot-configs#2875 (production - 837b184 : Andrei Obreja): The build was fixed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/274532392)
09:54nthomas|awaywhat was the issue ther e?
10:06mtabaraI think the issue still persists and most likely lies within that `range() + range()` sort of operation that attempts to leave some buildbot masters specifically out.
10:07mtabaraI see three pushes and backouts in the history so I assume it's not yet fixed but iterated on default to find the culprit
10:08mtabaraaobreja|lunch: let's try those changes in a PR first of https://github.com/mozilla-releng/build-buildbot-configs or running tests locally
10:08mtabaraI believe we've hit this as well last Friday
10:10mtabaramaybe it makes sense to change that range() + range logic to be something like range(1,277) and then check each of those instances separately in a conditional, something like if i == 102: continue # omit 102 for X reason bug Y, etc
10:10mtabarathat way we have a clear statement of what machines have been skipped and a corresponding bug number
10:10mtabaraand it's far easier to read rather than the sum of range()
10:45aobreja|builddutyit's a little bit strange because the same push in default failed first but after that went fine,exactly the same changes,I will push in production once these former win8 are also changed in Inventory and nagios
13:33PikeCallek: would you have some cycles to chat about bug 1397721 ?
13:33firebothttps://bugzil.la/1397721 NEW, nobody@mozilla.org [cross-channel] Build beta from l10n-central
13:40CallekPike: sure, let me skim the bug then vidyo?
13:40PikeCallek: great, thanks
13:41Callekhrm, reading the bug, might be worth bringing in releng releaseduty for 57, do you think we're at that stage?
13:42PikeCallek: I think we're at that stage, unless there's something in releng I don't see
13:43Callekok, wfm. let me find out who's releaseduty :-)
13:43bhearsumwhere are the nightly-only locales specified these days? i couldn't find it in shipped-locales, all-locales, mozharness, nor taskcluster graph code
13:45Pikebhearsum: I don't think we have anything right now not. It'd be a different set of all-locales and shipped-locales files
13:45bhearsumwe build a subset of locales on nightly though, don't we?
13:45Pikebhearsum: no, we started building all locales on Nightly with Dawn
13:45bhearsumoh!
13:45Pikeand moved our primary l10n target to Nightly
13:46bhearsumso does that mean that mozilla-central's all-locales file contains the list of locales we build on Nightly?
13:47CallekPike: so, https://bugzilla.mozilla.org/show_bug.cgi?id=1397721 will be chatted with me and kim, ~ now. My room?
13:47firebotBug 1397721 NEW, nobody@mozilla.org [cross-channel] Build beta from l10n-central
13:47* Callek pulls multiple channels to a place where everyone is :-)
13:48PikeCallek: 'll be right there
13:48Callekkmoir: my room? (incase you don't stalk `kim`)
13:48Pikealso, don't mind other folks joining
13:48kmoirsure
13:55travis-cibuild-tools#1964 (master - 677fd9f : Mihai Tabara): The build passed. (https://travis-ci.org/mozilla/build-tools/builds/274614366)
14:13flodbhearsum: yes, all-locales in m-c = list of locales we build nightly for
14:14bhearsumflod: thanks
14:25flodas for the original question, the list of "nightly only" languages is quite small these days (lo, ltg, ne-NP, tl), but it's not stored anywhere
14:27bhearsumflod: ahhh
14:27bhearsumi think i phrased my question very badly, i was looking for the list of nightly locales, not the list of nightly-only locales
14:27flodok, that's clearer :)
15:03Aryxhi, Windows buildbot test jobs are failing: https://treeherder.mozilla.org/logviewer.html#?job_id=130378439&repo=mozilla-inbound
15:03AryxCould not install python package: C:\slave\test\build\venv\Scripts\pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub psutil==3.1.1 failed after 5 tries!
15:03Aryxwill close trees for that
15:09Aryxfiled https://bugzilla.mozilla.org/show_bug.cgi?id=1399151
15:09firebotBug 1399151 NEW, nobody@mozilla.org Windows buildbot test fail: Could not install python package: C:\slave\test\build\venv\Scripts\pip i
15:09Aryxaobreja|buildduty: ^^
15:14aobreja|builddutyAryx: seems like machines that have been migrated from win8 are failing,I will disable them for the moment
15:15Aryxaobreja|buildduty: there are also win7 and win10 https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=1a96a71425be91b3121acf64e08188307a89b965&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable
15:25travis-cibuild-tools#1965 (master - b5cf81b : Mihai Tabara): The build passed. (https://travis-ci.org/mozilla/build-tools/builds/274653502)
15:47aobreja|buildduty arr:I did not saw any changes in the latest pushes that could cause error that Aryx mentioned above so I suspect an infrastructure change ,maybe markc.o can investigate more
15:49arraobreja|buildduty: there haven't been infrastructure changes
15:49arraobreja|buildduty: wre those new machines that you just enabled?
15:49aobreja|builddutyit seems that is also afecting win 7 and win 8 so it's not related with my changes
15:50aobreja|builddutyCould not install python package: C:\slave\test\build\venv\Scripts\pip install --timeout 120
15:50arrwhen did the error start?
15:51aobreja|builddutyfew hours ago I think
15:51Aryxhttps://treeherder.mozilla.org/logviewer.html#?job_id=130375694&repo=autoland is from 2:18pm UTC
15:52Aryxunfortunately, we don't have a system to query that information
15:53Aryxthe real error seems to be "error: Unable to find vcvarsall.bat". see also https://bugzilla.mozilla.org/show_bug.cgi?id=1351353
15:53firebotBug 1351353 REOPENED, nobody@mozilla.org Intermittent error: Unable to find vcvarsall.bat
15:54markco_I am looking into as well
15:55arrAryx: in that bug, spacurar said that that's normal (comment 1)
15:58arrAryx: are systems in AWS seeing this issue? are linux systems?
16:01Aryxarr: no, linux talos is fine and i see no issues with aws/tc instances
16:01arrAryx: we do w7 buildbot tests in AWS. Are you sure those are not impacted?
16:03Aryxwhat has the "tc-" prefix is fine, the jobs without are red, see https://treeherder.mozilla.org/#/jobs?repo=autoland&fromchange=9232bc78ae68dd7978bb0474af1ef8be581dd35a&group_state=expanded&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&filter-resultStatus=success&selectedJob=130389932&filter-searchStr=windows
16:03Aryxsorry, have to go afk for maybe 30 minutes
16:05arrso I wonder if that points to a change in the buildbot/mozharness configs or something
16:06aobreja|builddutyfrom what I see first fails appear 1h30 ago on win7,win8and win10
16:06aobreja|builddutyhttps://secure.pub.build.mozilla.org/buildapi/recent/t-w1064-ix?numbuilds=599
16:06aobreja|builddutyhttps://secure.pub.build.mozilla.org/buildapi/recent/t-w864-ix?numbuilds=599
16:06aobreja|builddutyhttps://secure.pub.build.mozilla.org/buildapi/recent/t-w732-ix?numbuilds=599
16:06arrhm, mochitest w8 x64 opt didn't have issues
16:06aobreja|builddutyand everything is red after that
16:06arraobreja|buildduty: does that match up with any in-tree changes?
16:07aobreja|builddutyI didn't found any changes that could be related
16:07aobreja|builddutythat's why I suspected infrastructure change
16:08arraobreja|buildduty: I don't think anything changed wrt the windows machines
16:08arrand it's really unlikely it would have changed across the board for all three OSes
16:10arrit might be network issues if they're having issues contacting the package mirror, but that should be impacting the linux machines, too
16:11aobreja|builddutylinux machines look fine:
16:11aobreja|builddutyhttps://secure.pub.build.mozilla.org/buildapi/recent/talos-linux64-ix?numbuilds=100
16:13arrand OS X looks okay
16:14arrmarkco_: are you able to run the command byhand and have it work?
16:15aobreja|builddutypypi download failures for Windows buildbot jobs
16:20markco_arr: I have to deconstructed it a bit to try
16:25markco_it won't work because all the venv pieces are not in place outside of the test running
16:27markco_but i am able to get to https://pypi.python.org/pypi and wget from it
16:27arrokay, so not firealls, then
16:29arrjmaher: know of anything in the tests that would have changed about 1.5-2h ago that would cause this? ^^
16:30arr(and a sanity check that the pip error is what's causing the actual failure)
16:32markco_Arr: 07:40:21 FATAL - Could not install python package: C:\slave\test\build\venv\Scripts\pip install --timeout 120 --no-index --find-links http://pypi.pvt.build.mozilla.org/pub --find-links http://pypi.pub.build.mozilla.org/pub psutil==3.1.1 failed after 5 tries! 07:40:21 FATAL - Running post_fatal callback...
16:33markco_That pip install is not finding the psutil version at http://pypi.pub.build.mozilla.org/pub/
16:33arrmarkco_: is there more info in the pip log?
16:34markco_https://pastebin.mozilla.org/9032232
16:34markco_and from the log
16:34markco_https://pastebin.mozilla.org/9032233
16:35arrhttp://pypi.pvt.build.mozilla.org/pub/ has things named 3.1.1, but I don't know exactly what format it would be looking for
16:36arrnone of that looks like it's changed since 2015, though
16:38arrpmoore: dustin: either of you folks have clues about that ^^ (I know it's buildbot infra, but I suspect both of you have more experience with tooltool and pip)
16:47arrmarkco_: if you do a pip -v is it more illuminating?
16:47travis-cibuild-tools#1966 (master - 558f6b0 : Mihai Tabara): The build passed. (https://travis-ci.org/mozilla/build-tools/builds/274688673)
16:48markco_arr: not really https://pastebin.mozilla.org/9032234
16:51arrdid someone recently change the version of pup that gets installed into the venv?
16:51arrer pip
16:53arrmarkco_: not sure about pip on windows... can you specify -vvv ?
16:53arr(for extra verboseness)
16:53arrnot sure that's really going to be more helpful
16:54akii think by default setuptools updates itself from pypi
16:55akihttps://bugzilla.mozilla.org/show_bug.cgi?id=1369250#c3 , hopefully that stuck and covered all cases
16:55firebotBug 1369250 FIXED, aki@mozilla.com "ImportError: No module named six" errors causing bustage across all trees
16:56markco_arr: https://pastebin.mozilla.org/9032236
17:05arrmarkco_: what if you try it without specifying the version?
17:06arrI don't actually know how pip determines the filename to match the version
17:07markco_arr: No matching distribution found for psutil
17:07arrI'm assuming the --find-links option is supposed tobe telling it something but I don't know pip well enought to know what
17:12bhearsumarr: --find-links is usually how you specify a pip mirror
17:12arrbhearsum: yeah, but how does it know the file name once you tell it the url?
17:12arrthat's the part I'm iffy on :}
17:13bhearsumi think it finds package names/versions based on filenames
17:13arrthere are definitely files with the name and version there
17:14arr"Skipping link http://pypi.pvt.build.mozilla.org/pub (from -f); not a file" seems weird
17:14arrbut I don't know if that's normal
17:15arrand the same for the pub address
17:15bhearsumweird, i've never seen that before =\
17:16arrthat was with verbose mode
17:17akiyeah, looks like a new pip
17:17akihttps://irccloud.mozilla.com/pastebin/YWS7Fexp/
17:17bhearsumahhh
17:17arrokay, so I was right
17:18arrhttp://pypi.pvt.build.mozilla.org/pub/pip-9.0.1.tar.gz 12-Sep-2017 07:14 1.1M
17:18arr
17:18arrso whoever uploaded that has broken things
17:19tedthe mozharness code for running pip already specifies that, FWIW: https://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/base/python.py#264
17:19ted(gps fixed that a while back)
17:20aki2 people sshed into puppet this morning
17:23arrI'm going to remove the new version of pip
17:27arrwaiting for it to sync to update the bug, then we should be able to reopen the trees
17:30gpsyes, there are several automation jobs that don't work with new pip
17:30gpsfor various reasons
17:30arrwish we were pinning the version of pip in tree
17:31arrthat would prevent things like this
17:31gpswe have pip vendored in tree
17:31gpsand if using a source checkout, that version gets used
17:31gpsthe problem is many jobs aren't using source checkouts
17:31dragromarr: then I'll need to find another way to update pup on OSX. Is strange that IT interfer with windows
17:31gpsarr: snapshotted package repos would prevent this type of problem :)
17:32arrgps: not really, we'd have the exact same probem
17:32gpsnot once things are pointed at a snapshot
17:32gpssnapshots are read-only
17:32gpsso if a snapshot has pip X, there's no way you can get pip Y
17:33gpsyou still have problems when upgrading things. but at least you can't break random stuff by modifying a snapshop... because snapshots are immutable
17:34arrgps: if you have a snapshot for every platform, that's a LOT of space you're taking up
17:34gpszfs/btrfs ftw
17:34arr(every platform, ever package)
17:34arrgps: I don't think AWS offers that :D
17:34gpsyou can format zfs/btrfs on an ebs volume
17:35gpsthis is outlined in my full proposal: https://docs.google.com/document/d/14ogSVp3KSMxrr2809-2Rrbic5DsVRtZdVJk85Ojie8U/edit
17:35arrAryx: so just for the record, this would have affected buildbot AWS as well
17:36arrgps: huh, zfs on linux? the last I checked that was... not good. admittedly I haven't looked recently. Haven't managed to read through your proposal yet
17:36gpsarr: it's good enough
17:36gpswe could use FreeBSD if you wanted
17:40arrAryx: the trees can be reopened
17:40Aryxok, thank you all for fixing this
17:42arrhopefully the workers clean up their venv between jobs
17:43arrgps: ^^ do you know if that's true?
17:43Callekclobber would be safer
17:43gpsarr: depends on the job
17:43CallekI know some jobs wouldn't succeed in downgrading pip...
17:43arrCallek: so how do you tell every machine to do a clobber build when it picks up a job?
17:44arrAryx: so until things clean themselves up with a clobber build, there might be some issues, still... I don't think there's anything I can do from a machine perspective for that, though
17:45CallekCLOBBERER for buildbot-based tasks, TC cache invalidation otherwise, (maybe https://tools.taskcluster.net/purge-caches I forget the way for sure)
17:45arrCallek: wouldn't be a problem for tc machines
17:45arrthey run a new enough version of python
17:46Callekhttps://api.pub.build.mozilla.org/clobberer/
17:47Aryxok, let me trigger it
17:47arrAryx: okay, will let you take over from here
17:47arrmarkco_: thanks for the debugging help
17:47markco_np
17:47arrgps: and aki, too
17:48akinp
17:48akii wish we had a better pinning story
17:48arryeah, me too
17:49arrback to my regularly scheduled meetings!
17:50gpswhich job actually failed? is there a link?
17:50gpsbecause i don't see a reason why we can't pin
17:52arrAryx: ^^
17:53Aryxgps: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=3867d0df5cafe68058139c4de8e47c4d6e79e567&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable has the earliest failures i could find
18:11jmaherarr: I am not aware of anything on tree today, been in meetings all day sorry
18:11arrjmaher: thanks, already solved
18:11jmaheroh, ok
18:14gpsInstalling pip>=1.5 into virtualenv C:\slave\test\build/venv
18:15gpshttps://dxr.mozilla.org/mozilla-central/source/testing/mozharness/mozharness/mozilla/testing/talos.py#525
18:15gpshttps://dxr.mozilla.org/mozilla-central/source/testing/mozharness/scripts/desktop_unittest.py#327
18:15gpsthose should be pinned to specific versions
18:17Callekgps: offhand is there a way to make pip (via global config) fail if its being used without a specific version pin?
18:17gpsi don't think so
18:17* Callek isn't aware of one, but its something I'd like to see us be able to enforce at that level
18:17gpswell, kinda
18:17gps"use requirements files with hash pinning"
18:18gpsyou could add --require-hashes to the global defaults
18:18gpsbut that would prevent you from doing a local package install
18:18gpsso it isn't practical
18:18Callekhrm requirements files is painful when you may want to do `pip install foo==1.1.1`
18:18Callekbut yea, would be nice to have a switch for forcing that piece alone :/
19:48Aryxmarkco_: hi, so i clobbered autoland windows machines 2h hours ago https://api.pub.build.mozilla.org/clobberer/ and the jobs still fail on a new push: https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=5951cc2057a14fb414194127feeb3d82afda9205&filter-resultStatus=runnable&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception
21:08nthomasfirebot: ping
21:08firebotnthomas: pong
21:20RyanVMAryx: I'm a bit confused - clobbering is only going to affect the builds
21:20RyanVMclobbering has nothing to do with the test machines?
21:22Aryxryanc-pto: http://logs.glob.uno/?c=mozilla%23releng#c312728
21:22Aryxsorry
21:22AryxRyanVM: http://logs.glob.uno/?c=mozilla%23releng#c312728
21:23RyanVMconsider me confused
21:23RyanVMhow is clobbering builds going to fix broken test machines?
21:23RyanVMCallek: ^
21:36RyanVMI rebooted t-w864-ix-071, we'll see if it goes green on the next job it runs
21:36RyanVMI'm thinking this may require mass-rebooting
21:59CallekI meant clobber the testers. Thought that was still a thing
22:02RyanVMI don't recall it ever being a thing
22:07arrKWierso: ^^ (since you NI me on the bug)
22:07arrso what's actually needed to fix things? Is it rebooting everything?
22:08arrare there just a few machines that are broken?
22:08KWiersothat seems like it would purge whatever's cached
22:08RyanVMAFAICT, it's all windows testers that aren't on AWS
22:08arrKWierso: so is this something you trigger through slavehealth?
22:08arror is it something you need assistance with?
22:09KWiersoarr: was hoping there was something we could hit to do them all as a batch rather than rebooting potentially hundreds of machines by hand
22:10arrKWierso: I thought slavehealth let you reboot a bunch?
22:10arrif not, I can just for loop a power cycle
22:10akiremoving pip may not have downgraded pip on those boxes
22:10arraki: I thought it was in a venv, so the venv would get recreated?
22:10arror do they not clean up the venv and recreate?
22:11akilooks like it should
22:11arrso, do I just reboot everything?
22:11akivenv is C:\\slave\\test\\build/venv , clobbers C:\\slave\\test\\build . not sure why it would be broken then, unless they managed to update system pip
22:11arraki: does clobber work on a tester?
22:11arrthat was the convo we were just having
22:12KWiersoarr: I think it's just the windows -ix machines, not necessarily everything
22:12akinot clobberer afaik, but looks like mozharness does 12:45:05 INFO - retry: Calling _rmtree_windows with args: ('C:\\slave\\test\\build',), kwargs: {}, attempt #1
22:12arrKWierso: sorry, by everythign, I meant w7, w8, w10, xp IX
22:13KWiersoarr: I see these failures for all of those, yes
22:13arraki: so are you saying they should self correct when picking up a new job?
22:13arror just when rebooted?
22:13akilooks like when picking up a new job
22:13arrso they should have self corrected already, then
22:13akiprobably. unless mozilla-build's python's pip got updated
22:13KWiersoaki,arr: https://secure.pub.build.mozilla.org/builddata/reports/slave_health/slave.html?class=test&type=t-w732-ix&name=t-w732-ix-055
22:14arrKWierso: are we still seeing a lot of errors, or just on machines that haven't picked up jobs since the pip upgrade?
22:14KWiersodoesn't seem like new jobs helped
22:14akidowngrade, no?
22:14arrhow do we force a downgrade?
22:14akii wonder if there's a ~/.pip
22:15arrthe upgrade happened in tree, so it's not something that was done as part of the management stuff
22:15aki~/.python-eggs
22:15akiyeah, pip caches locally on disk
22:16akihttps://stackoverflow.com/a/31807659
22:16aki<CSIDL_LOCAL_APPDATA>\pip\Cache
22:17arrwhat is pip caching that&#39;s bad?
22:17akipip 9.0.1 ?
22:18arrpip is caching the pip binary?
22:18akijust a guess
22:18arro.O
22:18arrso is there something that can be checked into tree to fix this?
22:18akifor when you install a new virtualenv
22:18akiwe can turn off caching with --no-cache-dir i believe
22:19akibut that means we&#39;ll hit the server for all requests
22:19akii suppose we could also land a &quot;nuke the cache dir&quot; puppet patch
22:19arrthey&#39;re not controlled by puppet
22:20arrso it sounds like the options are:
22:20arr1) reimage all the machines
22:20arr2) land something in tree to tell it not to use the cache
22:20arr3) put something in GPO to remove the file
22:20arrassuming that actually fixes things
22:20arr1 is... non optimal
22:20akican we access one of the machines? rdp?
22:21arraki: you should be able to
22:21akiok
22:21arrmarkco_: is afk, so he can&#39;t help
22:21akiit makes sense to verify whether the cache is actually at fault before most or all of those solutions
22:21arrif we determine that removing the file is the fix, I can ask Q to help if he&#39;s still around
22:21arryeah
22:25akii&#39;m not in the slave-pw list
22:25arrtalking to Q
22:26arrhe says #releng isn&#39;t updating for him, so having a convo in another channel
22:26akiok
22:26arrtold him your theory on the cache and asked him to take a look at t-w732-ix-055
22:27akiok
22:27* hwine just did a one step upgrade from Fx 31 to Fx 55.0.3 -- is that the magic of funsize I experienced?
22:28akihow big was the download?
22:28nthomasthat sounds ... not right
22:29hwineaki: didn&#39;t take long for either download or update
22:29akisurprised you didn&#39;t hit a watershed
22:29nthomasme too
22:29hwineyeah - I expected many steps
22:29akibut i&#39;d guess you downloaded a 70ish mb complete
22:30hwineOh, right - I forgot about the complete
22:30hwineand, in the office, 70MB is very quick
22:32arrhm http://pypi.pub.build.mozilla.org/pub/ is still showing pip 9.0.1 even though I removed it from the puppet server
22:33arrI wonder if there&#39;s a syncing job
22:34akii don&#39;t know where pypi.p.b.m.o is or what it syncs from
22:34akipretty sure it isn&#39;t puppet
22:34akithe backing server, i mean
22:35akibut that would explain the continued bustage
22:36* hwine thought our internal pypi was treated just like tooltool - manual uploads
22:37akido you know what server?
22:37arrit&#39;s on relengweb
22:37nthomasthat pypi might be the releng cluster ?
22:37akii&#39;m not sure i&#39;ve ever known
22:37akiok
22:39arrOkay, I&#39;ve manually removed it from the web cluster
22:40arrcan we force one machine to pick up a job?
22:41fox2mikearr: I have a question
22:42fox2mikethese&#39;s an SSL test somewhere for https://ssl-dv.mozqa.com/ within our tests
22:42catleeno questions allowed
22:42fox2mikeand that cert is with fucking thawte, who&#39;re being a PITA to getting it renewed
22:42arrfox2mike: sorry, don&#39;t know anything about mozq
22:42arra
22:42fox2mikeso question for you guys - is there a way we can skip those tests?
22:42fox2mikearr: know anyone who might?
22:42akiwe can retrigger a broken test for pip 9.0.1 testing
22:44arrfox2mike: hm, not sure off the top of my head... is there anyone listed as the owner of that in that spreadsheet corey has for all of the owners?
22:44arrryanvm might also have some historical context and might have a name
22:45arrverified that it&#39;s actually gone from the pypi mirror this time
22:45akii didn&#39;t see pip anywhere in my pip cache, so hopefully i&#39;m wrong about that
22:48fox2mikearr: it&#39;s whimboo
22:48fox2mikebut he&#39;s not in this timezone
22:48arrfox2mike: there you go :}
22:50arraki: btw, I tried to update your pw, and it failed. but the file you want is aspect-low-security.txt.gpg and you do appear to have rights to that
22:50arrQ says python 2.7 just got pip-1.5.5-py2.7.egg-info in its venv
22:51arrso... hopefully that fixes it?
22:51akiaha
22:51akiwoo
22:52akii see a green p https://treeherder.mozilla.org/#/jobs?repo=autoland&selectedJob=130474509
22:52arrKWierso: how quickly were jobs failing (aka, how soon can we tell whether or not it&#39;s fixed?)
22:52Qaki t-w732-ix-055 just picked up a job and got pip-1.5.5-py2.7.egg-info in its env
22:53arryay, Q got his irc working again :D
22:53Qinstead of 0.9
22:53akieach test failed around ~15min in
22:53KWiersoarr: seems like it&#39;s between 15-20 minutes into the job
22:53akiyeah
22:53QI am going to run a loop to make sure we purge 0.9
22:54Qjust to double check pip-1.5.5 should be the correct version ?
22:54akii believe so
22:54arrQ: okay, can I have you take over here?
22:54* arr needs to run
22:54Qarr: yes ma&#39;am
22:54akii wish we weren&#39;t 8 major versions behind, but it works. pip 8.x may be less broken for us, but i don&#39;t want to try right now
22:54arrQ++ thanks!
22:54arraki: I think anything modern is broken because of SSL
22:55arrbut not sure
22:55arronce nothing is on the buildbot machines anymore, we&#39;ll be in a better state
22:55akimaybe we should fix pypi.p.b.m.o ssl then
22:55akiyeah
22:56arraki: it&#39;s not the web site, it&#39;s the python client
22:56arron the windows machines
22:56arrthat&#39;s the thing that breaks every time you look at it sideways
22:56akii know, but 9.0.1 would work if it were pointing at a valid https site
22:56akiit was ignoring the link b/c it&#39;s http
22:57akiper the error msg, at least
22:57aki2nd green
22:57arrmaybe. we&#39;ve always found that the issue goes back to the version of pythog on windows
22:57arrmay not be the case here, I don&#39;t know
22:58akigetting rid of bb sgtm
22:58arryeah, will be happy when that happens
22:58arrso much old tooling we can leave behind
23:00akii see 4 greens where there was only burning. guessing we&#39;re good
23:00arrwoo
23:00arrQ ^^
23:00akithank you!
23:01arraki: thank you for your help!
23:01akinp, glad we didn&#39;t wipe everything
23:01arryeah, me, too... that would have been painful
23:02arrmy fault for not double checking the pypi mirror
23:03arrQ++ for jumping in, too
23:04Q\o/
23:04catleethanks Q, aki, arr!
23:21arrso the issue of making sure that gets pinned to a specific version in tree... who could do that? /cc gps, aki, ryanvm (starting with you guys)
23:30akihm
23:32akiit is pinned =\ =\ 16:18:02 INFO - Installing pip>=1.5 into virtualenv C:\slave\test\build/venv
23:32akiwe just have to change the >=1.5 to ==1.5.5 or w/e
23:33akihttps://dxr.mozilla.org/mozilla-central/search?q=pip%3E%3D1.5&redirect=true
23:35aki>=1.5,<2 may be preferable
23:35akithat may break non-bb
23:36akii&#39;d lean towards not touching pypi.p.b.m.o until we&#39;re off bb
23:36akihave a new pypi repo for tc
13 Sep 2017
No messages
   
Last message: 7 days and 15 hours ago