mozilla :: #taskcluster

21 Apr 2017
03:45philort-rex if i-011db77f9bb6877e4 hasn't already killed itself and its read-only filesystem, please kill it
03:59philor(it hasn't killed itself, burning 20-30 jobs every time I turn around, which is every couple of minutes)
04:25garndtI just killed it
04:32philorthanks
04:34philorit was a game 'un, took every job it possibly could :)
10:25tedpmoore: is there any way to run things as administrator in a generic worker?
10:26pmooreyes
10:26pmooreted: which worker type?
10:26pmoorewhat do you need to run?
10:26tedpmoore: i was looking at https://bugzilla.mozilla.org/show_bug.cgi?id=1270667
10:26pmoorewe also have an os groups feature that allows you to make a task user a member of an OS group
10:26firebotBug 1270667 NEW, nobody@mozilla.org Generate visual studio tooltool archive from Visual Studio C++ build tools installer
10:27tedthe VS installer always displays a UAC prompt
10:28pmooreted: do you know if it might work as a non-administrator, but with a subset of privileges, where we could create a specific OS group with just those privileges?
10:28tedi do not!
10:28tedi don't actually know what it needs auth for, tbh
10:28pmooreok
10:29pmoorewhich version of windows is this running on? windows server 2012 r2 ?
10:29pmooreor even 2016?
10:30pmooreis this something we should create a custom worker type for?
10:31pmoorealso, is this something that would run on many try pushes, or just something we would run from time-to-time?
10:31pmooreif it is an infrequent thing, we could always throw the worker away after running the job, to avoid an admin user maliciously soiling an environment
10:32tedthis should run rarely if ever
10:33tedideally only once every time we need to update the Visual C++ package
10:37pmooreted, so if you want to have a play around - have a look at https://docs.taskcluster.net/reference/workers/generic-worker/docs/payload - specifically "osGroups" part of the payload
10:37pmoorerun the task with the Administrators group (or is it Administrator group in windows? i forget)
10:37pmooreyou'll need some scopes for this, but the first time it fails, it will tell you what scopes you need, and we can grant them to you
10:38tedokay, thanks
10:38pmoorethis will give you a playground to test on, and when we roll out, we can decide if we want to go with a custom worker type or not
10:38pmoorebut this should at least give you a playground to see exactly what privileges you need etc
10:40pmooreted: and let me know if you get stuck
10:40tedokay
10:40tedi need to fiddle with the installer a little bit locally first, i'm not sure i have the commandlines 100% right
10:49pmooresure
10:57pmooregrenade: i've created a new (beta) worker type for win10 gpu
11:29grenadepmoore: ok. Need me to do anything?
11:31pmooregrenade: could you have a look at the PR on OCC? thanks!
11:32pmooregrenade: it doesn't block anything, it is just to avoid that the new worker type disappears when the old ami gets cleaned up in a few deploys time (as it is just a copy of the non-beta AMIs, which would eventually get deleted)
11:32pmoore(so no rush)
11:33pmoore(and even if they get deleted, i can always manually update the worker type)
11:33pmooregrenade: so the screenshots issue is solved, and this highlighted a python issue
11:34pmoorehttps://public-artifacts.taskcluster.net/X4CZTXqWToadLs3X1SOdEA/0/public/test_info/mozilla-test-fail-screenshot_sadien.png
11:35pmooregrenade: the good news is, on win10, it looks like we are getting the task bar
11:35pmoorewin10 is not running as current user, right?
11:35* pmoore checks the worker type definition
11:36pmooreoh - on beta it is - because this is used by generic worker CI - I'll switch that off for now
11:37pmoore... i've overloaded that worker type a little ...
12:57* garndt is just a few minutes too slow for dustin's repo resets
12:58dustinalways a bridesmaid...
13:10pmoore|afkgrenade: garndt: armenzg_afk: my latest win10 try push looking not too healthy: https://treeherder.allizom.org/#/jobs?repo=try&revision=8964896c28ac243af1755119abeae2116e4c2f29
13:11pmoore|afkis this worse than it used to be?
13:12pmoore|afkmaybe the beta worker type is missing some fixes that went into the non-beta worker type?
13:12pmoore|afki have to look after the kids for a bit, but i'll be back later
13:14pmoore|afkif anyone fancies having a look into this while i'm away - the manifests are here: https://github.com/mozilla-releng/OpenCloudConfig/tree/master/userdata/Manifest so might be worth checking how gecko-t-win10-64.json compares with gecko-t-win10-64-beta.json
13:14pmoore|afkand how gecko-t-win10-64-gpu.json compares with gecko-t-win10-64-gpu-b.json
13:27dustinthere should be a law that error messages from REST APIs should at least include the request URI path
13:28dustinthat, or node should figure out how to give useful tracebacks for async calls
13:28dustinbasically, I get "something went wrong": https://irccloud.mozilla.com/pastebin/6OsJtShG/
13:31dustintravis also helpfully doesn't log the rabbitmq version
14:10garndtpmoore|afk: thanks for the push. Yea, just briefly looking at it, it seems that there are some new greens, but also some new failures....
14:10garndtnot sure why yet
14:10garndtI wish there was a way to compare status of symbols on treeherder between pushes
14:19dustinapparently "dist: yakkety" in travis gets you .. trusty
14:19dustintravis kinda sucks too
14:19dustinit seems there's no way to use a modern rabbitmq
14:24dustinhm, http://packages.ubuntu.com/trusty/rabbitmq-server
14:24dustinyet the trusty image that "dist: yakkety" gets you has 3.6.6
14:24dustinwhat a mess
14:27garndtsounds all completely normal to me!
15:21pmooreTIL (the hard way) at least some nightly mozharness execution runs under python 2.6
15:22dustinreally :(
15:34garndtpython 2.6 for life!
15:37mkaplyI'm getting a "You do not have sufficient scopes" error - https://pastebin.mozilla.org/9019605 -
15:37mkaplyWhen scheduling jobs (I'm logged in and scheduling was working yesterday)
15:49dustinyou need to be logged in with the account that has scm access
16:09mkaplydustin: I'm logged in to the same account I was logged into yesterday (my mozilla okta account)
16:09mkaplyhttps://treeherder.mozilla.org/#/jobs?repo=try&revision=aec15ca1a15f10781f47828d4dbfbb46ab4c276f
16:09dustinyeah but I don't see "scm_level" in the list of scopes you have
16:09dustindid you lose your commit privs?
16:10dustin(just looking for 'scm_' in that pastebin)
16:10mkaplymkaply@mozilla.com (my logged in account) isn't my account used for checkin
16:10mkaplythat's mozilla@kaply.com.
16:10mkaplyI don't have LDAP login credentials for mozilla@kaply.com though
16:11dustinyou can use the forgot-password process for that account to get the password
16:13mkaplyI just logged in as mozilla@kaply.com and I get the same error
16:13dustinpastebin the message?
16:13mkaplyhttps://pastebin.mozilla.org/9019614
16:13dustinyou'll need to do a full username/password login with that user
16:14dustinemail's not enough :)
16:14mkaplyI don't understand why it worked yesterday. Weird
16:14dustinI'm not sure how this would have worked yesterday -- `queue:create-task:aws-provisioner-v1/gecko-decision` was required then, too
16:15dustinbtw I think the IAM team is working on the ability to "link" accounts
16:15dustinso you'd be able to use either one
16:15mkaplypasswordreset.mozilla.org requires login. That seems wrong
16:18dustinyeah, trying to remembre what the forgot-password process is..
16:21dustinmkaply: ah, found it - https://bugzilla.mozilla.org/enter_bug.cgi?component=MOC%3A%20Service%20Requests&form_name=enter_bug&product=Infrastructure%20%26%20Operations
16:22dustinmkaply: you might also just get the scm bits added to your @mozilla.com account
16:22dustinup to you really
16:22mkaplyI thought they work. I've been doing this for over a year. It stopped working today.
16:22mkaplySo something changed.
16:24mkaplyThat was horrible grammar. To get the scm bits would I open a bug in the same place?
16:26dustinI think so, yes
16:29bstackdustin: I had to rebase my fix for bug 1357673 does https://reviewboard.mozilla.org/r/132456/diff/1#index_header make sense to you?
16:29firebothttps://bugzil.la/1357673 ASSIGNED, bstack@mozilla.com Intermittent taskcluster/taskgraph/test/test_try_option_syntax.py | TestTryOptionSyntax.test_b_d, li
16:30bstacksorry to make this sync rather than async, but I'd like to get a fix out for that sooner rather than later
16:30dustinyep
16:30bstacksweet. ty
16:47markco___dustin: could you setup the provisioning bits for hardware windows 10 that you did for windows 7?
16:48dustinsure
16:48dustinwhat is that exactly?
16:49markco___i think it would just be setup a worker type gecko-t-win10-32-hw
16:50dustinsimilar to https://tools.taskcluster.net/auth/clients/#project%252freleng%252fworker%252fscl3-puppet%252fgecko-t-win7-32-hw
16:51markco___yes
17:11mkaplydustin: Thanks for your help!
17:35marcois there a way to get the task before the latest? that is, the task which was latest before https://tools.taskcluster.net/index/#gecko.v2.mozilla-central.latest.firefox/gecko.v2.mozilla-central.latest.firefox.linux64-ccov-opt
17:40dustinmarco: no, only the most recent, highest-ranked task is kept
17:40dustinyou can look by datestamp
17:40dustin(sibling to "latest")
17:43marcodustin: ok, thanks
17:43marcodustin: can I list all the tasks of the same type?
17:43mythmonis there a way to find all the tasks for a particular github repo? Or all tasks by a particular name?
17:43marco"linux64-ccov"
17:44marcootherwise, the datestamp won't be useful for me, as I don't know when the task is scheduled to run
17:44dustinmarco: the index lists datestamp first I think
17:44dustinso you'd have to check a few datestamps
17:44dustinmythmon: only if they are indexed
17:44dustinmythmon: kinda what marco's asking about :)
18:02marcodustin: would you mind triggering this hook? https://tools.taskcluster.net/hooks/#project-releng/services-staging-shipit-code-coverage-bot
18:02dustindone, https://tools.taskcluster.net/task-inspector/#LB5zn68UQh-fceir7mw7jw
18:07bstackmythmon: you can find all of the tasks for a github repo from the status reports they leave on the github commits
18:07marcodustin: thanks!
18:08mythmonbstack: sure, but i was hoping to do it a bit more directly then iterating over the github api
18:08dustinif you add some index routes to task.routes, you can use those
18:09bstackmythmon: you can also do https://github.taskcluster.net/v1/builds?organization=taskcluster&repository=taskcluster-github
18:09bstackwhere you replace the appropriate fields of course
18:09bstackthat goes back to ~October of last year I think?
18:09bstackperhaps a bit earlier
18:10dustinthat's just the most recent though, right?
18:10bstackmy link?
18:10bstackno, it's everything
18:11dustinoh, heh
18:16mythmonbstack: ooh, that's exactly what I was hoping for, thanks!
18:18bstacknp
18:18bstackmythmon: are you using it for automation or just a once-off? I have some thoughts about that endpoint going forward
18:19bstackbut it will continue to exist for a good while yet
18:19mythmonbstack: i'm thinking about making some visualization tools, nothing important
18:19mythmonare you thinking of removing the endpoint?
18:20bstackok, cool. I think it will work in general for that sort of thing. github itself will be the canonical source of information like this, but tc-gh might expose it via an endpoint like this eventually
18:20bstackit might change eventually or be deprecated
18:20bstackbut there will always be something like it
18:20bstackso if/when that day comes, porting should be easy
18:21mythmonok
18:22bstackalso I think Eli is thinking about this sort of thing now if I'm not mistaken? ^
18:22bstacka sort of taskcluster developer dashboard
18:22bstacknot sure how much it will know about github though
18:23Eliits on the radar, but nothing concrete or planned yet
18:23bstackkk
18:25mythmonEli: the thing i'm thinking about is to help visualize timing and dependencies of a taskgraph
18:26Elithat would be cool
18:27pmooregrenade: if you're still around (no worries if not) could you deploy gecko-t-win10-64-cu ? i think it hasn't been deployed since it was updated, and is only used by generic worker CI and taskcluster CI (so shouldn't impact anything in-tree)
19:48tedbleh, i wish we had a general-purpose task graph building/execution library available
19:49tedthe stuff in m-c is great but if you're doing anything outside of there you just have to roll your own
19:53dustinyeah
19:53dustinif you write such a thing, let me know :)
20:22tedhah
20:22teddustin: might be cool to figure out if we could factor some of what we have in m-c into a reusable python library
20:22tedif we could at least make it usable for, say, the NSS folks that seems like it'd be valuable
20:41philorare we intentionally doing TC Linux nightlies on beta and release, even though we don't do nightlies on beta and release?
21:13dustinted: yeah.. I think as the gecko stuff starts to "settle" we will have a much better notion of what works and what doesn't
21:13dustinturns out Task subclasses = BAD
21:13dustintransforms = GOOD
21:13dustinschemas, jury's still out
21:13dustinetc.
21:36garndttransforms++
23:57tedso i have a task that ran out of disk space
23:58tedin docker-worker
22 Apr 2017
No messages
   
Last message: 92 days and 11 hours ago