mozilla :: #taskcluster

16 May 2017
00:00dustinjryans: apt should be installed..
00:01jryansdustin: says not found... i happen to be on a loaner of a _build_ machine, rather than the more typical test machine...
00:08arrjryans: build machines are centos, not ubuntu
00:09arrso you'd be looking for yum/rpm not apt
00:09jryansarr: ah, thanks, it worked :)
12:37ochameaudustin: hi! it looks like I miss some scopes to be able to create tasks (I filed bug 1365238)
12:37firebothttps://bugzil.la/1365238 NEW, nobody@mozilla.org Allow github.com/ochameau/ff-dt to create tasks from a decision task
12:41garndtochameau: where is the code for creating that decision task? Are you setting the priority to "highest" anywhere in one of the tasks you're trying to create?
12:43garndthrmn
12:43garndtfound it
12:43garndtit doesn't look like it.....
12:43garndthttps://github.com/ochameau/ff-dt/commit/0e9e15db5df217ebe458c4e8f99581c9a0fbb723
12:45ochameaugarndt: I don't pass any explicit priority
12:45dustinhighest was only one of the options
12:45dustinI think the "you have []" is because task.scopes is empty
12:45dustinquestion is, which scope to put in there
12:45dustinI don't see the priority-less option as one of hte available options
12:47ochameauI don't really follow... so is it an issue regarding the priority?
12:47ochameauhow do I address that?
12:48ochameaunormandy doesn't seem to specify any priority either
12:48ochameauhttps://github.com/Osmose/normandy/blob/master/ci/taskcluster/bin/decision.py
12:48ochameaunor does it specify any scope
12:52dustinochameau: we just changed the priority handling last night -- greg and jonas worked on that
12:53dustinI don't think that broke anything, but I'm not entirely sure what the best choice is for you in the new system
12:54garndtif scopes is empty in that error, seems like the client and token being used within the decision task is wrong/non-existent
12:54ochameauah ok
12:54ochameauyes, that credential/token story is a mistery to me
12:55ochameauI don't understand how normandy handles that
12:55dustingarndt: the scopes is from task.scopes
12:55dustinhm
12:55dustinmaybe not
12:56dustintask.scopes has [queue:create-task:aws-provisioner-v1/github-worker]
12:57dustinhttps://github.com/ochameau/ff-dt/blob/task-decision/task_decision.js#L30 I see you're using the proxy
12:58dustinalthough it's not enabled
12:58dustinsomething's funny here
12:58garndtoh I looked at the wrong commit
12:58ochameauthat is one of my 20 tries. tbh, I don't know exactly what I'm doing here
12:58dustinah, ok
12:58ochameauit is not clear how I get authenticated
12:58dustinwhat's the latest try?
12:58ochameauhow the taskcluster-client node library is able to be authenticated
12:58dustinyou need to enable the taskclusterProxy feature in docker-worker
12:59dustinthen taskcluster-client talks to http://taskcluster (which points to the proxy via the magic of docker), wihch adds the authentication based on the decision task's .scopes
12:59ochameauah ok, so it sounds like using the proxy is the right thing to do?
12:59ochameauI just miss the taskclusterProxy in my yml?
12:59dustinyes
13:00dustinif you look at the payload for https://tools.taskcluster.net/task-inspector/#Y69w-FhFRdG1LEE-zJn7dQ/ you'll see it
13:00dustinthis line, too, is important - https://github.com/ochameau/ff-dt/blob/task-decision/task_decision.js#L30
13:00dustinsince that's what causes the client to use the proxy
13:00ochameauyes, I saw it in their taskcluster.yml
13:04ochameauis "queue:scheduler-id:taskcluster-github" important to call createTask? because my push get immediately rejected if I add it in the yml file
13:05ochameau\o/ it works, I see a new task!
13:23dustinyay!
13:23dustinno, that scheduler-id bit is part of the new stuff, but you can keep using the old stuff which will work fine without it
13:35garndtnice, so we can resolve that bug?
13:38dustinsounds like it, unless jonas wants to encourage a move to the new priorities :)
13:39garndtbased on what he said last night, he does :)
14:02ochameaucan we wait for a task to be finished? Is it the claimTask request?
14:04garbasis there a way to cancel all tasks sheduled for specific workertype?
14:05garbasfor "releng-task" we accedentaly created ~1300 tasks
14:18bastienand follow up on garbas : is there a way to another worker type for the shipit project , or ideally one for staging hooks / tasks to avoid this later on
14:23garbaswouuld a temporary solution be to for next 2 hours raise the capacity to process the tasks?
14:30dustingarbas: yes, tc-admin has a cancel task group option
14:30dustinsorry, tc-cli
14:30dustinhttps://github.com/taskcluster/taskcluster-cli
14:31dustinochameau: generally you want to do that with dependencies
14:31garbasdustin: but i need to know th taskid right?
14:31dustinbut if you are building a service waiting for tasks to finish you can listen to pulse
14:31dustinhaving a worker sit waiting for antother task is not a good use of resources
14:31dustingarbas: at least the taskGroupId, yes
14:31jmaherwcosta: did you figure out the esr52 vs beta vs trunk jobs for osx opt ?
14:32wcostajmaher: my first try on beta https://treeherder.mozilla.org/#/jobs?repo=try&revision=d9ea6785c556
14:32garbasdustin: ok, i only have a worker type right now
14:32garbasdustin: would it be possible to raise releng-task capacity for next 2hours. just to process this tasks
14:32jmaherwcosta: cool, I will await results
14:33dustingarndt: ^^ do you have a way to enumerate those taskIds?
14:33dustinotehrwise, yes, bumping capacity is OK
14:33dustinsorta like flaring off excess natural gas :(
14:33garbas:)
14:34garbastaskcluster is a natural resource? :)
14:34catleefraking for cpu cycles
14:36garbasdustin: also would it be possible to create a new workertype for code coverage effort that is being done by release management team (led by bastien)
14:36dustinyeah
14:37garbasmaybe it would be even better to first discuss this with releng/relman teams how do we plan to use taskcluster in regarding mozilla-releng/services
14:37garbasand then create needed worker types
14:37garbasto not bug you too much
14:40dustinthat sounds good :)
14:40bastiengarbas: +1
14:40dustinwe're happy to create workerTypes for you, and then modify them when you change your mind ;)
14:40* dustin will bump capacity now
14:41garbasdustin: for now just increase the capacity, i'll round up releng/relman teams shortly and get answers :)
14:44dustinyep
14:44dustinup to 100 now
14:44dustin(from 20)
14:56garbastnx!
14:57garbasdustin: we're still running at 20 parallel tasks, but i guess this will change soon once instances come up, right
14:57dustinit should
15:02pmoorejonasfj: https://github.com/mozilla-releng/OpenCloudConfig
15:10ckousikochameau: dustin
15:13ochameauckousik: yes?
15:15ckousikI'm sorry, I think I accidentally sent the message from my phone
15:16dustinhehe
15:53pmoore|mtgdustin: jonasfj: https://github.com/taskcluster/generic-worker/blob/master/worker_types/win2012r2/userdata
16:32pmoore|mtgjonasfj: https://github.com/petemoore/lsalogonuser
16:41dustinpmoore|mtg: any idea what might cause https://public-artifacts.taskcluster.net/ETEZDk_IRtOZVPa5rzrz9Q/0/public/test_info/mozilla-test-fail-screenshot_cijnpb.png
16:42dustinoperating hypothesis is that the dialog is somehow causing the test startup to hang
16:42garndtThere's some prefs in windows for default extension handling. I wonder if a rule needs to be added
16:46dustinyeah.. but why would firefox or marionette or mozrunner or whatever be opening a text file?
16:49* dustin wonders who to ask..
16:49garndtDoes Firefox have an option to "save as and open" when downloading a file? I wonder if the test is doing that
16:50garndtSorry, sitting in a doctors office so I can't help dig in too much
16:50garndtGood thing they're only running a hour behind!
17:07dustinno
17:07dustinI think there's a "start.exe" or something that will open a file in the shell
17:08garndtSo many warnings in that log, hard to pick things out
17:12dustinyeah
17:12dustinbut I compared it to a successful run
17:13garndtAh, so not perma fail. That makes things fun!
17:14dustinright
17:14dustinthere's some race here, maybe
17:14dustinoo, I have an idae
17:36Aprilhey there, taskcluster-ians
17:36Aprilquick question / favor
17:37Aprildid grafana.taskcluster.net recently go away (aka can it be removed from the bounty list) and could somebody, if they have a free moment, take a look gander at https://www.mozilla.org/en-US/security/bug-bounty/web-eligible-sites/ and let me know if the taskcluster list is up to date? Thanks! :)
17:38camdEli: hey man, I have another UX question for you wrt toggling infra and intermittent tests to show in this UI. Would you ping me when you have a couple mins? Probably quick. :)
17:40dustinApril: yeah, it's dead
17:40Aprilwoo!
17:41garndtThere's a telemetry one in the taskcluster list that isn't ours
17:41* April fives dustin a high five!
17:41dustinApril: lol, well, that explains why we've seen a 10x increase in traffic over the last week
17:41Aprildustin: <_<
17:41dustinApril: do these &quot;researchers&quot; have any idea that 90% of those sites are APIs and not &quot;websites&quot;?
17:41dustinscanning for a Joomla vuln .. probably not a good use of time
17:41dustinor XSS
17:42Aprilso
17:42Aprilthere are classes and groups in a lot of poorer countries
17:42Aprilwhere they teach this sort of bug bounty finding thing
17:42Aprilit&#39;s where we get a lot of very poor quality and/or wrong bug reports
17:42Aprilso my guess is that the answer is &quot;no&quot;
17:42dustinyeah
17:42dustinanyway, those scans don&#39;t hurt, just curious :)
17:43dustinso here&#39;s a categorized list of our sites I prepared earlier: https://public.etherpad-mozilla.org/p/b7RATyf84A
17:43Aprilyeah I get a lot of the same traffic on the observatory
17:43Aprildustin: awesome, thank you so much!
17:43dustin:)
17:43Aprilnot sure why loop is under taskcluster
17:43AprilI feel like somebody told me at some point that what it did was taskcluster related
17:43Aprilalternatively, I could be losing my sanity
17:43Aprilit&#39;s really hard to say at this point
17:44dustinit becomes a difficult distinction after a while at mozilla, yeah
17:44dustinanyway, I&#39;d say just the three &quot;sites&quot; are the ones most people should be paying attention to
17:45dustinIMHO anyone with the chops to do anything interesting with teh APIs will figure out what they&#39;re called (hint: they&#39;re all documented on docs.taskclsuter.net, and accessed from tools.taskcluster.net)
17:45Aprildustin: so it&#39;s your call, we have a lot of APIs on the list, simply because there are _some_ researchers that look for IDOR type issues in APIs
17:45dustinthe S3 buckets.. I&#39;m not sure
17:45dustinthey&#39;re all public
17:45AprilIDORs, information disclosures, SQL injections, etc.
17:45Aprilbut you&#39;re right, simply being on the list will get you a bunch of dumb scans
17:46Aprilso we can trim your list to just the three sites
17:46Aprilor leave it as is, adding these new API sites
17:46Aprilif it&#39;s no harm, I would be inclined to leave them on
17:46Aprilif the traffic is problematic, we can take it off
17:47Aprilit&#39;s totally your call
17:47dustinwell the list I just sent is largely a duplicate of what you&#39;ve got (except loop)
17:47dustinand the &quot;old&quot; stuff is deleted
17:47Aprilgot it
17:47dustinthe load isn&#39;t a big issue
17:47dustinso we can leave &#39;em on
17:47Aprilsounds good to me!
17:47Aprildustin: thanks for your help!
17:48dustin:)
17:53pmooredustin: just saw your highlight
17:54pmoorei recently highlighted on &quot;win&quot; and &quot;windows&quot; and now i&#39;m kinda flooded with highlights :D
17:54* dustin open to ideas
17:54dustinhaha, too much noise
17:54pmooreerm, so i guess the task wants to open something by just calling it with &quot;start&quot; (or whatever the command is)
17:55dustinwell, I don&#39;t think it *wants* to
17:55dustinbut something&#39;s doing so, I guess?
17:55pmooreyeah
17:55pmoorei guess we&#39;ll need to work out what is trying to launch it, and what the &quot;it&quot; is
17:55dustinyeah
17:56dustinhm, I said &quot;oo, I have an idae&quot; 42 minutes ago
17:56dustinI no longer have that idae
17:56pmoorehmmm &quot;Code Writer&quot; is suggested - so maybe it is program code that is meant to be executed, but isn&#39;t set as executable, so it tries to open it e.g. in an editor?
17:57dustin&quot;always use this app to open .txt files&quot; suggests it was a .txt file
17:57dustinright?
17:57pmooreah, well spotted
17:57pmoorei guess so
17:58pmooreso why would a test try to open a .txt file i wonder - maybe it tries to open it in firefox?
17:59dustinit&#39;s not even a test -- it hasn&#39;t started running the test suite yet
17:59dustinit&#39;s the marionette setup
18:00pmooreah, maybe whimboo might know what .txt file this is?
18:01dustinI was hoping so, but it seems not
18:01pmooreor maja_zf: do you know why marionette test setup might be opening a .txt file? see https://public-artifacts.taskcluster.net/ETEZDk_IRtOZVPa5rzrz9Q/0/public/test_info/mozilla-test-fail-screenshot_cijnpb.png
18:01pmooredustin: do you have an interactive session there?
18:01pmooredustin: could you just choose notepad, and then see what the .txt file is?
18:02dustinI don&#39;t
18:02dustinthis is just a screenshot from the failing task
18:02maja_zfhmmm, off the top of my head maybe it has something to do with file:// url tests?
18:02pmooremaja_zf: ah, good thinking
18:02pmooredustin: maybe run interactively?
18:03dustinI don&#39;t know how to do that :)
18:03pmooredustin: i will share with you my secret secrets of the most secret nature
18:04dustinoh dear
18:05pmoorewhen i was born, the doctors noticed something strange .... something quite .... abnormal .... and well, after some further investigation, they ..... oh actually it probably isn&#39;t relevant
18:05dustin&quot;well, first there were the dinosaurs..&quot;
18:07maja_zfoh wait, that&#39;s during setup for reftests. no idea why a txt file would be opened by marionette then.
18:19dustinI&#39;m trying to get a loaner..
18:19dustinwe&#39;ll see if I can tell
18:42garndthrm, I thought the error happened while trying to complete a reftest test, not during setup. I think I read it wrong
18:44dustinyeah, it&#39;s during setup
18:44dustinjust so I&#39;m clear:
18:44dustinit&#39;s during setup
18:44dustinlooks like it might be focus-related?
18:44dustinwe&#39;ll see
18:44garndtoh good, you said it two times more, so now I&quot;ll remember
18:47dustinhehe
18:47dustinI kinda love these investigations
18:48dustinbecause every time i learn something new and crazy about firefox
18:48dustinlike, on startup on a new system on windows, firefox opens an offscreen window and plays a built-in m4 in it to verify graphics sanity
18:50dustinI think the problem is here: https://dxr.mozilla.org/mozilla-central/source/testing/marionette/components/marionette.js?q=testing%2Fmarionette%2Fcomponents%2Fmarionette.js&redirect_type=direct#204
18:51garndtyea there was something aout that in the logs right before the timeout
18:54dustinoh, that was the idae -- add some logging to that loop :)
18:54dustinderp
18:54dustinit&#39;s hard being dumb
19:49pmooredustin: just had a win10 failure on generic-worker CI - https://public-artifacts.taskcluster.net/PyXfoqf-RmCypU7i0y_kYw/0/public/logs/live_backing.log
19:49pmoorehttps://irccloud.mozilla.com/pastebin/QjU5ENjI/
19:49pmoorewonder if there is a firewall problem
19:51pmooredustin: just thought i&#39;d let you know - otherwise i can look into it tomorrow
20:35* bstack is restarting most services to pick up new sentry keys
20:56jonasfjbstack: you can just leave them... they all restart tomorrow... and most have keys that timeout in an hour or so... they don&#39;t fetch keys until they need them anyways
20:56bstackthe only thing is that if something goes wrong in the next 24 hours they won&#39;t report to sentry
20:57bstackunless they get restarted, afaict
20:57jonasfjyes, they will... unless they already have credentials...
20:57bstackwhich most of them will have!
20:57bstacknot a big deal, they&#39;re all restarted now anyway
20:57jonasfjhow often do they send stuff?
20:57bstackthe ones I cared about
20:57jonasfjsure..
20:57jonasfjbstack: randomly is there rate limits on the internal sentry instance?
20:58bstackyep
20:58bstackand we&#39;re waaay under the rates that other services they have emit
20:58bstackwe&#39;re small fry
20:58jonasfjokay
21:01garndtyea, I think our load for most things are under cloudops limits for stuff they run
21:01garndtI know that to be the case when talking to them about other services they provide (like log aggregation)
21:11ochameauis there some docs about how &quot;dependencies&quot; attribute work? algolia search isn&#39;t very helpful.
21:12ochameauI was wondering, for example, if a task would run if its dependency fails?
21:14jonasfjochameau: look at task.dependencies and task.requires in schema explorer in API docs
21:14jonasfjochameau: https://docs.taskcluster.net/reference/platform/taskcluster-queue/references/api#createTask
21:16jonasfjochameau: by default tasks don&#39;t run if a dependency fails, setting task.requires = &#39;all-resolved&#39; instead of default (&#39;all-completed&#39;), then it will run even if dependencies fail..
21:17ochameaujonasfj: thanks! that&#39;s handy!
21:19ochameaunote that when searching for &quot;dependencies&quot;, this doesn&#39;t appear.
21:20jonasfjbstack: ^ your search-tech needs some help...
21:20bstackI think that&#39;s related to the anchors not really working either
21:21bstacksince we render that stuff with js client side
21:21jonasfjyeah
21:21* bstack pokes at docs for a few minutes
21:23jonasfjbstack: I think it&#39;s something we consider in some future... I don&#39;t see a good way to solve it.. we probably have to render schemas serverside to do this nicely
21:30garndtbstack..ummm....taskcluster-auth email from heroku
21:30garndtthat&#39;s new
21:31garndtI didnt&#39; know we had alerts setup there
21:32bstackhuh weird
21:32bstackI didn&#39;t know either
21:32bstackbut I didn&#39;t notice anything awry or in logs
21:33bstackgarndt: do you notice anything else wrong?
21:34bstackI don&#39;t even see the errors they&#39;re talking about in their monitoring thing yet
21:35bstackoh wait
21:35bstackthere they are
21:36bstackI see what&#39;s up
21:36bstackfixing
21:36bstacknothing other than sentry affected
21:37garndtah ok
21:37garndtthanks incident commander supremo!
21:37bstackerrr maybe not
21:37bstacklet&#39;s see here
21:37bstackI don&#39;t think anything has gone terribly wrong though
21:39bstackin either case, fixed now
21:40garndtthanks sir
21:55garndtEli: now that priority is going to become a Real Thing (tm)...I wonder if we should display it in the task inspector
21:57Eligarndt: like the priority of a task?
21:57garndtyes
21:58garndtah, I found where we can put it in, I probably can just push up a PR later
21:59Eli++
22:00garndt!t-rex adding priority specific routes to individual hg repos. shouldn&#39;t break anythign since it&#39;s on a per repo basis and &quot;lowest&quot; is the default right now anyways
22:47arrwcosta: the new yosemite machines are spinning up and all hitting the puppet error with the virtualenv package
22:47arrwcosta: did you have a patch for that?
22:48arrwe&#39;re going to have 78 machines failing to puppetize
22:48arrin addition to the ones you already have
23:34wcostaarr: can you send me logs? I wasn&#39;t aware of that
23:35arrwcosta: I opened a bug and cced you
17 May 2017
No messages
   
Last message: 67 days and 21 hours ago