mozilla :: #taskcluster

12 Aug 2017
00:01glandiumso maybe I should get my mozillians profile in the tc group...
00:48glandium\o/ https://queue.taskcluster.net/v1/task/LXjsn_ScRTWHJ6tlcwnAiQ/runs/0/artifacts/public/git-cinnabar-build.tar.zstd
01:03jonasfjnice..
01:04jonasfjglandium: found out how to send you an invite...
01:05jonasfjwe generally just add anyone who asks.. and seems reasonably familiar... there shouldn't be any super sensitive things in there... only mildly sensitive :) ie. stuff useful for running tests etc..
01:06glandiumUnsupported image file format. Expected tarball with extension: .tar.zst, .tar.lz4 or .tar
01:06glandiumzst vs zstd
01:06glandiumin 2017, we still limit extensions to 3 characters
01:07jonasfjlol, my bad...
01:07glandiumok, now I need to figure out how to create a task from a task
01:07jonasfjthe ext for zstd is .zst
01:08jonasfjin docker-worker -> https://docs.taskcluster.net/reference/workers/docker-worker/docs/features#feature-taskclusterproxy-
01:08jonasfjtask.payload.payload.taskclusterProxy = true
01:09jonasfjthen curl taskcluster/queue.taskcluster.net/v1/task/<taskId> ...
01:09jonasfjor just taskcluster/queue/v1/task/<taskId>
01:10jonasfjyou can effectively do: queue = new taskcluster.Queue({baseUrl: &#39;http://taskcluster/queue/v1&#39;})
01:10glandiumhow is the taskid supposed to be generated?
01:10jonasfjand queue will be a client client object, that you can use which will have task.scopes as authority (requests signed by proxy on the worker)
01:10jonasfjslugid.nice()
01:11jonasfjslugid == uuidv4 url-safe base64 encoded without the ==
01:11glandiumI&#39;ll figure out what the python code in gecko does :)
01:11jonasfjthe slugid.nice() function just eliminates options that starts with &quot;-&quot; because they aren&#39;t nice is cli tools
01:12jonasfjglandium: it doesn&#39;t use taskcluster client which is probably easier than raw rest
01:12glandiumsure, but I don&#39;t want to start writing js code :-p
01:12jonasfjhttps://github.com/taskcluster/taskcluster-client.py
01:13jonasfjgranted docs aren&#39;t as nice..
01:13jonasfjhelp() macro is useful though...
01:19glandiumis there a doc with the full schema for task definitions?
01:22jonasfjno, everything under task.payload is workerType dependent
01:22glandiumI&#39;m not talking about payload
01:22jonasfjfor docker-worker that means: https://docs.taskcluster.net/reference/workers/docker-worker/docs/payload
01:23jonasfjyeah. everything except task.payload is queue so in:
01:23jonasfjhttps://docs.taskcluster.net/reference/platform/taskcluster-queue/references/api#createTask
01:23jonasfjthe createTask input schema is the best...
01:23jonasfjI think that&#39;s also what we have here: https://docs.taskcluster.net/reference/platform/taskcluster-queue/docs/task-schema
01:24glandiumexactly what I was looking for. thanks
01:24jonasfjif consuming the queue.task() method response schema is better as we return defaults so some fields are always present there..
01:24glandiumI was looking for the right values for requires, for example
01:24jonasfjyeah...
01:24jonasfjthe hard part is what scopes to give the decision task in taskcluster.yml
01:24jonasfjI think you can grant:
01:24jonasfjhttps://irccloud.mozilla.com/pastebin/52NkMS9D/
01:25jonasfjas task.scopes in taskcluster.yml (for the decision task)
01:25jonasfjnote: task.scopes is important for the decision task, because the proxy feature that signs outgoing requests signs then with scopes from task.scopes
01:26glandiumI guess I don&#39;t actually need the create-task scopes
01:26jonasfjwe should talk to bstack about making a role for decision tasks.. so that we don&#39;t have list the scopes..
01:26jonasfjyou do...
01:26jonasfjyour user no...
01:26jonasfjbut your decision task does
01:26glandiumnot for the tasks I&#39;m going to create
01:26jonasfjyou can&#39;t create any tasks without scopes
01:27glandiumthe decision task needs scopes, but the tasks it creates don&#39;t, right?
01:27jonasfjtrue
01:27jonasfjbut when you specify the decision task in .taskcluster.yml that&#39;s where we have to add the scopes
01:28glandiumindeed
01:29glandiummmmm... the .taskcluster.yml file in gecko actually uses json-e
01:30jonasfjyeah, and it&#39;s should be doable in tc-gh if it&#39;s landed yet..
01:31glandiumand it doesn&#39;t have a lot of scopes
01:31jonasfjnope, it has a role-scope
01:31jonasfjscopes on the form: assume:<role> will expand to the role they assume
01:32jonasfjassume:<role> -> [&#39;assume:<role>&#39;, ... <scopes from role definition>]
01:32glandiumoic
01:32glandiumI guess I could just add assume the role for the repo, then?
01:32jonasfjunfortunately there is no role for github repos... well, there is...
01:32jonasfjthe role depends on the context, so:
01:33jonasfjassume:repo:github.com/<org>/<repo>:pull-request
01:33jonasfjassume:repo:github.com/<org>/<repo>:branch:<branch>
01:33jonasfjassume:repo:github.com/<org>/<repo>:release
01:33glandiummmmmmmm
01:33jonasfjand your task will only have one of these..
01:34glandiumwhat gecko does is assume:repo:${repoUrl[8:]}:*
01:34glandiumpresumably assume:repo:github.com/glandium/git-cinnabar:* would work?
01:34jonasfjyeah, maybe we can make something like: assume:repo:github.com/<org>/<repo>:base we should ask bstack
01:34jonasfjno because that assumes all of these roles
01:34jonasfjwhich your push wouldn&#39;t be allowed to do
01:35jonasfjthere is no way to express pass on exactly everything I have an nothing less
01:35glandiumI guess I&#39;ll just pass the list
01:35jonasfjI think tc-gh needs a variable that contains all the scopes... so people can substitute that into the decision task.scopes
01:36jonasfjyeah... and feel free to file a bug for variable or something else.. I&#39;m sure we can make something rather quickly
01:36jonasfjactually, I&#39;ll file a bug now..
01:38jonasfjhttps://bugzilla.mozilla.org/show_bug.cgi?id=1389714
01:38firebotBug 1389714 NEW, nobody@mozilla.org Allow variable with all scopes in expression language
01:39glandiumjonasfj: https://public-artifacts.taskcluster.net/MaY0_-XGTyKYuA0tsANIyg/0/public/logs/live_backing.log huh
01:40glandiumwhat version of zstd is the docker importer expecting?
01:40jonasfj1.0 something
01:40jonasfjI&#39;m sure >= 1.0.0 will do
01:41glandiumI built 1.1.4 as in your script...
01:41glandiumif I download the image and zstdcat | tar -tf it, it&#39;s happy
01:45jonasfjI don&#39;t think it&#39;s zstd
01:47jonasfjI wonder if the image contains a :latest tag... /me taking a look
01:47glandiumdoes it need to?
01:48glandiumhttps://pastebin.mozilla.org/9029525
01:51jonasfjit does: {&quot;git-cinnabar-build&quot;:{&quot;latest&quot;:&quot;c7d2d2d8f140f9b974b01d8e9b97345994ebae19a91851a5a9b5b43d6e31f6a9&quot;}}
01:51jonasfjfrom repositories file inside..
01:52glandiumcf. the paste, it works fine with docker load
01:57glandiumand there&#39;s no useful output from taskcluster :(
01:58jonasfjI&#39;ll file a bug... I sent you a private message with details... I think it&#39;s a format change...
02:12glandiumis there a way to list all the task groups associated with a repo?
02:14glandiumheh? I thought dependencies worked with any task, not only those from the current task group?
02:15gpsassuming all other things are equal, the differences in I/O perf between gecko-t-linux-medium and gecko-t-linux-large are night and day
02:16glandiumfacepalm, I took the wrong taskid
02:25glandiumwell, I should have enough to get seriously started now. Thanks jonasfj
02:25gpsdo you get windows support?
02:26glandiumgps: I&#39;ve seen github repos using it, so eventually, I guess I&#39;ll have that
02:26glandiummost of my CI is on linux anyways, and on travis, I can&#39;t do think efficiently
02:27glandiumnow I&#39;ll be able to have dependencies between tasks \o/
02:30jonasfjglandium: if in a week or two you ping me to become a guinea pig for windows under QEMU I&#39;m sure we can make that happen :)
02:30* jonasfj walking home...
13 Aug 2017
No messages
   
Last message: 10 days and 3 hours ago