mozilla :: #treeherder

14 Mar 2017
05:46cwillemorley or Eli (whenever you're online) - I'm working on cleaning up my neutrino commits now that yarn has landed, and now I can see the eslint errors emorley was talking about in the coments on the PR. I think the neutrino linter is configured correctly (I overwrite all of its settings to just use .eslintrc), but I get different linting results when running `yarn start` compoared to when I
05:46cwillrun eslint directly. That is, I see the indentation errors when eslint-loader checks the code, but not when I just run eslint directly on the same code.
05:46cwillDo you have any idea how this could be? I'm sort of at a loss.
08:30jgrahamI'm going to update prod to point at the same commit + my changes. Happy to do another deploy later for whereever we want to move prod to.
09:51herokubotdeployed bb5d4e5 to treeherder-prod
09:53emorleyjgraham: from what I can tell, the "deployment failed" email from Heroku is wrong - their mechinism for monitoring the deployment timed out. Papertrail looks fine
09:59emorleyjgraham: the deploy has finished, are we ok to resume ingestion?
10:00jgrahamemorley: Yep, will turn things on again
10:00jgrahamSorry was briefly afk
10:17emorleyjgraham: I've increased the store_pulse_jobs dyno count from 4 -> 30 to get the backlog (34K tasks) down
10:19emorleynow at 15K
10:24emorley0K, turning count back to 4
10:25emorley0, that is
10:27emorleyincreasing log_parser count from 5 to 20 to get that backlog down, now the jobs have added to it
10:37emorleybacklog gone, lowering to 5
11:24jgrahamemorley: Thanks for handling that
11:33davehuntemorley: log is gzipped, but I'm suspecting that I have an invalid assumption.. the file is a structured log from mozlog, does that even need parsing? how do I get Treeherder to present that appropriately?
11:37davehuntoh... https://github.com/mozilla/treeherder/blob/master/ui/plugins/job_details/main.html#L7-L9
11:37davehuntso the magical 'raw.log'? :)
12:00jgrahamdavehunt: So treeherder can't read raw logs (let's go with "historical reasons") You can upload them to blobber, but the log parser needs something formatted with --log-tbpl -
12:01jgraham(the other reason is that parsing all the json with python is rather slow. The tbpl log simply contains less
12:01davehuntjgraham: hmm.. interesting, I should be able to generate a tbpl log too
12:01jgraham)
12:01davehuntnot sure about blobber, I was just going to upload to an S3 bucket myself
12:03jgrahamOK, those two are equivalent, blobber is just easy to use from mozharness, but i guess you're doing your own thing here
12:03jgrahamAll treeherder wants is a url
12:05davehuntjgraham: thanks, I'll try this out
14:01wlach|afkemorley: jgraham: how long did the jobs table migration take, end-to-end?
14:01emorleylooking
14:03jgrahamemorley: The full process took 80 minutes (plus time for me to notice it was done); I think the migration itself was about 60
14:04jgrahamEr wlach^
14:04jgraham(I presume emorley will have more accurate numbers in a moment)
14:06emorleythe trees were closed 1 hour 29 minutes. `model.0005_autoclassify_status` took 57 mins, `model.0006_jobs_submit_time_repository_index` took ~2 mins
14:06emorley(from papertrail)
14:07jgrahamOh, look my inaccurate umbers were accurate :)
14:07jgraham*numbers
14:07emorleywhat I found interesting was how quick the index addition was
14:08jgrahamI didn't expect that there was another migration was there something other than my commits that hadn't been deployed before?
14:08emorleyyeah
14:09emorleysince it was a model migration after, so had a "depends on 0005..."
14:09emorleyso was excluded by Will when cherrypicking
14:09emorleybut it was absolutely fine to land it now
14:10jgrahamWell I'm just glad it didn't take another hour :)
14:10emorleythe index would have taken much longer than 2 mins to add if there was activity on the table hehe
14:10emorleytrue!
14:10jgrahamhttps://blogs.oracle.com/mysqlinnodb/entry/online_alter_table_in_mysql talks about online index creation
14:19emorleyI'm going to push master to prod
14:19wlach60 minutes isn't too bad
14:20wlachI wish the jobs table was more space efficient, i.e. not so many charfields
14:20wlachI bet half if not more of the rewrite time is just copying that junk
14:20emorleyagreed
14:21emorleyactually the table is much smaller than I expected in terms of size on disk
14:22emorleydata length: 6.5 GB, index length: 15.4GB
14:22emorley25 million rows
14:23emorleysmaller than job_detail's 475 million rows, 64GB data, 58GB indexes
14:24emorley(not surprised at that)
14:40herokubotdeployed c77f8ff to treeherder-prod
15:39emorleyhttp://www.tedunangst.com/flak/post/features-are-faults-redux
16:25davehuntokay, I'm using the tbpl output now and still the log parsing isn't working for me :/
16:25davehunthttps://pastebin.mozilla.org/8982015
16:26davehuntwhat else am I missing?
16:27camdemorley: haha, I thought that article was going to be about how bad "Redux" is... :)
16:27camdI'm liking redux myself, atm... So I was a bit nervous that I was "all wrong..."
16:27rwoodis anyone else having troubles running the treeherder unit tests with the latest master?
16:28rwoodor is it just me.... it's complaning about conftest import mismatcherror
16:28emorleyrwood: have you run a `vagrant provision` recently?
16:29rwoodemorley: no sir haha
16:31rwoodI did a vagrant destroy and vagrant up if that does the same?
16:31rwoodjust did provision and it didn't fix it hmmm
16:33davehuntEli: hey, do you know about Treeherder's log parser, and why it doesn't appear to be working for me? https://pastebin.mozilla.org/8982015
16:34Elidavehunt: unfortunately i dont, sorry
16:34Elinever used it
16:34davehuntEli: no problem, thanks :)
16:35davehuntjgraham: I'm using the TBPL log now, but still no parsing :( https://pastebin.mozilla.org/8982015
16:35emorley(in Treeherder meeting)
16:35davehuntah, right (thanks) :)
16:37jgrahamdavehunt: i think you need more "|"
16:37davehuntjgraham: this is using mozlog..
16:37jgrahamSpecifically I think, without looking that the log parser might expect 3
16:37jgrahamWell 2
16:37jgraham3 sections
16:38davehuntI see three sections
16:38davehuntoh, not for suite start/end
16:38jgrahamThat might be fine
16:38jgrahamI would need to read the code
16:39davehuntmaybe I can take a look at the code
16:39camdcwill: If you're free, feel free to join the treeherder meeting. :)
16:40camdTalking about your PR a bit. :)
17:05jfrenchwlach: emorley: camd: hi guys - I haven't found a bug for it yet; 'Show successful steps' in logviewer doesn't appear to work any more. part of the ongoing transition to the unified logviewer?
17:10wlachjfrench: let me check
17:12wlachjfrench: seems to wfm?
17:12jfrenchwlach: hm
17:13jfrenchcan you paste the link to the job your looking at?
17:15herokubotdeployed 534d29d to treeherder-prod
17:15jfrenchwlach: hm, ok I found one that works
17:15herokubotdeployed de18f1f to treeherder-prototype
17:16herokubotdeployed de18f1f to treeherder-stage
17:19jfrenchwlach: ya, I was just looking at a build job that happened to be in a TC job group
17:19jfrenchall good
17:27wlachjfrench: oh yeah logs for tc jobs are weird
17:45jgrahamdavehunt: Oh wait, all your tests pass
17:45jgrahamThe log parser doesn't do anything with passing tests
17:45davehuntjgraham: hmm.. why does Treeherder say the parsing is incomplete then?
17:49jgrahamdavehunt: Not sure. Grepping the source didn't immediately reveal complete being set to true anywhere
17:49emorleyjgraham: it does
17:50davehuntI was guessing that my log was too 'clean'.. no start/end markers
17:50emorley(since there can still be false positives for error lines, plus TinderboxPrints, plus perfherder lines etc)
17:51jgrahamemorley: OK, "nothing" isn't quite accurate
17:51jgrahamBut you wouldn't expect them to appear in the output
17:52jgrahamdavehunt: Oh yeah the step stuff
17:52jgrahamYOu might need to fake that :/
17:52emorleyyou don't any more
17:52emorleythe log parser will create a dummy step
17:53jgrahamAh, right
17:53* jgraham is reading the code as we speak
17:54jgrahamdavehunt: So you get a step in your output as expected
17:54jgrahamdavehunt: If you change a line from TEST-PASS to TEST-UNEXPECTED-FAIL, is that also in the output?
17:54davehuntI do? I'm not sure what I'm expecting :D
17:55davehuntthis is being generated by a suite and uploaded to S3 from Jenkins, so introducing a failure is a bit of a pain.. but I can give it a go
17:55jgrahamdavehunt: the "steps": [{"name": "Unnamed Step", []}] bit
17:55wlachsorry about all the lodash bustage all, I'm going to go through the upgrade guide and make sure there isn't anything else I missed
17:56jgrahamdavehunt: Right I'm not sure off the top of my head if you can just pass the parser a file, but at least a file:// url might work
17:56emorleyuse responses
17:56emorleyoh wait it's urllib for now
17:57emorleyI should finish that branch
17:57wlachok, double checked that there weren't any other deprecated methods in our source
17:57jgrahamYeah, a file:// url works with urllib iirc
17:57wlachsorry I didn't do this the first time around :(
18:04cwillcamd: sorry, I don't have a computer with a webcam anymore :(
18:05camdcwill: oh, didn't you get to keep that one from mozilla? Shoot, I thought that was part of it... :(
18:05cwillnah, it's on its way back
18:06camdBut no worries at all. You're certainly not obligated to attend our meetings. But you're always welcome. :)
18:06camdcwill: ahh, shoot. I had misunderstood that.
18:06cwillthanks, tbh I be I can rustle another laptop up from somwhere
18:07cwillre: my linting problem, I was thinking I could just eschew neutrino-lint-base for now, since we're still using the .eslintrc and may be switching to a preset later - thoughts?
18:07cwill(not sure if that will solve it, but I'll give it a try)
18:10Elicwill: what is your eslint command?
18:10cwilleslint -c .eslintrc ui/
18:12Eliso it isn't pulling in the neutrino eslint settings?
18:14cwillEli: right here, I thought I was just telling neutrino to use the .eslintrc: https://github.com/mozilla/treeherder/pull/2159/files/802cb5149926309d46026a031e9d8fdb41ff91aa#diff-fdb1e76e90d2a3641d04e1a3b1a6a613R15
18:14cwiller, sorry for the long line
18:14cwillbut it doesn't produce the same result as running eslint by itself (it finds several indentation errors)
18:15Elihmm I think that won't work as is. why not move your eslint config to the preset, then tell eslint to use the neutrino config?
18:15Elihttps://davidwalsh.name/neutrino-linting
18:15cwillthat sounds fine - I saw that page in the docs (about eslintrc.js) - I'll give it a try
18:15cwillthanks!
18:15Elinp, ping me if you need some help
18:15cwillwill do
18:16davidwalshBlog mention! Yesssss!
18:16ElixD
18:16cwill:) davidwalsh I recognized your name from your blog when I joined - I've enjoyed many articles!
18:16davidwalshcwill: Thanks :)
18:16Elidavidwalsh's blog has been so useful to me over the years ^_^
18:17* davidwalsh watches his ego grow
18:17Elilol
18:33herokubotdeployed ddfe69c to treeherder-stage
18:36herokubotdeployed ddfe69c to treeherder-prototype
18:47herokubotdeployed 9668c86 to treeherder-prod
18:47KWiersowlach: thanks :)
18:56wlachKWierso: yw :)
19:08herokubotdeployed 9f1cb4a to treeherder-prototype
19:08herokubotdeployed 9f1cb4a to treeherder-stage
20:10jfrenchwlach: got a sec to bounce a logviewer idea off you?
20:10wlachjfrench: sure, others in this channel may have ideas too
20:14jfrenchwlach: cool. so afaict there are no TC jobs yet which support 'Successful Steps' in logviewer (?) - if that's true I was wondering about disabling that checkbox for TC jobs, until some point later when it does http://pasteboard.co/Jr9pV3B34.jpg
20:15wlachjfrench: I like that idea, hide the option when it doesn't make sense
20:15wlachI don't know when/if we'll get around to fixing step parsing for taskcluster
20:16jfrenchwlach: ok, I have the change done already, I'll put it up
20:20wlachall this churn certainly has not endeared me to lodash :/
20:31herokubotdeployed 6116f12 to treeherder-prototype
20:31herokubotdeployed 6116f12 to treeherder-stage
20:32herokubotdeployed 6939104 to treeherder-prod
20:46camdEli: I think my first PR is ready for review by you: https://github.com/mozilla-rpweb/treeherder-manifest/pull/1
20:47camdI'll create a treeherder bug for this in a sec and assign it to you.
20:52bstackKWierso: looks like when I asked for bug 1333604 to be landed, I missed that the in-tree bits hadn't been landed yet. How hard is it to revert a commit and deploy a new treeherder?
20:53bstacktotally my fault
20:53firebothttps://bugzil.la/1333604 FIXED, bstack@mozilla.com make action tasks that interact with treeherder able to accept a treeherder api url
20:54rwoodwlach: hmm I still can't seem to mark an alert as downstream
20:56rwoodmaybe another instance of _.all somewhere?
20:57wlachrwood: argh, _.where this time
20:57wlachI could have *swore* I fixed and tested all this
20:58bstackwlach: maybe once you've got a sec you could help me figure out how hard it is to revert and deploy a new treeherder ^ ?
20:58bstackthanks for any help :|
21:00KWiersobstack: I can do a revert, but someone will need to cherrypick it to production
21:01bstackok, ty
21:01bstackis someone a person like w.lach?
21:01KWiersolikely
21:01wlachbstack: what do you need me to revert?
21:01KWiersohttps://github.com/mozilla/treeherder/commit/e280bcd618ffc1add861fe1eda2640842a4da880
21:01bstackshows me right for trying to land a 2 month old patch
21:01wlachrwood: cherrypicked a fix
21:02rwoodwlach: thanks! :)
21:02wlachbstack: ok I pushed a revert of that to production
21:02bstackok, ty
21:03bstackthanks very much to both of you
21:03wlachbstack: do you want me to revert that on master? or would you rather just send in a pr with the fix?
21:03bstackwell, eventually I'd like it in production, but the gecko bit has to land first and circulate through all of the branches before we can
21:03bstackso it probably makes sense to take it out of master for now
21:03bstackand I can come back later and get it in
21:07bstackthat looks correct to me
21:07bstackty
21:15herokubotdeployed 667a8d4 to treeherder-prod
21:17Elicamd: cool ty
21:18camdEli: you bet. thanks for taking a look. :) At least yarn kept me very honest on the eslint stuff. :)
21:18Elihehe
21:19camdI think after this the main next thing is a link back to treeherder for each test failure (or manifest, maybe)
21:23herokubotdeployed 86d0029 to treeherder-prototype
21:23herokubotdeployed 86d0029 to treeherder-stage
15 Mar 2017
No messages
   
Last message: 101 days and 5 hours ago