mozilla :: #perfherder

15 Mar 2017
18:59ShrutiJwlach: Hi! Are you free for sometime. I had a doubt
19:01wlachShrutiJ: sure, I'm here
19:02ShrutiJwlach: What will the x-axis and y-axis signify when we visualize the distribution?
19:03wlachShrutiJ: depends on the visualization :) what were you looking at?
19:05ShrutiJwlach: I was looking at bug 1164891. The visualization which Avi had proposed, y axis would be meaningless.
19:05firebothttps://bugzil.la/1164891 NEW, nobody@mozilla.org perfherder compare view shows averaged numbers + stddev over runs, we would like to visualize the di
19:05wlachlooking
19:06ShrutiJwlach: Whereas the one which you proposed, y-axis represents the result. So what will the x-axis represent?
19:08wlachShrutiJ: actually in the example I proposed, the y axis represents how many results support a particular value https://bugzilla.mozilla.org/show_bug.cgi?id=1164891#c1
19:08wlachthe x axis represents the range of values
19:09ShrutiJwlach: Ahh. Makes more sense now :)
19:09wlachthese are all just ways of visualizing statistical distributions
19:09ShrutiJwlach: So you want it to be a histogram?
19:09wlachyeah, I think we're basically talking about some variation of a histogram
19:11ShrutiJwlach: But in this comment https://bugzilla.mozilla.org/show_bug.cgi?id=1164891#c2 , Avi points out that we have less values compared to the example which you had put up
19:12wlachShrutiJ: yes, on reflection I am not completely sure if the approach I proposed is the right one
19:12ShrutiJwlach: Histogram doesn't works well if the data is not on a continous range.
19:14ShrutiJwlach: Is there any way by which can retrive this data?
19:14wlachyes, as I mentioned rwood made it so we dump the raw talos results on every job
19:14wlachlet me dig it up
19:15wlachhttps://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&revision=26ded80b0df562e15af98cdd3a15d5adac3ef426&filter-searchStr=talos&selectedJob=84059016
19:15wlachfor that job, navigate to job details, then look at "perfherder-data.json"
19:15wlachthat has the raw replicates for each test
19:15ShrutiJwlach: I found a few api links on git grepping "stddev". Not sure which one should I use.
19:15rwood:)
19:16wlachactually now that I look at it, I think we might have enough data to plot some kind of histogram
19:16wlachI guess it depends on whether you're comparing the test/subtest results or the replicates themselves
19:17wlachwe only have 5 or 6 results for each test/subtest, but many more for the replicates
19:17wlachfor the gsoc project I was thinking you'd be working at visualizing the replicates
19:17wlachif you have time left over I think it might be nice to work on visualizing the test/subtest results in a better way, perhaps using avih's suggestion. but that's a different task
19:18wlachsorry if I have been a bit confusing, haven't thought about this all that recently
19:22ShrutiJwlach: What exactly is replicates?
19:22wlachShrutiJ: it's just a term for the results of a particular performance test
19:23* ShrutiJ needs to read more about data representation in Perfherder :/
19:23ShrutiJokay
19:23wlachso tests produce "replicates". these are then aggregated into "subtest results". the subtest results are aggregated into "suite results"
19:23wlachperfherder currently only lets you visualize suite and subtest results
19:24wlachthe exact details of how the summarization works is described here: https://wiki.mozilla.org/Buildbot/Talos/Tests
19:24wlachwe currently certainly stand to have better docs here
19:25jmaherwlach: it is 30times better than 2 years ago
19:25ShrutiJwlach: And with the gsoc project we plan to extend the visualisation to "replicates"
19:25jmaherbut yeah, it could live with slightly better docs
19:26ShrutiJwlach: Documentation is better than any other open source community :D
19:26wlachjmaher: yup, absolutely agreed. didn't mean to knock the hard work you did :)
19:26wlachI think the test level documentation is good, we could just do with some explanation of the high-level concepts
19:27jmaheryeah
19:29ShrutiJwlach: I think things are more clear now. I'll work on the implementation part again and would probably write another mail to you in a day or two with more details.
19:29ShrutiJThanks for helping :)
16 Mar 2017
No messages
   
Last message: 104 days and 3 hours ago