w3 :: #testing

16 Mar 2017
09:00jgrahamMs2ger: You win the award for most implausible breakage of m-c
09:00Ms2gerOh dear
09:00jgrahamAdding Python 3 support to wptserve broke reftests
09:00jgrahamReftests don't use wptserve
09:01Ms2gerThey don't?
09:01jgrahamNot afaik
09:01jgrahamI mean normal dbaron reftests
09:03jgrahamBut they do use marionette to install an addon. And for some reason they import it using the marionette test harness rather than directly from the marionette client. And the marionette harness does use wptserve. And isn't configured to put six in the path on automation, so bustage.
09:04jgrahamSo happily at least part of this makes some sense
09:06Ms2gerFuck python packaging
09:09jgrahamWell quite
09:27Ms2gerI wonder what's up with the highlighting in https://github.com/gsnedders/csswg-test/blob/09674c9c828263779fd60401587b129c792265cf/css-cascade-3/reference/ref-filled-green-100px-square.xht
09:31MikeSmithMs2ger: maybe because of the CDATA thing
09:33MikeSmiththe > right after the ]] of the CDATA is red(dish)
10:28zcorpanMikeSmith: maybe that's something that would be fixed by https://github.com/textmate/html.tmbundle/pull/49
11:00MikeSmithzcorpan: oh man, a Browsers are of course only a small subset of the software that parse HTML
11:00MikeSmithfun thread
11:01Ms2gerThat sounds like my cue to tune out
11:21jgrahamI checked and browsers are an infintesimal fraction of all the lines of code in the universe, so to all intents and purposes they don't exist
11:35annevkjgraham: same goes for kernels
11:36annevkWho needs kernels anyway
11:39zcorpan(Also amusing that he puts more trust in his own experience of "I wrote a HTML parser many many years ago" than what is in the html standard and shipping in browsers for years)
14:07jgrahamgsnedders: Are you going to look at this font promise thing?
15:43gsneddersjgraham: what's there to do about it?
15:45jgrahamgsnedders: Well can you add the promise to the reftest wait condition in wptrunner, or does it only resolve if there are webfonts on the page?
15:47gsneddersjgraham: idk.
15:47gsneddersjgraham: it doesn't solve the harder background-image problems, though
15:50jgrahamI don't really know how to solve those if there's no generic way to tell that they loaded
15:50jgrahamApart from making each test solve it for itself somehow
15:53gsneddersthe web API doesn't expose any way to tell, sadly
15:53gsneddersand we have plenty of tests that rely on it
15:53gsneddershow does Mozilla's old reftest.list stuff deal with that?
16:01jgrahamNo idea
16:01jgrahamIt probably doesn't
16:05smcgruerWhats the problem with background-image?
16:06gsnedderssmcgruer: so the question is "when do you take screenshots for reftests?"
16:06gsnedderssmcgruer: the problem is "wait for the load event" in most browsers doesn't necessarily mean you have background-images loaded (as they don't block the load event)
16:06smcgruerAh, I see
16:07gsneddersthe grammar of that sentence was diabolical.
16:09smcgruerGoing for lunch now, but when I get back I'll try to see if Chromium does anything clever for background-image tests
16:17gsneddersjgraham: I&#39;m going to require all tests in css/ to have <link rel=help href=spec>, and try and drop that ASAP post-merge
16:17gsneddersrather than faffing about with this before the merge
16:17jgrahamgsnedders: Sure
16:21jugglinmikejgraham: Where is the best place to go for questions regarding spec interpretation?
16:22gsneddersjugglinmike: on what spec?
16:22jugglinmikewhat I have in mind is pretty basic--how the events that inherit from Event are initialized
16:22jugglinmikeI&#39;m sure plenty of people here could answer it, but I imagine there&#39;s a more appropriate place to drop questions like that
16:23jgrahamjugglinmike: #whatwg here or #content on irc.mozilla.org (mostly depending on how much bz you need in your life)
16:23jgraham(hint: more is better ;)
16:23jugglinmikeI&#39;ve been idling in #content for a few days, now. What is it, exactly?
16:23gsnedders(I feel far too wilhelm today, working in a suit)
16:24jgrahamgsnedders: Who died?
16:24jugglinmikedangerous joke, there
16:24jugglinmikeor, risky
16:24jgrahamjugglinmike: &quot;Content&quot; in mozilla parlance is pretty much &quot;anything that&#39;s not chrome (i.e. UI)&quot;
16:24gsneddersjgraham: neighbour
16:25jgrahamBut in practice it means anything dom related
16:25jugglinmikegot it. Thanks
16:25gsneddersalso, ugh, really need a way to distinguish in the lint reftest nodes from reftest tests
16:25jgrahamgsnedders: You know how I keep saying the lint should read the manifest
16:26gsneddersjgraham: next month, damnit
16:31gsneddersjgraham: so I think my plan for April is a) cleanup stuff post-merge; b) make ./manifest and ./lint quicker
16:33jgrahamgsnedders: sounds god
16:34gsneddersjgraham: lmk if there&#39;s anything else that you think should be a priority, I guess
16:34gsneddersjgraham: I have a meeting on Tuesday and probably ought to have some plan as to what I&#39;m doing after the merge
16:34jgrahamMy plan for april is to make this faster in m-c fwiw
16:35jgrahamwell reftests in wpt really
16:36jgrahamBut specifically with the goal of making CSS not take 6 hours per configuration, or whatever it currently is
16:37gsneddersthat&#39;s not so great
16:37gsneddersI presume that&#39;s based on your recent run?
16:38jgrahamI didn&#39;t actually do the sum
16:39jgrahamAnd it will be faster once the tests don&#39;t time out
16:39jgrahams/timeout/give an unexpected result/
16:39gsneddersoh, that definitely doens&#39;t help
16:40jgrahamI&#39;ll update the expectation data and give it another go when I have a free moment
16:42gsneddersoh, also we really need to sort out SET TIMEOUT so it doesn&#39;t trigger on files we know are reftests
16:43jgrahamSomething something, lint should use the manifest
16:44gsneddersdo we want to make it actually possible to run lint against staging?
16:44gsneddersIMO it&#39;d be useful, but depends how much complexity it adds
16:45gsneddersthe git index
16:45gsneddersthe git cache, whatever term we want to use for it
16:46jgrahamAh, yeah
16:46jgrahamWell you just removed support for using the committed git files, which is pretty close to what you would need for running against the staging area
16:47gsnedderswell yeah, except it didn&#39;t actually do that :)
16:47jgrahamSure, it was broken :p
16:48jgrahamAnyway I&#39;&#39;m obviously not opposed to it
16:52gsneddersergh, I should write tests
16:52gsnedderswhat loony decided we ought to require tests for our test infra
16:52gsneddersoh wait
17:28gsneddersjgraham: ^^
17:35smcgruergsnedders: I&#39;m no expert, but my first glance at background-image tests in Chromium makes it seem like they just assume the img will be loaded
17:39gsnedderssmcgruer: yeah, this matches my understanding
17:39gsnedderseveryone essentially uses non-public APIs to detect some notion of &quot;page fully loaded&quot;
17:42smcgruerYep :(. I don&#39;t know if there&#39;s a better approach than just creating a wpt &#39;api&#39; for this and requiring browsers to support it. Unfortunately means that the tests will only run in our test-runner mode, but that&#39;s already true today (our reftest-wait support is just a wrapper around the internal testrunner)
17:43gsneddersreftests are already awkward, but we really should make sure it&#39;s possible to run them with WebDriver
17:44gsneddersI mean ideally we also make it possible to write an extension that runs a single reftest
17:44smcgruerHow does WebDriver do the reftest-wait detection today?
17:45gsnedderssmcgruer: either through JS and a mutation observer or through polling the DOM
17:46smcgruerEh, thats how we do it in the layout test infra (the former, that is. DOM polling? Ew.)
17:46smcgruerI must admit I would have preferred the webkit layout test style javascript API, but thankfully MutationObserver is a thing these days so a class change is manageablke
17:48gsnedderswe need to also handle web font loading, which is what prompted this
17:48gsneddersand really also need to handle background-image
17:52jgrahamclass-change is fine in the wptrunner code
17:53jgrahamBut yeah things that you can&#39;t detect through a content API are hard
17:53jgrahamOne point of view is that it&#39;s the tests&#39; responsibiity to deal with this and use reftest-wait
17:53jgrahamI think that&#39;s a reasonable position for background image
17:53jgrahamLess so for fonts
17:54gsneddersthat means someone has to sit down and modify thousands of tests
17:54gsneddersgiven all the tests have been written under the current browser&#39;s impls which *do* block the screenshot on background-image, afaict
17:54gsneddersreftest.list tests, Blink and WebKit&#39;s reftests they all block the screenshot on background-image
17:55jgrahamHow sure are you about that?
17:55gsneddersproposing to go against that when there&#39;s no way for the test to even check if the background image is loaded seems not that useful
17:55gsneddersjgraham: certainly all the tests I can find rely on that being true
17:55gsneddersjgraham: and if they were literally all intermittent I&#39;d expect something to have been done by this point
17:56jgrahamI guess I should read the reftest harness. If it can do it I can make it work for gecko
17:56jgrahamBut not for other browsers
17:56smcgruerjeffcarp: ^ any idea who might know if blink layouttest infra already blocks on background image load?
17:56smcgruerOr if its just coincidental that our tests work...
17:56smcgruer*blocks the screenshot
18:00gsneddersjgraham: and I think Gecko does block load on background images?
18:02jgrahamhttps://dxr.mozilla.org/mozilla-central/source/layout/tools/reftest/reftest-content.js#448 seems to be what the reftest harness waits on
18:02jgrahamgsnedders: Not sure
18:02gsneddersjgraham: https://bugzilla.mozilla.org/show_bug.cgi?id=386620
18:03jgrahamgsnedders: OK, so maybe this works for gecko anyway
18:05jgraham(and in the worst case I can move logic into marionette to make it work using an nsIWebProgressListener or whatever)
18:07gsneddersfire_onload_after_image_background_loads seems to be gone now
18:07gsneddersbut I&#39;m pretty sure the behaviour is still there, somehow
18:10gsneddersjgraham: i mean it also contains logic to make sure the first paint after everything has happened which we can&#39;t do
18:16jgrahamgsnedders: Define &quot;we&quot;
18:16jgrahamI can, and I can add it to marionette and expose it through webdriver. Or write in the webdriver spec that you have to wait for a paint, or something
18:18jgrahamAnyway testing suggests that both Gecko and Blink wait for a background image for an inline stylesheet. But <link> is probably different
18:31annevkjgraham: do you see pings in w3c/web-platform-tests?
18:32annevkUnrelated: I&#39;ve two outstanding infra PRs that are super trivial https://github.com/w3c/web-platform-tests/pulls?q=is%3Apr+is%3Aopen+label%3Ainfra
18:33annevkjgraham: ping was in https://github.com/w3c/web-platform-tests/pull/5156 gotta go
18:35jgrahamannevk: In theory, although if you flag me for review then more probably (using the GH mechanism)
18:53jgrahamjugglinmike: What&#39;s the status of https://github.com/w3c/web-platform-tests/pull/4821 ?
18:56jgrahamHmm, I may have a code coverage report of gecko running wpt
18:56jgrahamIt is red
18:56jugglinmikejgraham: I thought you intended to push another commit based on your comment here https://github.com/w3c/web-platform-tests/pull/4821/files#r103947566
18:57jgrahamjugglinmike: Right. Is that all or are you waiting on something else?
19:00gsneddersjgraham: I mean, &quot;we&quot; being people trying to maintain wptrunner mostly using browser agnostic stuff
19:01gsneddersanyhow, /me -> home
19:08jgrahamgsnedders: Like I say we can perhaps specify that webdriver has to do the right thing here
19:22jgrahamjugglinmike: ^
19:29jgrahamgsnedders, Ms2ger: How would you feel about putting six.py directly into wptserve?
19:29jgrahamIt turns out that the existing situation isn&#39;t ideal for gecko for reasons
21:35jgrahambobholt, MikeSmith: ANy idea what&#39;s up with the mirroring of https://github.com/w3c/web-platform-tests/pull/5163 ? (it seems to not be mirrored)
21:45gsneddersjgraham: well we already vendor everything in wpt, so ehhh. I&#39;d rather we put it in wptserve._vendor.six rather than at the top level, but am okay
21:46jgrahamgsnedders: YEah, so the issue is that we have things that are using wptserve but are not wpt
21:47jgrahamAnd if you try to install six from the tree using pip when there&#39;s already a six installed, with the particular set of flags our ci harness uses, you get infinite recursion(!)
21:47jgrahamAnd for some reason I got objections to just installing it from a pypi mirror
22:50MikeSmithjgraham: dunno whats up the mirroring of that PR
22:50* MikeSmith checks the logs
22:51MikeSmith/dev/vdb1 29G 26G 654M 98% /u/www.w3c-test.org/web-platform-tests/submissions
22:52MikeSmithnot 100% at least
22:53MikeSmithhmm yeah but logs say No space left on device\nfatal: The remote end hung up unexpectedly
22:53MikeSmithtime to delete some stuff
23:02MikeSmith/dev/vdb1 29G 16G 11G 60% /u/www.w3c-test.org/web-platform-tests/submissions
23:06MikeSmithgrr for webhooks github really needs a redeliver all failed webhooks, in chronological order, from the last N days option
23:07MikeSmithor I need to train a monkey to push buttons for me
23:10MikeSmith...and now the webhook is just timing out
23:10MikeSmithjoy joy
23:11MikeSmithok finally
23:11MikeSmithhttps://w3c-test.org/submissions/5163/ now mirrored
17 Mar 2017
No messages
Last message: 9 days and 22 hours ago