w3 :: #testing

20 Mar 2017
12:03bobholtjgraham: i'm at a bit of an impasse getting web platform tests to run on Sauce Labs for Safari and Edge support
12:03bobholti'm running them on the web-platform.test domain on the Travis box with the hosts tunneled
12:04bobholti can prove to myself that it is loading the html and testrunner scripts, it is parsing those files, and it is running scripts inline in the HTML
12:04bobholthowever, it doesn't seem to run the tests, and i get a blank page
12:05bobholtany suggestions for debug vectors for me?
12:05bobholti'm a bit hamstrung by not having access to the browser consoles on sauce to see any errors it's throwing
13:08bobholtjgraham: i'm also having trouble tracking down the lint error logging change you made on 9 March
13:09bobholtyou referred to a merge message from gitbot and i just realized that logbot drops those in the logs
13:27jgrahambobholt: Is it set up to allow popup windows? The default use of window.open is a problem if you don't set the right pref or similar
13:27jgrahambobholt: Not sure what lint error logging change you mean
13:30bobholtjgraham: ah, i'll look into the popups, thanks
13:30bobholtyou had made a change somewhere to have wpt lint emit errors to be picked up by the prbuildbot
13:31bobholthttp://logs.glob.uno/?c=w3%23testing&s=9+Mar+2017&e=9+Mar+2017 at 14:59 and 17:11
13:32gsneddersOkay, I'm struggling to get Travis's allowed_failures to work
13:33gsneddershttps://github.com/w3c/web-platform-tests/blob/csswg_test_merged/.travis.yml is the file
13:33gsneddershttps://travis-ci.org/w3c/web-platform-tests/builds/212984048 is a build
13:33gsneddersthe apparently required "env:" is creating an extra build, and there is no allowed failures
13:33gsneddersoh wait
13:33gsneddersit's *allow* failure
13:36gsneddersno, still not getting picked up as an allowed failure :\
13:44jgrahambobholt: Hmm, I lost that too. I'll check my old laptop
13:46bobholti looked in tools/lint, but didn't see anything like that - wasn't sure where else it would have been
13:49jgrahambobholt: Not on my laptop unless it's in a stash somewhere :)
13:49bobholtno worries - i'll made something happen
13:49jgrahambobholt: Need to go afk for a moment, but I'll have another look. I remember writing the code :)
13:50bobholtyeah, i thought you were pointing at a merge event, so i figure it has to be somewhere
14:07mkwstjgraham, et al: Is the `?pipe=header(name, value)` syntax supposed to work with `.sub`?
14:08mkwstAh, I see. I need to add `sub` as one of the pipes explicitly. Nevermind!
14:14jgrahammkwst: That could be a bug
14:14jgrahamBut yeah it should work if you add it explicitly
15:00rbyersgsnedders: Is there a FAQ somewhere saying how different spec levels are supposed to be handled in WPT? I believe I know the answer (test only tip, use test snapshots at revision if necessary) but I can't find that written down anywhere in the README or docs...
15:01jgrahamrbyers: That one might be an "if we make this an unwritten rule no one can complain about it" (or we might just have forgotten to write it down ;)
15:05rbyersjgraham: Hah, yeah right. Perhaps that's also why I can't find our "blink prefers links to living-standard documents over versioned snapshots" written down anywhere (other than mail threads) either ;-)
15:05gsneddersrbyers: what jgraham said
15:06rbyersgsnedders: Which one? Unwritten rule, or just forgotten?
15:06gsneddersrbyers: part of this is down to some of the leveling being W3C v. WHATWG
15:06gsneddersrbyers: unwritten
15:06rbyersYep, understood
15:06gsneddersI mean, if you want to fight the fights that will inevitably happen
15:07rbyersI just wrote down the blink rule for living specs: https://docs.google.com/document/d/1vlTlsQKThwaX0-lj_iZbVTzyqY7LioqERU8DK3u3XjI/edit?pli=1#bookmark=id.w8j30a6lypz0
15:07gsneddersWhat's going to happen with CSS going forward is anyone's guess, and there's far more in the way of multiple levels of specs in active development compared with elsewhere
15:07gsneddersEsp. combined with the modularisation of 2.1, which often splits things up into more logical modules than 2.1 had sections
15:08* gsnedders away for 1.5 hours or so
15:09jgrahambobholt: Nope, I seem to have lost that code
15:09bobholtjgraham: no problem
15:09jgrahamWait wait wait
15:09jgrahamI must have pushed it because I got travis errors
15:11annevkrbyers: https://github.com/servo/servo/wiki/Relevant-spec-links is pretty good prior art
15:11jgrahambobholt: There you go, I just gave it a stupid branch title
15:12rbyersannevk: Thanks. I know we've said exactly those things on many blink-dev threads, but maybe never wrote it down anywhere official.
15:12bobholtjgraham: ha, thanks. sorry to make you dig up code weeks after the fact
15:13jgrahambobholt: Entirely my fault for not creating a PR
15:13annevkrbyers: in particular using platform.html5.org and encouraging folks to keep it up-to-date in that style seems like a good idea
15:13bobholti thought it had merged, so didn't prioritize keeping a link to it around
15:14bobholtjgraham: on my other (sauce) issue, popup blockers don't seem to be the problem. i've found a few more things to dig into, but i'll come back and haunt you if i run out of paths to go down
15:16jgrahambobholt: Sure. If you need me to investigate more deeply, put up a branch that I can try or something
16:07* gsnedders is back
16:45gsneddersjgraham: how expensive is clearing storage?
16:47jgrahamgsnedders: No idea? Probably not very but I don't want to do it for every test (in the gecko implementation it has to execute some code using marionette which is at least some overhead)
17:03gsneddersno ms2ger today? are we expecting him to be away?
17:18gsneddersI need https://github.com/w3c/csswg-test/pull/1239 reviewed, if you want to actually look at csswg-test for the first time in ages :P
17:18gsneddersbut it doesn't really matter when this week it happens
17:18gsneddersjust that it happens
17:21jgrahamgsnedders: r+
17:22jugglinmikeannevk jgraham So far, the best I have for this "204 with content" test is to *also* specify "Transfer-Encoding: chunked" and then provide a body that is malformed
17:24jugglinmikeThis request will fail in browsers that honor "Content-Length", and it will succeed in browsers that honor the 204 status code
17:25gsneddersdo browsers not handle Transfer-encoding: chunked differently wrt Content-length?
17:26jugglinmikeWell, I guess I should just omit Content-Length if I'm doing this
17:26jugglinmikewhich makes it a different test
17:26jugglinmikealong the same lines
17:28jugglinmikeBut it's still not precise enough. it's possible that the browser rejects because of the conflict (204 and Encoding: chunked). Because of that, we can't tell whether it read the "body" data from the connection
17:32gsneddersdo we have any documentation on testharnessreport.js anywhere?
17:38jugglinmikeannevk: jgraham This might sound like a cop out (or a challenge), but I'm starting to doubt that there is a deterministic way to check the contents of the connection buffer.
17:39jgrahamgsnedders: I see it's mentioned, but not really an explaination of why it exists
17:41annevkjugglinmike: so the main thing I'm wondering about is whether someone from Chrome's networking team can take a look at this
17:41annevkjugglinmike: since them being different from Firefox is what the issue is here
17:46gsneddersjgraham: or how to use it
17:47jgrahamgsnedders: Well that's sort of up to you, right?
17:48gsneddersjgraham: bah
17:48jugglinmikeannevk: Chrome-Firefox is easy to test. I will open a pull request in a moment
17:49annevkjugglinmike: you mean the difference we're observing?
17:49jugglinmikeWe're observing two differences
17:49annevkAside: what's a good place to ask whether making everything .any.js is actually a good idea as it will lead to a lot more tests?
17:50annevkI was thinking dev-platform, but maybe there's another place where I can also reach implementers?
17:50lyzadangerIm running into a brain-melting issue with testharness.js + the web-based test runner + safari, but Im thinking filing an issue in testharness.js might not make sense, as the affected tests are completing as expected if I run them directly. The problem is only happening when I run test(s) from the web runner. Any thoughts?
17:50jgrahamlyzadanger: The web runner might be horribly broken
17:50jugglinmikebut this other part differs between Chrome/Firefox (which both leave the connection buffer alone for 204 responses that specify Content-Length) and Safari (which attempts to read the specified number of bytes from the response)
17:50jgrahamI mean that's not so much a thought as a fact
17:50lyzadangerjgraham: Id like for that to be the answer here. Ive lostmore time than I wish to admit on this.
17:50annevklyzadanger: yeah, what jgraham said, I recommend running tests directly only
17:50lyzadangerOK. I should have asked hours ago :D
17:51annevkI never look at the test runner
17:51lyzadangerShould I open an issue _anywhere?_
17:51annevkMaybe we should stop recommending folks to go to the test runner?
17:51annevkIf we do that anywhere that is
17:51jgrahamI have never recommended that
17:51jgrahamIt's amazingly hard to get people *not* to use it
17:51jgrahamAnd because people do use it, it's hard to delete
17:52annevklyzadanger: thanks, I'll make a PR to delete that
17:52jgrahamYeah, happy to unreccommend that
17:52annevk(or move it to the end and recommend against it)
17:53jgrahamI'm hoping that once csswg-test merges we can make wptrunner much easier to use like |run --browser chrome path/to/test.html|
17:53lyzadangerjgraham: Yeah, TBH Ive beenprocrastinating on that.
17:53jgrahamWell without the --browser part perhaps
17:53lyzadangerGuess it needs to bubble up higher on my priority list
17:53jgrahamSince it's not really optional
17:55annevkjgraham: https://github.com/w3c/web-platform-tests/pull/5176
17:57annevkjgraham: https://github.com/w3c/web-platform-tests#publication was changed too right? We no longer require LGTM / mirror. Do you want another PR for that?
17:57gsnedderswe probably want some decent in-browser way of running tests, but that probably should be an extension
17:57* annevk reviews the entire README
17:57gsneddersgiven then it can actually do stuff like reftests, etc.
18:00jgrahamI"m pretty meh on that
18:00jgrahamExtensions are hard, idk what the advantage over an external runner is
18:01jgraham(hard because e.g. they have to be signed)
18:01* annevk is also still curious about his Aside earlier on...
18:02jgrahamannevk: The publication thing, or something else?
18:02gsneddersjgraham: because it's much easier when actively developing a reftest because running one page that's already open is quick v. having to start web server, start browser again, etc.
18:02jgrahamBut yeah the publication thing changed so that if there aren't any changes to .py files it's automatically mirrored
18:02annevkjgraham: the aside was where to best ask about doubling/tripling/quadrupling large number of tests and whether that's desired or not
18:03jgrahamannevk: Oh, that one yeah idk. I mean *I* don't object but I don't pay our aws bill :)
18:04annevkjgraham: I guess I'll email dev-platform for starters
18:04jgrahamI wonder how much it would ruin if .any ran both tests from a single html document
18:04annevkWell, now it's both, but there's requests for shared and service workers too
18:05annevkBut yeah, that might be better
18:05jgrahamMight be harder to do with service/shared worker if we get those
18:10gsneddersjgraham: but yeah, I for one would like to see an extension for when I'm writing tests
18:11jgrahamgsnedders: OK. I don't think it's a priority for me/us, so I can't imagine it will go on my Very Long List of Things To Do
18:12jgrahamBut of course if someone else writes it I"m happy to help out
18:16gsneddersjgraham: I want to get someone else to do it, but I might do it myself
18:44jugglinmikeuuhg annevk It's looking like this is all the same issue, after all.
18:44jugglinmikeI can make a demo for others to review, but it will not be safe for WPT
18:45jugglinmikeThe trouble is that you can only observe this when two requests are on the same connection, and (as far as I am aware) there is no way to force this circumstance using any standard API
18:45jugglinmikeSo the script will be consistent enough for demonstration purposes, but still fundamentally flaky
18:46jugglinmikeSo I guess I should make a standalone repository for it. Does that sound alright to you?
18:46annevkjugglinmike: flaky in both Firefox and Chrome or just Chrome?
18:47annevkCause I thought the weirdness was that it was only non-deterministic in Chrome
18:48annevkjugglinmike: in any event, I think I stand by my suggestion to let networking look into this, cause it seems like a bug
18:48jugglinmikeWell, my current explanation is deterministic--but it is only observable under non-deterministic conditions. does that make sense?
18:49annevkBut it might still be a bug
18:49jugglinmikeThis might be more clear when I make this standalone "demo" repository
18:49annevkI gotta go for the day
18:49annevkI mean, please do that if it's not too much trouble. I dunno what that would take
18:51jugglinmikeI will.
19:57fserbhey, qq. I've added a couple tests to web-platform-tests, and then I run ./manifest but the new tests don't show up there. :/ What am I missing?
19:58fserbactually, I'm getting an error now, because I've also deleted a test and it's complaining that the file doesn't exist. Where does it get the file list?
20:04jugglinmikefserb: via git. If you haven't created a commit from your local changes, you should run the manifest script with the `--work` flag
22:10gsneddersjeffcarp: what happens if the auto-sync fails to merge? does it try again later, or?
22:11gsneddersjeffcarp: (when doing the csswg-test merge, I ideally want some brief time when nobody pushes anything to wpt before the merge lands, because there is some limited manual work needed)
21 Mar 2017
No messages
Last message: 6 days and 18 hours ago