w3 :: #testing

15 Mar 2017
05:13annevkStill unclear why it's not intermittent in Fx then
05:14annevkTesting Content-Length makes sense as well
07:43zcorpanhttps://github.com/w3c/web-platform-tests/pull/5131#discussion_r106100064 - is this a non-issue these days?
07:45annevkNot sure, but there is some kind of /common/ JavaScript library that helps you get a unique token
09:18Ms2gergsnedders, do you need me to look at https://github.com/w3c/wpt-tools/pull/180 or is jgraham's review sufficient?
10:07jgrahamMs2ger: I'm happy to review that unless you want to
10:12Ms2gerjgraham, I'll leave it to you, then
15:13jgrahamjugglinmike: What's the expected usage of the HTTPRequest helper in webdriver tests?
15:14jgrahamI need a raw http connection to test new session, but it seems to be a decorator and a generator and I can't work out how you envison it being used
15:26jugglinmikejgraham: Just a moment; I need to refresh my memory on that
15:28jgrahamjugglinmike: OK, thanks!
15:33gsneddersbah, no ms2ger
15:34jugglinmikejgraham: It's implemented as a Python "fixture". You can see an example usage here https://github.com/w3c/web-platform-tests/blob/24d0d39ff586141a84685c2efee0cb7d62d53145/webdriver/contexts.py#L21-L25
15:34jgrahamI know. Shouldn't be allowed.
15:34jugglinmikebut I'm not sure it fits with our latest plans for methodology
15:35jugglinmikeWe've been discussing using the `transport` object attached to the `session` for this purpose
15:35jugglinmikeso that's what I updated it to here: https://github.com/w3c/web-platform-tests/pull/4756
15:36jugglinmikeThe "fixture" is still around on that branch because older tests (contexts.py) continue to rely on it
15:38gsneddersjgraham: https://github.com/w3c/wpt-tools/pull/180/commits/b859c36b71cd550b0968e1e2567499a925a0caf2 does that suffice?
15:38gsneddersidk what to say!
15:38jgrahamjugglinmike: OK, maybe I'll use a session but never actually start the session from the point of view of that object
15:38jgrahamjugglinmike: Thanks!
15:38jgrahams/session/Session/
15:39jgrahamgsnedders: Yeah, that's good
15:39jgrahamgsnedders: Want me to merge?
15:40jgrahamgsnedders: Or do you want to rewrite history?
15:40gsneddersI'll squash
15:41jgrahamgsnedders: OK, feel free to merge when you're done
15:42jugglinmikejgraham: Yes, that sounds right. "Session" isn't really a great name. It ought to be "Client", I think
15:43jgrahamjugglinmike: Well session makes sense in the context of a decorator
15:44jgrahamwith Session(host, port, blah) as session:
15:44jgraham # do stuff with the session
15:44jgrahamBut in this case not so much
15:44jgrahams/decorator/context manager/
15:45jugglinmikeI would think, `with Client(host, port, blah).connect() as session:`
15:46jugglinmikeJust because `Session(host, port, blah)` on its own doesn't create a session. So it's a little like, `Dog().changeIntoARealDog()`
15:49jgrahamYeah the original intent was that you would always use it as a contextmanager
15:49jgrahamBut that doesnt match the actual useage in tests
15:49jgrahamAnyway
15:52gsneddersgit rebase -i -x is such a life-saver when it comes to not introducing broken commits
15:53gsnedders(turns out my prior attempt at splitting up some of the early work on that branch was broken. whoops.)
15:59jgrahamgsnedders: \o/
15:59jgrahamgsnedders: What else is blocking you?
16:01gsneddersjgraham: they're all small
16:01gsneddersjgraham: I think I have one or two more PRs for wpt-tools, but both should be comparatively easy five-line fixes
16:01jgrahamgsnedders: Anything you need me to look at?
16:02gsneddersjgraham: should get those written today
16:02gsneddersjgraham: not yet :)
16:02jgrahamOK, let me know
16:16gsneddersjgraham: r? ^^
16:26gsneddersjgraham: see, I said it was easy :)
16:45annevklyzadanger: I'm assuming you'll file bugs for the failures you discovered with https://github.com/w3c/web-platform-tests/pull/5145?
16:45lyzadangerannevk: For sure. Im validating right now that one of the bugs I found applies to other keywords (`_parent`, `_top`, e.g.) so I can corral that all into one bug
16:46annevklyzadanger: I don't know if I stated it already, but it would be best if the bugs are mentioned in the pull request at some point, so folks digging through test additions and changes can find them easily
16:46annevklyzadanger: sounds good
16:46lyzadangerannevk: I have so far!
16:47annevkI love that you've found a bug in _blank
16:47annevkLiterally nothing in the platform is robust
16:47jgrahamlyzadanger++
16:47annevkconcur
16:47lyzadangerYeah, isnt that wild, annevk ? I was like, sure, Ill add coverage, but this stuff is ancient and solid...
16:48annevkI should no longer be surprised I think, but browsers get me every time
16:49lyzadangerIve reproduced the issue with `_parent` locally as well and bet itll apply to `_top` and maybe even `_self` when I get to those.
16:52jgrahamPretty sure that for annevk this is what job security looks like
16:54annevkHeh, and I find it not to be boring either, so great
16:58jugglinmikejgraham: You up for some more discussion about this EventSource test?
16:59jgrahamjugglinmike: Sure. Watching some talk now but I think the discussion might be more gripping
17:00jugglinmikeI'll do my best
17:00jugglinmikeWith this new framing, I've come to a different conclusion on why the test in `master` is flaky for Chrome and not for Firefox
17:03jugglinmikeactually, uploading example
17:03annevkjugglinmike: I think that's the key problem, the difference between browsers
17:04annevkjugglinmike: we should seek to resolve that somehow at the core
17:05jugglinmikeyeah annevk, I've found another way to resolve the flakiness, but I with you on making a standalone test that is dedicated to the fundamental difference
17:05annevkFor instance, if it was Content-Length, we should then test that, since it still seems wrong to get different results
17:05jugglinmikeannevk: jgraham here https://github.com/w3c/web-platform-tests/compare/master...bocoup:flaky-test-eventsource-demo2
17:06jugglinmikeThis passes in Firefox but times out in Chrome
17:07jugglinmikeThe important part is that garbage is initially written to the response stream. Firefox tolerates it, Chrome dismisses all the data and continues to wait for a well-formed response. That is the fundamental difference, as far as I can tell.
17:07annevkjugglinmike: can you make it a single write without the strip()?
17:08annevkjugglinmike: looks a bit like my tests here: https://github.com/w3c/web-platform-tests/pull/5102
17:08jugglinmikeannevk: The strip() takes place once statically prior to any request handling; it's just there for readability. I can remove it, but I don't think it will effect the behavior
17:09annevkjugglinmike: heh, I find it makes it less readable
17:10jugglinmikeCertainly noisier. Just trying to express the response in an isolated way
17:10annevkjugglinmike: anyway, this sounds reasonable, due to the lack of Content-Length both browsers treat it as part of the next request, and Firefox handles garbage before HTTP/ more gracefully here
17:11jgrahamYeah, this looks like a reasonable test, although I guess it's unclear what the pass condition is
17:11annevkIf this is what is going on we should add this test to https://github.com/w3c/web-platform-tests/pull/5102
17:12annevkPass conditions are not yet figured out since IETF <flips tables>
17:12annevkAnd then I think we should &quot;fix&quot; the current test by adding Content-Length
17:13annevkIt&#39;s still a little strange to me though that this is what&#39;s going on, since browsers generally support responses without Content-Length and handle them gracefully
17:14annevkThat they&#39;d just assume for 204 they can end the connection early seems suspicious and might lead to random failures in the wild of this nature
17:14jugglinmikeI need to check what is currently being written to the wire
17:15jugglinmikeThe test itself just says, &quot;the response is a 204 and it has this body&quot;. It&#39;s not clear what headers the Python code would define in this case
17:15jugglinmikesince those are two conflicting conditions when it comes to the &quot;content-length&quot; header
17:15jugglinmikejust a moment
17:16jugglinmikeIt is indeed adding a non-zero &quot;Content-Length&quot; header for the 204/205 responses
17:17jugglinmikeWhich Chrome ignores, which means that the body ends up as the initial response of the next request, which Chrome rejects
17:18jugglinmikehmm, given that, it&#39;s not clear what Firefox does with the initial response
17:18jugglinmikebecause (as demonstrated by that test I just shared), firefox is tolerant of invalid bytes that prefix a valid response
17:19jugglinmikeSo it&#39;s also possible that Firefox is ignoring the 204/205 body
17:19jugglinmikeSo the way to test for this, then
17:19jugglinmikeis to write a complete, valid HTTP response as the body
17:20jugglinmike...of a 204 response
17:22annevkgood times
17:23annevkI&#39;m pretty sure Firefox will eat the body and then discard it
17:23annevkNice thinking there btw of how to test it
17:29jugglinmikethanks!
17:59jugglinmikeannevk: When it comes to 204/205 responses with a Content-Length, the expected behavior is to consume the content but discard it. When it comes to extra bytes preceding a response, the expected behavior is to consume the invalid bytes until a valid response is found. Are those two statements correct?
17:59annevkjugglinmike: the second statement is not, for that we don&#39;t have agreed upon behavior yet
18:00jugglinmikeokay, good
18:00jugglinmikebecause that is tricky
18:00annevkjugglinmike: see the PR I referenced earlier about HTTP parsing
18:01jugglinmikeIt&#39;s also very tricky to test the first one, actually
18:01jugglinmikebut at least the intended behavior is clear
18:01annevkjugglinmike: I think even without Content-Length the expected behavior would be to consume and discard, since that&#39;s what happens for non-204/205
18:02annevkjugglinmike: but it&#39;s somewhat in the territory of undefined behavior (though not a lot)
18:02jugglinmikeannevk: if there were no content-length, how would the browser know when to stop consuming?
18:03annevkjugglinmike: connection close is what the browser uses I think
18:03annevkjugglinmike: https://github.com/w3c/ServiceWorker/issues/362#issuecomment-49011736 has some research on it
18:03jugglinmikeis there such an event if the &quot;Connection: close&quot; header isn&#39;t specified?
18:04annevkI&#39;m a little bit out of my depth since I never quite understand how TCP/IP works
18:04annevkAnd what is reliable and what is not
18:05annevkI would assume it&#39;s more reliable over TLS though
18:36jugglinmikeannevk: Neither browser consumes the body of a 204... I&#39;m just having trouble writing a deterministic test to prove it
18:38jugglinmikeI think I need to go back to wireshark for this
18:40annevkI&#39;m sad that the networking folks never pushed for the same kind of rigorous standardization that the content folks did
16 Mar 2017
No messages
   
Last message: 99 days and 21 hours ago