w3 :: #testing

19 Apr 2017
02:28MikeSmithgsnedders: Do you have figure for how many specs we have tests in wpt for?
02:29MikeSmithI count ~130 at the wpt root, plus about ~55 in the css dir
02:30MikeSmithhmm I guess we could get the actual number from the manifest
02:42MikeSmithOK based on the manifest, more like ~120 at the root, and ~54 in css
02:42MikeSmithso will make it about 175 specs
10:07jgrahamgsnedders: https://github.com/w3c/web-platform-tests/pull/5321 and https://github.com/w3c/web-platform-tests/pull/5327 plz. I think it's apparant that fantasai doesn't have an opinion, and the Hixie comment one in particular is a blocker
10:29jgrahamgsnedders: Reverted the submodule pointers in the tools import. I don't know what you want the note to say; afaict it's only an issue if you go back from a post-merge checkout to a pre-merge one?
10:31gsneddersjgraham: I still think that's common enough with people going from master to branches
10:36gsneddersjgraham: but that's also the reason why I'd rather do this all at once, and get rid of everything we're going to at once
10:42jgrahamgsnedders: I guess I can convert the remaining submodules to subtrees
10:42jgrahamgsnedders: jugglinmike will have to do the same for webidl2
11:30gsneddersjgraham: tho obviously we don't need the history for most of them
12:40jgrahamgsnedders: Thanks
12:40jgrahamfatal: no submodule mapping found in .gitmodules for path 'tools/html5lib/html5lib/tests/testdata'
12:54aenyHello, would this be the right place to get some pointers on an issue I'm having with the wptrunner?
12:54aygjgraham: This doesn't find the test: ./mach wpt testing/web-platform/mozilla/tests/classList.html --log-raw=/tmp/log
12:54aygThe test exists, I created it using the test creation script, and I ran wpt-manifest-update for good measure.
12:55aygI tried without the "testing/web-platform/" too.
12:57aygRunning with ./mach test instead of ./mach wpt works, but then --log-raw doesn't work.
12:59jgrahamayg: That seems odd. Did you check if it's included in the manifest?
12:59jgrahamaeny: yes
13:00aygI think it's not.
13:00aygWait, maybe it is.
13:00aygNo, it's not.
13:00aygHow do I add it?
13:00aygI'm sure I created it with the test creation script.
13:00jgrahamayg: |mach wpt-manifest-update| should add it
13:01jgrahamMaybe there's an issue with mozilla/ tests and the creation script
13:01aygjgraham: Wait, it is in the manifest, just it's in testing/web-platform/mozilla/meta/MANIFEST.json instead of testing/web-platform/meta/MANIFEST.json.
13:02jgrahamayg: What happens if you run |mach wpt /_mozilla/tests/classList.html|?
13:02aygjgraham: Doesn't work.
13:04jgrahamayg: Is it in the testharness section of the manifest?
13:04aygjgraham: Looks like it. "testharness": { "./classList.html": [
13:04ayg "testharness": { "./classList.html": [ [ "/_mozilla/./classList.html", {} ] ],
13:05aygMaybe it's somehow confused by the fact that it's in the root.
13:05aygThat's my bet.
13:05jgrahamayg: Ah yeah
13:05aygThere's a file in the root already, but it's a support file.
13:05jgrahamThat isn't supposed to work, even
13:05jgrahamBut it's not supposed to not work in that way :)
13:07aygNow it works, thanks.
13:07aygSome error messages here would have been helpful.
13:17aenySo, I've been following https://wptrunner.readthedocs.io/en/latest/usage.html and managed to get the the manifest file generated for web-platform-tests, but I'm stuck trying to run actual tests. First I'd like to ask what url_base should be set to in my .ini file? It didn't seem happy with the default / character in there.
13:25jgrahamIt should be /
13:25jgrahamYou aren't expected to change that
13:25jgrahamWhat are you running and what errors are you seeing?
13:26jgrahamayg: Yeah, agreed that the error handling here isn't superb
13:32aenyjgraham: to get the url_base error I'm running the following: wptrunner --config=../wptrunner.ini --ssl-type=none --binary=/usr/bin/firefox --manifest-update. The error message I get is: AttributeError: 'str' object has no attribute 'url_base'
13:34jgrahamaeny: Can you pastebin the ini file and the full traceback?
13:34gsneddersjgraham: did you figure out the submodule issue?
13:36jgrahamgsnedders: Maybe? I mean I added it by hand and git submodule status now doesn't error
13:38aenyjgraham: Sure can: https://pastebin.com/Qybkt2j3
13:51jgrahamgsnedders: ^
13:54jgrahamaeny: Something very similar looking wfm, do you have up to date wpt and up to date wptrunner?
13:57aenyjgraham: wpt is freshly cloned from github some hours ago, wptrunner came from pip.
13:58jgrahamaeny: Oh, wptrunner from pip probably isn't a good idea
13:58jgrahamWe haven't done a release for some time
13:59aenyjgraham: Thanks, will clone that from github as well and see how far I get :)
14:14jgrahamjugglinmike: So I added the resources changes to my branch
14:14jgrahamSince we want to land them together and there were conflicts and gsnedders talked me into removing all essential submodules
14:17gsneddersjgraham: I know we disagree, but what music is appropriate for merging submodules?
14:18jgrahamgsnedders: Given that github is telling me there's a merge conflict? Mogwai Fear Satan
14:24gsneddersjugglinmike: am I mistaken in thinking you guys are in Boston?
14:24* gsnedders forgets where on the east coast you are
14:25jugglinmikegsnedders: only partially mistaken. I'm in Boston, bobholt is farther north
14:25jugglinmikeHe *used* to be in Boston, though
14:27jugglinmikegsnedders: You might try O Fortuna
14:27jugglinmikejgraham: looking forward to it
14:28gsneddersjugglinmike: I'll likely be in Cambridge briefly in a few weeks to see plh, FWIW, if you want to meet up for lunch or something maybe, idk what works reasonably
14:29jugglinmikegsnedders: Totally! If you'd like a place to work, we're centrally located and have empty desks
14:29gsneddersjugglinmike: I'm literally going to be arriving on Saturday, wondering around on Sunday because flights are cheaper on Saturday, meeting plh for some/all of Monday then leaving that evening
14:29* jgraham figures out a way of resolving this merge conflict
14:29jgrahamDon't know that I can push this to the branch though
14:30gsneddersjugglinmike: I really do mean briefly :)
14:30jgrahamRebase doesn't really work
14:30gsneddersjgraham: yeah, for csswg-test I just wrote a script to do everything because it's virtually impossible to resolve conflicts
14:30jugglinmikegsnedders: Ah, I guess so!
14:32jgrahamhttps://github.com/w3c/web-platform-tests/pull/5606 is just to get the tools to run with the post-merge branch
14:32jgrahamIf you r+ the other one I will just directly
14:34gsneddersso what do you want me to look at?
14:34* gsnedders looks
14:36gsneddersjgraham: so what's the status of resources/webidl2/test/widlproc?
14:37jgrahamgsnedders: It's a submodule, but since it's under test/ I assume it isn't actually required to run anything
14:38gsneddersso what's the difference between 5606 and 5554? 5606 is more up to date?
14:41jgraham5606 is the branch merged into current master
14:41jgrahamBut merging that again produces an overcomplex history (possibly)
14:42gsnedderswhat are the check_stability.py changes?
14:45jgrahamTo prevent the stability checker running all tests when resources/ changed
14:45jgrahamWhich isn't feasible
14:49gsneddersthat's the effective diff v. master
14:49gsneddersat a filesystem level, rather than a git one
14:50gsneddersjgraham: how come there are chnages in idlharness.js?
14:51jgrahamgsnedders: GBood question
14:54gsneddershttps://github.com/w3c/testharness.js/commit/e7a80b151358184505514cc20e79444263656b92 isn't on master?
14:54gsneddersI didn't review https://github.com/w3c/web-platform-tests/commit/0a4479253b6968e957b6674b7826100bd96aa12b well enough to make sure it actually moved to up-to-date master
14:57gsneddersjgraham: can you add a commit prior to the merge that updates resources to the latest?
14:58gsneddersjgraham: https://github.com/w3c/web-platform-tests/pull/5554#issuecomment-295299805
14:59aenyjgraham: getting wptrunner from github seems to have fixed my issues, thanks. :)
14:59gsneddersjgraham: would still rather we said "if you want to checkout an old branch and get an error like x, you need to rm -rf that dir"
15:02jgrahamaeny: Ah, great, sorry for the problems
15:03jgrahamgsnedders: I will make a followup commit that updates the README with the actual SHA1 (although I think checkout -f will work?)
15:05gsneddersjgraham: I think the problem might be when you do a submodule update after?
15:05jgrahamgsnedders: Hmm, OK I guess I need to check
15:07* gsnedders checks too
15:08gsneddersgoing from master w. submodules to new-master -> error: The following untracked working tree files would be overwritten by checkout:
15:09gsneddersgit checkout -f ends up with a dirty checkout, with things like resources/.gitmodules surviving
15:09gsneddersand then going back to master I can't get the submodules init'd anymore
15:10gsneddersthough rm -rf'ing them makes them able to be so
15:11gsnedders(where master is the now the old master)
15:11gsneddersjgraham: ^^
15:12jgrahamgsnedders: Hmm, I don't see that
15:13jgrahamhttps://pastebin.mozilla.org/9019429 but git status looks reasonable afterwards
15:14gsneddersjgraham: is that with submodules init'd in wdspec_fixup?
15:15jgrahamAh, that makes a difference
15:16gsneddersalso try going from master with submodules init'd to wdspec_fixup
15:16gsneddersthis is git version 2.12.2
15:22jgrahamgsnedders: Does that sound right?
15:22jgrahamgsnedders: Going the other way I get warnings but nothing fatal
15:24gsneddersfatal: No url found for submodule path 'resources/webidl2/test/widlproc' in .gitmodules
15:24gsneddersFailed to recurse into submodule path 'resources/webidl2'
15:24gsneddersfatal: No url found for submodule path 'tools/html5lib' in .gitmodules
15:24gsneddersis what I get the other way
15:24gsneddersFailed to recurse into submodule path 'resources'
15:24gsneddersFailed to recurse into submodule path 'tools'
15:24gsneddersif submodules have been initalised
15:27jgrahamI have git 2.11.0
15:27jgrahamSo it could have changed
15:30gsneddersI can't repro any more
15:30gsneddersidk what exactly I did
15:35gsneddersjgraham: I'll claim you need a comma before i.e. though
15:37bobholtjgraham: my new PR on wptrunner solves my SauceConnect issues by doing what I believe you wanted in the first place - ensuring it's installed and running before running tests: https://github.com/w3c/wptrunner/pull/253
15:38jgrahambobholt: I like the sound of that :)
15:38bobholtif that ends up being the solution we go with, then https://github.com/w3c/web-platform-tests/pull/5231 is good to go - no changes required in that PR
15:44jgrahamgsnedders: Better?
15:45gsneddersjgraham: lgtm
18:06aygWhen trying to pull wpt, I get: error: The following untracked working tree files would be overwritten by merge: [lots of paths in resources/ and tools/] Please move or remove them before you can merge. Aborting
18:07aygAnyone have any ideas?
18:07aygjgraham, gsnedders?
18:07* ayg doesn't understand how this submodule thing works
18:08jgrahamayg: Happily I added instructions to the readme :)
18:08jgrahamayg: rm -r resources tools
18:08* ayg wound up doing that on his own
18:09jgrahamayg: This is a result of removing a whole pile of submodules to decrease future pain
18:09aygOkay, thanks.
22:30DomenicSo many bot comments, make it stop -_-
22:41DomenicFiled https://github.com/w3c/web-platform-tests/issues/5611
20 Apr 2017
No messages
Last message: 4 days and 21 hours ago