mozilla :: #balrog

12 Sep 2017
12:51bhearsumg'day folks
12:53jlorenzobhearsum: hi! how are you doing?
12:53bhearsumquite well, the coffee is working this morning
12:54jlorenzonice! coffee machine works in the Paris office too
12:54jlorenzoI'm testing https://github.com/mozilla/balrog/pull/394 out locally. I don't remember, how can one add a new user/role without touching the mysql directly?
12:54* jlorenzo doesn't recall if there's a way, actually
12:55bhearsumjlorenzo: you should be able to do it through the permissions ui
12:55bhearsumclick "Update" on a user, then there should be a tab for roles
12:55jlorenzoah right! now I remember :)
12:55jlorenzothank you
13:03bhearsumjlorenzo: np!
13:08cloudops-ansiblebalrog-admin #203: building master-f41081d06395dc39512c2a34c4261b29e6264084
13:08203BABRP1balrog-web #178: building mozilla/balrog:master-f41081d06395dc39512c2a34c4261b29e6264084
13:08cloudops-ansiblebalrog-web #179: building mozilla/balrog:master-f41081d06395dc39512c2a34c4261b29e6264084
13:16relud-trainingbhearsum miles: rds cpu is holding around 70% in prod. could you two look into that when you're up?
13:18cloudops-ansiblebalrog-admin #204: master-f41081d06395dc39512c2a34c4261b29e6264084 deployed to stage /cc relud bhearsum
13:18cloudops-ansiblebalrog-web #179: mozilla/balrog:master-f41081d06395dc39512c2a34c4261b29e6264084 deployed to stage /cc relud bhearsum
13:19cloudops-ansiblebalrog-admin #204: admin_scaledown in stage failed /cc relud
13:20cloudops-ansiblebalrog-admin #203: master-f41081d06395dc39512c2a34c4261b29e6264084 deployed to stage /cc relud bhearsum
13:28cloudops-ansiblebalrog-web #178: web_promote in stage failed /cc relud
13:44bhearsumrelud-training: will do
13:45bhearsumrelud-training: hm, looks like it went back down
13:45bhearsumvery strange though, there was no spike in requests
13:49bhearsumthere's really nothing unusual at all happening during that time
13:49bhearsumwe had the usual flood of nightly builds come in, which causes a lot of cache churn on the webheads, but that's been happening forever
14:49cloudops-ansiblebalrog-admin #205: building master-f705320369ef6c12d499e881906fcf9936c96f1c
15:00cloudops-ansiblebalrog-web #180: mozilla/balrog:master-f705320369ef6c12d499e881906fcf9936c96f1c deployed to stage /cc relud bhearsum
15:01cloudops-ansiblebalrog-admin #205: master-f705320369ef6c12d499e881906fcf9936c96f1c deployed to stage /cc relud bhearsum
15:25glassercIs it just me or does 'testForceParamWithBadInputs' fail on master? and/or is there a PR that I should be rebasing onto that fixes it
15:26bhearsumseems okay in CI: https://github.com/mozilla/balrog/commits/master
15:28bhearsumi didn't have local failures yesterday, let me try again with the latest master
15:28glassercThanks
15:37bhearsumglasserc: hmm, i think it's failing intermittently
15:37bhearsumit sounds like some of the inputs that hypothesis generates aren't handled well
15:38bhearsumwow, that failure traceback is awful
15:39glassercIt doesn't seem very intermittent to me, but it would make sense that a hypothesis test might not fail 100% of the time
15:39glassercIs it just me or does 47dd147b08ca308442915debea5018139751c5b3 not add any code, just a test?
15:40bhearsumyup
15:40bhearsumyour work on force ended up fixing the bug that person was working on, so we just took his new test
15:41allan-silvahey folks
15:41glassercOh, oops
15:41allan-silvaglasserc, connexion was updated to 1.1.15
15:41bhearsumglasserc: it's no problem
15:42allan-silvatry remove .tox e run ./run-test again
15:42glassercallan-silva: OK, thanks
15:42* bhearsum tries too
15:42bhearsumoh yeah, tox doesn't autoupgrade deps, does it?
15:43allan-silvavia ./run-tests i think not
15:43glassercOK, that fixed it, thanks
15:45bhearsumwell, i'm glad it's just test issues \o/
15:45allan-silvawhen you want test a single file and this happens, you can try something like this: "tox -r -e py27 -- auslib/test/web/test_client.py"
15:45allan-silva-r is to recreate
15:47glassercI've been running the tests through docker, isn't that the recommended way?
15:48glassercUsing docker run balrogtest ...
15:51bhearsumyeah
15:51bhearsumit uses a cache tox environment on the host though
15:51bhearsumfor speedyness
15:52glassercAh, so running tox on the host might update the cache?
15:52glassercOr do you mean running the tox command in the container somehow?
15:53allan-silvaNormally, I run test to files that has my changes, then "./run-tests.sh" to run all tests before open a PR.
15:53bhearsumyeah, running it on the host would
15:53bhearsumor just rm -rf .tox and then rerun run-tests.sh
15:55allan-silvaI setup a virtualenv locally, becase there are 938 tests hehe
15:55bhearsumyeah
15:55bhearsumawhile back, someone prototyped a way to speed them up with pytest fixtures in https://bugzilla.mozilla.org/show_bug.cgi?id=1339532
15:56bhearsumwe spend so much time rebuilding the database from scratch for each test
16:07glassercI'll try moving to a local virtualenv, because Ctrl-C doesn't seem to work in the docker container
16:09bhearsumyeah, that's known :(
16:09bhearsumallan-silva's method of running on the host until you think you're done is not a bad one
16:12glassercI'll try to add a couple pointers on how to do that to the docs
16:13glassercIt's something that would have made my development experience easier before :)
16:13bhearsumthanks for that!
16:13glassercThere's this annoying quirk where docker has to be run as sudo on Fedora, so there's lots of annoying permissions things I have to sort out too
16:15allan-silvaHave you already add your user to docker group?
16:16allan-silvaNeeds logoff after do it
16:17glassercThere is no docker group
16:19bhearsumis that selinux getting in the way again?
16:19bhearsumalthough, even on ubuntu, any files created by docker end up root owned
16:20glassercIt's http://www.projectatomic.io/blog/2015/08/why-we-dont-let-non-root-users-run-docker-in-centos-fedora-or-rhel/
16:21bhearsumahhh
16:21glassercI guess I could create a docker group and that might work OK
16:23allan-silvaI don't know how selinux works, but it is possible to create a docker group, logoff and restart docker daemon?
16:24glassercThis isn't really related to selinux
16:24bhavishya_bhearsum: Hi, I did a gsoc with cpython this year and have contributed very small fixes to balrog(https://github.com/mozilla/balrog/pulls?q=is%3Apr+is%3Aclosed+author%3Abhavishyagopesh), but would like to discuss things a bit further, that whether I should keep doing bugzilla-bugs or you'd suggest something else(may be as a build up for next yr's gsoc)....thanks.
16:24glassercThat might work, but if bhearsum is correct and files created by docker are still owned by root, then it doesn't seem worth the effort
16:30allan-silvaah, I got it! yeah, I always run "chown -r my_user ../balrog" when I want to test locally.
16:30bhearsumhey bhavishya_, i'm a bit busy for the remainder of the day, but if you send me an e-mail i can help you find something
16:31bhearsumfull disclosure though: i'm not 100% sure if Balrog will be part of GSoC next year
16:32bhavishya_bhearsum: ok, thanks...I'll contact you over mail then
17:26jlorenzobhearsum: sorry, I wanted to wrap the review up by today. I'm halfway through it. Would you mind if I finish it before tomorrow's morning ET?
17:27bhearsumjlorenzo: yeah, no problem
17:28jlorenzothanks!
17:28bhearsummy busted unit tests meant it wouldn't make this week's push anyways, so there's lots of time before the next :)
19:51catleebhearsum: is https://bugzilla.mozilla.org/show_bug.cgi?id=1294451&list_id= still valid?
19:51catleeI wonder how much it would help funsize
19:52bhearsumas far as i can tell it's still valid
19:53bhearsumi'm not sure how much it would help...the requests that funsize makes are pretty small typically
19:53bhearsumi can't imagine it hurting though
20:25glassercAny advice about what to do if I encounter "migrate.exceptions.DatabaseAlreadyControlledError" ?
20:25bhearsumthat usually means your database was only half initialized - usually removing .cache and starting the containers again will fix it
20:26bhearsumi assume you're getting it when starting the containers
20:26glassercThanks
20:26glassercYeah
20:27glassercI wonder if that's because I ctrl-c'd docker-compose up when it was taking too long
20:27glassercOK great, thanks
20:28glassercI think it's active now
20:30bhearsumyeah, that could do it if you ^C'ed at the wrong time
20:32glassercAnd there's something I can do to make it automatically import a recent dump of the live balrog state, right?
20:33bhearsumit should do that by default
20:33bhearsumit will download the latest dump prior to db init
20:36glassercI guess I know now where I was when I ^C'd
20:36bhearsumare you having trouble getting production data?
20:36glassercrm .cache/mysql/db.done scripts/prod_db_dump.sql might have fixed it
20:37bhearsumah, yeah, could be
20:37bhearsumwe only compare the timestamp on that file to the timestamp of the remote file before downloading
20:51glassercHmm, now when I try to start it up, I get "ERROR: for balrogui Container "128503db2f7c" is unhealthy."
20:52glassercMaybe that just means it's still downloading the dump
20:52bhearsumit's possible you might get that if the download takes too long...that's very annoying
20:53bhearsumwhat's in "docker-compose logs balrogadmin"?
20:53glassercIt ends with "Downloading latest database dum"
20:54bhearsumah
20:54glassercAnd my DSL is going as fast as it can
20:54glasserc1.5 million bits per second down
20:54glassercAny day now
20:54bhearsumoh, wow
20:54bhearsumyou might try increasing the timeout or retries for the healthchecks in docker-compose.yml
20:56glassercAh, once that finished, I was able to rerun docker-compose
20:56glassercI'm in now
20:56bhearsum\o/
20:56bhearsumhttps://github.com/mozilla/balrog/pull/397 will be shrinking the db dump by a considerable amount, so once that's all done it should be less likely in the future...
13 Sep 2017
No messages
   
Last message: 8 days and 6 hours ago