mozilla :: #balrog

17 Jul 2017
12:56bhearsumg'day folks
13:20203A9W1DWbalrog-web #121: building mozilla/balrog:master-6a864fdbbbc97bd018e393e9c8f7e6bf4497d91e
13:21cloudops-ansiblebalrog-admin #155: building master-6a864fdbbbc97bd018e393e9c8f7e6bf4497d91e
13:31cloudops-ansiblebalrog-web #122: mozilla/balrog:master-6a864fdbbbc97bd018e393e9c8f7e6bf4497d91e deployed to stage /cc relud bhearsum
13:31bhearsumnthomas, jlorenzo: thanks for all the reviews
13:31cloudops-ansiblebalrog-admin #154: master-6a864fdbbbc97bd018e393e9c8f7e6bf4497d91e deployed to stage /cc relud bhearsum
13:33jlorenzobhearsum: np!
13:40cloudops-ansiblebalrog-web #121: web_promote in stage failed /cc relud
14:58shivam2302Hii,I am Sudhanshu Bhardwaj from India .I am facing problems in installation of balrog in my system. Can someone plss help in solving these problems .Logs are here:
15:00bhearsumhey shivam2302, can you removing .cache and try again?
15:03shivam2302Sorry, But I m not able to get . How to remove cache ?
15:04bhearsumit's a hidden directory inside of the balrog directory - just remove it
15:04bhearsumeg, if you're on Linux or OS X: "rm -rf .cache"
15:15shivam2302I am getting same error even after clearing cache
15:21bhearsumare you sure it's the same error? can you paste again?
15:24bhearsumshivam2302: ^
15:26shivam2302my error has changed , it is now giving this error :
15:27bhearsumwhat does "docker-compose logs balrogadmin" say?
15:31shivam2302here are the logs for "docker-compose logs balrogadmin" :
15:31bhearsumthat's weird, it's still showing DatabaseAlreadyControlled
15:31bhearsumtry this though...
15:32bhearsumremove .cache again
15:32bhearsumchange "create" to "--version 24 create"
15:32bhearsumand then run docker-compose up one more time
15:39shivam2302After doing appropriate changes , my errors remain same
15:40bhearsumare you absolutely certain that all of .cache has been removed? the error you're getting usually means that it hasn't been
15:42shivam2302I have removed the .cache folder from my local balrog directory . But when i type "docker-compose up " , directory is formed again
15:43bhearsumyeah, that's normal
15:43bhearsumit's weird to get that error if .cache doesn't exist when you run docker-compose up
15:45shivam2302Now I m getting balrogui and balrogagent containers are unhealthy
15:46bhearsumthat's likely a symptom of the issue with balrogadmin
15:46bhearsumDatabaseControlledError is the root problem, but i'm not sure how to fix it if removing .cache isn't working :(
15:47shivam2302How , there is no .cache directory even after ruuning "docker-compose up "
16:07aksareenshivam2302: try executing $ docker-compose build
16:12shivam2302@aksareen successfully excecuted .
16:26bhearsumrelud, miles: can one of you have a look at it's currently blocking some release work
16:41reludbhearsum: grabbing the logs for you
16:44reludbhearsum: the agent log looks like this: &quot;root&quot;, &quot;Severity&quot;: 3, &quot;Fields&quot;: {&quot;message&quot;: &quot;Encountered exception:&quot;, &quot;traceback&quot;: &quot;Uncaught exception:\n File \&quot;/app/balrogagent/\&quot;, line 31, in run_agent\n sc = sorted(sc, key=lambda k: (k[\&quot;priority\&quot;], k[\&quot;when\&quot;]), reverse=True)\n<class &#39;TypeError&#39;>\nTypeError(&#39;unorderable types: NoneType() <
16:44reludint()&#39;,)\n&quot;, &quot;error&quot;: &quot;TypeError(&#39;unorderable types: NoneType() < int()&#39;,)&quot;}, &quot;Pid&quot;: 1, &quot;Hostname&quot;: &quot;ip-172-31-46-92&quot;, &quot;Logger&quot;: &quot;Balrog&quot;, &quot;Timestamp&quot;: 1500308445336223488}
16:44reludall identical
16:45bhearsumthanks - digging into that on my side
16:45reludstarted about an hour ago
16:45bhearsumyeah, that makes sense
16:45bhearsumi think i know what&#39;s happening, trying something
16:46bhearsumif i&#39;m right, should go away within ~1min
16:46bhearsumi think it&#39;s fixed
16:48bhearsumthanks relud, much appreciated
17:28cloudops-ansiblebalrog-web #123: failed in build step /cc relud
17:31aksareenbhearsum: need help docker-compose. Pulled in recent changes and rebuilt the image with updated dependencies but Here&#39;s the error log: . particularly fails at line 118 sqlalchemy.exc.OperationalError: (OperationalError) (1060, &quot;Duplicate column name &#39;memory&#39;&quot;) &#39;\nALTER TABLE rules ADD memory VARCHAR(100)&#39; ()
17:35bhearsumaksareen: ah, sorry - i meant to give you a heads up about that
17:35bhearsumi had to split up some of the work for one of my bugs - you&#39;ll need to downgrade your database to version 24 and try again
17:38bhearsumlet me know if you need a hand with that
17:39cloudops-ansiblebalrog-admin #156: master-bbe7976fa45e73ab13c9b9176141cb29a4c37733 deployed to stage /cc relud bhearsum
17:41bhearsumrelud: does need a kick? it&#39;s failed a couple of builds in a row
17:41reludbhearsum: i&#39;ll poke it
17:44reludbhearsum: built #122, just not 121 and 123. kicking it to rebuilding 123
17:45cloudops-ansiblebalrog-web #124: building mozilla/balrog:master-bbe7976fa45e73ab13c9b9176141cb29a4c37733
17:46aksareenbhearsum: yeah definitely need help. Trying to execute $python scripts/ --db mysql://balrogadmin:balrogadmin@balrogdb/balrog downgrade --version 24 . Cannot seem to connect with balrogdb . here&#39;s the error log :
17:46bhearsumaksareen: you&#39;re pretty close, but you need to run that from within the balrogadmin container
17:47bhearsumalternatively, you could rm -rf .cache, and set --version 24 in
17:48aksareenbhearsum: balrogadmin container isn&#39;t starting . here&#39;s the docker-compose log :
17:48bhearsumok, try the other method then
17:48bhearsumit&#39;s kindof a PITA when the current master and the current production aren&#39;t using the same database schema...
17:54cloudops-ansiblebalrog-web #124: mozilla/balrog:master-bbe7976fa45e73ab13c9b9176141cb29a4c37733 deployed to stage /cc relud bhearsum
18:40reludbhearsum: are you onboard for the ssl cert update for aus5 ?
18:43reludbhearsum: do we want to test the ssl cert before we deploy it? like, i could create a new elb, and put this cert on it, and someone could fake dns to the testing elb, and verify that firefox accepts the ssl cert and gets updates?
18:46* relud will test it, and then roll out after lunch
18:55reludokay, curl accepts the ssl cert
18:59* relud will figure out how to test that an actual firefox accepts it after lunch
19:13bhearsumrelud: yeah, that would be awesome
19:13bhearsumi can do that testing
19:13bhearsumoverriding the update server is...tricky these days
19:14reludbhearsum: if you locally modify dns to point to then you can test the new cert
19:14* relud lunches
19:15bhearsumoh that&#39;s a clever idea
19:33bhearsumrelud: can you check to see if you got some requests to with URLs like &#39;; in the past few minutes?
19:35reludI forgot to create elb access logs for the new elb
19:35reludi just set them up
19:36reludsince it&#39;s backed by the same ASG i can&#39;t differentiate requests post-elb
19:36bhearsumah, ok
19:36bhearsumi&#39;ll send a few more
19:36bhearsumoh, perfect
19:37* relud goes to lunch
19:44bhearsumaksareen: did you get your balrog instance working again?
19:46aksareenbhearsum: it&#39;s working sporadically. Sometimes docker-compose works perfectly. Sometimes it fails
19:48aksareennow it&#39;s not starting at all.
19:48bhearsumthat&#39;s really strange
19:52bhearsumwhat kind of errors are you getting?
20:11bhearsumrelud: ok, the new ssl certs look good - i need to give release-drivers a heads up before we enable it though. how do you feel about making the swap on Thursday? should be free of shipping releases, and it&#39;s probably best to keep it separate from the code push on wednesday
20:13aksareenbhearsum: it&#39;s working again for now. Cannot reproduce the issue
20:14bhearsumdo let me know if you see any more issues - it should be fully resolved by thursday
21:01reludbhearsum: doing the swap on thursday sounds good
21:07bhearsumok, cool
21:07bhearsumi&#39;ll give folks the heads up
21:07bhearsumgotta run now, talk to you later!
18 Jul 2017
No messages
Last message: 10 days and 2 hours ago