mozilla :: #releng

20 May 2017
09:59jcristaujlund: yesterday was the first time i actually saw those scripts :)
21:41philor!squirrel masses of command queue nagios alerts, of unclear origin unless it's the wine glass in https://hg.mozilla.org/integration/autoland/rev/f30e76c54839cc7c7635be21c104580fb593602d eating our liver again
21:41catleeruh roh
21:42philorI can't remember what problem wine gave us last time, a familiar state of affairs for me
21:43catleeupload host is unhappy on the first log I looked at
21:44catleeand second and third
21:46* catlee files a bug
21:47* philor considers how much he misses the missing things
21:48catleephilor: are uploads working ok?
21:49catleeor also busted?
21:50philorI think probably busted, there's a bunch of Mac and Windows builds that are "overdue" by hundreds of minutes, which also haven't spawned tests
21:50catleeprobably the same thing
21:51philoryeah; I think it's enough I need to close over it, though
21:51catleehttps://bugzilla.mozilla.org/show_bug.cgi?id=1366523
21:51firebotBug 1366523 NEW, nobody@mozilla.org upload.ffxbld.productdelivery.prod.mozaws.net not allowing ssh commands
21:52philora few missing tests, whatevs, but missing builds lead to huge pileups
21:53* philor does a quick cc while it's still public :)
21:53catleeare you going to close trees?
21:53philoryep
21:54* catlee pages cloudops
21:56philorclosed, that would go faster if treestatus had a way to check "all trees philor cares about, excluding Try" since I left it open
21:56catleeI wonder if Try is affected...
21:57catleetry looks ok
21:58catleeman, and I was just sitting down to enjoy a nice cold cider after yard work
21:59philornot much to see on try, other than the overdue things in https://treeherder.mozilla.org/#/jobs?repo=try&revision=2efde4cf5730cc5cfff09a43633f25cdf6a5cdfd&filter-resultStatus=running&filter-resultStatus=pending&filter-resultStatus=runnable
21:59KWierso|afkphilor: I could probably get a patch in to add a button to select those
21:59KWierso|afkmind filing a bug and assigning it to me?
22:05philorcatlee: never a good idea to have irc open on a weekend when I'm home, that's why I mostly don't publish or admit to my schedule
22:11nthomaswhats goin on ?
22:13philorupload.ffxbld.productdelivery.prod.mozaws.net hates us
22:15nthomasoh fun
22:19nthomasnothing obvious in datadog
22:21nthomasI let #moc know
22:26catleenthomas: I paged cloud ops already
22:26catleeNo response yet?
22:27nthomasyeah, I saw. just keeping moc in the loop, per memory of h.als requests
22:27nthomasnone Ive seen
22:30oremjbox should be back up very soon
22:33* philor starts retriggering the builds that didn't survive the shock of being denied
22:35oremjnthomas: catlee: back up now
22:35oremjcatlee: sorry, I saw the page from our monitoring but not your page
22:35nthomasyay, what was the issue ?
22:36oremjbunch of defunct ssh processes
22:36salty or
22:36saloremj:
22:38nthomasbm104 cleared the queue out fine, looking good
22:41* nthomas runs some retry_dead_queue with fabric
22:47nthomasphilor: the command queues are all green now, thanks for letting us know
22:48philornthomas: np, reopened, few enough dead builds I can stand the risk
22:50* nthomas goes back to afk
23:03philoryou know what would be fun? something other than "if taskcluster broke the same way it broke last night, while buildbot was being fixed"
23:48catleeThanks nthomas
21 May 2017
No messages
   
Last message: 1 day and 16 hours ago