mozilla :: #releng

11 Jul 2017
00:17rstrongCallek: I get an error triggering https://tools.taskcluster.net/hooks/project-releng/nightly-desktop-osx%2Foak
00:17rstrongIs there anything else I need to do to trigger it? Says not enough scopes
00:19Callekrstrong: can you create a bug with me needinfod with the scope error message. I suspect there is some minor issue here that won't be too hard to fix. In the meantime i suspect sheriff's or releng can trigger for you.
00:20KWiersorstrong/Callek: done
00:28rstrongthanks... I'll file soonish
00:32catleerstrong: we may be able to get rid of modules/libbz2 altogether
00:32rstrongyep and I mentioned that to mhowell... in another bug though
00:32catleeah, cool
00:49cpetersonRyanVM: who maintains Socorro's function name skip list for crash signatures? Looks like we should add Rust function `core::option::expect_failed`to the list, e.g. https://crash-stats.mozilla.com/report/index/03718a9c-9d98-4832-b8a6-026220170706
00:55RyanVMcpeterson: looks like people often submit PRs themselves for it
00:55cpetersondo you know where?
00:55RyanVMhttps://github.com/mozilla-services/socorro/tree/master/socorro/siglists#signatures-utilities-lists
00:55cpetersonRyanVM: thanks!
00:56RyanVMlooks like :adrian is someone you can ping for help if needed
01:06cpetersonI'll ask some Stylo developers if there are other Rust functions we should skip for crash signatures.
01:24travis-cibuild-tools#1865 (master - 05c3a47 : ffxbld): The build passed. (https://travis-ci.org/mozilla/build-tools/builds/252233289)
09:17jlorenzoTomcat|sheriffduty: good morning! I just rotated the windows signing keys for nightly. It'll be picked up by next nightly. I remain ready to revert the changes if something breaks.
09:17Tomcat|sheriffdutyok :) will keep an eye on this
09:20jlorenzoTomcat|sheriffduty: There's also another change made on Google Play. The APK might not be uploaded correctly today. If that's the case, jcristau and I know what's going on
09:21Tomcat|sheriffdutyok
09:49travis-cibuild-puppet#1633 (master - 933cb8d : Nick Thomas): The build was broken. (https://travis-ci.org/mozilla/build-puppet/builds/252334663)
09:50nthomasboo
09:57travis-cibuild-puppet#1634 (master - afd7e46 : Nick Thomas): The build was fixed. (https://travis-ci.org/mozilla/build-puppet/builds/252336867)
10:06travis-cibuild-puppet#1635 (production - e415d28 : Nick Thomas): The build passed. (https://travis-ci.org/mozilla/build-puppet/builds/252339500)
12:50catleeaselagea|buildduty: hey, can you merge bbconfigs to prod to trigger a reconfig?
12:51aselagea|builddutycatlee: on it
12:53aselagea|builddutydone
12:53travis-cibuild-buildbot-configs#2730 (master - 21bae0e : Chris AtLee): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/252389814)
12:57catleeaselagea|buildduty: thank you!
12:57travis-cibuild-buildbot-configs#2731 (production - 4ee31be : Alin Selagea): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/252391124)
12:57aselagea|builddutynp
13:08bhearsumTIL https://pypi.python.org/pypi/aioresponses/0.1.5
13:08bhearsumhelpful for async testing
13:53Callekbhearsum: ooo nice
13:53* Callek will have to try and remember all these asyncio things when i finally dive in to try and learn them better
13:53Callek:-)
14:20catleerwood: I think `if not branch.get('enable_talos')` is going to be safer in general than `if branch.get('enable_talos', False) is False`
14:21catleedo we know for sure that all branches with talos disabled have enable_talos set to False?
14:22catleelike maybe some branch has enable_talos set to None
14:25rwoodcatlee: ah, alright thanks I'll update it again sorry
14:26catleenp
14:26catleethanks!
14:26catleegenerally the only thing I use 'is' to compare to is None
14:33rwoodcool good point
14:34catleeand then we need to keep the new tc talos configs in sync :)
14:40rwoodcatlee: right, thanks for the review, I'll land the buildbot configs now, and the corresponding taskcluster patch after
14:41catleerwood: yeah, I mean that we're just enabling talos for windows on the date branch
14:41catleeso I'll port your changes over there for windows once we're ready
14:41catleewill probably need an entry here: https://hg.mozilla.org/projects/date/rev/14a74a96b5e3eaff292ba1ac3ca1efbccec2138a#l2.12
14:42rwoodcatlee: oh I see, what is the date branch
14:43catleerwood: that's where we're working on the windows tc migration
14:49rwoodah, I didn't know that, cool
14:51Callekrwood: we're targetting tier 1 on central before august, fwiw -- so we're being pretty aggressive there atm
14:52Callek* for builds
14:52Callek(incl nightlies)
14:58travis-cibuild-buildbot-configs#2732 (master - 08a9047 : Rob Wood): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/252438670)
15:00rwoodCallek: awesome thx, j.maher did mention that, I didn't know there was sep branch tho but that makes sense
16:09rwoodaobreja|buildduty: ping
16:10aobreja|builddutyrwood:pong
16:10rwoodaobreja|buildduty: looks like my patch broke autoland, can I bother you for backout of it please?
16:10rwoodhttps://treeherder.mozilla.org/#/jobs?repo=autoland&revision=feaa8a920f0687dcb9d79987587a2add8d811fb9
16:10rwoodsorry :(
16:11aobreja|builddutyok
16:11rwoodthanks!
16:22aobreja|builddutyrwood: I'm sorry I just figured out that I can't backout a patch from autoland as I don't have level3, ask please someone else,a sheriff or someone with these rights
16:23rwoodok thanks aobreja|buildduty
16:23KWiersorwood: hi
16:24rwoodKWierso: hey
16:24rwoodYou've probably seen the backscroll, are you the sheriff on duty
16:24KWiersojust getting online for the day, but yeah
16:25rwoodKWierso: sorry for the breakage, broke the gecko decision task on autoland :(
16:33rwoodKWierso: thanks for the backout, much appreciated
16:34KWiersonp
16:34KWiersonow to figure out why my bash shell is refusing to start on my main computer...
18:24catleehwine: hey, you've spent some time looking at windows signatures
18:24catleewould you mind looking at the signatures on our TC builds?
18:26travis-cibuild-buildbot-configs#2733 (master - 5fc0546 : Chris AtLee): The build passed. (https://travis-ci.org/mozilla-releng/build-buildbot-configs/builds/252520161)
18:50jlorenzocatlee: hwine: bhearsum is correct. I switched nightly today at around 10am UTC. I wanted to wait before changing release, but I don't mind doing it tomorrow if that helps
18:50hwinejlorenzo: no worries on timing
18:51jlorenzoso only the latest nightly got changed
18:51hwinejlorenzo: do we have the new cert (not the keys) anywhere on a bug?
18:51hwineit would be good to have that available for checking the serial number, etc.
18:51jlorenzohwine: no I haven't uploaded it on a bug. It's possible to do so, though
18:53hwinethat would be good, otherwise I have no way to trust the new serial number ;)
18:55catleehwine: so you should have seen the new cert for nightlies today
18:55catleeno?
18:55* catlee tries timezone math in his head
18:55hwineyes - that explains why all the installers "failed" :)
18:56hwinegreat test of the code -- could not have planned it better! \o/
18:56hwinebut also glad we haven't hooked up "page on failure" yet :D
18:59catleeoh, great
19:19jlorenzoI'm glad planets aligned that well :)
19:31gpsso, uh, https://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-27-03-02-04-mozilla-central/firefox-55.0a1.en-US.win64.json and https://archive.mozilla.org/pub/firefox/nightly/2017/05/2017-05-26-03-02-03-mozilla-central/firefox-55.0a1.en-US.win64.json are two different win64 nightlies (different dates on archive.mo URLs) but they have the same buildid. what's up with that?
19:32gpsyet the win64.installer.exe have different hashes
19:33RyanVMboth coming off the same rev is maybe relevant? still sounds like a bug somewhere, though
19:33bhearsumthat's probably just a timing issue...
19:33RyanVMfriday and saturday
19:33RyanVMI could see having two nightlies off the same rev if we were having issues on that Friday or something
19:33gpsbut sharing the same build id?
19:33RyanVMyeah, that's what seems weird
19:33gpsi thought buildid was determined at run time
19:34bhearsumbuildbot may do it differently....
19:34gpshaving multiple builds with the same buildid is not great :/
19:34gpsthere's no other (easy) way to disambiguate
19:35bhearsumhttps://github.com/mozilla-releng/build-buildbotcustom/blob/master/misc_scheduler.py#L82 seems like it tries to avoid it, so i'm not sure
19:35bhearsumi could see it being a duplicate if someone hit "rebuild"
19:36bhearsumbut that seems unlikely given the upload dirs
19:36gpsbut the builds were ~24 hours apart
19:36gpsaccording to the timestamps on archive.mo
19:39gpsi can get a list of builds that violate the unique (platform, buildid) constraint :)
19:40gpsin 2017, only that win64 nightly
19:42gpsin fact, going back to 2013, i can't find any other examples
19:42gpsout of curiosity, what is https://archive.mozilla.org/? does it hit a CDN?
19:42gpsi've noticed that subsequent HTTP requests are much faster than initial. i suspect some form of caching at the server.
19:43bhearsumit's all backed by cloudfront IIRC
19:43gpsyeah, that would explain it
19:44gpsthey definitely cache recent objects at the receiving server
19:44gpswin64 nightly for 20160111030207 also has multiple entries
19:44gpsas does win32 nightly for 20160411030231
19:45bhearsumi'm probably not the best person to help debug this - but i definitely thing it's worth filing
19:45gpswin32 for 20160723030206
19:45gpswhich component?
19:47bhearsumRelEng: General Automation
20:00catleehwine: what are you using the check the windows cert?
20:00catleepython module, or ???
20:01hwinePython lambda function, also usable from client
20:02hwinemeat of the code is from Google code
20:03gpsbug 1380123 filed
20:03hwineyou actually have a review request for it on GitHub ;)
20:03firebothttps://bugzil.la/1380123 NEW, nobody@mozilla.org Multiple Nightly releases on archive.mo sharing same buildid
20:04catleehwine: I do?
20:14catleehwine: point me to it again?
20:24hwinecatlee: my bad - I guess I only thought about it :( That said, I wouldn't mind a sanity check on the approach (i.e. is approach capable of doing what I claim)
20:25hwinecatlee: https://github.com/mozilla-services/fx-sig-verify
20:25akiyou wrote something in not-bash! :D
20:26hwineyeah -- and you thought that was an earthquake you felt -- it was the whole world shifting ;)
20:55catleewow, both these modules for checking sigs are horrible
20:55catleeI know, I'll write my own!
20:55catlee:P
20:55hwineancient code -- long unmaintained :)
20:55catleehwine: did you have to patch google's module much?
20:56hwinea couple of minor syntax, iirc
20:56catleeyeah, I found out that pefile is creating millions of copies of data in memory
20:56catleewhich is why it eats up 1.5G to validate a 60MB file
20:56hwinereally - that&#39;s odd - on lambda it runs in <50MB
20:57catleedifferent module
20:58catleewe&#39;re using https://github.com/erocarrera/pefile
21:00hwineah, I picked the right one, then :)
21:00akiyeah, and i had to create a special release of that to run on py3, because he landed a fix but didn&#39;t push to pypi
21:02hwine:( -- well, actually even better if I&#39;m using a completely different code base
21:02* hwine has possible plans to move to py3 now that lambda supports it
21:21tomprinceI have a couple of builds failing due to something like https://bugzilla.mozilla.org/show_bug.cgi?id=654538
21:21firebotBug 654538 NEW, nobody@mozilla.org Crash with SIGABRT incorrect checksum for freed object - object was probably modified after being fr
21:22akimight be a question for #build
21:24tomprinceOkay. I asked here first, since they both occur on the same slave and https://bugzilla.mozilla.org/show_bug.cgi?id=787281#c11 suggests that it might be a bad slave.
21:24firebotBug 787281 FIXED, nobody@mozilla.org remove slaves with hardware problems from the pool
21:30tomprince bld-lion-r5-027 failed with a malloc error twice on try-comm-central today.
21:42catleeaki, hwine: different python module can check for the signature in 0.02s and 9MB memory
21:42catleelet&#39;s use that one :)
21:44hwinemeh - won&#39;t change the lambda cost for me ;)
21:44hwinebut which one?
21:44catleeit&#39;s also a somewhat active project, although the pefile support is py2 only for now
21:44catleeconstruct
21:44catleehttps://github.com/construct/construct/blob/stable25/construct/formats/executable/pe32.py
21:46akihm, maybe py3 signtool can shell out for py2?
21:46hwinewhat about pe64?
21:46akior we can port it
21:47catleeyeah, I&#39;m going to submit a PR to fix up the py3 support
21:47catleepe64 is just an extension - it looks like this finds the signature on a 64-bit binary just fine
21:48catleepeplus
21:50akitomprince: might be a buildduty bug to take it out of the pool and fix
21:50* hwine doesn&#39;t have any 64bit in my test data yet
21:50hwine&quot;PE32+&quot; according to my version of &#39;file&#39;
21:51tomprinceaki: Should I file a bug for that then?
21:51akiyes please
21:56tomprinceaki: https://bugzilla.mozilla.org/show_bug.cgi?id=1380166
21:56firebotBug 1380166 NEW, nobody@mozilla.org bld-lion-r5-027 failing in malloc
12 Jul 2017
No messages
   
Last message: 71 days and 14 hours ago