mozilla :: #build

17 May 2017
00:32gandalfgps: do you want to be involved in the planning of the new langpack build system? (bug 1365440)?
00:32firebothttps://bugzil.la/1365440 NEW, nobody@mozilla.org Generate webextension language packs
01:15gandalfglandium: thanks for feedback! I'll let :pike respond as I don't understand fully what libs-% do :)
10:40decoderis it expected that our bootstrap system doesnt work on 32 bit?
10:40decoderfails to download rust compiler
10:41ted32-bit what
10:42decoderlinux
10:43decoderhttps://irccloud.mozilla.com/pastebin/8pRtFLiW/
10:46decodermust be a bug in our bootstrap
10:46decoderseems like installing it manually with the shell script that rust provides, it works
11:36froydnjI'm not sure we even support building on 32bit linux
12:18larsbergAt least the deps that servo uses build for 32 bit arm Linux fine. We've done a bunch of testing on pandaboard, etc.
12:19larsberg(Specifically the Linux panda distro, OT just their android image)
12:22froydnjI know people (e.g. Debian) do builds on 32-bit linux...I just don't know if we've really put effort into supporting that
12:28larsbergYeah pretty sure it's not tier 1 as I have memories of landing upstream SM build fixes for 32 bit Linux bustage.
12:29larsbergBecause we require arm32 and arm64 Linux http://build.servo.org/builders
12:29larsbergThough we don't test, just build :)
14:40Silne30I am using git cinnabar. What branch do I check out for beta?
14:45mshalSilne30: does 'git branch -a | grep beta' show anything?
14:53Silne30mshal: remotes/mozilla/beta
14:55mshalSilne30: so 'git checkout mozilla/beta' will get you there in a detached state, or 'git checkout --track mozilla/beta' if you want a local branch created to track beta
14:56mshaland maybe just double check to make sure it looks like you're at the latest beta commit according to https://hg.mozilla.org/releases/mozilla-beta/
14:58Silne30mshal: Thank you. Really appreciate it.
14:58mshalnp!
15:56froydnjmshal: I am planning on responding to your configure patches, just trying to get to a point where I can play around with it a little
15:56mshalty :)
15:56froydnjmshal: I think the primary case I cared about was having patches in the queue that modified old-configure.in or some of moz.configure, rebasing to squash patches, and then having to basically rebuild everything when the tree hadn't actually changed
15:58mshalfroydnj: I think that is mostly covered by this patchset, though I'm definitely interested to hear if it actually ends up useful in practice. There are still cases where changing old-configure.in and such would rebuild world
15:58mshalie: if a define is added or removed that ends up in mozilla-config.h
15:59mshaland this patchset only moves one define out of there as an example
17:08gandalfted: ping
17:25tedgandalf: pong
17:26gandalfted: can you help me with https://bugzilla.mozilla.org/show_bug.cgi?id=1365440#c2 ?
17:26firebotBug 1365440 NEW, nobody@mozilla.org Generate webextension language packs
17:26* ted looks
17:46tedgandalf: so i think glandium's only real request here is that we don't add more automation jobs that call directly into make
17:46tedif you wrap whatever you want to do into a mach target or something that should be fine
17:46gandalfI barely understand our build system, sorry
17:46tedeven if it's calling make targets under the hood, as long as that's internal to the build system
17:47tedgandalf: for the l10n side of things that makes you and everyone else :)
17:47gandalfas far as I understand, "libs-%" is just getting the right .dtd/.properties files into objdir/dist/xpi-stage/ ?
17:47gandalfand glandium doesn't want us to use it?
17:47gandalfand I'm not sure what should we use instead to traverse the jar.mn's and get their content in place?
17:52nalexandergandalf: you'll still invoke |make libs-...| under the hood; we just want the _entry point_ to be |mach build stuff-for-l10n|
17:52gandalfthe entry point is "make langpack-%"
17:52nalexandergandalf: so that all the paths in are accounted for in the mach commands, not {mach commands, Makefile targets, build scripts}
17:53gandalfahhh
17:53gandalfso you want ./mach langpack-% ?
17:59nalexandergandalf: something like that. Probably |mach build langpack-LANG|, since it'll just turn into a |mach build MAKE-TARGET| under the hood.
18:01tedyeah, or you could add a special command `mach langpack LANG` or something like that if you want to wrap it a little further
18:01tedmight make it easier for you to do nicer things in the future
18:01tedif you implement your own mach command you can write a bunch of stuff in python without invoking make directly
18:01tedeven if you have to shell out to make to do some of the work
18:02tedgandalf: so re: libs-%: https://dxr.mozilla.org/mozilla-central/source/browser/locales/Makefile.in#96
18:02tedthat's the makefile in firefox that includes l10n.mk
18:03gandalfmhm
18:03gandalfok, so there are two things:
18:03gandalf1) change what we do when we build a langpack
18:03gandalf2) move the command to be a mach command
18:03gandalfis that correct?
18:04tedthere's also http://searchfox.org/mozilla-central/source/toolkit/locales/Makefile.in#15 in toolkit/locales which does some other stuff for libs-%
18:04tedthat's where crashreporter.ini comes from
18:05tedgandalf: yeah, sounds like it
18:05tedgandalf: you can start out just wrapping the existing stuff in a mach command, and then change the bits you need to change as you go
18:05gandalfok
18:05tedgandalf: if you haven't, you should talk to rhelmer about webextension packaging
18:05gandalfhe's NI'ed in the bug
18:05tedhe's been doing a bunch of work with building system addons
18:05tedgreat :)
18:06gandalfthanks a lot!
18:06gandalfone issue we have is that libs-% do more than we need in a langpack
18:06tedyw! happy to help answer any quesitons i can
18:06gandalfis there any best-practice for that kind of scenario?
18:06gandalfshould I just add step to remove the files we don't need later?
18:06gandalfor some argument to libs-% to filter out what we don't need?
18:14tedi feel like it might sort of depend on how you actually wind up implementing the webextension build part
18:15tedif you do need to run `make libs-AB_CD` then yeah, you might need to give yourself some extra options there
18:15gandalfI'll use libs-% to get files in xpi-stage dir
18:15gandalfthen create manifest.json
18:15gandalfthen zip it
18:15gandalfbut somewhere in the middle I need to remove all the files I don't need :)
18:15gandalfok, thanks!
18:15tedyou could do something like `make libs-AB_CD WEBEXTENSION_LANGPACK=1` and then wrap bits you don't want in`ifndef WEBEXTENSION_LANGPACK`
18:16gandalfok, sweet, thanks!
18:39rhelmergandalf: we've got a bunch of build stuff in the tree that already works for making XPIs... for built-in add-ons I am working on making it possible to package them into and then load them from the omni jar
18:40rhelmerthere are a lot of assumptions around chrome registration etc. that probably won't be helpful for you
18:42gandalfrhelmer: the new langpacks will be webextensions
18:42rhelmeryeah, if this is for optional langpacks I think you could use the build system as-is
18:42gandalfrhelmer: I summarized the plan here: https://bugzilla.mozilla.org/show_bug.cgi?id=1365440#c4
18:42firebotBug 1365440 ASSIGNED, gandalf@aviary.pl Generate webextension language packs
18:42rhelmerand avoid the chrome registration stuff
18:43rhelmergandalf: cool. that's exciting, I'd love to get rid of the existing langpack stuff :)
18:43gandalfwe will make WebExtensions Gecko code load the manifest.json and register entries for ChromeRegistry
18:43gandalf:)
18:43gandalfand start planning to move away from chrome registry overall for l10n
18:43rhelmerheh yeah, it moves it into the webextension impl code but that's fine
18:43rhelmernice!
18:43rhelmerat least then we can improve the impl without having to fix every dang extension/langpack
18:44gandalfif you have any change suggestions for my plan, please put in the bug! :)
18:44rhelmerwill do
18:45rhelmergandalf: so moz.build already supports making XPIs if that's helpful... you need to call the output file "chrome.jar" which is kinda weird but it works
18:45gandalfI don't understand
18:45gandalfI'm not making XPI
18:45rhelmerhm, I am confused then too.
18:46gandalfI'm making a webextension .zip file
18:46gandalf(I know .xpi is a .zip, but I thought XPI indicates "old style extension")
18:46rhelmernope we still use xpi
18:46rhelmerthey are just zip (or jar) files though
18:46gandalfwhat is "xpi" then? :)
18:47rhelmerwe'll load zip that's fine, xpi is just so you can identify them (file handlers on OSes etc)
18:47gandalfyeah, so for the new langpack approach that I'm developing, we want to create .zip files
18:47gandalfregular webextensions
18:47gandalfno install.rdf, no chrome.manifest etc.
18:47rhelmerwebextensions as served from AMO use XPI
18:47gandalfno bootstrap.js :)
18:48rhelmeryeah gotcha, that's good :)
18:48gandalfok
18:48rhelmerwe didn't think it was worth changing extension, as virus scanners and file associations etc. all have XPI registered
18:48gandalfso how moz.build can help me building my webextensions?
18:48rhelmersimilarly chrome uses CRX but they are just zip files
18:49rhelmergandalf: I don't know if it'll be useful for your case (I need to read the bug etc) but I will give you an example
18:50rhelmeroops I said jar.mn but gave a moz.build example about (jar.mn is for chrome registration which you don't want)
18:51rhelmergandalf: so for instance for screenshots (which is a webextension plus a very minimal bootstrap.js and doesn't need to do chrome registration) just uses the moz.build http://searchfox.org/mozilla-central/source/browser/extensions/screenshots/moz.build#7-10
18:51rhelmerin a regular build that ends up as unpacked files (with symlinks etc) to make hacking easy
18:52gandalfhmm, is that better than my plan to just put everything in xpi-stage using libs-% and then zip it?
18:52rhelmerduring packaging it gets turned into a xpi (it's called jar internally everywhere, and technically we use jar signing format etc)
18:52rhelmergandalf: naw probably not
18:52gandalfok :)
18:52gandalfthen I think I'll just stick to it
18:52rhelmercool. I'll follow along, curious to see how it works out
18:53rhelmerthat is interesting that you will generate manifest.json
18:54gandalfyeah!
18:54gandalfI have no idea how mozbuild actions work, but I know python so I hope I'll be able to write it :)
18:54rhelmergandalf: the only concern I have with zip is if you host them somewhere it'll be confusing since we still use xpi everywhere else
18:54rhelmerit's not really a big deal though
18:54gandalfrhelmer: we'll host them on AMO
18:55rhelmeryeah just check w/ them then, I'll let you folks figure it out :)
18:55gandalfJohn-Galt and aswan are guiding me through the process of consuming those new langpacks
18:55rhelmergreat
18:55gandalfI'll file a bug for that today. First wanted to iron out how we'll build them
18:55gandalf:)
18:55gandalfrhelmer: I'd also love to get your take on bug 1365433
18:55firebothttps://bugzil.la/1365433 NEW, nobody@mozilla.org Decide on how to handle localization "packages"
19:08rhelmergandalf: I know AMO can take zip files for the input, they will probably serve the signed output as XPI files so I think it'll work fine from your pov
19:08gandalfcool! :)
19:08rhelmerwe want to make it easier for folks to make one .zip and upload it to chrome and firefox (and other!) stores
19:09gandalfI don't think it matters for langpacks yet. We're talking to google about our l10n API but I don't expect anything to happen anytime soon :)
19:09rhelmercool I will take a look, sorry going a little slow rn, lots of review requests and 57 related stuff going on.
19:10gandalfyeah, I totally understand
19:10rhelmeroh yeah for sure, I just mean you generating zip files for upload to AMO will probably be totally fine
19:10gandalfI'm trying to prepare as much as possible ahead of 57, since we want to start transitioning Firefox right after the quantum/photon craze cools down
19:10gandalfbut it's lower urgency than 57 of course :)
19:10rhelmerthat will be great to have more modernized langpack infra
19:11gandalfonce in 20 years I think it does make sense :D
19:11rhelmercool. yeah I have a lot of startup perf related stuff to look at (and then probably fallout to deal with in :P)
19:11rhelmerwhich is more quantum flow related I guess
19:11gandalfyep
19:12gandalfhigher urgency and priority :)
19:12rhelmerI am really glad you're working on this though! langpacks have been problematic forever
19:12gandalfI'll keep you up to date and you can just skim through the bugs in case you want to alter the direction
19:12rhelmerthanks!
19:14gandalfas you work on perf, can you plan for early langpack registering pls :)
19:14gandalfthe current model is pretty unfortunate perf wise since we have at least 5 stages of language negotiation during startup :(
20:18RyanVMfml, I just realized I can use vswhere without having to muck around with json at all
20:24Silne30RyanVM: Hey man. Do you use git-cinnabar?
20:24RyanVMno
20:24Silne30Dang.
20:24RyanVMwhat's your question?
20:25Silne30I am using git-cinnabar.
20:25Silne30Checked out the beta branch.
20:25Silne30git pull just hangs.
20:25RyanVMwhich OS?
20:26Silne30Linux.
20:26Silne30Doesn't happen on central.
20:26RyanVMhrm, dunno, sorry
20:38RyanVMted: I'm realizing now that I might be able to kill two birds with one stone after all here with respect to my need to detect visual c++ and winsdk during mozillabuild packaging
20:38dvanderI've asked this before I think, but I think I need a bit more guidance:
20:39RyanVMvcvars lives on in VC\Auxiliary\Build !
20:39dvanderGFX is looking to add more shaders, which basically will result in a 2MB text blob being checked into the source tree every time we change them. it's not ideal.
20:39dvanderwhat's the best way we can run the shader compiler as a build step? it's just a .sh script.
20:39froydnjdvander: is the shader compiler available everywhere?
20:40dvanderfroydnj: yup, pretty sure
20:40dvanderit comes with visual studio
20:40froydnjI feel like those statements contradict each other :)
20:40froydnjI guess you mean always available on the platforms we care about the shaders for
20:40froydnj?
20:40RyanVMwas wondering if tooltool is an option otherwise, depending on how often it changes
20:40dvanderfroydnj: right, sorry yes. these are d3d11 shaders.
20:41dvanderRyanVM: they change almost never
20:41dvanderbut when they do its at least a 200KB patch of just hex salad
20:41tedgps will probably not like you if you check them in
20:41froydnjtooltool would be an option, but that would be inconvenient for local builds
20:41froydnjrunning the shader compiler at build time is totally an option
20:41tedit's sort of a bummer to compile them if they almost never change, it means everyone gets to spend more build time doing that :-/
20:42dvanderit's about 10 seconds, it's not bad
20:42* RyanVM wonders what'll happen with clang-cl too (not to mention mingw)
20:42teddvander: it adds up!
20:43dvanderI guess, I would expect them to change a little more often now that we're doing more GPU stuff
20:43dvanderso it's a tradeoff between a ~2MB blob file changing infrequently and a small build cost on windows
20:44tedthere's no perfect answer here, tbh
20:45teddvander: are we importing them from some other place, or are these developed in-house and just compiled to that form for usage?
20:45dvanderted: they are developed in house. right now we run the compiler before checkin if we change these files
20:45tedgotcha
20:45tedthat would indicate that we should probably build them at build time
20:45tedwe've gone around on this before for generated files, it makes it more of a hassle to work with and people make mistakes etc
20:46dvanderyeah...
20:46dvanderif that's preferable, I'd love to do the integration but am not sure how. I couldn't easily find prior art
20:46froydnjtoo bad we don't have commit hooks to either (a) regenerate them at commit time or (b) ensure that you always change the source and generated file in a commit unless IKNOWWHATIMDOING is in the commit message or something
20:48teddvander: so the current state of the art is GENERATED_FILES
20:48tedwhich basically just lets you say "run this python script to generate this file"
20:49tedand in the python script you can do whatever
20:49tedfroydnj: ideally we'd just have a smart build system that pulled them from cache unless you changed them
20:49froydnjted: that too
20:49dvanderted: oh cool. can that handle dependencies?
20:50teddvander: yes
20:50dvanderperfect
20:51tedhttps://dxr.mozilla.org/mozilla-central/source/netwerk/dns/moz.build#63 for example
20:51tedplenty of examples in the tree
20:51dvandergreat, thanks
20:51tedhttp://gecko.readthedocs.io/en/latest/build/buildsystem/mozbuild-symbols.html#generated-files
20:51tedhas OK docs on it
20:52tedyour python function can return a set() of other files that should be listed as deps
20:54dvanderare moz.build scripts full-fledged python? are there restrictions/guidelines on what not to do in them?
20:55nalexanderdvander: very much not full-fledged Python. There's a very limited sandbox.
20:55nalexanderdvander: but the GENERATED_FILES script is full-fledged Python.
20:56rilliando we have a preference between CFLAGS and LOCAL_INCLUDES for -I lines in moz.build?
20:56tedrillian: prefer LOCAL_INCLUDES, it lets you use nice mozbuild paths
20:58rillianted: is that the magic / and jjj:q
20:58tedi see you're a vi user
20:59rillianand a mac user, where switching keyboard focus is hard :)
20:59tedhah
20:59tedbut yeah, that's 'relative/path', '/topsrcdir/relative', '@objdir/path' etc
20:59rillianthe magic / and whatever the other obsolute->objdir-relative path things
20:59rillianright
21:00rillianI guess python is more 'implicit is better than explicit'?
21:01rillianted: are LOCAL_INCLUDES passed to yasm as well and the C compiler?
21:01Silne30glandium: I am trying to use git-cinnabar to work with beta.
21:02Silne30fetching and pulling hangs on beta.
21:02Silne30But not on other branches.
21:03rillianSilne30: pulling from the unified repo on my mac has been extremely slow the past two months or so. It looks like a hang, but then succeeds 10-30 minutes later.
21:04gpsbeta has tons of heads
21:04gpsyou should also complain about VCS issues in #vcs, where more eyeballs are looking for them
21:06Silne30rillian: Oh wow. Thanks for the heads up.
21:06Silne30gps: Will do.
21:26Fallenwhat make target forces TEST_HARNESS_FILES to be copied? For some reason it is not working for me for a certain directory and a normal make doesn't seem to do it
21:33mshalFallen: they're installed via an install manifest, which should be done during export I believe
21:34mshalor maybe those are delayed until you run a test now?
21:35mshalFallen: oh, bug 1242051 looks like it changed that
21:35firebothttps://bugzil.la/1242051 FIXED, cmanchester@mozilla.com Only install test files as we need to run them
21:54Fallenhm so the entries correctly show up in the _build_manifest, but is still not copied. Do I understand correctly that if I run any kind of test then the manifest should be processed? Or do I need support-files and an .ini now instead of TEST_HARNESS_FILES/
21:59rillianlooks like LOCAL_INCLUDES doesn't propagate to yasm
22:07FallenMy issue may just be local issues. Even without my patches those files are not being copied while the same cset works on automation, so nevermind :)
22:27nalexanderFallen: at one point |./mach build install-tests| did the manifest install for me locally.
22:27nalexanderYou should see something like 0:15.68 Elapsed: 5.29s; From _tests: Kept 10 existing; Added/updated 26381; Removed 0 files and 0 directories.
22:43Silne30rillian: Hmmm...been a couple of hours.
22:43Silne30Almost.
22:46rillianSilne30: maybe a different issue then. I'm on 100 Mbps networks in case that affects the speed.
22:46rillianI ran one when you asked earlier and it took 7 minutes to complete.
22:51Silne30i am on a gigabit connection. But I normally get about 250Mbps down.
22:51Silne30Nothing else takes this long.
22:52Silne30I can clone mozilla-central in 10 minutes.
23:28glandiumSilne30: what commands are you running?
18 May 2017
No messages
   
Last message: 5 days and 4 hours ago