mozilla :: #build

17 May 2017
00:32gandalfgps: do you want to be involved in the planning of the new langpack build system? (bug 1365440)?
00:32firebot NEW, Generate webextension language packs
01:15gandalfglandium: thanks for feedback! I'll let :pike respond as I don't understand fully what libs-% do :)
10:40decoderis it expected that our bootstrap system doesnt work on 32 bit?
10:40decoderfails to download rust compiler
10:41ted32-bit what
10:46decodermust be a bug in our bootstrap
10:46decoderseems like installing it manually with the shell script that rust provides, it works
11:36froydnjI'm not sure we even support building on 32bit linux
12:18larsbergAt least the deps that servo uses build for 32 bit arm Linux fine. We've done a bunch of testing on pandaboard, etc.
12:19larsberg(Specifically the Linux panda distro, OT just their android image)
12:22froydnjI know people (e.g. Debian) do builds on 32-bit linux...I just don't know if we've really put effort into supporting that
12:28larsbergYeah pretty sure it's not tier 1 as I have memories of landing upstream SM build fixes for 32 bit Linux bustage.
12:29larsbergBecause we require arm32 and arm64 Linux
12:29larsbergThough we don't test, just build :)
14:40Silne30I am using git cinnabar. What branch do I check out for beta?
14:45mshalSilne30: does 'git branch -a | grep beta' show anything?
14:53Silne30mshal: remotes/mozilla/beta
14:55mshalSilne30: so 'git checkout mozilla/beta' will get you there in a detached state, or 'git checkout --track mozilla/beta' if you want a local branch created to track beta
14:56mshaland maybe just double check to make sure it looks like you're at the latest beta commit according to
14:58Silne30mshal: Thank you. Really appreciate it.
15:56froydnjmshal: I am planning on responding to your configure patches, just trying to get to a point where I can play around with it a little
15:56mshalty :)
15:56froydnjmshal: I think the primary case I cared about was having patches in the queue that modified or some of moz.configure, rebasing to squash patches, and then having to basically rebuild everything when the tree hadn't actually changed
15:58mshalfroydnj: I think that is mostly covered by this patchset, though I'm definitely interested to hear if it actually ends up useful in practice. There are still cases where changing and such would rebuild world
15:58mshalie: if a define is added or removed that ends up in mozilla-config.h
15:59mshaland this patchset only moves one define out of there as an example
17:08gandalfted: ping
17:25tedgandalf: pong
17:26gandalfted: can you help me with ?
17:26firebotBug 1365440 NEW, Generate webextension language packs
17:26* ted looks
17:46tedgandalf: so i think glandium's only real request here is that we don't add more automation jobs that call directly into make
17:46tedif you wrap whatever you want to do into a mach target or something that should be fine
17:46gandalfI barely understand our build system, sorry
17:46tedeven if it's calling make targets under the hood, as long as that's internal to the build system
17:47tedgandalf: for the l10n side of things that makes you and everyone else :)
17:47gandalfas far as I understand, "libs-%" is just getting the right .dtd/.properties files into objdir/dist/xpi-stage/ ?
17:47gandalfand glandium doesn't want us to use it?
17:47gandalfand I'm not sure what should we use instead to traverse the's and get their content in place?
17:52nalexandergandalf: you'll still invoke |make libs-...| under the hood; we just want the _entry point_ to be |mach build stuff-for-l10n|
17:52gandalfthe entry point is "make langpack-%"
17:52nalexandergandalf: so that all the paths in are accounted for in the mach commands, not {mach commands, Makefile targets, build scripts}
17:53gandalfso you want ./mach langpack-% ?
17:59nalexandergandalf: something like that. Probably |mach build langpack-LANG|, since it'll just turn into a |mach build MAKE-TARGET| under the hood.
18:01tedyeah, or you could add a special command `mach langpack LANG` or something like that if you want to wrap it a little further
18:01tedmight make it easier for you to do nicer things in the future
18:01tedif you implement your own mach command you can write a bunch of stuff in python without invoking make directly
18:01tedeven if you have to shell out to make to do some of the work
18:02tedgandalf: so re: libs-%:
18:02tedthat's the makefile in firefox that includes
18:03gandalfok, so there are two things:
18:03gandalf1) change what we do when we build a langpack
18:03gandalf2) move the command to be a mach command
18:03gandalfis that correct?
18:04tedthere's also in toolkit/locales which does some other stuff for libs-%
18:04tedthat's where crashreporter.ini comes from
18:05tedgandalf: yeah, sounds like it
18:05tedgandalf: you can start out just wrapping the existing stuff in a mach command, and then change the bits you need to change as you go
18:05tedgandalf: if you haven't, you should talk to rhelmer about webextension packaging
18:05gandalfhe's NI'ed in the bug
18:05tedhe's been doing a bunch of work with building system addons
18:05tedgreat :)
18:06gandalfthanks a lot!
18:06gandalfone issue we have is that libs-% do more than we need in a langpack
18:06tedyw! happy to help answer any quesitons i can
18:06gandalfis there any best-practice for that kind of scenario?
18:06gandalfshould I just add step to remove the files we don't need later?
18:06gandalfor some argument to libs-% to filter out what we don't need?
18:14tedi feel like it might sort of depend on how you actually wind up implementing the webextension build part
18:15tedif you do need to run `make libs-AB_CD` then yeah, you might need to give yourself some extra options there
18:15gandalfI'll use libs-% to get files in xpi-stage dir
18:15gandalfthen create manifest.json
18:15gandalfthen zip it
18:15gandalfbut somewhere in the middle I need to remove all the files I don't need :)
18:15gandalfok, thanks!
18:15tedyou could do something like `make libs-AB_CD WEBEXTENSION_LANGPACK=1` and then wrap bits you don't want in`ifndef WEBEXTENSION_LANGPACK`
18:16gandalfok, sweet, thanks!
18:39rhelmergandalf: we've got a bunch of build stuff in the tree that already works for making XPIs... for built-in add-ons I am working on making it possible to package them into and then load them from the omni jar
18:40rhelmerthere are a lot of assumptions around chrome registration etc. that probably won't be helpful for you
18:42gandalfrhelmer: the new langpacks will be webextensions
18:42rhelmeryeah, if this is for optional langpacks I think you could use the build system as-is
18:42gandalfrhelmer: I summarized the plan here:
18:42firebotBug 1365440 ASSIGNED, Generate webextension language packs
18:42rhelmerand avoid the chrome registration stuff
18:43rhelmergandalf: cool. that's exciting, I'd love to get rid of the existing langpack stuff :)
18:43gandalfwe will make WebExtensions Gecko code load the manifest.json and register entries for ChromeRegistry
18:43gandalfand start planning to move away from chrome registry overall for l10n
18:43rhelmerheh yeah, it moves it into the webextension impl code but that's fine
18:43rhelmerat least then we can improve the impl without having to fix every dang extension/langpack
18:44gandalfif you have any change suggestions for my plan, please put in the bug! :)
18:44rhelmerwill do
18:45rhelmergandalf: so already supports making XPIs if that's helpful... you need to call the output file "chrome.jar" which is kinda weird but it works
18:45gandalfI don't understand
18:45gandalfI'm not making XPI
18:45rhelmerhm, I am confused then too.
18:46gandalfI'm making a webextension .zip file
18:46gandalf(I know .xpi is a .zip, but I thought XPI indicates "old style extension")
18:46rhelmernope we still use xpi
18:46rhelmerthey are just zip (or jar) files though
18:46gandalfwhat is "xpi" then? :)
18:47rhelmerwe'll load zip that's fine, xpi is just so you can identify them (file handlers on OSes etc)
18:47gandalfyeah, so for the new langpack approach that I'm developing, we want to create .zip files
18:47gandalfregular webextensions
18:47gandalfno install.rdf, no chrome.manifest etc.
18:47rhelmerwebextensions as served from AMO use XPI
18:47gandalfno bootstrap.js :)
18:48rhelmeryeah gotcha, that's good :)
18:48rhelmerwe didn't think it was worth changing extension, as virus scanners and file associations etc. all have XPI registered
18:48gandalfso how can help me building my webextensions?
18:48rhelmersimilarly chrome uses CRX but they are just zip files
18:49rhelmergandalf: I don't know if it'll be useful for your case (I need to read the bug etc) but I will give you an example
18:50rhelmeroops I said but gave a example about ( is for chrome registration which you don't want)
18:51rhelmergandalf: so for instance for screenshots (which is a webextension plus a very minimal bootstrap.js and doesn't need to do chrome registration) just uses the
18:51rhelmerin a regular build that ends up as unpacked files (with symlinks etc) to make hacking easy
18:52gandalfhmm, is that better than my plan to just put everything in xpi-stage using libs-% and then zip it?
18:52rhelmerduring packaging it gets turned into a xpi (it's called jar internally everywhere, and technically we use jar signing format etc)
18:52rhelmergandalf: naw probably not
18:52gandalfok :)
18:52gandalfthen I think I'll just stick to it
18:52rhelmercool. I'll follow along, curious to see how it works out
18:53rhelmerthat is interesting that you will generate manifest.json
18:54gandalfI have no idea how mozbuild actions work, but I know python so I hope I'll be able to write it :)
18:54rhelmergandalf: the only concern I have with zip is if you host them somewhere it'll be confusing since we still use xpi everywhere else
18:54rhelmerit's not really a big deal though
18:54gandalfrhelmer: we'll host them on AMO
18:55rhelmeryeah just check w/ them then, I'll let you folks figure it out :)
18:55gandalfJohn-Galt and aswan are guiding me through the process of consuming those new langpacks
18:55gandalfI'll file a bug for that today. First wanted to iron out how we'll build them
18:55gandalfrhelmer: I'd also love to get your take on bug 1365433
18:55firebot NEW, Decide on how to handle localization "packages"
19:08rhelmergandalf: I know AMO can take zip files for the input, they will probably serve the signed output as XPI files so I think it'll work fine from your pov
19:08gandalfcool! :)
19:08rhelmerwe want to make it easier for folks to make one .zip and upload it to chrome and firefox (and other!) stores
19:09gandalfI don't think it matters for langpacks yet. We're talking to google about our l10n API but I don't expect anything to happen anytime soon :)
19:09rhelmercool I will take a look, sorry going a little slow rn, lots of review requests and 57 related stuff going on.
19:10gandalfyeah, I totally understand
19:10rhelmeroh yeah for sure, I just mean you generating zip files for upload to AMO will probably be totally fine
19:10gandalfI'm trying to prepare as much as possible ahead of 57, since we want to start transitioning Firefox right after the quantum/photon craze cools down
19:10gandalfbut it's lower urgency than 57 of course :)
19:10rhelmerthat will be great to have more modernized langpack infra
19:11gandalfonce in 20 years I think it does make sense :D
19:11rhelmercool. yeah I have a lot of startup perf related stuff to look at (and then probably fallout to deal with in :P)
19:11rhelmerwhich is more quantum flow related I guess
19:12gandalfhigher urgency and priority :)
19:12rhelmerI am really glad you're working on this though! langpacks have been problematic forever
19:12gandalfI'll keep you up to date and you can just skim through the bugs in case you want to alter the direction
19:14gandalfas you work on perf, can you plan for early langpack registering pls :)
19:14gandalfthe current model is pretty unfortunate perf wise since we have at least 5 stages of language negotiation during startup :(
20:18RyanVMfml, I just realized I can use vswhere without having to muck around with json at all
20:24Silne30RyanVM: Hey man. Do you use git-cinnabar?
20:24RyanVMwhat's your question?
20:25Silne30I am using git-cinnabar.
20:25Silne30Checked out the beta branch.
20:25Silne30git pull just hangs.
20:25RyanVMwhich OS?
20:26Silne30Doesn't happen on central.
20:26RyanVMhrm, dunno, sorry
20:38RyanVMted: I'm realizing now that I might be able to kill two birds with one stone after all here with respect to my need to detect visual c++ and winsdk during mozillabuild packaging
20:38dvanderI've asked this before I think, but I think I need a bit more guidance:
20:39RyanVMvcvars lives on in VC\Auxiliary\Build !
20:39dvanderGFX is looking to add more shaders, which basically will result in a 2MB text blob being checked into the source tree every time we change them. it's not ideal.
20:39dvanderwhat's the best way we can run the shader compiler as a build step? it's just a .sh script.
20:39froydnjdvander: is the shader compiler available everywhere?
20:40dvanderfroydnj: yup, pretty sure
20:40dvanderit comes with visual studio
20:40froydnjI feel like those statements contradict each other :)
20:40froydnjI guess you mean always available on the platforms we care about the shaders for
20:40RyanVMwas wondering if tooltool is an option otherwise, depending on how often it changes
20:40dvanderfroydnj: right, sorry yes. these are d3d11 shaders.
20:41dvanderRyanVM: they change almost never
20:41dvanderbut when they do its at least a 200KB patch of just hex salad
20:41tedgps will probably not like you if you check them in
20:41froydnjtooltool would be an option, but that would be inconvenient for local builds
20:41froydnjrunning the shader compiler at build time is totally an option
20:41tedit's sort of a bummer to compile them if they almost never change, it means everyone gets to spend more build time doing that :-/
20:42dvanderit's about 10 seconds, it's not bad
20:42* RyanVM wonders what'll happen with clang-cl too (not to mention mingw)
20:42teddvander: it adds up!
20:43dvanderI guess, I would expect them to change a little more often now that we're doing more GPU stuff
20:43dvanderso it's a tradeoff between a ~2MB blob file changing infrequently and a small build cost on windows
20:44tedthere's no perfect answer here, tbh
20:45teddvander: are we importing them from some other place, or are these developed in-house and just compiled to that form for usage?
20:45dvanderted: they are developed in house. right now we run the compiler before checkin if we change these files
20:45tedthat would indicate that we should probably build them at build time
20:45tedwe've gone around on this before for generated files, it makes it more of a hassle to work with and people make mistakes etc
20:46dvanderif that's preferable, I'd love to do the integration but am not sure how. I couldn't easily find prior art
20:46froydnjtoo bad we don't have commit hooks to either (a) regenerate them at commit time or (b) ensure that you always change the source and generated file in a commit unless IKNOWWHATIMDOING is in the commit message or something
20:48teddvander: so the current state of the art is GENERATED_FILES
20:48tedwhich basically just lets you say "run this python script to generate this file"
20:49tedand in the python script you can do whatever
20:49tedfroydnj: ideally we'd just have a smart build system that pulled them from cache unless you changed them
20:49froydnjted: that too
20:49dvanderted: oh cool. can that handle dependencies?
20:50teddvander: yes
20:51ted for example
20:51tedplenty of examples in the tree
20:51dvandergreat, thanks
20:51tedhas OK docs on it
20:52tedyour python function can return a set() of other files that should be listed as deps
20:54dvanderare scripts full-fledged python? are there restrictions/guidelines on what not to do in them?
20:55nalexanderdvander: very much not full-fledged Python. There's a very limited sandbox.
20:55nalexanderdvander: but the GENERATED_FILES script is full-fledged Python.
20:56rilliando we have a preference between CFLAGS and LOCAL_INCLUDES for -I lines in
20:56tedrillian: prefer LOCAL_INCLUDES, it lets you use nice mozbuild paths
20:58rillianted: is that the magic / and jjj:q
20:58tedi see you're a vi user
20:59rillianand a mac user, where switching keyboard focus is hard :)
20:59tedbut yeah, that's 'relative/path', '/topsrcdir/relative', '@objdir/path' etc
20:59rillianthe magic / and whatever the other obsolute->objdir-relative path things
21:00rillianI guess python is more 'implicit is better than explicit'?
21:01rillianted: are LOCAL_INCLUDES passed to yasm as well and the C compiler?
21:01Silne30glandium: I am trying to use git-cinnabar to work with beta.
21:02Silne30fetching and pulling hangs on beta.
21:02Silne30But not on other branches.
21:03rillianSilne30: pulling from the unified repo on my mac has been extremely slow the past two months or so. It looks like a hang, but then succeeds 10-30 minutes later.
21:04gpsbeta has tons of heads
21:04gpsyou should also complain about VCS issues in #vcs, where more eyeballs are looking for them
21:06Silne30rillian: Oh wow. Thanks for the heads up.
21:06Silne30gps: Will do.
21:26Fallenwhat make target forces TEST_HARNESS_FILES to be copied? For some reason it is not working for me for a certain directory and a normal make doesn't seem to do it
21:33mshalFallen: they're installed via an install manifest, which should be done during export I believe
21:34mshalor maybe those are delayed until you run a test now?
21:35mshalFallen: oh, bug 1242051 looks like it changed that
21:35firebot FIXED, Only install test files as we need to run them
21:54Fallenhm so the entries correctly show up in the _build_manifest, but is still not copied. Do I understand correctly that if I run any kind of test then the manifest should be processed? Or do I need support-files and an .ini now instead of TEST_HARNESS_FILES/
21:59rillianlooks like LOCAL_INCLUDES doesn't propagate to yasm
22:07FallenMy issue may just be local issues. Even without my patches those files are not being copied while the same cset works on automation, so nevermind :)
22:27nalexanderFallen: at one point |./mach build install-tests| did the manifest install for me locally.
22:27nalexanderYou should see something like 0:15.68 Elapsed: 5.29s; From _tests: Kept 10 existing; Added/updated 26381; Removed 0 files and 0 directories.
22:43Silne30rillian: Hmmm...been a couple of hours.
22:46rillianSilne30: maybe a different issue then. I'm on 100 Mbps networks in case that affects the speed.
22:46rillianI ran one when you asked earlier and it took 7 minutes to complete.
22:51Silne30i am on a gigabit connection. But I normally get about 250Mbps down.
22:51Silne30Nothing else takes this long.
22:52Silne30I can clone mozilla-central in 10 minutes.
23:28glandiumSilne30: what commands are you running?
18 May 2017
No messages
Last message: 5 days and 4 hours ago