mozilla :: #testpilot

15 May 2017
15:12fzzzyCaspy7: what do you mean other types of videos?
15:15Caspy7fzzzy: minvid currently supports youtube, vimeo (iirc) and pure HTML5 vids (there may be a few others, I don't remember) and it was planned to support many others last I knew
15:15Caspy7https://github.com/meandavejustice/min-vid/issues/331
15:15fzzzypure html5 vids seems to cover a lot of stuff, not sure what else there would be?
15:16Caspy7I feel like most of the videos I encounter nowadays are on custom players
15:16Caspy7other than youtube
15:16fzzzyoh, so it doesn't support any random html5 video yet?
15:16Caspy7correct
15:17fzzzyaha
15:17fzzzywell it sounds like a good thing to do, but I don't know the status of it :)
15:19Caspy7supporting more custom players gets a bit more complex than vanilla HTML
15:28vladikoffjgruen, xpi: https://github.com/Natim/hoverpad-sidebar/releases/download/1.0/hoverpad-sidebar-1.0.xpi
16:02_6a68clouserw: you are frozen-ish for me, super low frame rate
16:02_6a68I can hear you just fine though
16:02chuckIf clouserw was on Minecraft. https://irccloud.mozilla.com/file/Vnq9ihhS/Screen%20Shot%202017-05-15%20at%2010.01.30%20AM.png
16:03clouserw:(
16:03_6a68could the loud typer please mute? :-)
16:06fzzzynetwork issues
16:07clouserwJSON_voorhees: meeting in vidyo
16:17lorchardfzzzy: Oh btw, hope it didn't seem too abrupt / rude to hijack your PR :)
16:17fzzzylorchard: absolutely not! it looks great
16:24fzzzyi'm stoked that the gulpfile can use es6 now
16:24_6a68fzzzy: I guess we have a few minutes now before the internal meeting starts :-)
16:25_6a68To kinda summarize what I was thinking on the call, we can provide experiments (and the main addon) with a test pilot bootstrapped wrapper, until the APIs are available from the webextension framework directly
16:26fzzzy_6a68: sounds good to me
16:26_6a68Since we have to move the main addon away from the SDK, we get to take a step in that direction for free
16:26fzzzyso the telemetry stuff would be in the bootstrap part?
16:26_6a68Yeah, and we'd use the embedded webextension postmessage port for communication
16:26fzzzysounds great. I'll start work on that later this week.
16:26chuckWe'd also want to put the Heartbeat ratings in the wrapper.
16:27_6a68As well as the other stuff that wasn't quite ready in webextensions yet, I want to say it was the survey notification popups?
16:27_6a68fzzzy: cool, ping me if I can help with the code extraction, or just to talk through stuff
16:27fzzzywill do
16:27_6a68or, I can help with reviews if needed :-)
16:27fzzzychuck: is the hearbeat rating the survey notification popup?
16:27chuckI don't know what you mean by that.
16:28chuckIt's the 5-star rating prompt that pops you over to a SurveyGizmo survey.
16:28fzzzyyou said we'd want to put the Heartbeat ratings in the wrapper, and _6a68 mentioned the survey notification popups. are they the same thing?
16:28_6a68yup, that's what I was referring to as well
16:28fzzzycool
16:29_6a68lorchard: were there any other things in the test pilot addon that weren't webextensions yet? IIRC it was just surveys and metrics?
16:29_6a68and I guess, managing installation, which now is a website feature
16:30lorchardYeah, I think mainly it's metrics. Though, we are missing a webextension API to do the kind of notification bar UI - could adapt that to an embedded webextension though
16:31lorchardSeems like what's left is metrics, survey prompts, new experiment / news item badging or notification
16:31chuckYeah, I was suggesting that if we provide an Embedded WebExtension wrapper to experiments with the goal of getting rid of the Test Pilot extension that we'd want to make sure it included that.
16:31_6a68ah right, badging!
16:31lorchardOh and the installation management kind of boils down to removing experiments when Test PIlot is uninstalled
16:32_6a68if we put a star on the test pilot icon, instead of the "new" label, we could just toggle a different icon when there are new addons available
16:32chuckIf we get rid of the Test Pilot addon, you can't uninstall Test Pilot, so... :shrug:
16:32_6a68jgruen: wasn't there conversation recently about localizing that 'new' label?
16:32fzzzyhaha good point chuck
16:32lorchardWell, requiring a wrapper for *experiments* is a different set of issues
16:33chuckThat's what _6a68 was suggesting doing.
16:33lorchardBut also yeah, if we could make the test pilot addon just, like, evaporate... that would be a plus
16:33_6a68lorchard: the idea there is so that we don't have to do cross-addon communication to send metrics
16:33lorchardWe've avoided making requirements like that for *experiments* so as not to dictate how they're built, so that would be a change
16:33lorchardBut... maybe that's what we need to do
16:33_6a68lorchard: if we had a Telemetry API and a Survey API in the embedding bootstrapped part, the test pilot addon wouldn't be necessary
16:33lorchard(considering the coming SDK-pocalypse)
16:34chuckThat doesn't put many requirements on how experiments are built, does it? You can still do bootstrap-only, or you could do all WebExtension, or both.
16:34_6a68chuck: yeah, I'm thinking we could let our bootstrap.js provide hooks, so individual experiments could add their own bootstrap.js that fires as well
16:34lorchardMainly noting that we haven't imposed *any* requirements until now
16:34_6a68yeah, this is something I'd want to bring up at the next experiment dev meeting to gauge enthusiasm
16:35_6a68since they are either graduating or rewriting to remove the SDK, now's a good time to offer a change from our end
16:35lorchardSo imposing it as a requirement, we'd need to gin up examples, docs, etc and make sure everyone's on board
16:35lorchardKind of thinking about the logistics for coming experiments
16:35_6a68I think it would be pretty simple for experiments: instead of copying snippet A to submit metrics, they'd use snippet B
16:35lorchardbut if that made the main test pilot add-on go away, that would be a win IMO
16:36_6a68yup, the only slight weirdness might be multiple experiments trying to show surveys at the same time, since there won't be a global timer
16:36lorchardI also thought I read that we can't do things like switch a pure webextension to another add-on type as an upgrade
16:36lorchardi.e. if we wanted to take Snooze Tabs and turn it into an embedded webextension after release
16:36_6a68lorchard: good question, I can look into that
16:37lorchardCould maybe have a shared scratchpad convention between experiments as a soft lock on things like that
16:38chuck_6a68: I don&#39;t think that&#39;s possible if you use <notificationbox>
16:38chuckThere&#39;s a queue for them and you set priorities and such.
16:43_6a68I asked about moving from pure to embedded webextensions, and it sounds like there are things we could do. In particular, aswan mentioned we could flip a pref to not lose storage, then uninstall the pure webextension, reinstall an embedded webextension with the same addon ID
16:44lorchardWould be cool if we ended up with a create-test-pilot-experiment starter repo kind of thing as an end product of building required wrapper stuff
16:45lorchardHmm, what would flip the pref, though? :)
16:46_6a68Chatting with folks in teamaddons, it sounds like API experiments might be available sooner, if we really want them
16:47lorchardI kind of prefer the API experiments, if it turns out that they&#39;re a productive on-ramp to suggesting new APIs
16:48lorchardBut if it&#39;s not really helpful and embedded webext is easier...
16:53_6a68OK! change of plans, John-Galt says we could have APIs in 55, if we need them
16:53_6a68and that they could whitelist experiment addon IDs until signing is ready
16:53_6a68so we could have a Telemetry API experiment, and avoid all this bootstrapped craziness
16:53fzzzylorchard: it looks like https://github.com/mozilla/testpilot/pull/2425 will merge cleanly without rebasing. it looks good, should I just merge it?
16:53fzzzy_6a68: nice
16:53lorchard@fzzzy: Yeah sure
16:54lorchardthat should be completely dead & unused code for months now
16:54lorchardI may have been the only person who ever used it :)
16:54fzzzydone
16:54fzzzylorchard: the other pr doesn&#39;t merge cleanly, I&#39;ll review it once you rebase
16:55lorchard_6a68: I think the other thing was a mechanism for bundling both a webextension and a webextension API experiment in the same install
16:55lorchard(I guess we could finesse that on our end)
16:55_6a68lorchard: cool, I need to file a bugzilla bug about this. I&#39;ll ask what the upgrade path would look like, along with other questions
16:57fzzzy_6a68, lorchard, jgruen, clouserw: oh, I meant to mention in the meeting but forgot
16:58fzzzyhttps://github.com/mozilla/testpilot/pull/2380 has had a hard time getting the integration tests working on circle, but they work ok on travis. Can we get a clear answer on why we need to use circle? could we get some help from someone that knows circle?
16:59lorchardWell, we need to use circleci because ops asked us to, but I think the reason for it has gone away
16:59clouserwright, I think the reason was to build docker containers
16:59clouserwrelud is our ops support
16:59lorchardI&#39;ve been meaning to look into that PR, would prefer to figure out why it doesn&#39;t work on circleci than switch
16:59lorchardBecause the CI on circle does more than run tests - it also deploys to dev servers
16:59fzzzysure, either getting it working on circle or switching is fine, let&#39;s just do one or the other :)
17:00lorchardI just haven&#39;t carved out the time to look at it again yet, so maybe this week I&#39;ll look again
17:00fzzzythat&#39;d be sweet
17:00lorchardI suspect fixing that would be less work than uprooting the whole thing
17:02lorchardAlso, rebased 2411, looked like a small conflict
17:03fzzzythanks! i&#39;ll review!
17:12fzzzylorchard: switching the gulpfile to use es6 makes the build much faster. a++
17:13lorchardwoot
17:13lorchardThere&#39;s more ES6-ish things we could do in those tasks, but I kind of did just the minimal changes to get react working there
17:14fzzzyyeah obviously not a high priority to spiff things up that way
17:18_6a68Hmm. It sounds like webextension API experiments will likely be separate addons for 55. I guess the testpilot site could install them separately, but if users remove them, metrics and surveys will break for all the experiments
17:18_6a68I&#39;m looking but haven&#39;t found the tracking bug for this feature yet
17:23fzzzyhmm that&#39;s a monkey wrench
17:25lorchardYeah, they&#39;re separate dependent add-ons
17:26lorchardWondering if we can finesse that somewhat from the website / make experiments self-disable if they can&#39;t reach their associated webextension experiment APIs
17:27_6a68OK, this bug just got filed, sounds like it&#39;ll be a stretch for 55 https://bugzilla.mozilla.org/show_bug.cgi?id=1364976
17:27firebotBug 1364976 NEW, kmaglione+bmo@mozilla.com Allow API experiments to be bundled in the same XPI as the WebExtension that uses them
17:29_6a68Asking about options for detecting and showing some warning UI when users uninstall
17:29_6a68I guess we could show a warning popup that says, &quot;stuff is broken with this API missing, click here to return to testpilot.firefox.com and reinstall&quot;
17:31_6a68But wait! It turns out there is a bug to hide mozilla signed extensions from about:addons!
17:31_6a68And that could be done for 55
17:31_6a68https://bugzilla.mozilla.org/show_bug.cgi?id=1363624
17:31firebotBug 1363624 NEW, aswan@mozilla.com Add-on manifest field for hiding Mozilla Extensions from the add-on listing
17:31_6a68And shield wants it for 55 also
18:08dcoates_6a68: fzzzy: lmk when y&#39;all are going to talk txp addon
18:08_6a68dcoates: we did sort of talk about it earlier
18:09_6a68the outcome is that the webextensions team can make webextension API experiments invisible on the about:addons page for 55 (bug 1363624)
18:09firebothttps://bugzil.la/1363624 NEW, aswan@mozilla.com Add-on manifest field for hiding Mozilla Extensions from the add-on listing
18:09_6a68so we don&#39;t need to use a bootstrapped wrapper, we can just use the webextension API approach to provide Telemetry and Survey APIs
18:10dcoatescool. are we going to ditch the button then?
18:11_6a68dcoates: that wasn&#39;t decided, the question was more about whether to go embedded or not when moving away from the SDK. the main complication, aside from writing Gecko-ish code, is that there&#39;s not a clean upgrade path from webextension to embedded webextension
18:14dcoates_6a68: can you unpack &quot;not a clean upgrade path&quot; more?
18:14_6a68webextensions would need to be fully uninstalled to become embedded webextensions (though you can, I am told, flip a pref to avoid deleting existing storage when doing so)
18:15_6a68on the other hand, for webextensions to start using webextension API experiments, we&#39;d just need to add &quot;experiments.foo&quot; to the manifest for the webextensions
18:15lorchardThough also, I think our only pure webextension experiment right now is Snooze Tabs. Everything else is already embedded / SDK right?
18:16dcoates_6a68: you&#39;re talking about for txp experiments, not the txp addon, yeah?
18:18_6a68dcoates: I was thinking we could use the same API experiments for the main txp addon as well
18:20dcoatesyeah, ok. I don&#39;t think txp experiments should have to be embedded. api experiments seem better... what i&#39;ve been hoping for
18:20_6a68we&#39;d specifically have one experiment to provide a Telemetry API, another to provide a Survey API. we could keep a main txp addon to provide the button, for now
18:22dcoateswhich is basically what i was getting at with my POC. except that was a single addon that had the api and button
18:22_6a68yeah
18:23_6a68brb
18:36* _6a68 files a bug to summarize
18:46_6a68OK, another chat with teamaddons has corrected me again
18:46_6a68They want to just add the APIs to firefox, then whitelist addon access to them
18:54dcoatesi feel like we ought to still have a testpilot api
18:59_6a68dcoates: interesting. would that just wrap the telemetry and survey apis?
18:59lorchardI thought we were working toward whitelisting webext experiments for test pilot?
19:00lorchardBecause there will be other APIs besides replacements for the Test Pilot add-on that we want to build & work with
19:01lorchardFeeling like we might have to have a mtg or something to nail that down, since I think I&#39;ve heard different things over time as to what we should expect to be able to sue
19:01lorchards/sue/use/
19:02dcoates_6a68: yes, but i wouldn&#39;t say &quot;just&quot;
19:03_6a68lorchard: yeah, I was just looking and saw that the addons team has a webextensions API public meeting tomorrow at 1030, I might go to discuss this stuff https://wiki.mozilla.org/Add-ons/Contribute/Triage
19:05_6a68dcoates: what else would you want from a testpilot API experiment?
19:29dcoatesthe flexibility to not be bound to ff releases. so the next unknown feature that testpilot (the team) wants to offer experiments (A/B testing as a strawman example). where does it go? its own api experiment maybe, but then carry that forward a bit and now experiments depend on N other api experiments.
19:29lorchardYeah, that&#39;s what I&#39;m thinking about webext experiments
19:29lorchardIf we have to ride the trains to get APIs, then we&#39;re riding the trains
19:32dcoatesi think telemetry and survey api should *only* go into ff if release addons want to use them. otherwise it just slows us down imho
19:34dcoatesand if they do, great! but we still should have a place for newer or specialized apis
19:37dcoatesto be clear im not talking about api experiments in general, just api&#39;s that the testpilot infrastructure offers (not individual experiments and apis they might propose)
19:41chuckThe benefit to exposing them as a library instead of a Test Pilot experiment API: making backwards-incompatible changes becomes much less burdensome.
19:53dcoateschuck: what do you mean by library in this context?
19:56fzzzyA library that was loaded into a bootstrap extension could be individually updated in each bootstrap extension without depending on the firefox version. different bootstrap extensions could have different versions of the library installed without problems.
19:57chuckExactly.
19:57chuckvs an API provided by the browser to Test Pilot, which I think is what you were implying.
19:58lorchardWhat I was talking about is a WebExtension experiment - an API shipped as an add-on, which is another thing altogether :)
19:58dcoates1. it&#39;d kinda suck to make all experiments embedded extensions imo. 2. you can achieve the same result with versioned apis
19:59chuckShipping a Test Pilot WebExtension API provides the same versioning problems.
19:59chuckWhy not let NPM handle that for us? It&#39;s a solved problem.
20:02dcoatesi would agree if embedded webextensions didn&#39;t suck
20:02lorchardAlso, we don&#39;t get node-style packages in add-ons without dictating a build process
20:03lorchardOr at least requiring that there is one
20:03chuckdcoates: show your work on that statement?
20:04dcoateshttps://github.com/dannycoates/darkwing ?
20:05chuckI mean, why do you say they suck? You can&#39;t say something like that without backing it up.
20:09dcoatescommunicating between the bootstrap end and the webext end isn&#39;t as nice as using an api from a webext
20:09chuck&quot;Requires a few more lines of code&quot; != &quot;sucks&quot;
20:10dcoatesnot just &quot;a few more lines&quot; it changes how you need to structure the code
20:10fzzzyasync vs sync
20:10fzzzynot too onerous nowadays but a factor
20:10dcoatesmessaging vs function call
20:14chuck&quot;Isn&#39;t as nice&quot; and &quot;changes&quot; doesn&#39;t substantiate &quot;sucks&quot;, though.
20:14chuckThe former is subjective and the latter is not clearly a pro or con.
20:16dcoatesi meant it in a relative sense, when writing code, a plain webext api is more favorable. sucks was a poor choice of words
20:18dcoatesas far as i understand embedded webextensions aren&#39;t meant to be a permanent solution, more a stepping stone from sdk, etc, to webext
20:18chuckIn a vacuum, sure, but it requires us to either version every single API that makes a breaking change or tightly couple experiment versions to versions of the Test Pilot API.
20:18lorchardSomething else I think we need to figure out is whether the future of in-product feature development looks more like a webextension or a bootstrapped add-on
20:25lorchardSeems like if we build a library, and we need to change it, then we have to get every experiment to update & deploy a new release
20:26fzzzythat seems like less hassle than having to have a firefox that matches the api version an experiment expects
20:26lorchardIf we had a webextension experiment API add-on to update, we could minimize that - e.g. if the change is to fix a telemetry call or something that *doesn&#39;t* make an incompatible change
20:27lorchardI don&#39;t think having the API in firefox itself is really what we want anyway
20:27fzzzyit seems to me bootstrapped add-ons are going to have to be around for a while longer since webextensions are so limited. unless we restrict ourselves to experimenting with only things that are possible in webextensions. which seems really limiting
20:27lorchardUnless it&#39;s a very stable API that public webextensions shoul duse
20:27fzzzyaha, so the api would be provided by the webextension experiment add-on, i see
20:27lorchardfzzzy: WebExtension experiments are bridge in that scenario
20:28* fzzzy nods
20:29lorchardOf course then, we&#39;re juggling multiple add-ons for any given experiment. Kind of like we have now, but worse if an experiment needs test pilot + other experimental APIs
20:33lorchardOne of the things I&#39;m thinking is if a Test Pilot experiment surfaces the need for a new WebExtension API, building that API as a WebExtension Experiment could grease the skids to adopting the API in-product
20:33lorchard(assuming an in-product API looks at all code-wise like a WebExtension Experiment API)
20:35fzzzyyeah plus we dogfood webextension experiments
20:36lorchardBut, if in-product feature development looks more like a bootstrapped add-on, maybe that&#39;s a better base for test pilot experiments \(_o)/
20:37lorchardI think we&#39;re going to be doing some awkward splits between how FIrefox gets extended in the future vs how it gets developed under the hood :)
20:37_6a68wow such scrollback
20:45fzzzymuch discuss. wow
20:52fzzzywe should switch our jsx-using files to .jsx so github diffs aren&#39;t so ugly
20:54lorchardYeah, we have an ancient issue open for that
20:55lorchardhttps://github.com/mozilla/testpilot/issues/1376
20:57_6a68dcoates: good point about not wanting our APIs to ride the trains
20:59lorchardYeah, like I think our whole thing is to make things that don&#39;t ride the trains, so we can figure out if they&#39;re worth buying train tickets for in the first place
21:14_6a68That makes sense to me
21:14_6a68I&#39;m not clear, from the scrollback, whether we in general want to use API experiments or embedded webextensions
21:15_6a68API experiments have the advantage that the experiment code would be *just* a webextension, but it would require the APIs being backwards-compatible
21:16_6a68some kind of embedding bootstrapped wrapper around each experiment could pick up version changes whenever an individual experiment needed to ship a release, so that part seems nicer
21:16_6a68But the bootstrapped wrapper introduces complexity
21:17_6a68One nice thing about a bootstrapped wrapper is that experiments will need migration help to go from SDK to webextension
22:56fzzzy_6a68: I think a create-testpilot-test script that created a bootstrap extension that included the stuff we want to expose to webextensions, and exposes promise-based api to the webextension. but we should probably still discuss pros and cons.
22:57fzzzythe bootstrap extension can include a particular version of our library using npm. and people can get to work writing the webextension immediately without messing with the bootstrap at all
23:03_6a68fzzzy: yeah, think that&#39;s definitely the best option for individual experiments. We can dogfood the concept with the main test pilot addon rewrite
23:04fzzzy
23:04_6a68The only thing we&#39;d need would be to land a whitelist of test pilot addons before 57 hits nightly
23:05_6a68er, the only thing we&#39;d need from external teams
23:05fzzzyseems doable
23:05fzzzymaybe we can even preallocate random ids for new tests, so we have one already landed in firefox when we want to launch a test
23:05fzzzyassuming people are ok with random addon ids
23:06_6a68well, supposedly by 57, there will be magic powers associated with any addon that&#39;s signed with the mozilla key
23:06_6a68so hopefully we won&#39;t need to whitelist at all
23:06fzzzyah, mnice
23:06fzzzys/m//
23:06_6a68but as a worst case backup, it&#39;s not much work
23:07_6a68side note, I hate thrashing between projects, but there&#39;s so much happening all at once that needs discussion :-P
23:08fzzzyi hear that
23:08fzzzythrashing is the worst
23:08fzzzyjust get more ram
23:11_6a68heh, my brain has no more memory slots
23:11_6a68fzzzy: hey, did you ever get fluent localization sorted for the txp website?
23:11_6a68I think I&#39;ve finally cracked it for screenshots
23:12fzzzy_6a68: no, we&#39;re still on l20n and probably need a meeting about what&#39;s blocking fluent-react
23:12fzzzystas said something about not wanting to use dangerouslySetInnerHTML which would be required because of our experiment yaml files, or something.
23:12_6a68cool. hopefully I will have landed fluent-react (plus a getText-like workaround for non-react settings) by the time that meeting happens
23:13_6a68yeah, so there are spots in screenshots where a string is set inside, say, a model
23:13_6a68well, I should make sure the code fully works before I talk about it :-) but I&#39;m finally feeling optimistic
23:13fzzzythat&#39;s awesome that you&#39;ve integrated fluent-react into screenshots
23:14_6a68yeah, it&#39;s like one of six things I&#39;m trying to get done atm :-P
23:14fzzzyI don&#39;t really understand what the problem is. If there are things that aren&#39;t known until runtime currently we should preprocess them at build time instead i guess
23:14_6a68if you&#39;re not sure what the problem is, I guess a meeting is definitely necessary!
23:15fzzzyyeah that&#39;s why i want a meeting so everyone understands what the situation is
23:15JSON_voorheeschuck: I made a post in github, but sending here in case you miss it(notifications can get to be a lot in gh). Would you be open to have a meeting to discuss and make a decision on what do use for the news feed? as in wordpress or static generator etc..
16 May 2017
No messages
   
Last message: 9 days and 15 hours ago