mozilla :: #media

17 Jul 2017
02:43jw_wangalwu should know better.
02:43jw_wangabout audio channel.
02:45alwuyes, it's used for managing audio competing on b2g
03:01Caspy7so I didn't realize there were *3* forks of b2g now
03:01Caspy7https://en.wikipedia.org/wiki/Firefox_OS#Forks
03:02Caspy7a new feature phone just came out running KAI OS http://techpp.com/2017/07/13/reliance-jio-4g-volte-feature-phone/
03:03rocI think Acadine's one is pretty much dead, but Kai OS seems kind of alive
03:07Caspy7The specs and price point of that phone remind of the...I don't remember the code name. The $25 one.
03:07Caspy7except it was a smart phone and this is technically a feature phone
03:12kentuckyfriedtakahecpearce: I just noticed the existence of https://packages.ubuntu.com/artful/chromium-codecs-ffmpeg
03:13cpearcekentuckyfriedtakahe: presumably that's for the chromium browser, not the chrome browser.
03:14kentuckyfriedtakahecpearce: I'm guessing that makes Chromium able to play H.264/AAC
03:57cpearcejhlin: Hi! What technology do we use to run the android video decoders out of process? Is it IPDL based, or do we use some android framework that takes care of the IPC for us?
04:08kentuckyfriedtakahekinetik: how do we investigate issues like bug 1373829?
04:08firebothttps://bugzil.la/1373829 UNCONFIRMED, nobody@mozilla.org Audio/Video stutter while surfing on other Tabs
04:14kinetikkentuckyfriedtakahe: i guess working out if it's cpu or network related would be a start
04:16kentuckyfriedtakahekinetik: it is a fast machine with media.benchmark.vp9.fps: 213
04:18kinetikkentuckyfriedtakahe: that doesn't tell us much
04:32kinetikkentuckyfriedtakahe: the log from dxdiag might be useful, and knowing which specific machine this is also, since there might be something funny with the audio on it (https://twitter.com/x47nero/status/881939681600045057)
06:22bwucpearce: IIRC, jhlin created a lightweight Android service(process) in java layer to do that.
10:02achronoppadenot: when you can I would like to talk with you about webaudio channels
10:02padenotanytime !
10:02padenotI'm back now
10:02achronopnice
10:03achronopmy problem is that I nee to differentiate when an GraphDriver is called for webaudio and when for webrtc
10:03padenotwhy /
10:03padenot>?
10:03achronopthe issue is that webrtc does not accept anything more than 2 channels
10:04padenotwe're talking about audio output here
10:04achronopthe way that we do it provide all the available channels
10:04achronopyeah
10:05achronopso when soundflower (64) is use for webrtc we assert
10:05achronoplet me find the point
10:07achronoppadenot: here: http://searchfox.org/mozilla-central/source/dom/media/webrtc/MediaEngineWebRTCAudio.cpp#109
10:08achronopa comment in the definition of MAX_CHANNELS says "These are restrictions from the webrtc.org code"
10:10padenotdo you know what this method does ?
10:12achronopthe output data from the callback ends up there through MediaEngineWebRTCMicrophoneSource::NotifyOutputData
10:16padenotyeah
10:16padenotit's to feed the reverse stream to the echo canceler
10:16padenotI'd say, just downmix
13:30padenotachronop, does that sound doable ?
13:30achronoppadenot: yeah it works
13:30padenotcool
13:30achronopthe problem is we add processing
13:31achronopand when we create an echo gUM we open for example 8 channels but we use the 2 of them
13:34padenotin your test case yes
13:36achronopI will test it using a multichannel stream made in webaudio
15:14Caspy7Report of very poor WebRTC performance in a new macbook pro here: https://www.reddit.com/r/firefox/comments/6nt2p5/webrtc_on_latest_macbook_pro_is_a_disaster_high/
15:32jibCaspy7: There seem to be two problems reported there, sound only in one speaker, and the cpu issue. It's hard to comment without more info. We should get them to file a bug (two bugs) with more info. FWIW I have a new MBP and hear in both speakers (using headset) just fine. My CPu usage for appear.in is about ~45% in Activity Monitor column
15:32jibdrno: ^
15:33achronopCaspy7: panning to right speaker is intentional, in older MBPs mic is located right next or above the left speaker, this is a way to avoid distortion when internal speakers are on
15:33jibah
15:33* jib was using headset
15:35achronopyeah, we do not pan on external speakers
15:37jibmakes sense. AFAICT the mic is next to the left speaker on my new MBP as well
15:38jibIs someone commenting on the reddit, or shall I?
15:43Caspy7jib: sorry, was away. I can do some copy and pasting, but followup questions may have me back here :P
15:43Caspy7achronop: what about newer MBPs?
15:44achronopjib commented just above that they have it in the same place
15:46achronopsource wise we do not differentiate between MBPs models so the same applies there
15:47drno|irccloudjib: the speaker thing sounds like the panning we do for the mic over the speaker. Does anyone know if the mic is still over the speaker in the most recent MBP?
15:47achronopit was an old patch, jesup can explain more about it
15:47jibdrno: yes I have a new MBP and the mic seems to be on the left only
15:47Caspy7jib, achronop: so good action points here would be to inform about intentional speaker behavior and encourage to file a bug for the CPU usage?
15:48jibyes
15:48jibwith version and platform info
15:48drno|irccloudjib: best is probably to double check with our new friends at Apple about hands free mode on modern MBP
15:49padenotdrno|irccloud, jib, we're investigating
15:51jibcool thanks
15:51drno|irccloudpadenot: I'll ping our new friends for a quick reference
15:51padenotdrno|irccloud, I think we have everything we need but time
15:51padenotas always
15:51jibhow would hands-free mode interfere?
15:54jib"unfortunately, it's the only browser that supports WebRTC on mac" - hmm
16:00jibachronop: Btw, unrelated while testing, I'm hearing distorted multi-second delayed audio in nightly just joining same appear.in call
16:00Caspy7jib, achronop: I left a comment, you're welcome to amend or elaborate
16:00jibCaspy7: cool thanks!
16:01achronopjib: do yo test between 2 tabs?
16:01jibI tested both
16:01padenotsame or different process ?
16:02jibin two tabs, and between lone tab and my phone
16:02padenotI suppose different if it's on different machines
16:02jibyeah it also affects simple fiddles once appear.in is connected to another peer
16:03jibI get crackling noises at first, and then there's several seconds of delay
16:03* jib checks if it's specific to appear.in
16:04jiboh it's even simpler to reproduce with apprtc because it turns on mic even before connecting to a peer,
16:05jibso all I have to do is open e.g. https://appr.tc/r/10415681 and the audio goes south, even in my plain https://jsfiddle.net/jib1/pz5pynyf/ in a different tab
16:06mjfdrno|irccloud: ping
16:06jibachronop: ^
16:06achronopyeah I'm testing
16:06drno|irccloudmjf: pong
16:07jibI'll do a mozregression
16:08achronopjib: I do not have it, do you use a headset?
16:09jibno
16:09* jib tries with headset
16:10jibsame problem
16:11achronoplet me update to latest Nightly
16:11jibI'm on 56.0a1 (2017-07-16) (64-bit)
16:14jibhmm, odd, not happening with mozregression. Let me doublecheck my assumptions
16:18jibachronop: It does not happen in a clean profile. I'm looking into what the difference is
16:39jibachronop: Ok, the problem only appears when "dom.ipc.processCount.web" is defined in about:config
16:40jibit doesn't seem to matter what value I set it to, the sound is bad, but if I "Reset" it then it fixes it
16:42achronopjib: what value did you have it?
16:43jibI've tried 4, I've tried 1, and I've tried 0, even empty string and they all cause problems
16:44jibit seems to be the very presence of the field that causes it
16:44jibwhen reproducing it, I make sure to close down to a single tab inbetween, and refresh that tab
16:45* jib tries to provoke it in a clean profile
16:47achronopjib: I have a dom.ipc.processCount and dom.ipc.processCount.webLargeAllocation
16:47jibnot those
16:47achronopnot a dom.ipc.processCount.web
16:48achronopshould I create it?
16:48jibyes
16:48jibinteger
16:48jibgive it e.g. a 1 value
16:48achronopshould I restart?
16:49jibno just close all but one tab and refresh it
16:50jibthen open e.g. https://appr.tc/r/336851128 in one tab, share, and open https://jsfiddle.net/jib1/pz5pynyf/ in another and share
16:50achronopapprtc is very slow now even without the fiddle
16:51achronopvideo is delayed
16:53jibYou don't need to actually connect
16:54jibjust open the tab to see self view
16:54jibjust share camera and mic
16:55achronopI do not get audio with appr.tc
16:56achronopeven it asks for the mic perm
16:56jibnot audible no
16:56jibbut open https://jsfiddle.net/jib1/pz5pynyf/ hear there
16:56jibin a second tab
16:56jibit's the presence of the appr.tc tab that does it
16:58jibthis is weird, the lone reference to that pref is https://dxr.mozilla.org/mozilla-central/source/toolkit/xre/nsAppRunner.cpp#5193 - looks like an override
16:58achronopso, audio is fine video is considerable slow
17:02jibachronop: what do you mean video is slow? are you connected to a peer?
17:02jesupjib: is that the number-of-content-processes config?
17:02jibactually, I can also reproduce by changing "dom.ipc.processCount" from the default 4 to 1
17:03jibjesup: yes, though I happened to be modifying an old override value at first
17:03jibbecause that's what I happened to have in my profile
17:03jibso it seems we work with multiple processes but not when the limit is lowered to 1
17:04jesupCaspy7: as jib and achronop mentioned, the panning on mbp's is quite intentional to reduce internal echo. drno's mic/speaker would vibrate and distort, for example, and in general it leads to a ton of "echo" making harder to get real signal for the person speaking
17:04jesupworks with 4, but not with 1??? Do you have something in another tab chewing CPU? CC/GC pain?
17:05jesupThis makes me think it's *something* in another tab/window that's causing the problem
17:05jib2 works, 4 works, 1 produces crackle/delay symptom
17:05jesupjib: calling between tabs? or between machines?
17:05jibno calls, just gum in two tabs
17:06jesupah, sharing gum.... I know they've been more-aggressively turning down setTimeout/etc in background tabs, but that shouldn't affect gum... I think
17:07jibThe STRs are: 1) start with no tabs (or in lone single tab) 2) open https://appr.tc/r/492910907 and allow cam+mic, then in a second tab open https://jsfiddle.net/jib1/pz5pynyf/ and allow cam+mic. And that's it
17:07achronopjib I have it with dom.ipc.processCount = 1
17:07jibyou can reproduce?
17:07achronopyeah
17:07jesupunless it's somehow related to billm's mainthread-queue-per-document stuff, which I didn't think was landed
17:07jibcool
17:07* jib tries mozregression again with this info
17:08jesupThis didn't use to happen that I know of... try Beta? Release?
17:08jesupmight be simpler/faster as a check before mozregression
17:08achronopjib: slow video means big latency when I move my hand and expecting to see it on screen
17:08jibwell I want the range ;)
17:08jesupsure, but that gives you a starting point
17:09jesupi.e. a --good :-)
17:17achronopjib: I have it also with my debug build
17:17jibgood
17:18achronopjib: I have to run for now but if you post here I will check later
17:18jibI have the range down to 2017-06-30, 2017-07-01 almost done
17:18jibok I'll file a bug
17:19achronopI have it only with dom.ipc.processCount = 1
17:19jibyes that's what I'm regression testing with
17:20jibI have it down to https://hg.mozilla.org/mozilla-central/pushloghtml?fromchange=d9a144b7b6d994fc9a497c53b13f51a2a654d85e&tochange=587daa4bdc4b40b7053f4ca3b74723ca747f3b52
17:20achronopsetting the dom.ipc.processCount.web did not repro
17:20jibI see some patches from padenot
17:20jibyes ignore that bit
17:20jibit's an override setting, another way to change the process count that happened to be how my profile changed it
17:20jibdom.ipc.processCount = 1 is the right repro step
17:21jibnot sure which patch it is yet, continuing
17:21jibjsut wanted to give an update if you're done for the day :)
17:22achronopmedia wise I see the channelCount constraint and a number of padenot's patches
17:22* jib is bisecting
17:23jibany bets?
17:23jib:)
17:24achronopit's a land in the middle of work week ... everything is open :)
17:25achronopI could backout media stuff and retry it
17:26jibI'll have more info in a sec (repro steps are a bit arduous)
17:26jesup 5de53323f3ad Paul Adenot Bug 1330360 - Create new MSGs for each nsPIDOMWindow. r=jesup
17:27firebothttps://bugzil.la/1330360 FIXED, padenot@mozilla.com Use multiple MediaStreamGraph in the same process
17:27jesupseparate MSG for each tab...
17:28achronophmm that's why I see a new cubeb stream with every new tab call
17:29achronopI though UpdateSingleSource would operate here and keep just one stream
17:30jesuppadenot: ^
17:32jibachronop, padenot: Final result: https://hg.mozilla.org/integration/mozilla-inbound/pushloghtml?fromchange=15cc8dd663627ec12ecd4be11a69f0caad8e4e11&tochange=29ac12c81bcda10a981dfe3371f70a9b17a19edd
17:32jib23:36.79 INFO: Last good revision: 15cc8dd663627ec12ecd4be11a69f0caad8e4e11
17:32jib23:36.79 INFO: First bad revision: 29ac12c81bcda10a981dfe3371f70a9b17a19edd
17:36jibyeah it reproduces more simply by just opening https://jsfiddle.net/jib1/pz5pynyf/ in two tabs actually
17:40jibachronop: yeah unsure how this interacts with UpdateSingleSource actually. https://hg.mozilla.org/mozilla-central/rev/5de53323f3ad seems to force a new MSG per window regardless
17:40jesupYup. exactly what I suspected
17:41jibhttps://hg.mozilla.org/mozilla-central/rev/5de53323f3ad#l5.90
17:44jibachronop: So this happens without modifying dom.ipc.procesCount as well (with the default value of 4). You just have to open https://jsfiddle.net/jib1/pz5pynyf/ in enough tabs. I get the problem after the 4th fiddle tab :(
17:45jibSo with separate MSGs, does that mean we can have e.g. echoCancellation in one and not another?
17:46jibtab
17:46* jib is unsure what the constraints modification model is here now
17:47jibi.e. where are device settings kept, and is there a singular set of them?
17:48jiblooks like we're being saved here by multiple processes (up to a point)
17:49* jib files a bug
18:12jesupjib: aec could in theory be on for some tabs, and off for others (pass through processing or not) -- but dividing all the sub-options (NS, no NS, AGC, no AGC) isn't likely worthwhile. In general, if any tab needs AEC all get it. With multi-process, this might not occur until lots of tabs are using it (very unusual)
18:23jesupI'm not shocked we had a problem like this, though I know padenot looked at perf issues.
18:23* jesup thinks he should have asked padenot to have a pref, however
18:24jesupthough perhaps a pref might add a fair bit of code, given some aspects of this. Perhaps not, so worth looking at
18:25jesupwe might need to disable this in beta, and a pref is much nicer than a backout, if it's a safe switch
18:26jesupthe restructuring is such that without testing with it preffed off it might not be safe to just flip the pref, so landing preffed off, then preffing on after we verify no regressions is a nice way to do this. But we can also add code to pref off now, pref it off, and then work to fix it and pref on
18:53jibjesup: "In general, if any tab needs AEC all get it." This isn't quite right, I never got my constraints tests green and landed (bug 1088621) but as I recall, if two tabs want a boolean audio setting off and one wants it on, then it's off
18:53firebothttps://bugzil.la/1088621 NEW, jib@mozilla.com Need constraint support for fake audio/video on test machines to test camera constraints & capabilit
18:54jibbut yeah multi process likely throws many of those assumptions out the window
18:55jibwell, a pref isn't necessarily safer in hindsight, the uplift patch is just smaller ;)
18:58jibsafest might be to revert the patch
18:58jibbut let's see what padenot and achronop say tomorrow
19:04jyajesup: : seeing the phershon isn't here. What's the rational behind webRTC with a media element firing loadedmetadata when it doesn't have yet a video resolution?
19:05jyaHmmm I now recall something like you don't yet know until it's too late if we're going to have only audio or both.
19:07jyaStill the specs are clear. Loadedmetadata is to be fired if we know the video dimensions.
19:15jibjya: I thought that worked. e.g. http://jsfiddle.net/jib1/9t3szw0L/57/ - when does it not work?
19:15jibor did you mean a remote peer connection track?
19:27jyajib: it's how it works
19:27jibwhat do you mean?
19:27jyaIt will call the code that check the readyState multiple times until it decides it's okay to change.
19:28* jib is not following
19:29jibDo you have a reproducible case where width and height are not ready?
19:29jesupjya: there were a lot of issues around loadedmetadata... I'm not sure I remember all the issues without rechecking old bugs (and IRC logs)
19:29jesupI know (ok IIRC) we used to fire it before sizes were available
19:30jesupbut I think that was changed....
19:30jyajib: that area of the code managing readyState has often been a sore point. With regression being introduced, simply because no one from media could understand why those special conditions were put in place.
19:31jyaSo I'm looking to clean that up. Plus it will make HTMLMediaElement more content agnostic, making it easier to rewrite (as the plan is to rewrite those doms in rust)
19:32* jib has not seen a case where it fired without width and height in years
19:33jyajib: a bit less than 3 years.
19:33jyaI've never been keen on the solution adopted.
19:34fippojesup: i recall a chrome issue where it refuses to play audio before receiving video data because of the "no loadedmetadata before video size is known" rule. which has quite nasty sideeffects...
19:34jesupjib: jya: yeah, we fixed that years ago
19:35jesupfippo: chrome does refuse to play audio until the video arrives, because they gate audio on loadedmetadata (or did as of last summer)
19:35jesupsome SFU/MCU services force a black frame through just to keep chrome happy
19:35jyabug 879717
19:35jesupIIRC
19:35firebothttps://bugzil.la/879717 FIXED, apehrson@mozilla.com drawImage on MediaStream assigned to <video> stopped working again
19:35jesupyeah, that was a pain (especially for captureStream)
19:36jyaI had that one marked as to be redone for years. So now that I&#39;m hovering over webRTC, I&#39;m going to scratch a itch.
19:36jesupI&#39;d talk to pehrsons (and perhaps roc)
19:36jesuproc: ping (not about this topic, but I see you&#39;re on)
19:37jyafippo: that seems to me that chrome behaviour is the technically correct one.
19:37jyaCertainly the most spec compliant.
19:37fippojesup: yeah. combine that with chromes behaviour to return a video track even when the device can not be opened... (which may be compliant but... I think they changed that finally)
19:38jesupanother old bug on this: bug 926753
19:38firebothttps://bugzil.la/926753 FIXED, nobody@mozilla.org videoWidth and videoHeight should be set in loadedmetadata for getUserMedia & PeerConnection-connect
19:38jyaaudio is to play once loadeddata is fired. Which can only happen after loadedmetadata, so if no video yet, we can&#39;t be in readyState HAVE_METADATA yet
19:39jesupjya: the problem is that a video stream may not have a frame for a Long Time (if ever - think captureStream on a canvas), but you may have audio
19:39jyajesup: should adjust the spec accordingly then.
19:39jesupyes
19:40jesupBut let&#39;s not screw people over for years waiting for that.... or force horrible hoop jumping in the meantime
19:40jesup:-)
19:41* jib is confused. What does the spec say here, and where are we non-compliant?
19:42jesupThe spec was written without any idea it would be used for playing anything other than containerized streaming video
19:42fippojesup: as long as you update width/height before firing the event... (guess who does not...)
19:42jibhence it says what?
19:44jesuphttps://html.spec.whatwg.org/multipage/media.html#event-media-loadedmetadata
19:44jiband which spec?
19:44jesupThe user agent has just determined the duration and dimensions of the media resource and the text tracks are ready.
19:44jesupand related: HAVE_METADATA (numeric value 1) Enough of the resource has been obtained that the duration of the resource is available. In the case of a video element, the dimensions of the video are also available. No media data is available for the immediate current playback position.
19:45jesup(readyState)
19:45jibjesup: ok good, and we&#39;re doing that right then yes?
19:46jibI mean wfm https://jsfiddle.net/jib1/9t3szw0L/57
19:46jesupI believe so, after pehrsons&#39; fix. We will play audio even if we haven&#39;t fired onloadedmetadata I believe. (note: all this landed a long time ago, and I didn&#39;t do it)
19:46jibah
19:47jesupI think the audio part is what jya was referring to
19:47fippojesup: at least it matches waht i recall :-)
19:47jibso the problem is we should not be playing audio yet?
19:48jibthough this assumes autoplay or play() having been called of course
19:48jesupWell, the spec should say explicitly that we can, IMHO, or the entire thing should be redesigned to better support p2p/dynamic video (not gonna happen)
19:48jesupyes
19:48jesupreadyState makes it clear they are only thinking about containerized playback
19:49jibwell it says for loadedmetadata: &quot;Preconditions: readyState is newly equal to HAVE_METADATA or greater for the first time. &quot;
19:49jibsince it says OR GREATER, is there a problem?
19:52fippojib: the reason for not playing audio is that you have an arbitrary point to lip-sync. but if you dont get video.... what lip movement will you sync with?
19:53* jib will brb
19:56fippouhm... why is there a &quot;remember this decision&quot; on the screensharing picker?
20:05jibfippo: try clicking it. It&#39;s fake-out UX
20:07* jib is personally not a fan of fake &quot;detonate&quot; buttons
20:08fippojib: haha!
20:10jibbut my point about the spec was I THINK it says it&#39;s ok to delay loadedmetadata until e.g. HAVE_CURRENT_DATA if that&#39;s what it takes to have dimensions
20:11jibas long as loadedmetadata fires before loadeddata (e.g. one after the other)
20:13jibjesup: ^
20:38jibwhich can also be delayed
21:07jesuppehrsons: read the threads above re loadedmetadata
21:07jesupwhen you&#39;re back :-)
21:26cpearcepadenot: ping
21:32rocjesup: hi
21:48mjfroc: Howdy! Are you still in North America?
21:48rocno
21:51mjfAre you home, or still traveling?
21:53rocI&#39;m at home!
21:54mjfThat has to feel great! :-)
23:12rocyeah :-)
18 Jul 2017
No messages
   
Last message: 68 days and 20 hours ago