mozilla :: #datapipeline

12 Jul 2017
00:21jgauntwhat the heck? &quot;TypeError: Can not merge type <class &#39;pyspark.sql.types.StringType&#39;> and <class &#39;pyspark.sql.types.LongType&#39;>&quot;
00:22jgauntgot this error when I tried .toDF() method on an RDD
00:25jgauntthis would suggest I need to specify the schema verbosely
00:26jgauntI&#39;ve seen Frank do that in his SHIELD+Data+Join notebook
00:26jgaunta bit frustrating not to understand when that&#39;s a necessity and when it&#39;s not
15:34Dextermreid, FYI -> <- (plots at the end of the page)
15:36frankjgaunt: it&#39;s a bit of magic for spark to infer the schema, basically it&#39;s &quot;we won&#39;t know whether we can infer the schema until we try&quot;
15:51gfritzscheharter: were there any interesting new articles added that we should call out?
15:51gfritzschethats what i was trying to ask :)
16:02gfritzscheRaFromBRC: whats the vidyo room?
16:02gfritzscheor mreid: ^
16:03gfritzsche... ok, rmillers apparently
16:43Fallenbmiroglio: heads up for your queries:
18:50bmiroglioFallen: thanks for the heads up
20:24hartergfritzsche: There are a few coming, but they&#39;re still under review
20:31kitcambridge|sfmreid: hi! i&#39;m trying to run the telemetry-batch-view tests in a docker container, and i see they&#39;re aborted early. any tips? (or if i can just run the sync view tests, i don&#39;t really need to run the entire test suite)
13 Jul 2017
No messages
Last message: 15 days and 2 hours ago