0

I'm creating a dataframe in spark like this:

DF: DataFrame = df.sqlContext.createDataFrame(myTypeRDD, getMyTypeSchema())

MyType is a complex data type.

For testing purposes I want to use it as a MyType collection.

My attempts:

  1. Trying to cast on the dataFrame:

DF.as[MyType] & DF.map(row => row.asInstanceOf[MyType] gave:

No Encoder found for org.joda.time.DateTime

  1. Trying to cast after collecting:

DF.collect().asInstanceOf[MyType] gave:

org.apache.spark.sql.catalyst.expressions.GenericRowWithSchema cannot be cast to MyType

Any suggestions would be appreciated

Dotan
  • 6,602
  • 10
  • 34
  • 47
  • Possible duplicate: http://stackoverflow.com/questions/33688945/convert-string-with-form-mm-dd-yyyy-hhmm-to-joda-datetime-in-dataframe-in-spa – T. Gawęda Apr 27 '17 at 13:02
  • Possible duplicate of [Convert string with form "MM/dd/yyyy HH:mm" to joda datetime in dataframe in Spark](http://stackoverflow.com/questions/33688945/convert-string-with-form-mm-dd-yyyy-hhmm-to-joda-datetime-in-dataframe-in-spa) – Rajat Mishra Apr 27 '17 at 13:18
  • How about `DF.rdd.collect` ?? – sarveshseri Apr 27 '17 at 13:36
  • http://stackoverflow.com/questions/34796176/unsupportedoperationexception-error-converting-string-to-datetime-using-joda-tim – pasha701 Apr 27 '17 at 13:42

0 Answers0