版权声明:本文为博主原创文章,未经博主允许不得转载。 https://blog.csdn.net/kaaosidao/article/details/82462435
一、dataFrame转为DataSet
第一种方式:通过class
val s = spark.sql(
"""
|select "1" id,1.1 f2,1.2 f3
|union all
|select "2" id,1.1 f2,1.1 f3
""".stripMargin).as[Bean](Encoders.bean(classOf[Bean])).show()
class Bean extends Serializable{
@BeanProperty var id:String = _
@BeanProperty var f2:Double = _
@BeanProperty var f3:Double = _
}
如果不加@BeanProperty 会报错:
Exception in thread "main" java.lang.UnsupportedOperationException: Cannot infer type for class com.***.***.Bean because it is not bean-compliant
可见scala的bean与javaBean
第二种:通过cass class
这种方式只需
import spark.implicits._
supported by importing spark.implicits._
二、dataFrame转为DataSet
DataSet.toDF即可