方法一:在linux命令行提交spark-sql,将执行结果写入文件
全量刷写:
spark-sql -S -e "select * from ldltmp.mars_smb_ba" > result.txt
追加写:
spark-sql -S -e "select * from ldltmp.mars_smb_ba" >> result.txt
方法二:在spark-shell,将df写入到本地文件
在集群模式下,最好将结果写入到HDFS,因为你不知道rdd的数据在哪台机器上,所以当save时,执行save操作的机器没有数据可能会导致结果为空。
可以创建sparkSession时指定master为local。此处不演示,推荐使用方法一。
Spark context Web UI available at http://utility02.loreal.com:4060
Spark context available as 'sc' (master = yarn, app id = application_1593419766907_555160).
Spark session available as 'spark'.
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/___/ .__/\_,_/_/ /_/\_\ version 2.3.0.cloudera4
/_/
Using Scala version 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_141)
Type in expressions to have them evaluated.
Type :help for more information.
#进行查询
scala> var df = spark.sql("select * from ldltmp.mars_smb_ba")
df: org.apache.spark.sql.DataFrame = [BaCode: string, StoreCode: string ... 8 more fields]
#将结果写入hdfs
scala> df.coalesce(1).rdd.saveAsTextFile("hdfs://*****")