SPARK 学习资源汇总（1）

1. 书籍，在线文档

• Learning Spark
• Advanced.Analytics.with.Spark
• Mastering Apache Spark
• Official Guide
• Spark Guide by Cloudera

2. 网站

• official site
• user mailing list
• spark channel on youtube
• spark summit
• spark technology center
• sparkhub
• meetup
• spark third party packages
• databricks blog
• http://blog.madhukaraphatak.com/
• databricks docs
• databricks training
• cloudera blog about spark
• https://0x0fff.com
• http://techsuppdiva.github.io/
• csdn spark 知识库
• 过往记忆

3. Databricks Blog

• Apache Spark 1.5 DataFrame API Highlights: Date/Time/String Handling, Time Intervals, and UDAFs
• Achieving End-to-end Security for Apache Spark with Databricks

上面两篇是 databricks 出的关于 databricks 专业版的描述，虽然没有从根本上解决问题，但是读起来还是挺有说服力的，哈哈，因为采用了很多很细节的方案。不错不错，各位有在做云产品的，在宣传自己的安全方案时可用参考参考哦。

• Deep Dive into Spark SQL’s Catalyst Optimizer
• A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets
• Understanding your Apache Spark application through visualization
• New Visualizations for Understanding Apache Spark Streaming Applications
• An Introduction to Writing Apache Spark Applications on Databricks
• A Gentle Introduction to Apache Spark on Databricks
• Apache Spark on Databricks for Data Scientists
• Import Notebook Apache Spark on Databricks for Data Engineers
• Structured Streaming In Apache Spark: A new high-level API for streaming
• Spark SQL Supported Syntax
• Combining Machine Learning Frameworks with Apache Spark
• Deep Dive: Memory Management in Apache Spark
• https://databricks.com/blog/2017/07/11/introducing-apache-spark-2-2.html
• https://spark-summit.org/east-2017/events/improving-python-and-spark-performance-and-interoperability/
• http://www.slideshare.net/wesm/high-performance-python-on-apache-spark
• https://www.slideshare.net/SparkSummit/spark-and-online-analytics-spark-summit-east-talky-by-shubham-chopra
• https://www.slideshare.net/SparkSummit/spark-parquet-in-depth-spark-summit-east-talk-by-emily-curtin-and-robbie-strickland
• https://www.slideshare.net/databricks/sparksql-a-compiler-from-queries-to-rdds

4. 文章，博客

• RDD论文英文版
• RDD论文中文版
• An Architecture for Fast and General Data Processing on Large Clusters
• How-to: Tune Your Apache Spark Jobs (Part 1)
• How-to: Tune Your Apache Spark Jobs (Part 2)
• By 0x0fff Spark Misconceptions
• By 0x0fff Spark Architecture
• By 0x0fff Spark DataFrames are faster, aren’t they?
• By 0x0fff Spark Architecture: Shuffle
• By 0x0fff Modern Data Architecture
• By 0x0fff Spark Architecture Talk
• By 0x0fff Apache Spark Future
• By 0x0fff Data Industry Trends
• By 0x0fff Spark Memory Management
• By 0x0fff Spark Architecture Video
• 借助 Redis ，让 Spark 提速 45 倍！
• 量化派基于Hadoop、Spark、Storm的大数据风控架构
• 基于Spark的异构分布式深度学习平台
• 你对Hadoop和Spark生态圈了解有几许？
• Hadoop vs Spark
• 雅虎开源CaffeOnSpark：基于Hadoop/Spark的分布式深度学习
• 2016 上海第二次 spark meetup: 1. spark_meetup.pdf
• 2016 上海第二次 spark meetup: 2. Flink_ An unified stream engine.pdf
• 2016 上海第二次 spark meetup: 3. Spark在计算广告领域的应用实践.pdf
• 2016 上海第二次 spark meetup: 4. splunk_spark.pdf
• 基于Spark的医疗和金融大数据
• Monitoring Spark with Graphite and Grafana
• Spark使用CombineTextInputFormat缓解小文件过多导致Task数目过多的问题
• Databricks Empowers Enterprises to Secure Their Apache Spark Workloads
• Spark配置参数
• Running Spark Python Applications
• How-to: Prepare Your Apache Hadoop Cluster for PySpark Jobs
• Apache Spark’s Hidden REST API
• [SQL, sqlContext, hiveContext]
  • What is the difference between Apache Spark SQLContext vs HiveContext?
  • SQL Window Functions
  • Performance Tuning SQL Queries
  • HIVE-vs-RDBMS
  • Windowing and Analytics Functions
• [Spark Memory Issues]
  • http://www.cnblogs.com/wrencai/p/4231934.html
  • http://stackoverflow.com/questions/32349611/what-should-be-the-optimal-value-for-spark-sql-shuffle-partitions-or-how-do-we-i
  • http://stackoverflow.com/questions/21138751/spark-java-lang-outofmemoryerror-java-heap-space
  • A Beginner’s Guide on Troubleshooting Spark Applications
  • http://www.mincoder.com/article/2381.shtml
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-1/
• http://blog.cloudera.com/blog/2015/03/how-to-tune-your-apache-spark-jobs-part-2/
• http://blog.cloudera.com/blog/2014/05/apache-spark-resource-management-and-yarn-app-models/
• https://spark.apache.org/docs/latest/job-scheduling.html
• https://databricks.com/blog/2015/06/22/understanding-your-spark-application-through-visualization.html
• http://blog.csdn.net/u012684933/article/details/50378725

5. 视频

• YouTube: what is apache spark

• Introduction to Spark Architecture
• Top 5 Mistakes When Writing Spark Applications

• slide Top 5 mistakes when writing Spark applications

• Tuning and Debugging Apache Spark

• slide Tuning and Debugging Apache Spark

• A Deeper Understanding of Spark Internals - Aaron Davidson (Databricks)

• slide A Deeper Understanding of Spark Internals - Aaron Davidson (Databricks)

• Building, Debugging, and Tuning Spark Machine Learning Pipelines - Joseph Bradley (Databricks)

• slide Building, Debugging, and Tuning Spark Machine Learning Pipelines

• Spark DataFrames Simple and Fast Analysis of Structured Data - Michael Armbrust (Databricks)

• slide Spark DataFrames Simple and Fast Analysis of Structured Data - Michael Armbrust (Databricks)

• Spark Tuning for Enterprise System Administrators

• slide Spark Tuning for Enterprise System Administrators

• Structuring Spark: DataFrames, Datasets, and Streaming

• slide Structuring Spark: DataFrames, Datasets, and Streaming

• Spark in Production: Lessons from 100+ Production Users

• slide Spark in Production: Lessons from 100+ Production Users

• Production Spark and Tachyon use Cases

• slide Production Spark and Tachyon use Cases

• SparkUI Visualization

• slide SparkUI Visualization

• Everyday I’m Shuffling - Tips for Writing Better Spark Programs, Strata San Jose 2015

• slide Everyday I’m Shuffling - Tips for Writing Better Spark Programs, Strata San Jose 2015

• Large Scale Distributed Machine Learning on Apache Spark

• Securing your Spark Applications

• slide Securing your Spark Applications

• Building a REST Job Server for Interactive Spark as a Service

• slide Building a REST Job Server for Interactive Spark as a Service

• Exploiting GPUs for Columnar DataFrame Operations

• slide Exploiting GPUs for Columnar DataFrame Operations

• Easy JSON Data Manipulation in Spark - Yin Huai (Databricks)

• slide Easy JSON Data Manipulation in Spark - Yin Huai (Databricks)

• Sparkling: Speculative Partition of Data for Spark Applications - Peilong Li

• slide Sparkling: Speculative Partition of Data for Spark Applications - Peilong Li

• Advanced Spark Internals and Tuning – Reynold Xin

• slide Advanced Spark Internals and Tuning – Reynold Xin

• The Future of Real Time in Spark

• slide The Future of Real Time in Spark

• Spark 2 0

• slide Spark 2 0

• Democratizing Access to Data

• slide Democratizing Access to Data

• Not Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture

• slide Not Your Father’s Database: How to Use Apache Spark Properly in Your Big Data Architecture

• Disrupting Big Data with Apache Spark in the Cloud
• slide Disrupting Big Data with Apache Spark in the Cloud

我一直很欣赏 databricks 出的 video 和 slide，结构非常清晰，这个是其中一个非常好的演讲，里面有很多值得借鉴的地方，特别是当你像别人介绍你的工作，产品的时候。[我有一个感受，很少有人能清晰，有条理的介绍自己正在做的产品，对于一些小众的产品，甚至一些职业的销售也难以做到清晰，简明的叙述。这个 video 和 slide 有很大的参考价值。我自己感觉仔细研究这些 video 和 slide 有时候比看上一两本专业讲销售的书还要管用。]

• Getting The Best Performance With PySpark

• slide Getting The Best Performance With PySpark

• Disrupting Big Data with Apache Spark in the Cloud

• slide Disrupting Big Data with Apache Spark in the Cloud

• Large Scale Deep Learning with TensorFlow

• slide Large Scale Deep Learning with TensorFlow

• 700 Queries Per Second with Updates: Spark As A Real-Time Web Service
• slide 700 Queries Per Second with Updates: Spark As A Real-Time Web Service
• blog 700 SQL Queries per Second in Apache Spark with FiloDB
• Operational Tips for Deploying Spark

• slide Operational Tips for Deploying Spark

• Connecting Python To The Spark Ecosystem

• slide Connecting Python To The Spark Ecosystem

• Livy: A REST Web Service For Apache Spark

• slide Livy: A REST Web Service For Apache Spark

• Understanding Memory Management In Spark For Fun And Profit

• slide Understanding Memory Management In Spark For Fun And Profit

• Deep Dive: Memory Management in Apache Spark

• slide Deep Dive: Memory Management in Apache Spark

• High Performance Python on Apache Spark

• slide High Performance Python on Apache Spark

• A Deep Dive into Structured Streaming

• slide A Deep Dive into Structured Streaming

• slide Spark Job Server and Spark as a Query Engine (Spark Meetup 5/14)

• Apache spark job server example with installation

• Productionizing Spark and the Spark REST Job Server

• slide Productionizing Spark and the Spark REST Job Server

• Data Profiling and Pipeline Processing with Spark

• slide Data Profiling and Pipeline Processing with Spark

• Spark at Bloomberg: Dynamically Composable Analytics

• slide Spark at Bloomberg: Dynamically Composable Analytics

• Data Storage Tips for Optimal Spark Performance - Vida Ha (Databricks)

• slide Data Storage Tips for Optimal Spark Performance-(Vida Ha, Databricks)

• Sparkling Pandas - using Apache Spark to scale Pandas - Holden Karau and Juliet Hougland