【SparkAPI JAVA版】JavaPairRDD——cartesian（三）

版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/sdut406/article/details/88096334

JavaPairRDD的cartesian方法讲解

官方文档说明

Return the Cartesian product of this RDD and another one, that is, the RDD of all pairs of elements (a, b) where a is in `this` and b is in `other`.

中文含义

该函数返回的是Pair类型的RDD，计算结果是当前RDD和other RDD中每个元素进行笛卡儿计算的结果。最后返回的是CartesianRDD。

方法原型

//scala
def cartesian[U: ClassTag](other: RDD[U]): RDD[(T, U)]
//java
public static <U> JavaPairRDD<T,U> cartesian(JavaRDDLike<U,?> other)

实例

public class Cartesian {

    public static void main(String[] args) {
        System.setProperty("hadoop.home.dir","F:\\hadoop-2.7.1");
        SparkConf conf = new SparkConf().setMaster("local").setAppName("TestSpark");
        JavaSparkContext sc = new JavaSparkContext(conf);


        //JavaPairRDD cartesian示例
        JavaPairRDD<Integer,Integer> javaPairRDD1 = sc.parallelizePairs(Lists.newArrayList(
                new Tuple2<Integer, Integer>(1,11),
                new Tuple2<Integer, Integer>(2,22),
                new Tuple2<Integer, Integer>(3,33)
        ));
        JavaPairRDD<Integer,Integer> javaPairRDD2 = sc.parallelizePairs(Lists.newArrayList(
                new Tuple2<Integer, Integer>(7,71),
                new Tuple2<Integer, Integer>(8,82),
                new Tuple2<Integer, Integer>(9,93)
        ));
        JavaPairRDD<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>> javaPairRDD3 = javaPairRDD1.cartesian(javaPairRDD2);
        javaPairRDD3.foreach(new VoidFunction<Tuple2<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>>>() {
            public void call(Tuple2<Tuple2<Integer, Integer>, Tuple2<Integer, Integer>> tuple2Tuple2Tuple2) throws Exception {
                System.out.println(tuple2Tuple2Tuple2);
                /*System.out.print("key:"+tuple2Tuple2Tuple2._1+",");
                System.out.println("value:"+tuple2Tuple2Tuple2._2);*/
            }
        });

        //JavaRDD cartesian示例
        JavaRDD<Integer> javaRDD1 = sc.parallelize(Lists.newArrayList(1,2,3));
        JavaRDD<Integer> javaRDD2 = sc.parallelize(Lists.newArrayList(7,8,9));
        JavaPairRDD<Integer,Integer> javaPairRDD = javaRDD1.cartesian(javaRDD2);
        javaPairRDD.foreach(new VoidFunction<Tuple2<Integer, Integer>>() {
            public void call(Tuple2<Integer, Integer> integerIntegerTuple2) throws Exception {
                System.out.println(integerIntegerTuple2);
            }
        });
    }
}

结果

//JavaPairRDD cartesian示例
((1,11),(7,71))
((1,11),(8,82))
((1,11),(9,93))
((2,22),(7,71))
((2,22),(8,82))
((2,22),(9,93))
((3,33),(7,71))
((3,33),(8,82))
((3,33),(9,93))
19/03/03 23:37:52 INFO Executor: Finished task 0.0 in stage 0.0 (TID 0). 751 bytes result sent to driver
//JavaRDD cartesian示例
(1,7)
(1,8)
(1,9)
(2,7)
(2,8)
(2,9)
(3,7)
(3,8)
(3,9)

注意：计算笛卡尔积很消耗内存，不要在大量数据的时候随便使用。