首页 > pyspark执行reduce时报错:port out of range:1885434979

pyspark执行reduce时报错:port out of range:1885434979

刚装完spark-1.6.0-bin-hadoop2.6.tgz,
Java 1.8.0_65
Python 2.7

运行pyspark:

dd = sc.parallelize([1,2,3,4,5])
dd.reduce(lambda a, b:a+b)

报错:

16/01/14 18:35:28 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0, localhost, partition 0,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1, localhost, partition 1,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 2.0 in stage 0.0 (TID 2, localhost, partition 2,PROCESS_LOCAL, 2083 bytes)
16/01/14 18:35:28 INFO TaskSetManager: Starting task 3.0 in stage 0.0 (TID 3, localhost, partition 3,PROCESS_LOCAL, 2102 bytes)
16/01/14 18:35:28 INFO Executor: Running task 3.0 in stage 0.0 (TID 3)
16/01/14 18:35:28 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/01/14 18:35:28 INFO Executor: Running task 2.0 in stage 0.0 (TID 2)
16/01/14 18:35:28 INFO Executor: Running task 1.0 in stage 0.0 (TID 1)
16/01/14 18:35:28 ERROR Executor: Exception in task 3.0 in stage 0.0 (TID 3)
java.lang.IllegalArgumentException: port out of range:1885434979
        at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
        at java.net.InetSocketAddress.<init>(InetSocketAddress.java:188)
        at java.net.Socket.<init>(Socket.java:244)

尝试了其他方式,map可以执行,reduce/collection等都是报这个错。

“port out of range:1885434979” 字面上是端口数太大,没有找到任何设置的地方。

求解。

【热门文章】
【热门文章】