org.apache.spark.streaming.flume

Class FlumePollingInputDStream<T>

Object

org.apache.spark.streaming.dstream.DStream<T>

org.apache.spark.streaming.dstream.InputDStream<T>

org.apache.spark.streaming.dstream.ReceiverInputDStream<SparkFlumeEvent>

org.apache.spark.streaming.flume.FlumePollingInputDStream<T>

All Implemented Interfaces:

java.io.Serializable, Logging

public class FlumePollingInputDStream<T>
extends ReceiverInputDStream<SparkFlumeEvent>

A ReceiverInputDStream that can be used to read data from several Flume agents running SparkSinks.

See Also:

Serialized Form

Constructor Summary

Constructors

Constructor and Description
FlumePollingInputDStream(StreamingContext _ssc, scala.collection.Seq<java.net.InetSocketAddress> addresses, int maxBatchSize, int parallelism, StorageLevel storageLevel, scala.reflect.ClassTag<T> evidence$1)

Method Summary

Methods

Modifier and Type	Method and Description
scala.collection.Seq<java.net.InetSocketAddress>	addresses()
Receiver<SparkFlumeEvent>	getReceiver() Gets the receiver object that will be sent to the worker nodes to receive data.
int	maxBatchSize()
int	parallelism()

Methods inherited from class org.apache.spark.streaming.dstream.ReceiverInputDStream

compute, id, start, stop

Methods inherited from class org.apache.spark.streaming.dstream.InputDStream

dependencies, isTimeValid, lastValidTime, slideDuration

Methods inherited from class org.apache.spark.streaming.dstream.DStream

cache, checkpoint, checkpointData, checkpointDuration, clearCheckpointData, clearMetadata, context, count, countByValue, countByValueAndWindow, countByWindow, creationSite, filter, flatMap, foreach, foreach, foreachRDD, foreachRDD, generatedRDDs, generateJob, getCreationSite, getOrCompute, glom, graph, initialize, isInitialized, map, mapPartitions, mustCheckpoint, parentRememberDuration, persist, persist, print, print, reduce, reduceByWindow, reduceByWindow, register, remember, rememberDuration, repartition, restoreCheckpointData, saveAsObjectFiles, saveAsTextFiles, setContext, setGraph, slice, slice, ssc, storageLevel, toPairDStreamFunctions, transform, transform, transformWith, transformWith, union, updateCheckpointData, validate, window, window, zeroTime

Methods inherited from class Object

equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Methods inherited from interface org.apache.spark.Logging

initializeIfNecessary, initializeLogging, isTraceEnabled, log_, log, logDebug, logDebug, logError, logError, logInfo, logInfo, logName, logTrace, logTrace, logWarning, logWarning

Constructor Detail

FlumePollingInputDStream

public FlumePollingInputDStream(StreamingContext _ssc,
                        scala.collection.Seq<java.net.InetSocketAddress> addresses,
                        int maxBatchSize,
                        int parallelism,
                        StorageLevel storageLevel,
                        scala.reflect.ClassTag<T> evidence$1)

Method Detail

addresses

public scala.collection.Seq<java.net.InetSocketAddress> addresses()

maxBatchSize

public int maxBatchSize()

parallelism

public int parallelism()

getReceiver

public Receiver<SparkFlumeEvent> getReceiver()

Description copied from class: ReceiverInputDStream

Gets the receiver object that will be sent to the worker nodes to receive data. This method needs to defined by any specific implementation of a ReceiverInputDStream.

Specified by:

getReceiver in class ReceiverInputDStream<SparkFlumeEvent>