Overall: akka-stream from the data source is a Source, the circulation flow member nodes and three framework Flow data stream end Sink (stream components) thereof. This wherein: Source and Sink stream endpoints are two separate, but the Source and Sink Flow in the intermediate stream may be provided by a plurality of channel-node, each node represents some of the elements of the data stream conversion processing, they are sequentially linked It may represent the overall operation of the process. A complete data stream (data stream may be run) must be closed to a data flow, namely: from the outside, two data streams must connect a Source and a Sink. We can put a direct connection to a Source Sink to get one of the easiest to run the data stream, as follows:
Source(1 to 10).runWith(Sink.foreach(println))
From another angle: akka-stream data flow and comprises two portions, and FIG Graph calculator Materializer. Graph represent the operation program, Materializer responsible for preparing the operational environment and placed into the program Graph Actor actual operation system going to produce results (effects) and get the operation result. So: akka-stream must have a description Graph functions and processes. Each Graph but also by a number of representatives finer sub-Graph function composition. An operating data stream must be represented by a data flow diagram closed (closed graph), and this in turn represent different data ClosedGraph conversion sub-picture processing functions (sub-graph) composition. Customized data stream function is to press for the Graph function needs to be customized.
Graph may be a two part GraphShape and GraphStage described: GraphShape describes the number of input and output ports of Graph, GraphStage described data conversion process in the circulation. We first analyze GraphShape, which is the base class Shape:
/**
* A Shape describes the inlets and outlets of a [[Graph]]. In keeping with the
* philosophy that a Graph is a freely reusable blueprint, everything that
* matters from the outside are the connections that can be made with it,
* otherwise it is just a black box.
*/
abstract class Shape {
/**
* Scala API: get a list of all input ports
*/
def inlets: immutable.Seq[Inlet[_]]
/**
* Scala API: get a list of all output ports
*/
def outlets: immutable.Seq[Outlet[_]]
/**
* Create a copy of this Shape object, returning the same type as the
* original; this constraint can unfortunately not be expressed in the
* type system.
*/
def deepCopy(): Shape
...}
Shape subclasses must implement the abstract above three functions. akka-stream previously provided some basic shapes, including SourceShape / FlowShape / SinkShape:
/ **
* the Source A [[the Shape]] One has exactly Inputs Output and NO, IT A Models Source
* of Data.
* /
Final Case class SourceShape [+ T] (OUT: Outlet [T @uncheckedVariance]) {the extends the Shape
the override Val Inlets: immutable.Seq [Inlet, [_]] = EmptyImmutableSeq
the override Val Outlets: immutable.Seq [Outlet [_]] = OUT :: nil
the override the deepCopy DEF (): SourceShape [T] = SourceShape (out.carbonCopy ())
}
Object SourceShape {
/ * ** the API the Java /
DEF of [T] (Outlet: Outlet [T @uncheckedVariance]): SourceShape [T ] =
SourceShape (Outlet)
}
/ **
* A Flow [[Shape]] has exactly one input and one output, it looks from the
* outside like a pipe (but it can be a complex topology of streams within of
* course).
*/
final case class FlowShape[-I, +O](in: Inlet[I @uncheckedVariance], out: Outlet[O @uncheckedVariance]) extends Shape {
override val inlets: immutable.Seq[Inlet[_]] = in :: Nil
override val outlets: immutable.Seq[Outlet[_]] = out :: Nil
override def deepCopy(): FlowShape[I, O] = FlowShape(in.carbonCopy(), out.carbonCopy())
}
object FlowShape {
/** Java API */
def of[I, O](inlet: Inlet[I @uncheckedVariance], outlet: Outlet[O @uncheckedVariance]): FlowShape[I, O] =
FlowShape(inlet, outlet)
}
还有一个稍微复杂点的双向流形状BidiShape：
//#bidi-shape
/**
* A bidirectional flow of elements that consequently has two inputs and two
* outputs, arranged like this:
*
* {{{
* +------+
* In1 ~>| |~> Out1
* | bidi |
* Out2 <~| |<~ In2
* +------+
* }}}
*/
final case class BidiShape[-In1, +Out1, -In2, +Out2](
in1: Inlet[In1 @uncheckedVariance],
out1: Outlet[Out1 @uncheckedVariance],
in2: Inlet[In2 @uncheckedVariance],
out2: Outlet[Out2 @uncheckedVariance]) extends Shape {
//#implementation-details-elided
override val inlets: immutable.Seq[Inlet[_]] = in1 :: in2 :: Nil
override val outlets: immutable.Seq[Outlet[_]] = out1 :: out2 :: Nil
/**
* Java API for creating from a pair of unidirectional flows.
*/
def this(top: FlowShape[In1, Out1], bottom: FlowShape[In2, Out2]) = this(top.in, top.out, bottom.in, bottom.out)
override def deepCopy(): BidiShape[In1, Out1, In2, Out2] =
BidiShape(in1.carbonCopy(), out1.carbonCopy(), in2.carbonCopy(), out2.carbonCopy())
//#implementation-details-elided
}
//#bidi-shape
object BidiShape {
def fromFlows[I1, O1, I2, O2](top: FlowShape[I1, O1], bottom: FlowShape[I2, O2]): BidiShape[I1, O1, I2, O2] =
BidiShape(top.in, top.out, bottom.in, bottom.out)
/** Java API */
def of[In1, Out1, In2, Out2](
in1: Inlet[In1 @uncheckedVariance],
out1: Outlet[Out1 @uncheckedVariance],
in2: Inlet[In2 @uncheckedVariance],
out2: Outlet[Out2 @uncheckedVariance]): BidiShape[In1, Out1, In2, Out2] =
BidiShape(in1, out1, in2, out2)
}
There are many-to-many UniformFanOutShape and UniformFanInShape. The following is a Shape our custom-many:
Case class TwoThreeShape [the I, I2, O, the O2, O3] (
IN1: Inlet, [the I],
IN2: Inlet, [I2],
OUT1 of: Outlet [O],
OUT2 of the : Outlet [the O2],
OUT3 of: Outlet [O3]) {the extends the Shape
the override Inlets DEF: immutable.Seq [Inlet, [_]] IN1 = IN2 :: :: Nil
the override DEF Outlets: immutable.Seq [Outlet [_]] :: :: OUT3 OUT2 OUT1 = Nil ::
the override DEF deepCopy (): the Shape = TwoThreeShape (
in1.carbonCopy (),
in2.carbonCopy (),
out1.carbonCopy(),
out2.carbonCopy(),
out3.carbonCopy()
)
}
This is a binary three shapes. We only need to implement inlets, outlets and deepCopy these three functions.
GraphStage describes the behavior of the data flow component, out of the flow pattern and the flow transition process in the component data flow through the element to define the specific function of the flow member. The following is a type definition GraphStage of:
/**
* A GraphStage represents a reusable graph stream processing stage. A GraphStage consists of a [[Shape]] which describes
* its input and output ports and a factory function that creates a [[GraphStageLogic]] which implements the processing
* logic that ties the ports together.
*/
abstract class GraphStage[S <: Shape] extends GraphStageWithMaterializedValue[S, NotUsed] {
final override def createLogicAndMaterializedValue(inheritedAttributes: Attributes): (GraphStageLogic, NotUsed) =
(createLogic(inheritedAttributes), NotUsed)
@throws(classOf[Exception])
def createLogic(inheritedAttributes: Attributes): GraphStageLogic
}
每个构件都需要根据需求通过实现createLogic来设计GraphStageLogic功能。GraphStageLogic定义如下：
/**
* Represents the processing logic behind a [[GraphStage]]. Roughly speaking, a subclass of [[GraphStageLogic]] is a
* collection of the following parts:
* * A set of [[InHandler]] and [[OutHandler]] instances and their assignments to the [[Inlet]]s and [[Outlet]]s
* of the enclosing [[GraphStage]]
* * Possible mutable state, accessible from the [[InHandler]] and [[OutHandler]] callbacks, but not from anywhere
* else (as such access would not be thread-safe)
* * The lifecycle hooks [[preStart()]] and [[postStop()]]
* * Methods for performing stream processing actions, like pulling or pushing elements
*
* The stage logic is completed once all its input and output ports have been closed. This can be changed by
* setting `setKeepGoing` to true.
*
* The `postStop` lifecycle hook on the logic itself is called once all ports are closed. This is the only tear down
* callback that is guaranteed to happen, if the actor system or the materializer is terminated the handlers may never
* see any callbacks to `onUpstreamFailure`, `onUpstreamFinish` or `onDownstreamFinish`. Therefore stage resource
* cleanup should always be done in `postStop`.
*/
abstract class GraphStageLogic private[stream] (val inCount: Int, val outCount: Int) {...}
GraphStageLogic output is mainly responsible for the event input port responds by InHandler and OutHandler, to change the way the elements and flow control on a port:
/**
* Collection of callbacks for an input port of a [[GraphStage]]
*/
trait InHandler {
/**
* Called when the input port has a new element available. The actual element can be retrieved via the
* [[GraphStageLogic.grab()]] method.
*/
@throws(classOf[Exception])
def onPush(): Unit
/**
* Called when the input port is finished. After this callback no other callbacks will be called for this port.
*/
@throws(classOf[Exception])
def onUpstreamFinish(): Unit = GraphInterpreter.currentInterpreter.activeStage.completeStage()
/**
* Called when the input port has failed. After this callback no other callbacks will be called for this port.
*/
@throws(classOf[Exception])
def onUpstreamFailure(ex: Throwable): Unit = GraphInterpreter.currentInterpreter.activeStage.failStage(ex)
}
/**
* Collection of callbacks for an output port of a [[GraphStage]]
*/
trait OutHandler {
/**
* Called when the output port has received a pull, and therefore ready to emit an element, i.e. [[GraphStageLogic.push()]]
* is now allowed to be called on this port.
*/
@throws(classOf[Exception])
def onPull(): Unit
/**
* Called when the output port will no longer accept any new elements. After this callback no other callbacks will
* be called for this port.
*/
@throws(classOf[Exception])
def onDownstreamFinish(): Unit = {
GraphInterpreter
.currentInterpreter
.activeStage
.completeStage()
}
}
We can see: we need to implement InHandler.onPush () and OutHandler.onPull. akka-stream in the link data streams are achieved Reactive-Stream-Specification, it is required in terms of the input ports in response to the upstream push InHandler signal onPush, output port in response to a read signal onPull OutHandler downstream. It would need member itself: from the input port pull (in), the output port push (out).
Here we demonstrate a design cycle to generate Source string of the specified character. Source only one output port, we need only observe the read signal output port downstream. So in this case we only need to rewrite the function OutHandler:
class AlphaSource(chars: Seq[String]) extends GraphStage[SourceShape[String]] {
val outport = Outlet[String]("output")
val shape = SourceShape(outport)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
new GraphStageLogic(shape) {
var pos: Int = 0
setHandler(outport,new OutHandler {
override def onPull(): Unit = {
push(outport,chars(pos))
pos += 1
if (pos == chars.length) pos = 0
}
})
}
}
GraphStage class is a subclass Graph:
GraphStage class abstract [S <: the Shape] the extends GraphStageWithMaterializedValue [S, NotUsed] {...}
abstract class GraphStageWithMaterializedValue [S + <: the Shape, + M] the extends Graph [S, M] {...}
so we can AlphaSource as Graph then Source.fromGraph to construct Source components:
val sourceGraph: Graph[SourceShape[String],NotUsed] = new AlphaSource(Seq("A","B","C","D"))
val alphaSource = Source.fromGraph(sourceGraph).delay(1.second,DelayOverflowStrategy.backpressure)
alphaSource.runWith(Sink.foreach(println))
Also for Sink: we only need to look at and read data upstream push signal:
class UppercaseSink the extends GraphStage [SinkShape [String]] {
Val InPort = Inlet, [String] ( "INPUT")
Val SinkShape Shape = (InPort)
the override DEF createLogic ( inheritedAttributes: the Attributes): GraphStageLogic =
new new GraphStageLogic (Shape) {InHandler with
the override DEF PRESTART (): Unit = pull (InPort)
the override the OnPush DEF (): Unit = {
the println (Grab (InPort) .toUpperCase)
pull (InPort)
}
setHandler (InPort, the this)
}
}
From the above AlphaSource, UppercaseSink we try a slightly stream flow control element, the main input port is output to take a passive state changes in response to: pull port to be operated by push,. Here are some commonly used methods of operation and port status event:
Output port status change events by OutHandler callback function (callback) to capture. With setHandler (out, outHandler) to register OutHandler instance. The following operation is a function for the output port:
1, push (out, elem): launch data port, allowing only put forward only after reading the data requires the use of pull downstream, before this can not be invoked many times
2, complete (out): normal manual close ports
3, fail (out, exeption): abnormal manually close the port
Output port in response to the event include:
1, onPull (): may receive downstream data, this time data can be transmitted to the output port with a push (out, elem)
2, onDownStreamFinish (): read data downstream of the stop, after which the event will not receive any onPull
The following functions can obtain the current state of the output port:
1, isAvailable (out): true representative may use the push (out, elem)
2, isClosed (out): true representative of the output port has been shut down, unable to listen to events or push data
Similarly, the input port callback state capture is achieved by treatment with setHandler (in, inHandler) registered in InHandler. Input port manipulation functions comprising:
1, pull (in): read the data requirements presented to the upstream, allowing only upstream has been completed in order to use the data push, many times before not to call
2, grab (in): Port read from the current data, only the upstream completed before you can use data push, which many times not to call
3, cancel (in): off manually input port
Input port events:
1, onPush (): an upstream data has been transmitted to the input port, this time can grab (in) to read the current data, a pull (in) next to the upstream data requirements
2, onUpstreamFinish (): upstream data transmission has been terminated, after which events are not captured onPush not use pull (in) the requested data to the upstream
3, onUpstreamFalure (): abend upstream
Acquiring an input port status Method:
1, isAvailable (in): true representatives can now use the grab (in) reading the current data
2, hasBeenPulled (in): true representative of pull has been used (in) data were read requests, in this state does not allow the use of pull again (in)
3, isClosed (in): true representative of the port has been closed, after which the administration can not pull (in) and can not be captured onPush event
From the above functional description pull (in) and a push (out, elem) can be derived which are strictly interdependent, with each cycle, namely: downstream pull (in) must push the upstream front (OUT), and the upstream push (out, elem) downstream before must pull (in). This is readily appreciated, as is akka-stream Reactive-Stream, a push, a pull downstream of the binding mode communicate with each other. But such a scenario is inconvenient in some applications, such as data flow control. akka-stream also provides a simpler API allows the user a more flexible operation of the port. The functions of this API include the following:
1, emit (out, elem): temporary replacement OutHandler, transmitted to port elem, then restore OutHandler
2, emitMultiple (out, Iterable (e1, e2, e3 ...)): OutHandler temporary replacement, send a string of data to the port, and then restore OutHandler
3, read (in) (andThen): temporary replacement InHandler, reading a data element from the port, and then restore InHandler
4, readN (in) (andThen): InHandler temporary replacement, read data elements from the port n, and then restore InHandler
5, abortEmitting (): cancel outstanding output port data push
6, abortReading (): read operation to cancel outstanding input port
The API also supports the fact reactive-stream-backpressure, we can draw from emitMultiple function source code:
/**
* Emit a sequence of elements through the given outlet and continue with the given thunk
* afterwards, suspending execution if necessary.
* This action replaces the [[OutHandler]] for the given outlet if suspension
* is needed and reinstalls the current handler upon receiving an `onPull()`
* signal (before invoking the `andThen` function).
*/
final protected def emitMultiple[T](out: Outlet[T], elems: Iterator[T], andThen: () ⇒ Unit): Unit =
if (elems.hasNext) {
if (isAvailable(out)) {
push(out, elems.next())
if (elems.hasNext)
setOrAddEmitting(out, new EmittingIterator(out, elems, getNonEmittingHandler(out), andThen))
else andThen()
} else {
setOrAddEmitting(out, new EmittingIterator(out, elems, getNonEmittingHandler(out), andThen))
}
} else andThen()
Here we customize a Flow GraphStage, using the read / emit allow user-defined function can control the flow of data flow elements and screening. For Flow, while the need to focus on an upstream push data input port and output port on the downstream status read request status:
trait Row
trait Move
case object Stand extends Move
case class Next(rows: Iterable[Row]) extends Move
class FlowValve(controller: Row => Move) extends GraphStage[FlowShape[Row,Row]] {
val inport = Inlet[Row]("input")
val outport = Outlet[Row]("output")
val shape = FlowShape.of(inport,outport)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
new GraphStageLogic(shape) with InHandler with OutHandler {
override def onPush(): Unit = {
controller(grab(inport)) match {
case Next(rows) => emitMultiple(outport,rows)
case _ => pull(inport)
}
}
override def onPull(): Unit = pull(inport)
setHandlers(inport,outport,this)
}
}
This FlowValve above type is specially designed for administering a user-defined set of functions and controller. The controller functions to push upstream data elements to determine the content of the current data element or Stand across the Next (...) transmitting one or more elements downstream. When receiving data sent downstream can pull request FlowValve when it will pass directly to the upstream. The following is an example of a user-defined function:
Case class the Order (Burger: String, the qty: Int) the extends Row
Case class Burger (MSG: String) the extends Row
DEF orderDeliver: Row => the Move = Order => {
Order match {
Case the Order (name, the qty) =>
IF (the qty> 0) {
Val Burgers: the Iterable [Burger] =
(to the qty. 1) .foldLeft (the Iterable [Burger] ()) {(B, A) =>
B ++ the Iterable (Burger (S "name $ $ $ A of the qty {}"))
}
the Next (Burgers)
} Stand the else
}
}
val flowGraph: Graph[FlowShape[Row,Row],NotUsed] = new FlowValve(orderDeliver)
val deliverFlow: Flow[Row,Row,NotUsed] = Flow.fromGraph(flowGraph)
val orders = List(Order("cheeze",2),Order("beef",3),Order("pepper",1),Order("Rice",0)
,Order("plain",1),Order("beef",2))
Source(orders).via(deliverFlow).to(Sink.foreach(println)).run()
Test calculation results are shown below:
Burger (cheeze. 1 of 2)
Burger (cheeze 2 of 2)
Burger (Beef. 1 of. 3)
Burger (Beef 2 of. 3)
Burger (Beef. 3 of. 3)
Burger (Pepper. 1 of. 1)
Burger ( . 1. 1 of Plain)
Burger (Beef. 1 of 2)
Burger (Beef 2 of 2)
Exactly what we expected result. For stream type diffusion member many and many-to-shaped combined akka-stream and provided UniformFanIn UniformFanOut two kinds GraphStage. These two in combination may be constructed many-shaped member, it has been preset GraphStage enough.
The following is an exemplary source code according to:
import akka.NotUsed
import akka.actor._
import akka.stream.ActorMaterializer
import akka.stream.scaladsl._
import akka.stream.stage._
import akka.stream._
import scala.concurrent.duration._
import scala.collection.immutable.Iterable
class AlphaSource(chars: Seq[String]) extends GraphStage[SourceShape[String]] {
val outport = Outlet[String]("output")
val shape = SourceShape(outport)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
new GraphStageLogic(shape) {
var pos: Int = 0
setHandler(outport,new OutHandler {
override def onPull(): Unit = {
push(outport,chars(pos))
pos += 1
if (pos == chars.length) pos = 0
}
})
}
}
class UppercaseSink extends GraphStage[SinkShape[String]] {
val inport = Inlet[String]("input")
val shape = SinkShape(inport)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
new GraphStageLogic(shape) with InHandler {
override def preStart(): Unit = pull(inport)
override def onPush(): Unit = {
println(grab(inport).toUpperCase)
pull(inport)
}
setHandler(inport,this)
}
}
trait Row
trait Move
case object Stand extends Move
case class Next(rows: Iterable[Row]) extends Move
class FlowValve(controller: Row => Move) extends GraphStage[FlowShape[Row,Row]] {
val inport = Inlet[Row]("input")
val outport = Outlet[Row]("output")
val shape = FlowShape.of(inport,outport)
override def createLogic(inheritedAttributes: Attributes): GraphStageLogic =
new GraphStageLogic(shape) with InHandler with OutHandler {
override def onPush(): Unit = {
controller(grab(inport)) match {
case Next(rows) => emitMultiple(outport,rows)
case _ => pull(inport)
}
}
override def onPull(): Unit = pull(inport)
setHandlers(inport,outport,this)
}
}
object GraphStages extends App {
implicit val sys = ActorSystem("demoSys")
implicit val ec = sys.dispatcher
implicit val mat = ActorMaterializer(
ActorMaterializerSettings(sys)
.withInputBuffer(initialSize = 16, maxSize = 16)
)
val sourceGraph: Graph[SourceShape[String],NotUsed] = new AlphaSource(Seq("a","b","c","d"))
val alphaSource = Source.fromGraph(sourceGraph).delay(1.second,DelayOverflowStrategy.backpressure)
// alphaSource.runWith(Sink.foreach(println))
val sinkGraph: Graph[SinkShape[String],NotUsed] = new UppercaseSink
val upperSink = Sink.fromGraph(sinkGraph)
alphaSource.runWith(upperSink)
case class Order(burger: String, qty: Int) extends Row
case class Burger(msg: String) extends Row
def orderDeliver: Row => Move = order => {
order match {
case Order(name,qty) =>
if (qty > 0) {
val burgers: Iterable[Burger] =
(1 to qty).foldLeft(Iterable[Burger]()) { (b, a) =>
b ++ Iterable(Burger(s"$name $a of ${qty}"))
}
Next(burgers)
} else Stand
}
}
val flowGraph: Graph[FlowShape[Row,Row],NotUsed] = new FlowValve(orderDeliver)
deliverFlow Val: Flow [Row, Row, NotUsed] = Flow.fromGraph (flowGraph)
Val = Orders List (the Order ( "cheeze", 2), the Order ( "Beef",. 3), the Order ( "Pepper",. 1), the Order ( "Rice", 0)
, the Order ( "Plain",. 1), the Order ( "Beef", 2))
the Source (Orders) .via (deliverFlow) .to (Sink.foreach (the println)). RUN ()
the Source // (. 1 to 10) .runWith (Sink.foreach (the println))
scala.io.StdIn.readLine ()
sys.terminate ()
}
----------------- ----
author: TIGER_XC
source: CSDN
original: https: //blog.csdn.net/TIGER_XC/article/details/77841179
copyright: This article is a blogger original article, reproduced, please attach Bowen link!
