We used many chapters earlier to discuss how to move data from the background database to memory and then perform line-by-line operations. The solution we chose was to convert background data into data streams in memory. Whether it is opening database tables or reading data from the database, it involves the safe use of database tables as a resource: at the very least, we should ensure that the occupied resources can be released when we complete the use or when there are errors and abnormal exits in the middle of use. When it comes to resource security, we have to think of the bracket function that is commonly used in functional programming. fs2 also provides this function:
def bracket[F[_],R,A](r: F[R])(use: R => Stream[F,A], release: R => F[Unit]): Stream[F,A] = Stream.mk {
StreamCore.acquire(r, release andThen (Free.eval)) flatMap { case (_, r) => use(r).get }
}
The input parameters r, use and release of this function involve resource occupancy processing: R is generally open file or library table operation, use is the process of resource use such as reading data, release is the process of resource release cleaning after normal completion of resource use. Function bracket ensures that these processes are referenced correctly.
Let's use a few examples to analyze the function of this function:
val s = Stream.bracket(Task.delay(throw new Exception("Oh no!")))(
_ => Stream(1,2,3) ++ Stream.fail(new Exception("boom!")) ++ Stream(3,4),
_ => Task.delay(println("normal end")))
s.runLog.unsafeRun //> java.lang.Exception: Oh no!
//| at demo.ws.streams$$anonfun$main$1$$anonfun$1.apply(demo.ws.streams.scal
//| a:4)
//| at demo.ws.streams$$anonfun$main$1$$anonfun$1.apply(demo.ws.streams.scal
//| a:4)
In the example above, we created anomalies in two places. We can use onError to intercept these anomalies:
val s1 = s.map(_.toString).onError {e => Stream.emit(e.getMessage)}
s1.runLog.unsafeRun //> res0: Vector[String] = Vector(Oh no!)
The Stream element type must be converted with toString before the intercepted exception information can be put into Stream. Note that release is not invoked because resources are not occupied yet. But if there are other cleanups besides releasing resources, we can use onFinalize to ensure that the cleanup program can be invoked:
val s5 = s1.onFinalize(Task.delay{println("finally end!")})
s5.runLog.unsafeRun //> finally end!
//| res1: Vector[String] = Vector(Oh no!)
What happens if there are exceptions in the use of resources?
val s3 = Stream.bracket(Task.delay())(
_ => Stream(1,2,3) ++ Stream.fail(new Exception("boom!")) ++ Stream(3,4),
_ => Task.delay(println("normal end")))
val s4 = s3.map(_.toString).onError {e => Stream.emit(e.getMessage)}
.onFinalize(Task.delay{println("finally end!")})
s4.runLog.unsafeRun //> normal end
//| finally end!
//| res2: Vector[String] = Vector(1, 2, 3, boom!)
Return result res2 correctly records the location of the error, and all cleanup processes are running. Of course, instead of changing Stream element types, we can use attempt:
val s6 = s3.attempt.onError {e => Stream.emit(e.getMessage)}
.onFinalize(Task.delay{println("finally end!")})
s6.runLog.unsafeRun //> normal end
//| finally end!
//| res3: Vector[Object] = Vector(Right(1), Right(2), Right(3), Left(java.lang.Exception: boom!))
We discussed earlier in FunDA(1) that the slick Query Action run return result type is Future[Iterable[ROW]. Slick acquires data by reading it into memory at one time, so the Static-Source mentioned in the title of this issue refers to such a collection in memory. Then we don't have to consider opening and occupying database tables. We just need to use the FunDA DataRowType.getTypedRow function to get the Iterable[ROW] result and pass it directly to bracket. Now the most important thing is how to convert Seq[ROW] into Stream [F[], ROW]. We can use Seq's fold function to construct Stream:
val data = Seq(1,2,3,4) //> data : Seq[Int] = List(1, 2, 3, 4)
val s8 = data.foldLeft(Stream[Task,Int]())((s,a) => s ++ Stream.emit(a))
def log[A](prompt: String): Pipe[Task,A,A] =
_.evalMap {row => Task.delay{ println(s"$prompt> $row"); row }}
//> log: [A](prompt: String)fs2.Pipe[fs2.Task,A,A]
s8.through(log("")).run.unsafeRun //> > 1
//| > 2
//| > 3
//| > 4
On the surface, there seems to be no problem, but careful analysis: Seq[ROW] can be a huge collection, and foldLeft is a recursive function, whether or not tail recursion can cause stack Overflow Error. It seems to use freemonad, which can store every operation in the memory structure and operate in a fixed stack space. The following function converts Seq[ROW] into Stream [F[], ROW] using fs2.Pull type structure:
def pullSeq[ROW](h: Seq[ROW]): Pull[Task, ROW, Unit] = {
val it = h.iterator
def go(it: Iterator[ROW]): Pull[Task, ROW, Unit] = for {
res <- Pull.eval(Task.delay({ if (it.hasNext) Some(it.next()) else None }))
next <- res.fold[Pull[Task, ROW, Unit]](Pull.done)(o => Pull.output1(o) >> go(it))
} yield next
go(it)
}
def streamSeq[ROW](h: Seq[ROW]): Stream[Task, ROW] =
pullSeq(h).close
Although go is a recursive function, because Pull is a freemonad, each flapMap loop (>) stores the new Iterable it state in heap memory. Because each step is stored in the memory structure, and the mode of computing these steps is to drag downstream step by step, that is to say, dragging downstream produces one element at a time. Pull Seq returns to Pull, Pull. close >> Stream, which is what the streamSeq function does. Now we can use bracket directly to build Stream safely:
val s9 = Stream.bracket(Task.delay(data))(streamSeq, _ => Task.delay())
s9.through(log("")).run.unsafeRun //> > 1
//| > 2
//| > 3
//| > 4
Now you can rest assured. But our goal is to provide a minimum threshold tool library for popular programmers who don't need to know Task, onError, onFinalize... We must make the use of bracket functions more straightforward so that users can call them more easily:
type FDAStream[A] = Stream[Task,A]
implicit val strategy = Strategy.fromFixedDaemonPool(4)
//> strategy : fs2.Strategy = Strategy
def fda_staticSource[ROW](acquirer: => Seq[ROW],
releaser: => Unit = (),
errhandler: Throwable => FDAStream[ROW] = null,
finalizer: => Unit = ()): FDAStream[ROW] = {
val s = Stream.bracket(Task(acquirer))(r => streamSeq(r), r => Task(releaser))
if (errhandler != null)
s.onError(errhandler).onFinalize(Task.delay(finalizer))
else
s.onFinalize(Task.delay(finalizer))
} //> fda_staticSource: [ROW](acquirer: => Seq[ROW], releaser: => Unit, errhandle
//| r: Throwable => demo.ws.streams.FDAStream[ROW], finalizer: => Unit)demo.ws.
//| streams.FDAStream[ROW]
If fda_staticSource is fully invoked, the following can be done:
val s10 = fda_staticSource(data,
println("endofuse"), e => { println(e.getMessage);Stream.emit(-99) },
println("finallyend"))
s10.through(log("")).run.unsafeRun //> > 1
//| > 2
//| > 3
//| > 4
//| endofuse
//| finallyend
The simplest and most direct way is as follows:
val s11 = fda_staticSource(acquirer = data)
s11.through(log("")).run.unsafeRun //> > 1
//| > 2
//| > 3
//| > 4
Or call methods with exception handling procedures:
val s12 = fda_staticSource(acquirer = data, errhandler = {e => println(e.getMessage);Stream()})
s12.through(log("")).run.unsafeRun //> > 1
//| > 2
//| > 3
//| > 4
Following is the source code for this discussion demonstration:
import fs2._
object streams {
val s = Stream.bracket(Task.delay(throw new Exception("Oh no!")))(
_ => Stream(1,2,3) ++ Stream.fail(new Exception("boom!")) ++ Stream(3,4),
_ => Task.delay(println("normal end")))
//s.runLog.unsafeRun
val s1 = s.map(_.toString).onError {e => Stream.emit(e.getMessage)}
s1.runLog.unsafeRun
val s5 = s1.onFinalize(Task.delay{println("finally end!")})
s5.runLog.unsafeRun
val s3 = Stream.bracket(Task.delay())(
_ => Stream(1,2,3) ++ Stream.fail(new Exception("boom!")) ++ Stream(3,4),
_ => Task.delay(println("normal end")))
val s4 = s3.map(_.toString).onError {e => Stream.emit(e.getMessage)}
.onFinalize(Task.delay{println("finally end!")})
s4.runLog.unsafeRun
val s6 = s3.attempt.onError {e => Stream.emit(e.getMessage)}
.onFinalize(Task.delay{println("finally end!")})
s6.runLog.unsafeRun
val data = Seq(1,2,3,4)
val s8 = data.foldLeft(Stream[Task,Int]())((s,a) => s ++ Stream.emit(a))
def log[A](prompt: String): Pipe[Task,A,A] =
_.evalMap {row => Task.delay{ println(s"$prompt> $row"); row }}
s8.through(log("")).run.unsafeRun
def pullSeq[ROW](h: Seq[ROW]): Pull[Task, ROW, Unit] = {
val it = h.iterator
def go(it: Iterator[ROW]): Pull[Task, ROW, Unit] = for {
res <- Pull.eval(Task.delay({ if (it.hasNext) Some(it.next()) else None }))
next <- res.fold[Pull[Task, ROW, Unit]](Pull.done)(o => Pull.output1(o) >> go(it))
} yield next
go(it)
}
def streamSeq[ROW](h: Seq[ROW]): Stream[Task, ROW] =
pullSeq(h).close
val s9 = Stream.bracket(Task.delay(data))(streamSeq, _ => Task.delay())
s9.through(log("")).run.unsafeRun
type FDAStream[A] = Stream[Task,A]
implicit val strategy = Strategy.fromFixedDaemonPool(4)
def fda_staticSource[ROW](acquirer: => Seq[ROW],
releaser: => Unit = (),
errhandler: Throwable => FDAStream[ROW] = null,
finalizer: => Unit = ()): FDAStream[ROW] = {
val s = Stream.bracket(Task(acquirer))(r => streamSeq(r), r => Task(releaser))
if (errhandler != null)
s.onError(errhandler).onFinalize(Task.delay(finalizer))
else
s.onFinalize(Task.delay(finalizer))
}
val s10 = fda_staticSource(data,
println("endofuse"), e => { println(e.getMessage);Stream.emit(-99) },
println("finallyend"))
s10.through(log("")).run.unsafeRun
val s11 = fda_staticSource(acquirer = data)
s11.through(log("")).run.unsafeRun
val s12 = fda_staticSource(acquirer = data, errhandler = {e => println(e.getMessage);Stream()})
s12.through(log("")).run.unsafeRun
}