Basic lambda prototype :)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Basic lambda prototype :)

Gyula Fóra-2
Hey guys,

I am trying contain my excitement, but I am failing :)

So today I implemented a basic prototype for connecting batch processing
and streaming jobs as a good starting point for a 'lambda architecture
api'. And most surprisingly it works :) It is very simple so far (currently
only works for map operator and chaining is turned off) and some hacking
along the way, but I hope you will like it.

You can see the commit here:
https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637

And a sample job:
https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java

And for the others, it looks something like this:

 ExecutionEnvironment batchEnv = ExecutionEnvironment.
*createLocalEnvironment*(1);
StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment.
*createLocalEnvironment*(1);

DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new*
Increment());

DataStream<Long> dataStream = streamEnv.generateSequence(10, 20);

  *dataStream.lambdaJoin(dataSet).map(new Increment()).print();*
dataSet.print();

 * LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv,
streamEnv);*

* lambdaEnv.execute(":)");*


I hope I could get you excited too :)

Cheers,

Gyula
Reply | Threaded
Open this post in threaded view
|

Re: Basic lambda prototype :)

Márton Balassi-3
Wow, that is in fact quite impressive. :)

On Fri, Sep 26, 2014 at 11:21 PM, Gyula Fóra <[hidden email]> wrote:

> Hey guys,
>
> I am trying contain my excitement, but I am failing :)
>
> So today I implemented a basic prototype for connecting batch processing
> and streaming jobs as a good starting point for a 'lambda architecture
> api'. And most surprisingly it works :) It is very simple so far (currently
> only works for map operator and chaining is turned off) and some hacking
> along the way, but I hope you will like it.
>
> You can see the commit here:
>
> https://github.com/mbalassi/incubator-flink/commit/7bfc16a7803e1f2cc4b38c746af069aab4450637
>
> And a sample job:
>
> https://github.com/mbalassi/incubator-flink/blob/7bfc16a7803e1f2cc4b38c746af069aab4450637/flink-addons/flink-streaming/flink-streaming-core/src/test/java/org/apache/flink/streaming/api/LambdaTest.java
>
> And for the others, it looks something like this:
>
>  ExecutionEnvironment batchEnv = ExecutionEnvironment.
> *createLocalEnvironment*(1);
> StreamExecutionEnvironment streamEnv = StreamExecutionEnvironment.
> *createLocalEnvironment*(1);
>
> DataSet<Long> dataSet = batchEnv.generateSequence(1, 10).map(*new*
> Increment());
>
> DataStream<Long> dataStream = streamEnv.generateSequence(10, 20);
>
>   *dataStream.lambdaJoin(dataSet).map(new Increment()).print();*
> dataSet.print();
>
>  * LambdaEnvironment lambdaEnv = new LambdaEnvironment(batchEnv,
> streamEnv);*
>
> * lambdaEnv.execute(":)");*
>
>
> I hope I could get you excited too :)
>
> Cheers,
>
> Gyula
>
Reply | Threaded
Open this post in threaded view
|

Re: Basic lambda prototype :)

Ufuk Celebi-2

On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote:

>> I hope I could get you excited too :)

+1 :-)
Reply | Threaded
Open this post in threaded view
|

Re: Basic lambda prototype :)

Stephan Ewen
Wow, amazing :-) And so quickly done!
Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]>:

>
> On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote:
>
> >> I hope I could get you excited too :)
>
> +1 :-)
Reply | Threaded
Open this post in threaded view
|

Re: Basic lambda prototype :)

Kostas Tzoumas-2
Wow so cool!

On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <[hidden email]> wrote:

> Wow, amazing :-) And so quickly done!
> Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]>:
>
> >
> > On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]> wrote:
> >
> > >> I hope I could get you excited too :)
> >
> > +1 :-)
>
Reply | Threaded
Open this post in threaded view
|

Re: Basic lambda prototype :)

Paris Carbone
+1
Looks nice!

On 27 Sep 2014, at 21:04, Kostas Tzoumas <[hidden email]<mailto:[hidden email]>> wrote:

Wow so cool!

On Sat, Sep 27, 2014 at 2:52 PM, Stephan Ewen <[hidden email]<mailto:[hidden email]>> wrote:

Wow, amazing :-) And so quickly done!
Am 27.09.2014 11:14 schrieb "Ufuk Celebi" <[hidden email]<mailto:[hidden email]>>:


On 27 Sep 2014, at 08:00, Márton Balassi <[hidden email]<mailto:[hidden email]>> wrote:

I hope I could get you excited too :)

+1 :-)