[DISCUSS]: Integrating Flink Table API & SQL with CEP

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Hi Till,

Could you grant me the edit permission of Flink WIKI? My id is imjark.

Thanks,
Jark Wu

2017-06-15 0:07 GMT+08:00 Till Rohrmann <[hidden email]>:

> I think that the integration of SQL and CEP would make a good FLIP.
>
> Cheers,
> Till
>
> On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu <[hidden email]> wrote:
>
> > Hi,
> >
> > Do you think whether we should create a FLIP for this proposal to track
> > progress?
> >
> > Regards,
> > Jark
> >
> > 2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:
> >
> > > Hi Fabian,
> > >
> > > Thanks a lot. Agree that we can start working by adding the missing
> > > features of the CEP library.
> > >
> > > Best regards,
> > > Dian
> > >
> > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]>
> > wrote:
> > >
> > > > Hi,
> > > >
> > > > @Dian Fu: I gave you contributor permissions. :-)
> > > >
> > > > I don't think we have to wait for Calcite 1.13 to start working on
> the
> > > > missing features of the CEP library and extending the prototype.
> > > > We might want to wait with the integration into flink-table until
> > Calcite
> > > > 1.13 is out and we updated the dependency though.
> > > >
> > > > Best, Fabian
> > > >
> > > > 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:
> > > >
> > > > > Hi Jark, Dian,
> > > > >
> > > > > Thanks for bring up this discuss and share the prototype.
> > > > >
> > > > > +1 to push this great feature forward!
> > > > >
> > > > > Cheers,
> > > > > SunJincheng
> > > > >
> > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > > > >
> > > > > > Thank you Yueting for pointing out the mistake in the prototype.
> I
> > > > > > accidentally introduced it when merge code.
> > > > > >
> > > > > > I'm so glad to see so many people are interested in the feature.
> > > Let's
> > > > > work
> > > > > > out together to push it forward!
> > > > > >
> > > > > > Cheers,
> > > > > > Jark Wu
> > > > > >
> > > > > >
> > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> > > > > >
> > > > > > > +1 for the feature. Myself was a user of Siddhi, this is pretty
> > > user
> > > > > > > friendly feature to provide to user.
> > > > > > >
> > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> [hidden email]>
> > > > > wrote:
> > > > > > >
> > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > >
> > > > > > > > Very glad that you're interested in this feature and you're
> > > > > definitely
> > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Dian
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > [hidden email]> wrote:
> > > > > > > >
> > > > > > > > > Hi all,
> > > > > > > > >
> > > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > > Unfortunately I
> > > > > > > > had
> > > > > > > > > time just for a brief look at the design doc, but it looks
> > > really
> > > > > > cool
> > > > > > > > and
> > > > > > > > > thorough. Also will have a second run tomorrow and will try
> > to
> > > > > > provide
> > > > > > > > more
> > > > > > > > > comments. Anyway will be glad to help pushing the
> initiative
> > > > > forward.
> > > > > > > > >
> > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > >
> > > > > > > > > Dawid Wysakowicz
> > > > > > > > >
> > > > > > > > > *Data/Software Engineer*
> > > > > > > > >
> > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > >
> > > > > > > > > <http://getindata.com/>
> > > > > > > > >
> > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> [hidden email]
> > >:
> > > > > > > > >
> > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > >
> > > > > > > > > > I checked out your prototype code, but it didn't pass the
> > > > > CEPITCase
> > > > > > > > test
> > > > > > > > > in
> > > > > > > > > > the flink-table component.
> > > > > > > > > > It turns out that in the `MatchCodeGenerator.scala` file,
> > > line
> > > > 74
> > > > > > > > should
> > > > > > > > > > use `${classOf[IterativeCondition.
> > > > Context[_]].getCanonicalName}`
> > > > > > > > instead
> > > > > > > > > > of
> > > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > > >
> > > > > > > > > > I've also read your desgin document and it looks fine to
> > me.
> > > > > > > Actually,
> > > > > > > > I
> > > > > > > > > am
> > > > > > > > > > working on the same thing recently, I think maybe we can
> > work
> > > > > > > together
> > > > > > > > to
> > > > > > > > > > push this forward.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Yueting Chen
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Fabian,
> > > > > > > > > > >
> > > > > > > > > > > We have evaluated the missing features of Flink CEP
> > > roughly,
> > > > it
> > > > > > > > should
> > > > > > > > > > not
> > > > > > > > > > > be quite difficult to support them. Kostas, Dawid,
> what's
> > > > your
> > > > > > > > thought?
> > > > > > > > > > >
> > > > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we could
> > > create
> > > > > the
> > > > > > > > JIRAs
> > > > > > > > > > and
> > > > > > > > > > > start to work right now or we should wait until the
> > release
> > > > of
> > > > > > > > calcite
> > > > > > > > > > > 1.13?
> > > > > > > > > > >
> > > > > > > > > > > Btw, could you help to add me(dian.fu) to the
> contributor
> > > > list,
> > > > > > > then
> > > > > > > > I
> > > > > > > > > > can
> > > > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Dian
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Jark,
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks for updating the design doc and sharing your
> > > > > prototype!
> > > > > > > > > > > > I didn't look at the code in detail, but the fact
> that
> > it
> > > > is
> > > > > > less
> > > > > > > > > than
> > > > > > > > > > 1k
> > > > > > > > > > > > LOC is very promising. It seems that most of the
> > complex
> > > > CEP
> > > > > > > logic
> > > > > > > > > can
> > > > > > > > > > be
> > > > > > > > > > > > reused :-)
> > > > > > > > > > > > Adding a dependency on flink-cep should be fine, IMO.
> > It
> > > > is a
> > > > > > > very
> > > > > > > > > slim
> > > > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding the missing features of Flink CEP that you
> > > listed
> > > > > in
> > > > > > > the
> > > > > > > > > > design
> > > > > > > > > > > > doc, it would be good to get some in put from Kostas
> > and
> > > > > Dawid
> > > > > > > > which
> > > > > > > > > > are
> > > > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > > > Do you have already plans regarding some of the
> missing
> > > > > > features
> > > > > > > or
> > > > > > > > > can
> > > > > > > > > > > you
> > > > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers, Fabian
> > > > > > > > > > > >
> > > > > > > > > > > > Btw. The Calcite community started a discussion about
> > > > > releasing
> > > > > > > > > Calcite
> > > > > > > > > > > > 1.13. So, the missing features might soon be
> available.
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]
> >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Good news! We have made a prototype for integrating
> > CEP
> > > > and
> > > > > > > SQL.
> > > > > > > > > See
> > > > > > > > > > > this
> > > > > > > > > > > > > link
> > > > > > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > You can check CepITCase to try the simple CQL
> > example.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Meanwhile, we updated our design doc with
> additional
> > > > > > > > implementation
> > > > > > > > > > > > detail,
> > > > > > > > > > > > > including how
> > > > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the
> > > > features
> > > > > > > > needed
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > > > >
> > > > > > > > > > > > > In the prototype, we make flink-table dependency on
> > > > > > flink-cep.
> > > > > > > Do
> > > > > > > > > you
> > > > > > > > > > > > think
> > > > > > > > > > > > > that is fine?
> > > > > > > > > > > > >
> > > > > > > > > > > > > What do you think about the prototype and the
> design
> > > doc
> > > > ?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for sharing your ideas with the
> community. I
> > > > > really
> > > > > > > like
> > > > > > > > > the
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > document and think that it's a good approach to
> > > follow
> > > > > > > Oracle's
> > > > > > > > > SQL
> > > > > > > > > > > > > > extension for pattern matching. Looking forward
> to
> > > > having
> > > > > > > > support
> > > > > > > > > > for
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Till
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your
> support.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > @Fabian, I totally agree with you that we
> should
> > > > focus
> > > > > on
> > > > > > > SQL
> > > > > > > > > > > first.
> > > > > > > > > > > > > > Let's
> > > > > > > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure
> > > about
> > > > > > > that. I
> > > > > > > > > > think
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > makes sense to make it required in streaming
> > > > > mode(either
> > > > > > > > order
> > > > > > > > > by
> > > > > > > > > > > > > rowtime
> > > > > > > > > > > > > > > or order by proctime). But CEP also works in
> > batch
> > > > > mode,
> > > > > > > and
> > > > > > > > > not
> > > > > > > > > > > > > > necessary
> > > > > > > > > > > > > > > to order by some column. Nevertheless, we can
> > > support
> > > > > CEP
> > > > > > > on
> > > > > > > > > > batch
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > later.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > We are estimating how to implement
> > MATCH_RECOGNIZE
> > > > with
> > > > > > CEP
> > > > > > > > > > library
> > > > > > > > > > > > > (with
> > > > > > > > > > > > > > > NFA, CEP operator). And we will output a
> detailed
> > > doc
> > > > > > and a
> > > > > > > > > > > prototype
> > > > > > > > > > > > > in
> > > > > > > > > > > > > > > the next days.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have
> been
> > > > > thinking
> > > > > > > > about
> > > > > > > > > > > this
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > >> some time but haven't had time to work on this
> > > > > feature.
> > > > > > > > > > > > > > >> I think it would be a great addition and value
> > add
> > > > for
> > > > > > > > Flink's
> > > > > > > > > > SQL
> > > > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> I read the proposal and think it is very good.
> > We
> > > > > might
> > > > > > > need
> > > > > > > > > to
> > > > > > > > > > > add
> > > > > > > > > > > > a
> > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > >> more details, esp. when planning the concrete
> > > steps
> > > > of
> > > > > > the
> > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > > > >> - IMO, the development should start focusing
> on
> > > SQL
> > > > > and
> > > > > > > its
> > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > >> Pattern support for the Table API should be
> > added
> > > > > later.
> > > > > > > We
> > > > > > > > > > > followed
> > > > > > > > > > > > > > that
> > > > > > > > > > > > > > >> approach for the OVER windows and I think it
> > > worked
> > > > > > quiet
> > > > > > > > > well.
> > > > > > > > > > > > > > >> - We probably want to reuse as much as
> possible
> > > from
> > > > > the
> > > > > > > CEP
> > > > > > > > > > > > library.
> > > > > > > > > > > > > > >> That means we need to check if the semantics
> of
> > > the
> > > > > CEP
> > > > > > > > > library
> > > > > > > > > > > and
> > > > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we
> > can
> > > > > > express
> > > > > > > > the
> > > > > > > > > > > > PATTERN
> > > > > > > > > > > > > > >> semantics with the CEP library). This should
> be
> > > one
> > > > of
> > > > > > the
> > > > > > > > > first
> > > > > > > > > > > > > steps,
> > > > > > > > > > > > > > IMO.
> > > > > > > > > > > > > > >> - I would make the orderBy() clause required.
> In
> > > > > regular
> > > > > > > SQL
> > > > > > > > > > rows
> > > > > > > > > > > > have
> > > > > > > > > > > > > > no
> > > > > > > > > > > > > > >> order, so we need to make that explicit (this
> > > would
> > > > > also
> > > > > > > be
> > > > > > > > > > > > consistent
> > > > > > > > > > > > > > with
> > > > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> This is a really interesting idea that has
> been
> > > in
> > > > > our
> > > > > > > > heads
> > > > > > > > > > > > > > >>> since the first implementation of the CEP
> > > library.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> And as for the design document, I will
> > definitely
> > > > > have
> > > > > > a
> > > > > > > > look
> > > > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
> @Timo,
> > > > @Till,
> > > > > > > > @Kostas
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > > > [hidden email]
> > > > > >:
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Dian and me and our teammates have
> > investigated
> > > > this
> > > > > > > for a
> > > > > > > > > > long
> > > > > > > > > > > > > time.
> > > > > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP is
> an
> > > > > > exciting
> > > > > > > > > thing
> > > > > > > > > > > for
> > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > >>>> It'll make SQL more powerful and give users
> > the
> > > > > > ability
> > > > > > > to
> > > > > > > > > > > easily
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > >>>> quickly build CEP applications.  And I find
> > > Flink
> > > > > > > > community
> > > > > > > > > > has
> > > > > > > > > > > > also
> > > > > > > > > > > > > > talked
> > > > > > > > > > > > > > >>>> about this idea before, such as the mailing
> > list
> > > > [1]
> > > > > > and
> > > > > > > > [2]
> > > > > > > > > > and
> > > > > > > > > > > > > > Fabian &
> > > > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this
> > topic
> > > > > > again.
> > > > > > > > > > Because
> > > > > > > > > > > we
> > > > > > > > > > > > > > >>>> already have pattern matching foundation in
> > > Flink
> > > > > CEP
> > > > > > > > > library,
> > > > > > > > > > > and
> > > > > > > > > > > > > > Stream
> > > > > > > > > > > > > > >>>> SQL is ready now and Calcite has partially
> > > > supported
> > > > > > > > pattern
> > > > > > > > > > > > > matching
> > > > > > > > > > > > > > >>>> syntax!  We also drafted a design doc about
> > how
> > > to
> > > > > > > > integrate
> > > > > > > > > > SQL
> > > > > > > > > > > > and
> > > > > > > > > > > > > > CEP,
> > > > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > > > sharing
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you
> > > into
> > > > > this
> > > > > > > > > > > discussion,
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > > > list-archive.1008284.n3
> > > > > > > .
> > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-
> td9743.html#a9787
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > > > list-archive.1008284.n3
> > > > > > > .
> > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > >>>> om/Effort-to-add-SQL-
> > StreamSQL-to-Flink-td9727.
> > > > > > > html#a9790
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > > > tillrohrmann/streaming-
> > > > > > > > > > > analytics-
> > > > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Flink's CEP library is a great library for
> > > > complex
> > > > > > > event
> > > > > > > > > > > > > processing,
> > > > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > > > >>>>> and more customers are expressing their
> > > interests
> > > > > in
> > > > > > > it.
> > > > > > > > > But
> > > > > > > > > > it
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > > > >>>>> some limitations that users usually have to
> > > > write a
> > > > > > lot
> > > > > > > > of
> > > > > > > > > > code
> > > > > > > > > > > > > even
> > > > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > > > >>>>> very simple pattern match use case as it
> > > > currently
> > > > > > only
> > > > > > > > > > > supports
> > > > > > > > > > > > > the
> > > > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > > > >>>>> We have investigated some popular CEP
> > products
> > > > such
> > > > > > as
> > > > > > > > > esper
> > > > > > > > > > > [1]
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > > > >>>>> [2] and found that most of these CEP
> products
> > > > > support
> > > > > > > > > > SQL-like
> > > > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > > > >>>>> such as EPL to describe the match pattern.
> > But
> > > > > these
> > > > > > > > > > solutions
> > > > > > > > > > > > also
> > > > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > > > >>>>> the drawbacks that the pattern match
> > languages
> > > > are
> > > > > > not
> > > > > > > > > > standard
> > > > > > > > > > > > > SQL,
> > > > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > > > >>>>> learn curve is steep for users and it's
> > > > impossible
> > > > > to
> > > > > > > > > > integrate
> > > > > > > > > > > > > them
> > > > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3]
> > > > > supports a
> > > > > > > new
> > > > > > > > > > > pattern
> > > > > > > > > > > > > > >>>>> recognition clause match_recognize which
> is a
> > > > > pattern
> > > > > > > > > > > recognition
> > > > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a
> set
> > > of
> > > > > new
> > > > > > > > > syntaxes
> > > > > > > > > > > to
> > > > > > > > > > > > > > define
> > > > > > > > > > > > > > >>>>> match pattern in sql expression. Calcite
> > > already
> > > > > > > supports
> > > > > > > > > > part
> > > > > > > > > > > of
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > >>>>> standard [5].  I think it will be of great
> > > value
> > > > to
> > > > > > > > support
> > > > > > > > > > > > > > expressing
> > > > > > > > > > > > > > >>>>> pattern recognition clause with
> > match_recognize
> > > > > > clause
> > > > > > > by
> > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink
> CEP
> > > > > library.
> > > > > > > Any
> > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > > > > > eventprocessing/cql-
> > > > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > > > CQLLR1531
> > > > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > > > classes/winter17/cs240B/notes/
> > > > > > > > > row-pat
> > > > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Till Rohrmann
@Jark: You should now have the permissions to create pages in the Flink
wiki.

Cheers,
Till

On Thu, Jun 15, 2017 at 5:11 AM, Jark Wu <[hidden email]> wrote:

> Hi Till,
>
> Could you grant me the edit permission of Flink WIKI? My id is imjark.
>
> Thanks,
> Jark Wu
>
> 2017-06-15 0:07 GMT+08:00 Till Rohrmann <[hidden email]>:
>
> > I think that the integration of SQL and CEP would make a good FLIP.
> >
> > Cheers,
> > Till
> >
> > On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu <[hidden email]> wrote:
> >
> > > Hi,
> > >
> > > Do you think whether we should create a FLIP for this proposal to track
> > > progress?
> > >
> > > Regards,
> > > Jark
> > >
> > > 2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:
> > >
> > > > Hi Fabian,
> > > >
> > > > Thanks a lot. Agree that we can start working by adding the missing
> > > > features of the CEP library.
> > > >
> > > > Best regards,
> > > > Dian
> > > >
> > > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > @Dian Fu: I gave you contributor permissions. :-)
> > > > >
> > > > > I don't think we have to wait for Calcite 1.13 to start working on
> > the
> > > > > missing features of the CEP library and extending the prototype.
> > > > > We might want to wait with the integration into flink-table until
> > > Calcite
> > > > > 1.13 is out and we updated the dependency though.
> > > > >
> > > > > Best, Fabian
> > > > >
> > > > > 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:
> > > > >
> > > > > > Hi Jark, Dian,
> > > > > >
> > > > > > Thanks for bring up this discuss and share the prototype.
> > > > > >
> > > > > > +1 to push this great feature forward!
> > > > > >
> > > > > > Cheers,
> > > > > > SunJincheng
> > > > > >
> > > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > >
> > > > > > > Thank you Yueting for pointing out the mistake in the
> prototype.
> > I
> > > > > > > accidentally introduced it when merge code.
> > > > > > >
> > > > > > > I'm so glad to see so many people are interested in the
> feature.
> > > > Let's
> > > > > > work
> > > > > > > out together to push it forward!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jark Wu
> > > > > > >
> > > > > > >
> > > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> > > > > > >
> > > > > > > > +1 for the feature. Myself was a user of Siddhi, this is
> pretty
> > > > user
> > > > > > > > friendly feature to provide to user.
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > > >
> > > > > > > > > Very glad that you're interested in this feature and you're
> > > > > > definitely
> > > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Dian
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > > [hidden email]> wrote:
> > > > > > > > >
> > > > > > > > > > Hi all,
> > > > > > > > > >
> > > > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > > > Unfortunately I
> > > > > > > > > had
> > > > > > > > > > time just for a brief look at the design doc, but it
> looks
> > > > really
> > > > > > > cool
> > > > > > > > > and
> > > > > > > > > > thorough. Also will have a second run tomorrow and will
> try
> > > to
> > > > > > > provide
> > > > > > > > > more
> > > > > > > > > > comments. Anyway will be glad to help pushing the
> > initiative
> > > > > > forward.
> > > > > > > > > >
> > > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > > >
> > > > > > > > > > Dawid Wysakowicz
> > > > > > > > > >
> > > > > > > > > > *Data/Software Engineer*
> > > > > > > > > >
> > > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > > >
> > > > > > > > > > <http://getindata.com/>
> > > > > > > > > >
> > > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> > [hidden email]
> > > >:
> > > > > > > > > >
> > > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > > >
> > > > > > > > > > > I checked out your prototype code, but it didn't pass
> the
> > > > > > CEPITCase
> > > > > > > > > test
> > > > > > > > > > in
> > > > > > > > > > > the flink-table component.
> > > > > > > > > > > It turns out that in the `MatchCodeGenerator.scala`
> file,
> > > > line
> > > > > 74
> > > > > > > > > should
> > > > > > > > > > > use `${classOf[IterativeCondition.
> > > > > Context[_]].getCanonicalName}`
> > > > > > > > > instead
> > > > > > > > > > > of
> > > > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > > > >
> > > > > > > > > > > I've also read your desgin document and it looks fine
> to
> > > me.
> > > > > > > > Actually,
> > > > > > > > > I
> > > > > > > > > > am
> > > > > > > > > > > working on the same thing recently, I think maybe we
> can
> > > work
> > > > > > > > together
> > > > > > > > > to
> > > > > > > > > > > push this forward.
> > > > > > > > > > >
> > > > > > > > > > > Thanks,
> > > > > > > > > > > Yueting Chen
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Fabian,
> > > > > > > > > > > >
> > > > > > > > > > > > We have evaluated the missing features of Flink CEP
> > > > roughly,
> > > > > it
> > > > > > > > > should
> > > > > > > > > > > not
> > > > > > > > > > > > be quite difficult to support them. Kostas, Dawid,
> > what's
> > > > > your
> > > > > > > > > thought?
> > > > > > > > > > > >
> > > > > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we could
> > > > create
> > > > > > the
> > > > > > > > > JIRAs
> > > > > > > > > > > and
> > > > > > > > > > > > start to work right now or we should wait until the
> > > release
> > > > > of
> > > > > > > > > calcite
> > > > > > > > > > > > 1.13?
> > > > > > > > > > > >
> > > > > > > > > > > > Btw, could you help to add me(dian.fu) to the
> > contributor
> > > > > list,
> > > > > > > > then
> > > > > > > > > I
> > > > > > > > > > > can
> > > > > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > > > > >
> > > > > > > > > > > > Best regards,
> > > > > > > > > > > > Dian
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Jark,
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for updating the design doc and sharing your
> > > > > > prototype!
> > > > > > > > > > > > > I didn't look at the code in detail, but the fact
> > that
> > > it
> > > > > is
> > > > > > > less
> > > > > > > > > > than
> > > > > > > > > > > 1k
> > > > > > > > > > > > > LOC is very promising. It seems that most of the
> > > complex
> > > > > CEP
> > > > > > > > logic
> > > > > > > > > > can
> > > > > > > > > > > be
> > > > > > > > > > > > > reused :-)
> > > > > > > > > > > > > Adding a dependency on flink-cep should be fine,
> IMO.
> > > It
> > > > > is a
> > > > > > > > very
> > > > > > > > > > slim
> > > > > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding the missing features of Flink CEP that
> you
> > > > listed
> > > > > > in
> > > > > > > > the
> > > > > > > > > > > design
> > > > > > > > > > > > > doc, it would be good to get some in put from
> Kostas
> > > and
> > > > > > Dawid
> > > > > > > > > which
> > > > > > > > > > > are
> > > > > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > > > > Do you have already plans regarding some of the
> > missing
> > > > > > > features
> > > > > > > > or
> > > > > > > > > > can
> > > > > > > > > > > > you
> > > > > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers, Fabian
> > > > > > > > > > > > >
> > > > > > > > > > > > > Btw. The Calcite community started a discussion
> about
> > > > > > releasing
> > > > > > > > > > Calcite
> > > > > > > > > > > > > 1.13. So, the missing features might soon be
> > available.
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <
> [hidden email]
> > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Good news! We have made a prototype for
> integrating
> > > CEP
> > > > > and
> > > > > > > > SQL.
> > > > > > > > > > See
> > > > > > > > > > > > this
> > > > > > > > > > > > > > link
> > > > > > > > > > > > > > https://github.com/apache/flink/compare/master.
> ..
> > > > > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > You can check CepITCase to try the simple CQL
> > > example.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Meanwhile, we updated our design doc with
> > additional
> > > > > > > > > implementation
> > > > > > > > > > > > > detail,
> > > > > > > > > > > > > > including how
> > > > > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and
> the
> > > > > features
> > > > > > > > > needed
> > > > > > > > > > to
> > > > > > > > > > > > add
> > > > > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > In the prototype, we make flink-table dependency
> on
> > > > > > > flink-cep.
> > > > > > > > Do
> > > > > > > > > > you
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > that is fine?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > What do you think about the prototype and the
> > design
> > > > doc
> > > > > ?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for sharing your ideas with the
> > community. I
> > > > > > really
> > > > > > > > like
> > > > > > > > > > the
> > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > document and think that it's a good approach to
> > > > follow
> > > > > > > > Oracle's
> > > > > > > > > > SQL
> > > > > > > > > > > > > > > extension for pattern matching. Looking forward
> > to
> > > > > having
> > > > > > > > > support
> > > > > > > > > > > for
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your
> > support.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > @Fabian, I totally agree with you that we
> > should
> > > > > focus
> > > > > > on
> > > > > > > > SQL
> > > > > > > > > > > > first.
> > > > > > > > > > > > > > > Let's
> > > > > > > > > > > > > > > > keep Table API in mind and discuss that
> later.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not
> sure
> > > > about
> > > > > > > > that. I
> > > > > > > > > > > think
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > makes sense to make it required in streaming
> > > > > > mode(either
> > > > > > > > > order
> > > > > > > > > > by
> > > > > > > > > > > > > > rowtime
> > > > > > > > > > > > > > > > or order by proctime). But CEP also works in
> > > batch
> > > > > > mode,
> > > > > > > > and
> > > > > > > > > > not
> > > > > > > > > > > > > > > necessary
> > > > > > > > > > > > > > > > to order by some column. Nevertheless, we can
> > > > support
> > > > > > CEP
> > > > > > > > on
> > > > > > > > > > > batch
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > later.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > We are estimating how to implement
> > > MATCH_RECOGNIZE
> > > > > with
> > > > > > > CEP
> > > > > > > > > > > library
> > > > > > > > > > > > > > (with
> > > > > > > > > > > > > > > > NFA, CEP operator). And we will output a
> > detailed
> > > > doc
> > > > > > > and a
> > > > > > > > > > > > prototype
> > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > the next days.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > > > > > [hidden email]
> > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have
> > been
> > > > > > thinking
> > > > > > > > > about
> > > > > > > > > > > > this
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > >> some time but haven't had time to work on
> this
> > > > > > feature.
> > > > > > > > > > > > > > > >> I think it would be a great addition and
> value
> > > add
> > > > > for
> > > > > > > > > Flink's
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> I read the proposal and think it is very
> good.
> > > We
> > > > > > might
> > > > > > > > need
> > > > > > > > > > to
> > > > > > > > > > > > add
> > > > > > > > > > > > > a
> > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > >> more details, esp. when planning the
> concrete
> > > > steps
> > > > > of
> > > > > > > the
> > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > > > > >> - IMO, the development should start focusing
> > on
> > > > SQL
> > > > > > and
> > > > > > > > its
> > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > >> Pattern support for the Table API should be
> > > added
> > > > > > later.
> > > > > > > > We
> > > > > > > > > > > > followed
> > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > >> approach for the OVER windows and I think it
> > > > worked
> > > > > > > quiet
> > > > > > > > > > well.
> > > > > > > > > > > > > > > >> - We probably want to reuse as much as
> > possible
> > > > from
> > > > > > the
> > > > > > > > CEP
> > > > > > > > > > > > > library.
> > > > > > > > > > > > > > > >> That means we need to check if the semantics
> > of
> > > > the
> > > > > > CEP
> > > > > > > > > > library
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how
> we
> > > can
> > > > > > > express
> > > > > > > > > the
> > > > > > > > > > > > > PATTERN
> > > > > > > > > > > > > > > >> semantics with the CEP library). This should
> > be
> > > > one
> > > > > of
> > > > > > > the
> > > > > > > > > > first
> > > > > > > > > > > > > > steps,
> > > > > > > > > > > > > > > IMO.
> > > > > > > > > > > > > > > >> - I would make the orderBy() clause
> required.
> > In
> > > > > > regular
> > > > > > > > SQL
> > > > > > > > > > > rows
> > > > > > > > > > > > > have
> > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > >> order, so we need to make that explicit
> (this
> > > > would
> > > > > > also
> > > > > > > > be
> > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> This is a really interesting idea that has
> > been
> > > > in
> > > > > > our
> > > > > > > > > heads
> > > > > > > > > > > > > > > >>> since the first implementation of the CEP
> > > > library.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> And as for the design document, I will
> > > definitely
> > > > > > have
> > > > > > > a
> > > > > > > > > look
> > > > > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > > > > [hidden email]>
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
> > @Timo,
> > > > > @Till,
> > > > > > > > > @Kostas
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> Dian and me and our teammates have
> > > investigated
> > > > > this
> > > > > > > > for a
> > > > > > > > > > > long
> > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP
> is
> > an
> > > > > > > exciting
> > > > > > > > > > thing
> > > > > > > > > > > > for
> > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > >>>> It'll make SQL more powerful and give
> users
> > > the
> > > > > > > ability
> > > > > > > > to
> > > > > > > > > > > > easily
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >>>> quickly build CEP applications.  And I
> find
> > > > Flink
> > > > > > > > > community
> > > > > > > > > > > has
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > talked
> > > > > > > > > > > > > > > >>>> about this idea before, such as the
> mailing
> > > list
> > > > > [1]
> > > > > > > and
> > > > > > > > > [2]
> > > > > > > > > > > and
> > > > > > > > > > > > > > > Fabian &
> > > > > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this
> > > topic
> > > > > > > again.
> > > > > > > > > > > Because
> > > > > > > > > > > > we
> > > > > > > > > > > > > > > >>>> already have pattern matching foundation
> in
> > > > Flink
> > > > > > CEP
> > > > > > > > > > library,
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > Stream
> > > > > > > > > > > > > > > >>>> SQL is ready now and Calcite has partially
> > > > > supported
> > > > > > > > > pattern
> > > > > > > > > > > > > > matching
> > > > > > > > > > > > > > > >>>> syntax!  We also drafted a design doc
> about
> > > how
> > > > to
> > > > > > > > > integrate
> > > > > > > > > > > SQL
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > CEP,
> > > > > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > > > > sharing
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include
> you
> > > > into
> > > > > > this
> > > > > > > > > > > > discussion,
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > > > > list-archive.1008284.n3
> > > > > > > > .
> > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-
> > td9743.html#a9787
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > > > > list-archive.1008284.n3
> > > > > > > > .
> > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > >>>> om/Effort-to-add-SQL-
> > > StreamSQL-to-Flink-td9727.
> > > > > > > > html#a9790
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > > > > tillrohrmann/streaming-
> > > > > > > > > > > > analytics-
> > > > > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> Flink's CEP library is a great library
> for
> > > > > complex
> > > > > > > > event
> > > > > > > > > > > > > > processing,
> > > > > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > > > > >>>>> and more customers are expressing their
> > > > interests
> > > > > > in
> > > > > > > > it.
> > > > > > > > > > But
> > > > > > > > > > > it
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > > > > >>>>> some limitations that users usually have
> to
> > > > > write a
> > > > > > > lot
> > > > > > > > > of
> > > > > > > > > > > code
> > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > > > > >>>>> very simple pattern match use case as it
> > > > > currently
> > > > > > > only
> > > > > > > > > > > > supports
> > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > > > > >>>>> We have investigated some popular CEP
> > > products
> > > > > such
> > > > > > > as
> > > > > > > > > > esper
> > > > > > > > > > > > [1]
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > > > > >>>>> [2] and found that most of these CEP
> > products
> > > > > > support
> > > > > > > > > > > SQL-like
> > > > > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > > > > >>>>> such as EPL to describe the match
> pattern.
> > > But
> > > > > > these
> > > > > > > > > > > solutions
> > > > > > > > > > > > > also
> > > > > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > > > > >>>>> the drawbacks that the pattern match
> > > languages
> > > > > are
> > > > > > > not
> > > > > > > > > > > standard
> > > > > > > > > > > > > > SQL,
> > > > > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > > > > >>>>> learn curve is steep for users and it's
> > > > > impossible
> > > > > > to
> > > > > > > > > > > integrate
> > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL
> [3]
> > > > > > supports a
> > > > > > > > new
> > > > > > > > > > > > pattern
> > > > > > > > > > > > > > > >>>>> recognition clause match_recognize which
> > is a
> > > > > > pattern
> > > > > > > > > > > > recognition
> > > > > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a
> > set
> > > > of
> > > > > > new
> > > > > > > > > > syntaxes
> > > > > > > > > > > > to
> > > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > >>>>> match pattern in sql expression. Calcite
> > > > already
> > > > > > > > supports
> > > > > > > > > > > part
> > > > > > > > > > > > of
> > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > >>>>> standard [5].  I think it will be of
> great
> > > > value
> > > > > to
> > > > > > > > > support
> > > > > > > > > > > > > > > expressing
> > > > > > > > > > > > > > > >>>>> pattern recognition clause with
> > > match_recognize
> > > > > > > clause
> > > > > > > > by
> > > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink
> > CEP
> > > > > > library.
> > > > > > > > Any
> > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > > > > > > eventprocessing/cql-
> > > > > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > > > > CQLLR1531
> > > > > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > > > > classes/winter17/cs240B/notes/
> > > > > > > > > > row-pat
> > > > > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Fabian Hueske-2
Hi everybody,

I just stumbled over this blog post [1] which discusses new features in SQL
2016.
Apparently the match_recognize clause is part of that. The blogpost also
contains a slide set that presents the pattern matching feature and a link
to a 90 page technical report.

I thought this might be helpful as a references.

Cheers, Fabian

[1] http://modern-sql.com/blog/2017-06/whats-new-in-sql-2016

2017-06-15 11:53 GMT+02:00 Till Rohrmann <[hidden email]>:

> @Jark: You should now have the permissions to create pages in the Flink
> wiki.
>
> Cheers,
> Till
>
> On Thu, Jun 15, 2017 at 5:11 AM, Jark Wu <[hidden email]> wrote:
>
> > Hi Till,
> >
> > Could you grant me the edit permission of Flink WIKI? My id is imjark.
> >
> > Thanks,
> > Jark Wu
> >
> > 2017-06-15 0:07 GMT+08:00 Till Rohrmann <[hidden email]>:
> >
> > > I think that the integration of SQL and CEP would make a good FLIP.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu <[hidden email]> wrote:
> > >
> > > > Hi,
> > > >
> > > > Do you think whether we should create a FLIP for this proposal to
> track
> > > > progress?
> > > >
> > > > Regards,
> > > > Jark
> > > >
> > > > 2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > Thanks a lot. Agree that we can start working by adding the missing
> > > > > features of the CEP library.
> > > > >
> > > > > Best regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > @Dian Fu: I gave you contributor permissions. :-)
> > > > > >
> > > > > > I don't think we have to wait for Calcite 1.13 to start working
> on
> > > the
> > > > > > missing features of the CEP library and extending the prototype.
> > > > > > We might want to wait with the integration into flink-table until
> > > > Calcite
> > > > > > 1.13 is out and we updated the dependency though.
> > > > > >
> > > > > > Best, Fabian
> > > > > >
> > > > > > 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]
> >:
> > > > > >
> > > > > > > Hi Jark, Dian,
> > > > > > >
> > > > > > > Thanks for bring up this discuss and share the prototype.
> > > > > > >
> > > > > > > +1 to push this great feature forward!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > SunJincheng
> > > > > > >
> > > > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > >
> > > > > > > > Thank you Yueting for pointing out the mistake in the
> > prototype.
> > > I
> > > > > > > > accidentally introduced it when merge code.
> > > > > > > >
> > > > > > > > I'm so glad to see so many people are interested in the
> > feature.
> > > > > Let's
> > > > > > > work
> > > > > > > > out together to push it forward!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jark Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]
> >:
> > > > > > > >
> > > > > > > > > +1 for the feature. Myself was a user of Siddhi, this is
> > pretty
> > > > > user
> > > > > > > > > friendly feature to provide to user.
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > > > >
> > > > > > > > > > Very glad that you're interested in this feature and
> you're
> > > > > > > definitely
> > > > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Dian
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > > > [hidden email]> wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi all,
> > > > > > > > > > >
> > > > > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > > > > Unfortunately I
> > > > > > > > > > had
> > > > > > > > > > > time just for a brief look at the design doc, but it
> > looks
> > > > > really
> > > > > > > > cool
> > > > > > > > > > and
> > > > > > > > > > > thorough. Also will have a second run tomorrow and will
> > try
> > > > to
> > > > > > > > provide
> > > > > > > > > > more
> > > > > > > > > > > comments. Anyway will be glad to help pushing the
> > > initiative
> > > > > > > forward.
> > > > > > > > > > >
> > > > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > > > >
> > > > > > > > > > > Dawid Wysakowicz
> > > > > > > > > > >
> > > > > > > > > > > *Data/Software Engineer*
> > > > > > > > > > >
> > > > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > > > >
> > > > > > > > > > > <http://getindata.com/>
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> > > [hidden email]
> > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > > > >
> > > > > > > > > > > > I checked out your prototype code, but it didn't pass
> > the
> > > > > > > CEPITCase
> > > > > > > > > > test
> > > > > > > > > > > in
> > > > > > > > > > > > the flink-table component.
> > > > > > > > > > > > It turns out that in the `MatchCodeGenerator.scala`
> > file,
> > > > > line
> > > > > > 74
> > > > > > > > > > should
> > > > > > > > > > > > use `${classOf[IterativeCondition.
> > > > > > Context[_]].getCanonicalName}`
> > > > > > > > > > instead
> > > > > > > > > > > > of
> > > > > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > > > > >
> > > > > > > > > > > > I've also read your desgin document and it looks fine
> > to
> > > > me.
> > > > > > > > > Actually,
> > > > > > > > > > I
> > > > > > > > > > > am
> > > > > > > > > > > > working on the same thing recently, I think maybe we
> > can
> > > > work
> > > > > > > > > together
> > > > > > > > > > to
> > > > > > > > > > > > push this forward.
> > > > > > > > > > > >
> > > > > > > > > > > > Thanks,
> > > > > > > > > > > > Yueting Chen
> > > > > > > > > > > >
> > > > > > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <
> > > > [hidden email]
> > > > > >
> > > > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Fabian,
> > > > > > > > > > > > >
> > > > > > > > > > > > > We have evaluated the missing features of Flink CEP
> > > > > roughly,
> > > > > > it
> > > > > > > > > > should
> > > > > > > > > > > > not
> > > > > > > > > > > > > be quite difficult to support them. Kostas, Dawid,
> > > what's
> > > > > > your
> > > > > > > > > > thought?
> > > > > > > > > > > > >
> > > > > > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we
> could
> > > > > create
> > > > > > > the
> > > > > > > > > > JIRAs
> > > > > > > > > > > > and
> > > > > > > > > > > > > start to work right now or we should wait until the
> > > > release
> > > > > > of
> > > > > > > > > > calcite
> > > > > > > > > > > > > 1.13?
> > > > > > > > > > > > >
> > > > > > > > > > > > > Btw, could you help to add me(dian.fu) to the
> > > contributor
> > > > > > list,
> > > > > > > > > then
> > > > > > > > > > I
> > > > > > > > > > > > can
> > > > > > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > Dian
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Jark,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Thanks for updating the design doc and sharing
> your
> > > > > > > prototype!
> > > > > > > > > > > > > > I didn't look at the code in detail, but the fact
> > > that
> > > > it
> > > > > > is
> > > > > > > > less
> > > > > > > > > > > than
> > > > > > > > > > > > 1k
> > > > > > > > > > > > > > LOC is very promising. It seems that most of the
> > > > complex
> > > > > > CEP
> > > > > > > > > logic
> > > > > > > > > > > can
> > > > > > > > > > > > be
> > > > > > > > > > > > > > reused :-)
> > > > > > > > > > > > > > Adding a dependency on flink-cep should be fine,
> > IMO.
> > > > It
> > > > > > is a
> > > > > > > > > very
> > > > > > > > > > > slim
> > > > > > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding the missing features of Flink CEP that
> > you
> > > > > listed
> > > > > > > in
> > > > > > > > > the
> > > > > > > > > > > > design
> > > > > > > > > > > > > > doc, it would be good to get some in put from
> > Kostas
> > > > and
> > > > > > > Dawid
> > > > > > > > > > which
> > > > > > > > > > > > are
> > > > > > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > > > > > Do you have already plans regarding some of the
> > > missing
> > > > > > > > features
> > > > > > > > > or
> > > > > > > > > > > can
> > > > > > > > > > > > > you
> > > > > > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Cheers, Fabian
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Btw. The Calcite community started a discussion
> > about
> > > > > > > releasing
> > > > > > > > > > > Calcite
> > > > > > > > > > > > > > 1.13. So, the missing features might soon be
> > > available.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <
> > [hidden email]
> > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Good news! We have made a prototype for
> > integrating
> > > > CEP
> > > > > > and
> > > > > > > > > SQL.
> > > > > > > > > > > See
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > link
> > > > > > > > > > > > > > > https://github.com/apache/flink/compare/master
> .
> > ..
> > > > > > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > You can check CepITCase to try the simple CQL
> > > > example.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Meanwhile, we updated our design doc with
> > > additional
> > > > > > > > > > implementation
> > > > > > > > > > > > > > detail,
> > > > > > > > > > > > > > > including how
> > > > > > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and
> > the
> > > > > > features
> > > > > > > > > > needed
> > > > > > > > > > > to
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > In the prototype, we make flink-table
> dependency
> > on
> > > > > > > > flink-cep.
> > > > > > > > > Do
> > > > > > > > > > > you
> > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > that is fine?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > What do you think about the prototype and the
> > > design
> > > > > doc
> > > > > > ?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > > > > > [hidden email]
> > > > > > > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Thanks for sharing your ideas with the
> > > community. I
> > > > > > > really
> > > > > > > > > like
> > > > > > > > > > > the
> > > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > document and think that it's a good approach
> to
> > > > > follow
> > > > > > > > > Oracle's
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > extension for pattern matching. Looking
> forward
> > > to
> > > > > > having
> > > > > > > > > > support
> > > > > > > > > > > > for
> > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > > > > > [hidden email]
> > > > > > > >
> > > > > > > > > > wrote:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your
> > > support.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > @Fabian, I totally agree with you that we
> > > should
> > > > > > focus
> > > > > > > on
> > > > > > > > > SQL
> > > > > > > > > > > > > first.
> > > > > > > > > > > > > > > > Let's
> > > > > > > > > > > > > > > > > keep Table API in mind and discuss that
> > later.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not
> > sure
> > > > > about
> > > > > > > > > that. I
> > > > > > > > > > > > think
> > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > makes sense to make it required in
> streaming
> > > > > > > mode(either
> > > > > > > > > > order
> > > > > > > > > > > by
> > > > > > > > > > > > > > > rowtime
> > > > > > > > > > > > > > > > > or order by proctime). But CEP also works
> in
> > > > batch
> > > > > > > mode,
> > > > > > > > > and
> > > > > > > > > > > not
> > > > > > > > > > > > > > > > necessary
> > > > > > > > > > > > > > > > > to order by some column. Nevertheless, we
> can
> > > > > support
> > > > > > > CEP
> > > > > > > > > on
> > > > > > > > > > > > batch
> > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > later.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > We are estimating how to implement
> > > > MATCH_RECOGNIZE
> > > > > > with
> > > > > > > > CEP
> > > > > > > > > > > > library
> > > > > > > > > > > > > > > (with
> > > > > > > > > > > > > > > > > NFA, CEP operator). And we will output a
> > > detailed
> > > > > doc
> > > > > > > > and a
> > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > the next days.
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > > > > > > [hidden email]
> > > > > > > > > > >:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have
> > > been
> > > > > > > thinking
> > > > > > > > > > about
> > > > > > > > > > > > > this
> > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > >> some time but haven't had time to work on
> > this
> > > > > > > feature.
> > > > > > > > > > > > > > > > >> I think it would be a great addition and
> > value
> > > > add
> > > > > > for
> > > > > > > > > > Flink's
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> I read the proposal and think it is very
> > good.
> > > > We
> > > > > > > might
> > > > > > > > > need
> > > > > > > > > > > to
> > > > > > > > > > > > > add
> > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > >> more details, esp. when planning the
> > concrete
> > > > > steps
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > > > > > >> - IMO, the development should start
> focusing
> > > on
> > > > > SQL
> > > > > > > and
> > > > > > > > > its
> > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > >> Pattern support for the Table API should
> be
> > > > added
> > > > > > > later.
> > > > > > > > > We
> > > > > > > > > > > > > followed
> > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > >> approach for the OVER windows and I think
> it
> > > > > worked
> > > > > > > > quiet
> > > > > > > > > > > well.
> > > > > > > > > > > > > > > > >> - We probably want to reuse as much as
> > > possible
> > > > > from
> > > > > > > the
> > > > > > > > > CEP
> > > > > > > > > > > > > > library.
> > > > > > > > > > > > > > > > >> That means we need to check if the
> semantics
> > > of
> > > > > the
> > > > > > > CEP
> > > > > > > > > > > library
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or
> how
> > we
> > > > can
> > > > > > > > express
> > > > > > > > > > the
> > > > > > > > > > > > > > PATTERN
> > > > > > > > > > > > > > > > >> semantics with the CEP library). This
> should
> > > be
> > > > > one
> > > > > > of
> > > > > > > > the
> > > > > > > > > > > first
> > > > > > > > > > > > > > > steps,
> > > > > > > > > > > > > > > > IMO.
> > > > > > > > > > > > > > > > >> - I would make the orderBy() clause
> > required.
> > > In
> > > > > > > regular
> > > > > > > > > SQL
> > > > > > > > > > > > rows
> > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > >> order, so we need to make that explicit
> > (this
> > > > > would
> > > > > > > also
> > > > > > > > > be
> > > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas
> <
> > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> This is a really interesting idea that
> has
> > > been
> > > > > in
> > > > > > > our
> > > > > > > > > > heads
> > > > > > > > > > > > > > > > >>> since the first implementation of the CEP
> > > > > library.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> And as for the design document, I will
> > > > definitely
> > > > > > > have
> > > > > > > > a
> > > > > > > > > > look
> > > > > > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > > > > > [hidden email]>
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
> > > @Timo,
> > > > > > @Till,
> > > > > > > > > > @Kostas
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> Dian and me and our teammates have
> > > > investigated
> > > > > > this
> > > > > > > > > for a
> > > > > > > > > > > > long
> > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP
> > is
> > > an
> > > > > > > > exciting
> > > > > > > > > > > thing
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > >>>> It'll make SQL more powerful and give
> > users
> > > > the
> > > > > > > > ability
> > > > > > > > > to
> > > > > > > > > > > > > easily
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >>>> quickly build CEP applications.  And I
> > find
> > > > > Flink
> > > > > > > > > > community
> > > > > > > > > > > > has
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > talked
> > > > > > > > > > > > > > > > >>>> about this idea before, such as the
> > mailing
> > > > list
> > > > > > [1]
> > > > > > > > and
> > > > > > > > > > [2]
> > > > > > > > > > > > and
> > > > > > > > > > > > > > > > Fabian &
> > > > > > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up
> this
> > > > topic
> > > > > > > > again.
> > > > > > > > > > > > Because
> > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > >>>> already have pattern matching foundation
> > in
> > > > > Flink
> > > > > > > CEP
> > > > > > > > > > > library,
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > Stream
> > > > > > > > > > > > > > > > >>>> SQL is ready now and Calcite has
> partially
> > > > > > supported
> > > > > > > > > > pattern
> > > > > > > > > > > > > > > matching
> > > > > > > > > > > > > > > > >>>> syntax!  We also drafted a design doc
> > about
> > > > how
> > > > > to
> > > > > > > > > > integrate
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > CEP,
> > > > > > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > > > > > sharing
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include
> > you
> > > > > into
> > > > > > > this
> > > > > > > > > > > > > discussion,
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > > > > > list-archive.1008284.n3
> > > > > > > > > .
> > > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-
> > > td9743.html#a9787
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > > > > > list-archive.1008284.n3
> > > > > > > > > .
> > > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > > >>>> om/Effort-to-add-SQL-
> > > > StreamSQL-to-Flink-td9727.
> > > > > > > > > html#a9790
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > > > > > tillrohrmann/streaming-
> > > > > > > > > > > > > analytics-
> > > > > > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> Flink's CEP library is a great library
> > for
> > > > > > complex
> > > > > > > > > event
> > > > > > > > > > > > > > > processing,
> > > > > > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > > > > > >>>>> and more customers are expressing their
> > > > > interests
> > > > > > > in
> > > > > > > > > it.
> > > > > > > > > > > But
> > > > > > > > > > > > it
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > > > > > >>>>> some limitations that users usually
> have
> > to
> > > > > > write a
> > > > > > > > lot
> > > > > > > > > > of
> > > > > > > > > > > > code
> > > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > > > > > >>>>> very simple pattern match use case as
> it
> > > > > > currently
> > > > > > > > only
> > > > > > > > > > > > > supports
> > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > > > > > >>>>> We have investigated some popular CEP
> > > > products
> > > > > > such
> > > > > > > > as
> > > > > > > > > > > esper
> > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > > > > > >>>>> [2] and found that most of these CEP
> > > products
> > > > > > > support
> > > > > > > > > > > > SQL-like
> > > > > > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > > > > > >>>>> such as EPL to describe the match
> > pattern.
> > > > But
> > > > > > > these
> > > > > > > > > > > > solutions
> > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > > > > > >>>>> the drawbacks that the pattern match
> > > > languages
> > > > > > are
> > > > > > > > not
> > > > > > > > > > > > standard
> > > > > > > > > > > > > > > SQL,
> > > > > > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > > > > > >>>>> learn curve is steep for users and it's
> > > > > > impossible
> > > > > > > to
> > > > > > > > > > > > integrate
> > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL
> > [3]
> > > > > > > supports a
> > > > > > > > > new
> > > > > > > > > > > > > pattern
> > > > > > > > > > > > > > > > >>>>> recognition clause match_recognize
> which
> > > is a
> > > > > > > pattern
> > > > > > > > > > > > > recognition
> > > > > > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > > > > > >>>>> proposed in this paper [4]. It
> proposes a
> > > set
> > > > > of
> > > > > > > new
> > > > > > > > > > > syntaxes
> > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > >>>>> match pattern in sql expression.
> Calcite
> > > > > already
> > > > > > > > > supports
> > > > > > > > > > > > part
> > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > >>>>> standard [5].  I think it will be of
> > great
> > > > > value
> > > > > > to
> > > > > > > > > > support
> > > > > > > > > > > > > > > > expressing
> > > > > > > > > > > > > > > > >>>>> pattern recognition clause with
> > > > match_recognize
> > > > > > > > clause
> > > > > > > > > by
> > > > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > > > > > >>>>> with Flink Table API & SQL and the
> Flink
> > > CEP
> > > > > > > library.
> > > > > > > > > Any
> > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > > > > > >>>>> https://docs.oracle.com/
> middleware/1213/
> > > > > > > > > > > eventprocessing/cql-
> > > > > > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > > > > > CQLLR1531
> > > > > > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > > > > > classes/winter17/cs240B/notes/
> > > > > > > > > > > row-pat
> > > > > > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > > > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Thanks Fabian for the standard Row Pattern Recognition in SQL!

I just created a FLIP for this feature.

FILP-20:
https://cwiki.apache.org/confluence/display/FLINK/FLIP-20%3A+Integration+of+SQL+and+CEP

Cheers,
Jark Wu

2017-06-16 0:49 GMT+08:00 Fabian Hueske <[hidden email]>:

> Hi everybody,
>
> I just stumbled over this blog post [1] which discusses new features in SQL
> 2016.
> Apparently the match_recognize clause is part of that. The blogpost also
> contains a slide set that presents the pattern matching feature and a link
> to a 90 page technical report.
>
> I thought this might be helpful as a references.
>
> Cheers, Fabian
>
> [1] http://modern-sql.com/blog/2017-06/whats-new-in-sql-2016
>
> 2017-06-15 11:53 GMT+02:00 Till Rohrmann <[hidden email]>:
>
> > @Jark: You should now have the permissions to create pages in the Flink
> > wiki.
> >
> > Cheers,
> > Till
> >
> > On Thu, Jun 15, 2017 at 5:11 AM, Jark Wu <[hidden email]> wrote:
> >
> > > Hi Till,
> > >
> > > Could you grant me the edit permission of Flink WIKI? My id is imjark.
> > >
> > > Thanks,
> > > Jark Wu
> > >
> > > 2017-06-15 0:07 GMT+08:00 Till Rohrmann <[hidden email]>:
> > >
> > > > I think that the integration of SQL and CEP would make a good FLIP.
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu <[hidden email]> wrote:
> > > >
> > > > > Hi,
> > > > >
> > > > > Do you think whether we should create a FLIP for this proposal to
> > track
> > > > > progress?
> > > > >
> > > > > Regards,
> > > > > Jark
> > > > >
> > > > > 2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:
> > > > >
> > > > > > Hi Fabian,
> > > > > >
> > > > > > Thanks a lot. Agree that we can start working by adding the
> missing
> > > > > > features of the CEP library.
> > > > > >
> > > > > > Best regards,
> > > > > > Dian
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Hi,
> > > > > > >
> > > > > > > @Dian Fu: I gave you contributor permissions. :-)
> > > > > > >
> > > > > > > I don't think we have to wait for Calcite 1.13 to start working
> > on
> > > > the
> > > > > > > missing features of the CEP library and extending the
> prototype.
> > > > > > > We might want to wait with the integration into flink-table
> until
> > > > > Calcite
> > > > > > > 1.13 is out and we updated the dependency though.
> > > > > > >
> > > > > > > Best, Fabian
> > > > > > >
> > > > > > > 2017-06-13 9:45 GMT+02:00 jincheng sun <
> [hidden email]
> > >:
> > > > > > >
> > > > > > > > Hi Jark, Dian,
> > > > > > > >
> > > > > > > > Thanks for bring up this discuss and share the prototype.
> > > > > > > >
> > > > > > > > +1 to push this great feature forward!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > SunJincheng
> > > > > > > >
> > > > > > > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > > >
> > > > > > > > > Thank you Yueting for pointing out the mistake in the
> > > prototype.
> > > > I
> > > > > > > > > accidentally introduced it when merge code.
> > > > > > > > >
> > > > > > > > > I'm so glad to see so many people are interested in the
> > > feature.
> > > > > > Let's
> > > > > > > > work
> > > > > > > > > out together to push it forward!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <
> [hidden email]
> > >:
> > > > > > > > >
> > > > > > > > > > +1 for the feature. Myself was a user of Siddhi, this is
> > > pretty
> > > > > > user
> > > > > > > > > > friendly feature to provide to user.
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > > > > > >
> > > > > > > > > > > Very glad that you're interested in this feature and
> > you're
> > > > > > > > definitely
> > > > > > > > > > > welcome to join this work and also anyone else.:)
> > > > > > > > > > >
> > > > > > > > > > > Best regards,
> > > > > > > > > > > Dian
> > > > > > > > > > >
> > > > > > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > > > > > [hidden email]> wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi all,
> > > > > > > > > > > >
> > > > > > > > > > > > Integrating SQL with CEP seems like a really nice
> idea.
> > > > > > > > > Unfortunately I
> > > > > > > > > > > had
> > > > > > > > > > > > time just for a brief look at the design doc, but it
> > > looks
> > > > > > really
> > > > > > > > > cool
> > > > > > > > > > > and
> > > > > > > > > > > > thorough. Also will have a second run tomorrow and
> will
> > > try
> > > > > to
> > > > > > > > > provide
> > > > > > > > > > > more
> > > > > > > > > > > > comments. Anyway will be glad to help pushing the
> > > > initiative
> > > > > > > > forward.
> > > > > > > > > > > >
> > > > > > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > > > > > >
> > > > > > > > > > > > Dawid Wysakowicz
> > > > > > > > > > > >
> > > > > > > > > > > > *Data/Software Engineer*
> > > > > > > > > > > >
> > > > > > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > > > > > >
> > > > > > > > > > > > <http://getindata.com/>
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <
> > > > [hidden email]
> > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi Dian & Jark,
> > > > > > > > > > > > >
> > > > > > > > > > > > > I checked out your prototype code, but it didn't
> pass
> > > the
> > > > > > > > CEPITCase
> > > > > > > > > > > test
> > > > > > > > > > > > in
> > > > > > > > > > > > > the flink-table component.
> > > > > > > > > > > > > It turns out that in the `MatchCodeGenerator.scala`
> > > file,
> > > > > > line
> > > > > > > 74
> > > > > > > > > > > should
> > > > > > > > > > > > > use `${classOf[IterativeCondition.
> > > > > > > Context[_]].getCanonicalName}`
> > > > > > > > > > > instead
> > > > > > > > > > > > > of
> > > > > > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > > > > > >
> > > > > > > > > > > > > I've also read your desgin document and it looks
> fine
> > > to
> > > > > me.
> > > > > > > > > > Actually,
> > > > > > > > > > > I
> > > > > > > > > > > > am
> > > > > > > > > > > > > working on the same thing recently, I think maybe
> we
> > > can
> > > > > work
> > > > > > > > > > together
> > > > > > > > > > > to
> > > > > > > > > > > > > push this forward.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Thanks,
> > > > > > > > > > > > > Yueting Chen
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <
> > > > > [hidden email]
> > > > > > >
> > > > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi Fabian,
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We have evaluated the missing features of Flink
> CEP
> > > > > > roughly,
> > > > > > > it
> > > > > > > > > > > should
> > > > > > > > > > > > > not
> > > > > > > > > > > > > > be quite difficult to support them. Kostas,
> Dawid,
> > > > what's
> > > > > > > your
> > > > > > > > > > > thought?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we
> > could
> > > > > > create
> > > > > > > > the
> > > > > > > > > > > JIRAs
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > start to work right now or we should wait until
> the
> > > > > release
> > > > > > > of
> > > > > > > > > > > calcite
> > > > > > > > > > > > > > 1.13?
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Btw, could you help to add me(dian.fu) to the
> > > > contributor
> > > > > > > list,
> > > > > > > > > > then
> > > > > > > > > > > I
> > > > > > > > > > > > > can
> > > > > > > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Best regards,
> > > > > > > > > > > > > > Dian
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > > > > > > [hidden email]>
> > > > > > > > > > > > > wrote:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Hi Jark,
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Thanks for updating the design doc and sharing
> > your
> > > > > > > > prototype!
> > > > > > > > > > > > > > > I didn't look at the code in detail, but the
> fact
> > > > that
> > > > > it
> > > > > > > is
> > > > > > > > > less
> > > > > > > > > > > > than
> > > > > > > > > > > > > 1k
> > > > > > > > > > > > > > > LOC is very promising. It seems that most of
> the
> > > > > complex
> > > > > > > CEP
> > > > > > > > > > logic
> > > > > > > > > > > > can
> > > > > > > > > > > > > be
> > > > > > > > > > > > > > > reused :-)
> > > > > > > > > > > > > > > Adding a dependency on flink-cep should be
> fine,
> > > IMO.
> > > > > It
> > > > > > > is a
> > > > > > > > > > very
> > > > > > > > > > > > slim
> > > > > > > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Regarding the missing features of Flink CEP
> that
> > > you
> > > > > > listed
> > > > > > > > in
> > > > > > > > > > the
> > > > > > > > > > > > > design
> > > > > > > > > > > > > > > doc, it would be good to get some in put from
> > > Kostas
> > > > > and
> > > > > > > > Dawid
> > > > > > > > > > > which
> > > > > > > > > > > > > are
> > > > > > > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > > > > > > Do you have already plans regarding some of the
> > > > missing
> > > > > > > > > features
> > > > > > > > > > or
> > > > > > > > > > > > can
> > > > > > > > > > > > > > you
> > > > > > > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Cheers, Fabian
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > Btw. The Calcite community started a discussion
> > > about
> > > > > > > > releasing
> > > > > > > > > > > > Calcite
> > > > > > > > > > > > > > > 1.13. So, the missing features might soon be
> > > > available.
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <
> > > [hidden email]
> > > > >:
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Hi guys,
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Good news! We have made a prototype for
> > > integrating
> > > > > CEP
> > > > > > > and
> > > > > > > > > > SQL.
> > > > > > > > > > > > See
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > link
> > > > > > > > > > > > > > > > https://github.com/apache/
> flink/compare/master
> > .
> > > ..
> > > > > > > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > You can check CepITCase to try the simple CQL
> > > > > example.
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Meanwhile, we updated our design doc with
> > > > additional
> > > > > > > > > > > implementation
> > > > > > > > > > > > > > > detail,
> > > > > > > > > > > > > > > > including how
> > > > > > > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API,
> and
> > > the
> > > > > > > features
> > > > > > > > > > > needed
> > > > > > > > > > > > to
> > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > In the prototype, we make flink-table
> > dependency
> > > on
> > > > > > > > > flink-cep.
> > > > > > > > > > Do
> > > > > > > > > > > > you
> > > > > > > > > > > > > > > think
> > > > > > > > > > > > > > > > that is fine?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > What do you think about the prototype and the
> > > > design
> > > > > > doc
> > > > > > > ?
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > > > > > > [hidden email]
> > > > > > > > > > >:
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Thanks for sharing your ideas with the
> > > > community. I
> > > > > > > > really
> > > > > > > > > > like
> > > > > > > > > > > > the
> > > > > > > > > > > > > > > > design
> > > > > > > > > > > > > > > > > document and think that it's a good
> approach
> > to
> > > > > > follow
> > > > > > > > > > Oracle's
> > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > extension for pattern matching. Looking
> > forward
> > > > to
> > > > > > > having
> > > > > > > > > > > support
> > > > > > > > > > > > > for
> > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > > > > > Till
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > > > > > > [hidden email]
> > > > > > > > >
> > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your
> > > > support.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > @Fabian, I totally agree with you that we
> > > > should
> > > > > > > focus
> > > > > > > > on
> > > > > > > > > > SQL
> > > > > > > > > > > > > > first.
> > > > > > > > > > > > > > > > > Let's
> > > > > > > > > > > > > > > > > > keep Table API in mind and discuss that
> > > later.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regarding to the orderBy() clause, I'm
> not
> > > sure
> > > > > > about
> > > > > > > > > > that. I
> > > > > > > > > > > > > think
> > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > > makes sense to make it required in
> > streaming
> > > > > > > > mode(either
> > > > > > > > > > > order
> > > > > > > > > > > > by
> > > > > > > > > > > > > > > > rowtime
> > > > > > > > > > > > > > > > > > or order by proctime). But CEP also works
> > in
> > > > > batch
> > > > > > > > mode,
> > > > > > > > > > and
> > > > > > > > > > > > not
> > > > > > > > > > > > > > > > > necessary
> > > > > > > > > > > > > > > > > > to order by some column. Nevertheless, we
> > can
> > > > > > support
> > > > > > > > CEP
> > > > > > > > > > on
> > > > > > > > > > > > > batch
> > > > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > > later.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > We are estimating how to implement
> > > > > MATCH_RECOGNIZE
> > > > > > > with
> > > > > > > > > CEP
> > > > > > > > > > > > > library
> > > > > > > > > > > > > > > > (with
> > > > > > > > > > > > > > > > > > NFA, CEP operator). And we will output a
> > > > detailed
> > > > > > doc
> > > > > > > > > and a
> > > > > > > > > > > > > > prototype
> > > > > > > > > > > > > > > > in
> > > > > > > > > > > > > > > > > > the next days.
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske
> <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> As you wrote, Till and I (and Kostas)
> have
> > > > been
> > > > > > > > thinking
> > > > > > > > > > > about
> > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > >> some time but haven't had time to work
> on
> > > this
> > > > > > > > feature.
> > > > > > > > > > > > > > > > > >> I think it would be a great addition and
> > > value
> > > > > add
> > > > > > > for
> > > > > > > > > > > Flink's
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> I read the proposal and think it is very
> > > good.
> > > > > We
> > > > > > > > might
> > > > > > > > > > need
> > > > > > > > > > > > to
> > > > > > > > > > > > > > add
> > > > > > > > > > > > > > > a
> > > > > > > > > > > > > > > > > bit
> > > > > > > > > > > > > > > > > >> more details, esp. when planning the
> > > concrete
> > > > > > steps
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > > > > > > >> - IMO, the development should start
> > focusing
> > > > on
> > > > > > SQL
> > > > > > > > and
> > > > > > > > > > its
> > > > > > > > > > > > > > > semantics.
> > > > > > > > > > > > > > > > > >> Pattern support for the Table API should
> > be
> > > > > added
> > > > > > > > later.
> > > > > > > > > > We
> > > > > > > > > > > > > > followed
> > > > > > > > > > > > > > > > > that
> > > > > > > > > > > > > > > > > >> approach for the OVER windows and I
> think
> > it
> > > > > > worked
> > > > > > > > > quiet
> > > > > > > > > > > > well.
> > > > > > > > > > > > > > > > > >> - We probably want to reuse as much as
> > > > possible
> > > > > > from
> > > > > > > > the
> > > > > > > > > > CEP
> > > > > > > > > > > > > > > library.
> > > > > > > > > > > > > > > > > >> That means we need to check if the
> > semantics
> > > > of
> > > > > > the
> > > > > > > > CEP
> > > > > > > > > > > > library
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or
> > how
> > > we
> > > > > can
> > > > > > > > > express
> > > > > > > > > > > the
> > > > > > > > > > > > > > > PATTERN
> > > > > > > > > > > > > > > > > >> semantics with the CEP library). This
> > should
> > > > be
> > > > > > one
> > > > > > > of
> > > > > > > > > the
> > > > > > > > > > > > first
> > > > > > > > > > > > > > > > steps,
> > > > > > > > > > > > > > > > > IMO.
> > > > > > > > > > > > > > > > > >> - I would make the orderBy() clause
> > > required.
> > > > In
> > > > > > > > regular
> > > > > > > > > > SQL
> > > > > > > > > > > > > rows
> > > > > > > > > > > > > > > have
> > > > > > > > > > > > > > > > > no
> > > > > > > > > > > > > > > > > >> order, so we need to make that explicit
> > > (this
> > > > > > would
> > > > > > > > also
> > > > > > > > > > be
> > > > > > > > > > > > > > > consistent
> > > > > > > > > > > > > > > > > with
> > > > > > > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas
> Kloudas
> > <
> > > > > > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >>> Thanks a lot for opening the
> discussion!
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> This is a really interesting idea that
> > has
> > > > been
> > > > > > in
> > > > > > > > our
> > > > > > > > > > > heads
> > > > > > > > > > > > > > > > > >>> since the first implementation of the
> CEP
> > > > > > library.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> And as for the design document, I will
> > > > > definitely
> > > > > > > > have
> > > > > > > > > a
> > > > > > > > > > > look
> > > > > > > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > > > > > > [hidden email]>
> > > > > > > > > > > > wrote:
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian,
> > > > @Timo,
> > > > > > > @Till,
> > > > > > > > > > > @Kostas
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> Dian and me and our teammates have
> > > > > investigated
> > > > > > > this
> > > > > > > > > > for a
> > > > > > > > > > > > > long
> > > > > > > > > > > > > > > > time.
> > > > > > > > > > > > > > > > > >>>> We think consolidating Flink SQL and
> CEP
> > > is
> > > > an
> > > > > > > > > exciting
> > > > > > > > > > > > thing
> > > > > > > > > > > > > > for
> > > > > > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > > > > > >>>> It'll make SQL more powerful and give
> > > users
> > > > > the
> > > > > > > > > ability
> > > > > > > > > > to
> > > > > > > > > > > > > > easily
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >>>> quickly build CEP applications.  And I
> > > find
> > > > > > Flink
> > > > > > > > > > > community
> > > > > > > > > > > > > has
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > talked
> > > > > > > > > > > > > > > > > >>>> about this idea before, such as the
> > > mailing
> > > > > list
> > > > > > > [1]
> > > > > > > > > and
> > > > > > > > > > > [2]
> > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > Fabian &
> > > > > > > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up
> > this
> > > > > topic
> > > > > > > > > again.
> > > > > > > > > > > > > Because
> > > > > > > > > > > > > > we
> > > > > > > > > > > > > > > > > >>>> already have pattern matching
> foundation
> > > in
> > > > > > Flink
> > > > > > > > CEP
> > > > > > > > > > > > library,
> > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > Stream
> > > > > > > > > > > > > > > > > >>>> SQL is ready now and Calcite has
> > partially
> > > > > > > supported
> > > > > > > > > > > pattern
> > > > > > > > > > > > > > > > matching
> > > > > > > > > > > > > > > > > >>>> syntax!  We also drafted a design doc
> > > about
> > > > > how
> > > > > > to
> > > > > > > > > > > integrate
> > > > > > > > > > > > > SQL
> > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > CEP,
> > > > > > > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > > > > > > sharing
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I
> include
> > > you
> > > > > > into
> > > > > > > > this
> > > > > > > > > > > > > > discussion,
> > > > > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > > > > > > list-archive.1008284.n3
> > > > > > > > > > .
> > > > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-
> > > > td9743.html#a9787
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > > > > > > list-archive.1008284.n3
> > > > > > > > > > .
> > > > > > > > > > > > > > nabble.c
> > > > > > > > > > > > > > > > > >>>> om/Effort-to-add-SQL-
> > > > > StreamSQL-to-Flink-td9727.
> > > > > > > > > > html#a9790
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > > > > > > tillrohrmann/streaming-
> > > > > > > > > > > > > > analytics-
> > > > > > > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > > > > > > [hidden email]
> > > > > > > > > >:
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> Flink's CEP library is a great
> library
> > > for
> > > > > > > complex
> > > > > > > > > > event
> > > > > > > > > > > > > > > > processing,
> > > > > > > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > > > > > > >>>>> and more customers are expressing
> their
> > > > > > interests
> > > > > > > > in
> > > > > > > > > > it.
> > > > > > > > > > > > But
> > > > > > > > > > > > > it
> > > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > > > > > > >>>>> some limitations that users usually
> > have
> > > to
> > > > > > > write a
> > > > > > > > > lot
> > > > > > > > > > > of
> > > > > > > > > > > > > code
> > > > > > > > > > > > > > > > even
> > > > > > > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > > > > > > >>>>> very simple pattern match use case as
> > it
> > > > > > > currently
> > > > > > > > > only
> > > > > > > > > > > > > > supports
> > > > > > > > > > > > > > > > the
> > > > > > > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > > > > > > >>>>> We have investigated some popular CEP
> > > > > products
> > > > > > > such
> > > > > > > > > as
> > > > > > > > > > > > esper
> > > > > > > > > > > > > > [1]
> > > > > > > > > > > > > > > > and
> > > > > > > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > > > > > > >>>>> [2] and found that most of these CEP
> > > > products
> > > > > > > > support
> > > > > > > > > > > > > SQL-like
> > > > > > > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > > > > > > >>>>> such as EPL to describe the match
> > > pattern.
> > > > > But
> > > > > > > > these
> > > > > > > > > > > > > solutions
> > > > > > > > > > > > > > > also
> > > > > > > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > > > > > > >>>>> the drawbacks that the pattern match
> > > > > languages
> > > > > > > are
> > > > > > > > > not
> > > > > > > > > > > > > standard
> > > > > > > > > > > > > > > > SQL,
> > > > > > > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > > > > > > >>>>> learn curve is steep for users and
> it's
> > > > > > > impossible
> > > > > > > > to
> > > > > > > > > > > > > integrate
> > > > > > > > > > > > > > > > them
> > > > > > > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > > > > > > >>>>> We find that Oracle's CEP solution
> CQL
> > > [3]
> > > > > > > > supports a
> > > > > > > > > > new
> > > > > > > > > > > > > > pattern
> > > > > > > > > > > > > > > > > >>>>> recognition clause match_recognize
> > which
> > > > is a
> > > > > > > > pattern
> > > > > > > > > > > > > > recognition
> > > > > > > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > > > > > > >>>>> proposed in this paper [4]. It
> > proposes a
> > > > set
> > > > > > of
> > > > > > > > new
> > > > > > > > > > > > syntaxes
> > > > > > > > > > > > > > to
> > > > > > > > > > > > > > > > > define
> > > > > > > > > > > > > > > > > >>>>> match pattern in sql expression.
> > Calcite
> > > > > > already
> > > > > > > > > > supports
> > > > > > > > > > > > > part
> > > > > > > > > > > > > > of
> > > > > > > > > > > > > > > > > this
> > > > > > > > > > > > > > > > > >>>>> standard [5].  I think it will be of
> > > great
> > > > > > value
> > > > > > > to
> > > > > > > > > > > support
> > > > > > > > > > > > > > > > > expressing
> > > > > > > > > > > > > > > > > >>>>> pattern recognition clause with
> > > > > match_recognize
> > > > > > > > > clause
> > > > > > > > > > by
> > > > > > > > > > > > > > > > integrating
> > > > > > > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > > > > > > >>>>> with Flink Table API & SQL and the
> > Flink
> > > > CEP
> > > > > > > > library.
> > > > > > > > > > Any
> > > > > > > > > > > > > > > thoughts?
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > > > > > > >>>>> https://docs.oracle.com/
> > middleware/1213/
> > > > > > > > > > > > eventprocessing/cql-
> > > > > > > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > > > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > > > > > > CQLLR1531
> > > > > > > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > > > > > > classes/winter17/cs240B/notes/
> > > > > > > > > > > > row-pat
> > > > > > > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > > > > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>>
> > > > > > > > > > > > > > > > > >>
> > > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > > >
> > > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
12