[DISCUSS]: Integrating Flink Table API & SQL with CEP

classic Classic list List threaded Threaded
24 messages Options
12
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[DISCUSS]: Integrating Flink Table API & SQL with CEP

Dian Fu
Hi everyone,

Flink's CEP library is a great library for complex event processing, more
and more customers are expressing their interests in it. But it also has
some limitations that users usually have to write a lot of code even for a
very simple pattern match use case as it currently only supports the Java
API.
We have investigated some popular CEP products such as esper [1] and siddhi
[2] and found that most of these CEP products support SQL-like expressions
such as EPL to describe the match pattern. But these solutions also have
the drawbacks that the pattern match languages are not standard SQL, the
learn curve is steep for users and it's impossible to integrate them into
the Flink Table API & SQL.
We find that Oracle's CEP solution CQL [3] supports a new pattern
recognition clause match_recognize which is a pattern recognition clause
proposed in this paper [4]. It proposes a set of new syntaxes to define
match pattern in sql expression. Calcite already supports part of this
standard [5].  I think it will be of great value to support expressing
pattern recognition clause with match_recognize clause by integrating it
with Flink Table API & SQL and the Flink CEP library. Any thoughts?

[1] http://www.espertech.com
[2] https://github.com/wso2/siddhi
[3]
https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
[4]
http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pattern-recogniton-11.pdf
[5] https://issues.apache.org/jira/browse/CALCITE-1570

Best Regards,
Dian
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Hi devs,

Dian and me and our teammates have investigated this for a long time. We
think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll
make SQL more powerful and give users the ability to easily and quickly
build CEP applications.  And I find Flink community has also talked about
this idea before, such as the mailing list [1] and [2] and Fabian & Till's
talk in Flink Forward 2016 [3].

I think THIS IS THE POINT to bring up this topic again. Because we already
have pattern matching foundation in Flink CEP library, and Stream SQL is
ready now and Calcite has partially supported pattern matching syntax!  We
also drafted a design doc about how to integrate SQL and CEP, and how to
support CEP on Table API.
https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing


@Fabian, @Timo, @Till, @Kostas I include you into this discussion, it would
be great to hear your response.


What do others think?


[1]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787

[2]
http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790

[3]
https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin


Regards,

Jark Wu



2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:

> Hi everyone,
>
> Flink's CEP library is a great library for complex event processing, more
> and more customers are expressing their interests in it. But it also has
> some limitations that users usually have to write a lot of code even for a
> very simple pattern match use case as it currently only supports the Java
> API.
> We have investigated some popular CEP products such as esper [1] and siddhi
> [2] and found that most of these CEP products support SQL-like expressions
> such as EPL to describe the match pattern. But these solutions also have
> the drawbacks that the pattern match languages are not standard SQL, the
> learn curve is steep for users and it's impossible to integrate them into
> the Flink Table API & SQL.
> We find that Oracle's CEP solution CQL [3] supports a new pattern
> recognition clause match_recognize which is a pattern recognition clause
> proposed in this paper [4]. It proposes a set of new syntaxes to define
> match pattern in sql expression. Calcite already supports part of this
> standard [5].  I think it will be of great value to support expressing
> pattern recognition clause with match_recognize clause by integrating it
> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>
> [1] http://www.espertech.com
> [2] https://github.com/wso2/siddhi
> [3]
> https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/
> GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
> [4]
> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/
> row-pattern-recogniton-11.pdf
> [5] https://issues.apache.org/jira/browse/CALCITE-1570
>
> Best Regards,
> Dian
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas

2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:

> Hi devs,
>
> Dian and me and our teammates have investigated this for a long time. We
> think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll
> make SQL more powerful and give users the ability to easily and quickly
> build CEP applications.  And I find Flink community has also talked about
> this idea before, such as the mailing list [1] and [2] and Fabian & Till's
> talk in Flink Forward 2016 [3].
>
> I think THIS IS THE POINT to bring up this topic again. Because we already
> have pattern matching foundation in Flink CEP library, and Stream SQL is
> ready now and Calcite has partially supported pattern matching syntax!  We
> also drafted a design doc about how to integrate SQL and CEP, and how to
> support CEP on Table API. https://docs.google.com/document/d/
> 1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
>
>
> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it
> would be great to hear your response.
>
>
> What do others think?
>
>
> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787
>
> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
>
> [3] https://www.slideshare.net/tillrohrmann/streaming-
> analytics-cep-two-sides-of-the-same-coin
>
>
> Regards,
>
> Jark Wu
>
>
>
> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
>
>> Hi everyone,
>>
>> Flink's CEP library is a great library for complex event processing, more
>> and more customers are expressing their interests in it. But it also has
>> some limitations that users usually have to write a lot of code even for a
>> very simple pattern match use case as it currently only supports the Java
>> API.
>> We have investigated some popular CEP products such as esper [1] and
>> siddhi
>> [2] and found that most of these CEP products support SQL-like expressions
>> such as EPL to describe the match pattern. But these solutions also have
>> the drawbacks that the pattern match languages are not standard SQL, the
>> learn curve is steep for users and it's impossible to integrate them into
>> the Flink Table API & SQL.
>> We find that Oracle's CEP solution CQL [3] supports a new pattern
>> recognition clause match_recognize which is a pattern recognition clause
>> proposed in this paper [4]. It proposes a set of new syntaxes to define
>> match pattern in sql expression. Calcite already supports part of this
>> standard [5].  I think it will be of great value to support expressing
>> pattern recognition clause with match_recognize clause by integrating it
>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>>
>> [1] http://www.espertech.com
>> [2] https://github.com/wso2/siddhi
>> [3]
>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
>> [4]
>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-
>> pattern-recogniton-11.pdf
>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
>>
>> Best Regards,
>> Dian
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Kostas Kloudas
Thanks a lot for opening the discussion!

This is a really interesting idea that has been in our heads
since the first implementation of the CEP library.

A big +1 for moving forward with this.

And as for the design document, I will definitely have a look
and comment there.

Kostas

> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
>
> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas  
>
> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email] <mailto:[hidden email]>>:
> Hi devs,
>
> Dian and me and our teammates have investigated this for a long time. We think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll make SQL more powerful and give users the ability to easily and quickly build CEP applications.  And I find Flink community has also talked about this idea before, such as the mailing list [1] and [2] and Fabian & Till's talk in Flink Forward 2016 [3].
>
> I think THIS IS THE POINT to bring up this topic again. Because we already have pattern matching foundation in Flink CEP library, and Stream SQL is ready now and Calcite has partially supported pattern matching syntax!  We also drafted a design doc about how to integrate SQL and CEP, and how to support CEP on Table API. https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing <https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing>
>
> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it would be great to hear your response.
>
>
>
> What do others think?
>
>
>
> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787 <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Add-CEP-library-to-Flink-td9743.html#a9787>
> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790 <http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790>
> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin <https://www.slideshare.net/tillrohrmann/streaming-analytics-cep-two-sides-of-the-same-coin>
>
> Regards,
>
> Jark Wu
>
>
>
>
> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email] <mailto:[hidden email]>>:
> Hi everyone,
>
> Flink's CEP library is a great library for complex event processing, more
> and more customers are expressing their interests in it. But it also has
> some limitations that users usually have to write a lot of code even for a
> very simple pattern match use case as it currently only supports the Java
> API.
> We have investigated some popular CEP products such as esper [1] and siddhi
> [2] and found that most of these CEP products support SQL-like expressions
> such as EPL to describe the match pattern. But these solutions also have
> the drawbacks that the pattern match languages are not standard SQL, the
> learn curve is steep for users and it's impossible to integrate them into
> the Flink Table API & SQL.
> We find that Oracle's CEP solution CQL [3] supports a new pattern
> recognition clause match_recognize which is a pattern recognition clause
> proposed in this paper [4]. It proposes a set of new syntaxes to define
> match pattern in sql expression. Calcite already supports part of this
> standard [5].  I think it will be of great value to support expressing
> pattern recognition clause with match_recognize clause by integrating it
> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>
> [1] http://www.espertech.com <http://www.espertech.com/>
> [2] https://github.com/wso2/siddhi <https://github.com/wso2/siddhi>
> [3]
> https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531 <https://docs.oracle.com/middleware/1213/eventprocessing/cql-reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531>
> [4]
> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pattern-recogniton-11.pdf <http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pattern-recogniton-11.pdf>
> [5] https://issues.apache.org/jira/browse/CALCITE-1570 <https://issues.apache.org/jira/browse/CALCITE-1570>
>
> Best Regards,
> Dian
>
>

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Fabian Hueske-2
Thanks Dian and Jark for this proposal!

As you wrote, Till and I (and Kostas) have been thinking about this for
some time but haven't had time to work on this feature.
I think it would be a great addition and value add for Flink's SQL support
and Table API.

I read the proposal and think it is very good. We might need to add a bit
more details, esp. when planning the concrete steps of the implementation.

A few comments to the proposal:
- IMO, the development should start focusing on SQL and its semantics.
Pattern support for the Table API should be added later. We followed that
approach for the OVER windows and I think it worked quiet well.
- We probably want to reuse as much as possible from the CEP library. That
means we need to check if the semantics of the CEP library and Oracle's
PATTERN syntax are aligned (or how we can express the PATTERN semantics
with the CEP library). This should be one of the first steps, IMO.
- I would make the orderBy() clause required. In regular SQL rows have no
order, so we need to make that explicit (this would also be consistent with
the OVER windows).

Let me know what you think.

Best, Fabian

2017-06-07 11:41 GMT+02:00 Kostas Kloudas <[hidden email]>:

> Thanks a lot for opening the discussion!
>
> This is a really interesting idea that has been in our heads
> since the first implementation of the CEP library.
>
> A big +1 for moving forward with this.
>
> And as for the design document, I will definitely have a look
> and comment there.
>
> Kostas
>
> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
>
> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
>
> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
>
>> Hi devs,
>>
>> Dian and me and our teammates have investigated this for a long time. We
>> think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll
>> make SQL more powerful and give users the ability to easily and quickly
>> build CEP applications.  And I find Flink community has also talked about
>> this idea before, such as the mailing list [1] and [2] and Fabian & Till's
>> talk in Flink Forward 2016 [3].
>>
>> I think THIS IS THE POINT to bring up this topic again. Because we
>> already have pattern matching foundation in Flink CEP library, and Stream
>> SQL is ready now and Calcite has partially supported pattern matching
>> syntax!  We also drafted a design doc about how to integrate SQL and CEP,
>> and how to support CEP on Table API. https://docs.google.com/docume
>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
>>
>>
>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it
>> would be great to hear your response.
>>
>>
>> What do others think?
>>
>>
>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.
>> com/Add-CEP-library-to-Flink-td9743.html#a9787
>>
>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.
>> com/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
>>
>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
>> cep-two-sides-of-the-same-coin
>>
>>
>> Regards,
>>
>> Jark Wu
>>
>>
>>
>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
>>
>>> Hi everyone,
>>>
>>> Flink's CEP library is a great library for complex event processing, more
>>> and more customers are expressing their interests in it. But it also has
>>> some limitations that users usually have to write a lot of code even for
>>> a
>>> very simple pattern match use case as it currently only supports the Java
>>> API.
>>> We have investigated some popular CEP products such as esper [1] and
>>> siddhi
>>> [2] and found that most of these CEP products support SQL-like
>>> expressions
>>> such as EPL to describe the match pattern. But these solutions also have
>>> the drawbacks that the pattern match languages are not standard SQL, the
>>> learn curve is steep for users and it's impossible to integrate them into
>>> the Flink Table API & SQL.
>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
>>> recognition clause match_recognize which is a pattern recognition clause
>>> proposed in this paper [4]. It proposes a set of new syntaxes to define
>>> match pattern in sql expression. Calcite already supports part of this
>>> standard [5].  I think it will be of great value to support expressing
>>> pattern recognition clause with match_recognize clause by integrating it
>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>>>
>>> [1] http://www.espertech.com
>>> [2] https://github.com/wso2/siddhi
>>> [3]
>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
>>> [4]
>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
>>> tern-recogniton-11.pdf
>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
>>>
>>> Best Regards,
>>> Dian
>>>
>>
>>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Hi  @Kostas, @Fabian, thank you for your support.

@Fabian, I totally agree with you that we should focus on SQL first. Let's
keep Table API in mind and discuss that later.

Regarding to the orderBy() clause, I'm not sure about that. I think it
makes sense to make it required in streaming mode(either order by rowtime
or order by proctime). But CEP also works in batch mode, and not necessary
to order by some column. Nevertheless, we can support CEP on batch SQL
later.

We are estimating how to implement MATCH_RECOGNIZE with CEP library (with
NFA, CEP operator). And we will output a detailed doc and a prototype in
the next days.

Regards,
Jark Wu


2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:

> Thanks Dian and Jark for this proposal!
>
> As you wrote, Till and I (and Kostas) have been thinking about this for
> some time but haven't had time to work on this feature.
> I think it would be a great addition and value add for Flink's SQL support
> and Table API.
>
> I read the proposal and think it is very good. We might need to add a bit
> more details, esp. when planning the concrete steps of the implementation.
>
> A few comments to the proposal:
> - IMO, the development should start focusing on SQL and its semantics.
> Pattern support for the Table API should be added later. We followed that
> approach for the OVER windows and I think it worked quiet well.
> - We probably want to reuse as much as possible from the CEP library. That
> means we need to check if the semantics of the CEP library and Oracle's
> PATTERN syntax are aligned (or how we can express the PATTERN semantics
> with the CEP library). This should be one of the first steps, IMO.
> - I would make the orderBy() clause required. In regular SQL rows have no
> order, so we need to make that explicit (this would also be consistent with
> the OVER windows).
>
> Let me know what you think.
>
> Best, Fabian
>
> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <[hidden email]>:
>
>> Thanks a lot for opening the discussion!
>>
>> This is a really interesting idea that has been in our heads
>> since the first implementation of the CEP library.
>>
>> A big +1 for moving forward with this.
>>
>> And as for the design document, I will definitely have a look
>> and comment there.
>>
>> Kostas
>>
>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
>>
>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
>>
>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
>>
>>> Hi devs,
>>>
>>> Dian and me and our teammates have investigated this for a long time. We
>>> think consolidating Flink SQL and CEP is an exciting thing for Flink. It'll
>>> make SQL more powerful and give users the ability to easily and quickly
>>> build CEP applications.  And I find Flink community has also talked about
>>> this idea before, such as the mailing list [1] and [2] and Fabian & Till's
>>> talk in Flink Forward 2016 [3].
>>>
>>> I think THIS IS THE POINT to bring up this topic again. Because we
>>> already have pattern matching foundation in Flink CEP library, and Stream
>>> SQL is ready now and Calcite has partially supported pattern matching
>>> syntax!  We also drafted a design doc about how to integrate SQL and CEP,
>>> and how to support CEP on Table API. https://docs.google.com/docume
>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
>>>
>>>
>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it
>>> would be great to hear your response.
>>>
>>>
>>> What do others think?
>>>
>>>
>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
>>>
>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
>>>
>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
>>> cep-two-sides-of-the-same-coin
>>>
>>>
>>> Regards,
>>>
>>> Jark Wu
>>>
>>>
>>>
>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
>>>
>>>> Hi everyone,
>>>>
>>>> Flink's CEP library is a great library for complex event processing,
>>>> more
>>>> and more customers are expressing their interests in it. But it also has
>>>> some limitations that users usually have to write a lot of code even
>>>> for a
>>>> very simple pattern match use case as it currently only supports the
>>>> Java
>>>> API.
>>>> We have investigated some popular CEP products such as esper [1] and
>>>> siddhi
>>>> [2] and found that most of these CEP products support SQL-like
>>>> expressions
>>>> such as EPL to describe the match pattern. But these solutions also have
>>>> the drawbacks that the pattern match languages are not standard SQL, the
>>>> learn curve is steep for users and it's impossible to integrate them
>>>> into
>>>> the Flink Table API & SQL.
>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
>>>> recognition clause match_recognize which is a pattern recognition clause
>>>> proposed in this paper [4]. It proposes a set of new syntaxes to define
>>>> match pattern in sql expression. Calcite already supports part of this
>>>> standard [5].  I think it will be of great value to support expressing
>>>> pattern recognition clause with match_recognize clause by integrating it
>>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>>>>
>>>> [1] http://www.espertech.com
>>>> [2] https://github.com/wso2/siddhi
>>>> [3]
>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
>>>> [4]
>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
>>>> tern-recogniton-11.pdf
>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
>>>>
>>>> Best Regards,
>>>> Dian
>>>>
>>>
>>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Till Rohrmann
Thanks for sharing your ideas with the community. I really like the design
document and think that it's a good approach to follow Oracle's SQL
extension for pattern matching. Looking forward to having support for SQL
with CEP capabilities :-)

Cheers,
Till

On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:

> Hi  @Kostas, @Fabian, thank you for your support.
>
> @Fabian, I totally agree with you that we should focus on SQL first. Let's
> keep Table API in mind and discuss that later.
>
> Regarding to the orderBy() clause, I'm not sure about that. I think it
> makes sense to make it required in streaming mode(either order by rowtime
> or order by proctime). But CEP also works in batch mode, and not necessary
> to order by some column. Nevertheless, we can support CEP on batch SQL
> later.
>
> We are estimating how to implement MATCH_RECOGNIZE with CEP library (with
> NFA, CEP operator). And we will output a detailed doc and a prototype in
> the next days.
>
> Regards,
> Jark Wu
>
>
> 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
>
>> Thanks Dian and Jark for this proposal!
>>
>> As you wrote, Till and I (and Kostas) have been thinking about this for
>> some time but haven't had time to work on this feature.
>> I think it would be a great addition and value add for Flink's SQL
>> support and Table API.
>>
>> I read the proposal and think it is very good. We might need to add a bit
>> more details, esp. when planning the concrete steps of the implementation.
>>
>> A few comments to the proposal:
>> - IMO, the development should start focusing on SQL and its semantics.
>> Pattern support for the Table API should be added later. We followed that
>> approach for the OVER windows and I think it worked quiet well.
>> - We probably want to reuse as much as possible from the CEP library.
>> That means we need to check if the semantics of the CEP library and
>> Oracle's PATTERN syntax are aligned (or how we can express the PATTERN
>> semantics with the CEP library). This should be one of the first steps, IMO.
>> - I would make the orderBy() clause required. In regular SQL rows have no
>> order, so we need to make that explicit (this would also be consistent with
>> the OVER windows).
>>
>> Let me know what you think.
>>
>> Best, Fabian
>>
>> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <[hidden email]>:
>>
>>> Thanks a lot for opening the discussion!
>>>
>>> This is a really interesting idea that has been in our heads
>>> since the first implementation of the CEP library.
>>>
>>> A big +1 for moving forward with this.
>>>
>>> And as for the design document, I will definitely have a look
>>> and comment there.
>>>
>>> Kostas
>>>
>>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
>>>
>>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
>>>
>>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
>>>
>>>> Hi devs,
>>>>
>>>> Dian and me and our teammates have investigated this for a long time.
>>>> We think consolidating Flink SQL and CEP is an exciting thing for Flink.
>>>> It'll make SQL more powerful and give users the ability to easily and
>>>> quickly build CEP applications.  And I find Flink community has also talked
>>>> about this idea before, such as the mailing list [1] and [2] and Fabian &
>>>> Till's talk in Flink Forward 2016 [3].
>>>>
>>>> I think THIS IS THE POINT to bring up this topic again. Because we
>>>> already have pattern matching foundation in Flink CEP library, and Stream
>>>> SQL is ready now and Calcite has partially supported pattern matching
>>>> syntax!  We also drafted a design doc about how to integrate SQL and CEP,
>>>> and how to support CEP on Table API. https://docs.google.com/docume
>>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
>>>>
>>>>
>>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it
>>>> would be great to hear your response.
>>>>
>>>>
>>>> What do others think?
>>>>
>>>>
>>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
>>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
>>>>
>>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
>>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
>>>>
>>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
>>>> cep-two-sides-of-the-same-coin
>>>>
>>>>
>>>> Regards,
>>>>
>>>> Jark Wu
>>>>
>>>>
>>>>
>>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Flink's CEP library is a great library for complex event processing,
>>>>> more
>>>>> and more customers are expressing their interests in it. But it also
>>>>> has
>>>>> some limitations that users usually have to write a lot of code even
>>>>> for a
>>>>> very simple pattern match use case as it currently only supports the
>>>>> Java
>>>>> API.
>>>>> We have investigated some popular CEP products such as esper [1] and
>>>>> siddhi
>>>>> [2] and found that most of these CEP products support SQL-like
>>>>> expressions
>>>>> such as EPL to describe the match pattern. But these solutions also
>>>>> have
>>>>> the drawbacks that the pattern match languages are not standard SQL,
>>>>> the
>>>>> learn curve is steep for users and it's impossible to integrate them
>>>>> into
>>>>> the Flink Table API & SQL.
>>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
>>>>> recognition clause match_recognize which is a pattern recognition
>>>>> clause
>>>>> proposed in this paper [4]. It proposes a set of new syntaxes to define
>>>>> match pattern in sql expression. Calcite already supports part of this
>>>>> standard [5].  I think it will be of great value to support expressing
>>>>> pattern recognition clause with match_recognize clause by integrating
>>>>> it
>>>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
>>>>>
>>>>> [1] http://www.espertech.com
>>>>> [2] https://github.com/wso2/siddhi
>>>>> [3]
>>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
>>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
>>>>> [4]
>>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
>>>>> tern-recogniton-11.pdf
>>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
>>>>>
>>>>> Best Regards,
>>>>> Dian
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Hi guys,

Good news! We have made a prototype for integrating CEP and SQL. See this
link
https://github.com/apache/flink/compare/master...wuchong:cep-on-sql?expand=1


You can check CepITCase to try the simple CQL example.

Meanwhile, we updated our design doc with additional implementation detail,
including how
to translate MATCH_RECOGNIZE into CEP API, and the features needed to add
to Flink CEP,
and the implementation plan. See the document
https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit#heading=h.4oas4koy8qu3

In the prototype, we make flink-table dependency on flink-cep. Do you think
that is fine?

What do you think about the prototype and the design doc ?

Any feedbacks are welcome!

Cheers,
Jark Wu


2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:

> Thanks for sharing your ideas with the community. I really like the design
> document and think that it's a good approach to follow Oracle's SQL
> extension for pattern matching. Looking forward to having support for SQL
> with CEP capabilities :-)
>
> Cheers,
> Till
>
> On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
>
> > Hi  @Kostas, @Fabian, thank you for your support.
> >
> > @Fabian, I totally agree with you that we should focus on SQL first.
> Let's
> > keep Table API in mind and discuss that later.
> >
> > Regarding to the orderBy() clause, I'm not sure about that. I think it
> > makes sense to make it required in streaming mode(either order by rowtime
> > or order by proctime). But CEP also works in batch mode, and not
> necessary
> > to order by some column. Nevertheless, we can support CEP on batch SQL
> > later.
> >
> > We are estimating how to implement MATCH_RECOGNIZE with CEP library (with
> > NFA, CEP operator). And we will output a detailed doc and a prototype in
> > the next days.
> >
> > Regards,
> > Jark Wu
> >
> >
> > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> >
> >> Thanks Dian and Jark for this proposal!
> >>
> >> As you wrote, Till and I (and Kostas) have been thinking about this for
> >> some time but haven't had time to work on this feature.
> >> I think it would be a great addition and value add for Flink's SQL
> >> support and Table API.
> >>
> >> I read the proposal and think it is very good. We might need to add a
> bit
> >> more details, esp. when planning the concrete steps of the
> implementation.
> >>
> >> A few comments to the proposal:
> >> - IMO, the development should start focusing on SQL and its semantics.
> >> Pattern support for the Table API should be added later. We followed
> that
> >> approach for the OVER windows and I think it worked quiet well.
> >> - We probably want to reuse as much as possible from the CEP library.
> >> That means we need to check if the semantics of the CEP library and
> >> Oracle's PATTERN syntax are aligned (or how we can express the PATTERN
> >> semantics with the CEP library). This should be one of the first steps,
> IMO.
> >> - I would make the orderBy() clause required. In regular SQL rows have
> no
> >> order, so we need to make that explicit (this would also be consistent
> with
> >> the OVER windows).
> >>
> >> Let me know what you think.
> >>
> >> Best, Fabian
> >>
> >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <[hidden email]
> >:
> >>
> >>> Thanks a lot for opening the discussion!
> >>>
> >>> This is a really interesting idea that has been in our heads
> >>> since the first implementation of the CEP library.
> >>>
> >>> A big +1 for moving forward with this.
> >>>
> >>> And as for the design document, I will definitely have a look
> >>> and comment there.
> >>>
> >>> Kostas
> >>>
> >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
> >>>
> >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> >>>
> >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> >>>
> >>>> Hi devs,
> >>>>
> >>>> Dian and me and our teammates have investigated this for a long time.
> >>>> We think consolidating Flink SQL and CEP is an exciting thing for
> Flink.
> >>>> It'll make SQL more powerful and give users the ability to easily and
> >>>> quickly build CEP applications.  And I find Flink community has also
> talked
> >>>> about this idea before, such as the mailing list [1] and [2] and
> Fabian &
> >>>> Till's talk in Flink Forward 2016 [3].
> >>>>
> >>>> I think THIS IS THE POINT to bring up this topic again. Because we
> >>>> already have pattern matching foundation in Flink CEP library, and
> Stream
> >>>> SQL is ready now and Calcite has partially supported pattern matching
> >>>> syntax!  We also drafted a design doc about how to integrate SQL and
> CEP,
> >>>> and how to support CEP on Table API. https://docs.google.com/docume
> >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
> >>>>
> >>>>
> >>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion, it
> >>>> would be great to hear your response.
> >>>>
> >>>>
> >>>> What do others think?
> >>>>
> >>>>
> >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> >>>>
> >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> >>>>
> >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
> >>>> cep-two-sides-of-the-same-coin
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Jark Wu
> >>>>
> >>>>
> >>>>
> >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> >>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> Flink's CEP library is a great library for complex event processing,
> >>>>> more
> >>>>> and more customers are expressing their interests in it. But it also
> >>>>> has
> >>>>> some limitations that users usually have to write a lot of code even
> >>>>> for a
> >>>>> very simple pattern match use case as it currently only supports the
> >>>>> Java
> >>>>> API.
> >>>>> We have investigated some popular CEP products such as esper [1] and
> >>>>> siddhi
> >>>>> [2] and found that most of these CEP products support SQL-like
> >>>>> expressions
> >>>>> such as EPL to describe the match pattern. But these solutions also
> >>>>> have
> >>>>> the drawbacks that the pattern match languages are not standard SQL,
> >>>>> the
> >>>>> learn curve is steep for users and it's impossible to integrate them
> >>>>> into
> >>>>> the Flink Table API & SQL.
> >>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
> >>>>> recognition clause match_recognize which is a pattern recognition
> >>>>> clause
> >>>>> proposed in this paper [4]. It proposes a set of new syntaxes to
> define
> >>>>> match pattern in sql expression. Calcite already supports part of
> this
> >>>>> standard [5].  I think it will be of great value to support
> expressing
> >>>>> pattern recognition clause with match_recognize clause by integrating
> >>>>> it
> >>>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
> >>>>>
> >>>>> [1] http://www.espertech.com
> >>>>> [2] https://github.com/wso2/siddhi
> >>>>> [3]
> >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
> >>>>> [4]
> >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> >>>>> tern-recogniton-11.pdf
> >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> >>>>>
> >>>>> Best Regards,
> >>>>> Dian
> >>>>>
> >>>>
> >>>>
> >>>
> >>>
> >>
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Fabian Hueske-2
Hi Jark,

Thanks for updating the design doc and sharing your prototype!
I didn't look at the code in detail, but the fact that it is less than 1k
LOC is very promising. It seems that most of the complex CEP logic can be
reused :-)
Adding a dependency on flink-cep should be fine, IMO. It is a very slim
library with almost none external dependencies.

Regarding the missing features of Flink CEP that you listed in the design
doc, it would be good to get some in put from Kostas and Dawid which are
the main contributors to CEP.
Do you have already plans regarding some of the missing features or can you
assess how hard it would be to integrate them?

Cheers, Fabian

Btw. The Calcite community started a discussion about releasing Calcite
1.13. So, the missing features might soon be available.

2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:

> Hi guys,
>
> Good news! We have made a prototype for integrating CEP and SQL. See this
> link
> https://github.com/apache/flink/compare/master...
> wuchong:cep-on-sql?expand=1
>
>
> You can check CepITCase to try the simple CQL example.
>
> Meanwhile, we updated our design doc with additional implementation detail,
> including how
> to translate MATCH_RECOGNIZE into CEP API, and the features needed to add
> to Flink CEP,
> and the implementation plan. See the document
> https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
>
> In the prototype, we make flink-table dependency on flink-cep. Do you think
> that is fine?
>
> What do you think about the prototype and the design doc ?
>
> Any feedbacks are welcome!
>
> Cheers,
> Jark Wu
>
>
> 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
>
> > Thanks for sharing your ideas with the community. I really like the
> design
> > document and think that it's a good approach to follow Oracle's SQL
> > extension for pattern matching. Looking forward to having support for SQL
> > with CEP capabilities :-)
> >
> > Cheers,
> > Till
> >
> > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
> >
> > > Hi  @Kostas, @Fabian, thank you for your support.
> > >
> > > @Fabian, I totally agree with you that we should focus on SQL first.
> > Let's
> > > keep Table API in mind and discuss that later.
> > >
> > > Regarding to the orderBy() clause, I'm not sure about that. I think it
> > > makes sense to make it required in streaming mode(either order by
> rowtime
> > > or order by proctime). But CEP also works in batch mode, and not
> > necessary
> > > to order by some column. Nevertheless, we can support CEP on batch SQL
> > > later.
> > >
> > > We are estimating how to implement MATCH_RECOGNIZE with CEP library
> (with
> > > NFA, CEP operator). And we will output a detailed doc and a prototype
> in
> > > the next days.
> > >
> > > Regards,
> > > Jark Wu
> > >
> > >
> > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> > >
> > >> Thanks Dian and Jark for this proposal!
> > >>
> > >> As you wrote, Till and I (and Kostas) have been thinking about this
> for
> > >> some time but haven't had time to work on this feature.
> > >> I think it would be a great addition and value add for Flink's SQL
> > >> support and Table API.
> > >>
> > >> I read the proposal and think it is very good. We might need to add a
> > bit
> > >> more details, esp. when planning the concrete steps of the
> > implementation.
> > >>
> > >> A few comments to the proposal:
> > >> - IMO, the development should start focusing on SQL and its semantics.
> > >> Pattern support for the Table API should be added later. We followed
> > that
> > >> approach for the OVER windows and I think it worked quiet well.
> > >> - We probably want to reuse as much as possible from the CEP library.
> > >> That means we need to check if the semantics of the CEP library and
> > >> Oracle's PATTERN syntax are aligned (or how we can express the PATTERN
> > >> semantics with the CEP library). This should be one of the first
> steps,
> > IMO.
> > >> - I would make the orderBy() clause required. In regular SQL rows have
> > no
> > >> order, so we need to make that explicit (this would also be consistent
> > with
> > >> the OVER windows).
> > >>
> > >> Let me know what you think.
> > >>
> > >> Best, Fabian
> > >>
> > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> [hidden email]
> > >:
> > >>
> > >>> Thanks a lot for opening the discussion!
> > >>>
> > >>> This is a really interesting idea that has been in our heads
> > >>> since the first implementation of the CEP library.
> > >>>
> > >>> A big +1 for moving forward with this.
> > >>>
> > >>> And as for the design document, I will definitely have a look
> > >>> and comment there.
> > >>>
> > >>> Kostas
> > >>>
> > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
> > >>>
> > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > >>>
> > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > >>>
> > >>>> Hi devs,
> > >>>>
> > >>>> Dian and me and our teammates have investigated this for a long
> time.
> > >>>> We think consolidating Flink SQL and CEP is an exciting thing for
> > Flink.
> > >>>> It'll make SQL more powerful and give users the ability to easily
> and
> > >>>> quickly build CEP applications.  And I find Flink community has also
> > talked
> > >>>> about this idea before, such as the mailing list [1] and [2] and
> > Fabian &
> > >>>> Till's talk in Flink Forward 2016 [3].
> > >>>>
> > >>>> I think THIS IS THE POINT to bring up this topic again. Because we
> > >>>> already have pattern matching foundation in Flink CEP library, and
> > Stream
> > >>>> SQL is ready now and Calcite has partially supported pattern
> matching
> > >>>> syntax!  We also drafted a design doc about how to integrate SQL and
> > CEP,
> > >>>> and how to support CEP on Table API. https://docs.google.com/docume
> > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=sharing
> > >>>>
> > >>>>
> > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion,
> it
> > >>>> would be great to hear your response.
> > >>>>
> > >>>>
> > >>>> What do others think?
> > >>>>
> > >>>>
> > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > >>>>
> > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > >>>>
> > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
> > >>>> cep-two-sides-of-the-same-coin
> > >>>>
> > >>>>
> > >>>> Regards,
> > >>>>
> > >>>> Jark Wu
> > >>>>
> > >>>>
> > >>>>
> > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > >>>>
> > >>>>> Hi everyone,
> > >>>>>
> > >>>>> Flink's CEP library is a great library for complex event
> processing,
> > >>>>> more
> > >>>>> and more customers are expressing their interests in it. But it
> also
> > >>>>> has
> > >>>>> some limitations that users usually have to write a lot of code
> even
> > >>>>> for a
> > >>>>> very simple pattern match use case as it currently only supports
> the
> > >>>>> Java
> > >>>>> API.
> > >>>>> We have investigated some popular CEP products such as esper [1]
> and
> > >>>>> siddhi
> > >>>>> [2] and found that most of these CEP products support SQL-like
> > >>>>> expressions
> > >>>>> such as EPL to describe the match pattern. But these solutions also
> > >>>>> have
> > >>>>> the drawbacks that the pattern match languages are not standard
> SQL,
> > >>>>> the
> > >>>>> learn curve is steep for users and it's impossible to integrate
> them
> > >>>>> into
> > >>>>> the Flink Table API & SQL.
> > >>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
> > >>>>> recognition clause match_recognize which is a pattern recognition
> > >>>>> clause
> > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes to
> > define
> > >>>>> match pattern in sql expression. Calcite already supports part of
> > this
> > >>>>> standard [5].  I think it will be of great value to support
> > expressing
> > >>>>> pattern recognition clause with match_recognize clause by
> integrating
> > >>>>> it
> > >>>>> with Flink Table API & SQL and the Flink CEP library. Any thoughts?
> > >>>>>
> > >>>>> [1] http://www.espertech.com
> > >>>>> [2] https://github.com/wso2/siddhi
> > >>>>> [3]
> > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#CQLLR1531
> > >>>>> [4]
> > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> > >>>>> tern-recogniton-11.pdf
> > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > >>>>>
> > >>>>> Best Regards,
> > >>>>> Dian
> > >>>>>
> > >>>>
> > >>>>
> > >>>
> > >>>
> > >>
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Dian Fu
Hi Fabian,

We have evaluated the missing features of Flink CEP roughly, it should not
be quite difficult to support them. Kostas, Dawid, what's your thought?

For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs and
start to work right now or we should wait until the release of calcite 1.13?

Btw, could you help to add me(dian.fu) to the contributor list, then I can
assign the JIRAs to myself? Thanks a lot.

Best regards,
Dian

On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]> wrote:

> Hi Jark,
>
> Thanks for updating the design doc and sharing your prototype!
> I didn't look at the code in detail, but the fact that it is less than 1k
> LOC is very promising. It seems that most of the complex CEP logic can be
> reused :-)
> Adding a dependency on flink-cep should be fine, IMO. It is a very slim
> library with almost none external dependencies.
>
> Regarding the missing features of Flink CEP that you listed in the design
> doc, it would be good to get some in put from Kostas and Dawid which are
> the main contributors to CEP.
> Do you have already plans regarding some of the missing features or can you
> assess how hard it would be to integrate them?
>
> Cheers, Fabian
>
> Btw. The Calcite community started a discussion about releasing Calcite
> 1.13. So, the missing features might soon be available.
>
> 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
>
> > Hi guys,
> >
> > Good news! We have made a prototype for integrating CEP and SQL. See this
> > link
> > https://github.com/apache/flink/compare/master...
> > wuchong:cep-on-sql?expand=1
> >
> >
> > You can check CepITCase to try the simple CQL example.
> >
> > Meanwhile, we updated our design doc with additional implementation
> detail,
> > including how
> > to translate MATCH_RECOGNIZE into CEP API, and the features needed to add
> > to Flink CEP,
> > and the implementation plan. See the document
> > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> >
> > In the prototype, we make flink-table dependency on flink-cep. Do you
> think
> > that is fine?
> >
> > What do you think about the prototype and the design doc ?
> >
> > Any feedbacks are welcome!
> >
> > Cheers,
> > Jark Wu
> >
> >
> > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
> >
> > > Thanks for sharing your ideas with the community. I really like the
> > design
> > > document and think that it's a good approach to follow Oracle's SQL
> > > extension for pattern matching. Looking forward to having support for
> SQL
> > > with CEP capabilities :-)
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
> > >
> > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > >
> > > > @Fabian, I totally agree with you that we should focus on SQL first.
> > > Let's
> > > > keep Table API in mind and discuss that later.
> > > >
> > > > Regarding to the orderBy() clause, I'm not sure about that. I think
> it
> > > > makes sense to make it required in streaming mode(either order by
> > rowtime
> > > > or order by proctime). But CEP also works in batch mode, and not
> > > necessary
> > > > to order by some column. Nevertheless, we can support CEP on batch
> SQL
> > > > later.
> > > >
> > > > We are estimating how to implement MATCH_RECOGNIZE with CEP library
> > (with
> > > > NFA, CEP operator). And we will output a detailed doc and a prototype
> > in
> > > > the next days.
> > > >
> > > > Regards,
> > > > Jark Wu
> > > >
> > > >
> > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> > > >
> > > >> Thanks Dian and Jark for this proposal!
> > > >>
> > > >> As you wrote, Till and I (and Kostas) have been thinking about this
> > for
> > > >> some time but haven't had time to work on this feature.
> > > >> I think it would be a great addition and value add for Flink's SQL
> > > >> support and Table API.
> > > >>
> > > >> I read the proposal and think it is very good. We might need to add
> a
> > > bit
> > > >> more details, esp. when planning the concrete steps of the
> > > implementation.
> > > >>
> > > >> A few comments to the proposal:
> > > >> - IMO, the development should start focusing on SQL and its
> semantics.
> > > >> Pattern support for the Table API should be added later. We followed
> > > that
> > > >> approach for the OVER windows and I think it worked quiet well.
> > > >> - We probably want to reuse as much as possible from the CEP
> library.
> > > >> That means we need to check if the semantics of the CEP library and
> > > >> Oracle's PATTERN syntax are aligned (or how we can express the
> PATTERN
> > > >> semantics with the CEP library). This should be one of the first
> > steps,
> > > IMO.
> > > >> - I would make the orderBy() clause required. In regular SQL rows
> have
> > > no
> > > >> order, so we need to make that explicit (this would also be
> consistent
> > > with
> > > >> the OVER windows).
> > > >>
> > > >> Let me know what you think.
> > > >>
> > > >> Best, Fabian
> > > >>
> > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > [hidden email]
> > > >:
> > > >>
> > > >>> Thanks a lot for opening the discussion!
> > > >>>
> > > >>> This is a really interesting idea that has been in our heads
> > > >>> since the first implementation of the CEP library.
> > > >>>
> > > >>> A big +1 for moving forward with this.
> > > >>>
> > > >>> And as for the design document, I will definitely have a look
> > > >>> and comment there.
> > > >>>
> > > >>> Kostas
> > > >>>
> > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
> > > >>>
> > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > > >>>
> > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > >>>
> > > >>>> Hi devs,
> > > >>>>
> > > >>>> Dian and me and our teammates have investigated this for a long
> > time.
> > > >>>> We think consolidating Flink SQL and CEP is an exciting thing for
> > > Flink.
> > > >>>> It'll make SQL more powerful and give users the ability to easily
> > and
> > > >>>> quickly build CEP applications.  And I find Flink community has
> also
> > > talked
> > > >>>> about this idea before, such as the mailing list [1] and [2] and
> > > Fabian &
> > > >>>> Till's talk in Flink Forward 2016 [3].
> > > >>>>
> > > >>>> I think THIS IS THE POINT to bring up this topic again. Because we
> > > >>>> already have pattern matching foundation in Flink CEP library, and
> > > Stream
> > > >>>> SQL is ready now and Calcite has partially supported pattern
> > matching
> > > >>>> syntax!  We also drafted a design doc about how to integrate SQL
> and
> > > CEP,
> > > >>>> and how to support CEP on Table API.
> https://docs.google.com/docume
> > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=
> sharing
> > > >>>>
> > > >>>>
> > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this discussion,
> > it
> > > >>>> would be great to hear your response.
> > > >>>>
> > > >>>>
> > > >>>> What do others think?
> > > >>>>
> > > >>>>
> > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > >>>>
> > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.c
> > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > >>>>
> > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-analytics-
> > > >>>> cep-two-sides-of-the-same-coin
> > > >>>>
> > > >>>>
> > > >>>> Regards,
> > > >>>>
> > > >>>> Jark Wu
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > >>>>
> > > >>>>> Hi everyone,
> > > >>>>>
> > > >>>>> Flink's CEP library is a great library for complex event
> > processing,
> > > >>>>> more
> > > >>>>> and more customers are expressing their interests in it. But it
> > also
> > > >>>>> has
> > > >>>>> some limitations that users usually have to write a lot of code
> > even
> > > >>>>> for a
> > > >>>>> very simple pattern match use case as it currently only supports
> > the
> > > >>>>> Java
> > > >>>>> API.
> > > >>>>> We have investigated some popular CEP products such as esper [1]
> > and
> > > >>>>> siddhi
> > > >>>>> [2] and found that most of these CEP products support SQL-like
> > > >>>>> expressions
> > > >>>>> such as EPL to describe the match pattern. But these solutions
> also
> > > >>>>> have
> > > >>>>> the drawbacks that the pattern match languages are not standard
> > SQL,
> > > >>>>> the
> > > >>>>> learn curve is steep for users and it's impossible to integrate
> > them
> > > >>>>> into
> > > >>>>> the Flink Table API & SQL.
> > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new pattern
> > > >>>>> recognition clause match_recognize which is a pattern recognition
> > > >>>>> clause
> > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes to
> > > define
> > > >>>>> match pattern in sql expression. Calcite already supports part of
> > > this
> > > >>>>> standard [5].  I think it will be of great value to support
> > > expressing
> > > >>>>> pattern recognition clause with match_recognize clause by
> > integrating
> > > >>>>> it
> > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> thoughts?
> > > >>>>>
> > > >>>>> [1] http://www.espertech.com
> > > >>>>> [2] https://github.com/wso2/siddhi
> > > >>>>> [3]
> > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> CQLLR1531
> > > >>>>> [4]
> > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> > > >>>>> tern-recogniton-11.pdf
> > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > >>>>>
> > > >>>>> Best Regards,
> > > >>>>> Dian
> > > >>>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

yueting chen
Hi Dian & Jark,

I checked out your prototype code, but it didn't pass the CEPITCase test in
the flink-table component.
It turns out that in the `MatchCodeGenerator.scala` file, line 74 should
use `${classOf[IterativeCondition.Context[_]].getCanonicalName}` instead of
`${classOf[IterativeCondition.Context[_]]}`.

I've also read your desgin document and it looks fine to me. Actually, I am
working on the same thing recently, I think maybe we can work together to
push this forward.

Thanks,
Yueting Chen

On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]> wrote:

> Hi Fabian,
>
> We have evaluated the missing features of Flink CEP roughly, it should not
> be quite difficult to support them. Kostas, Dawid, what's your thought?
>
> For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs and
> start to work right now or we should wait until the release of calcite
> 1.13?
>
> Btw, could you help to add me(dian.fu) to the contributor list, then I can
> assign the JIRAs to myself? Thanks a lot.
>
> Best regards,
> Dian
>
> On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi Jark,
> >
> > Thanks for updating the design doc and sharing your prototype!
> > I didn't look at the code in detail, but the fact that it is less than 1k
> > LOC is very promising. It seems that most of the complex CEP logic can be
> > reused :-)
> > Adding a dependency on flink-cep should be fine, IMO. It is a very slim
> > library with almost none external dependencies.
> >
> > Regarding the missing features of Flink CEP that you listed in the design
> > doc, it would be good to get some in put from Kostas and Dawid which are
> > the main contributors to CEP.
> > Do you have already plans regarding some of the missing features or can
> you
> > assess how hard it would be to integrate them?
> >
> > Cheers, Fabian
> >
> > Btw. The Calcite community started a discussion about releasing Calcite
> > 1.13. So, the missing features might soon be available.
> >
> > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> >
> > > Hi guys,
> > >
> > > Good news! We have made a prototype for integrating CEP and SQL. See
> this
> > > link
> > > https://github.com/apache/flink/compare/master...
> > > wuchong:cep-on-sql?expand=1
> > >
> > >
> > > You can check CepITCase to try the simple CQL example.
> > >
> > > Meanwhile, we updated our design doc with additional implementation
> > detail,
> > > including how
> > > to translate MATCH_RECOGNIZE into CEP API, and the features needed to
> add
> > > to Flink CEP,
> > > and the implementation plan. See the document
> > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > >
> > > In the prototype, we make flink-table dependency on flink-cep. Do you
> > think
> > > that is fine?
> > >
> > > What do you think about the prototype and the design doc ?
> > >
> > > Any feedbacks are welcome!
> > >
> > > Cheers,
> > > Jark Wu
> > >
> > >
> > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
> > >
> > > > Thanks for sharing your ideas with the community. I really like the
> > > design
> > > > document and think that it's a good approach to follow Oracle's SQL
> > > > extension for pattern matching. Looking forward to having support for
> > SQL
> > > > with CEP capabilities :-)
> > > >
> > > > Cheers,
> > > > Till
> > > >
> > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
> > > >
> > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > >
> > > > > @Fabian, I totally agree with you that we should focus on SQL
> first.
> > > > Let's
> > > > > keep Table API in mind and discuss that later.
> > > > >
> > > > > Regarding to the orderBy() clause, I'm not sure about that. I think
> > it
> > > > > makes sense to make it required in streaming mode(either order by
> > > rowtime
> > > > > or order by proctime). But CEP also works in batch mode, and not
> > > > necessary
> > > > > to order by some column. Nevertheless, we can support CEP on batch
> > SQL
> > > > > later.
> > > > >
> > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP library
> > > (with
> > > > > NFA, CEP operator). And we will output a detailed doc and a
> prototype
> > > in
> > > > > the next days.
> > > > >
> > > > > Regards,
> > > > > Jark Wu
> > > > >
> > > > >
> > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> > > > >
> > > > >> Thanks Dian and Jark for this proposal!
> > > > >>
> > > > >> As you wrote, Till and I (and Kostas) have been thinking about
> this
> > > for
> > > > >> some time but haven't had time to work on this feature.
> > > > >> I think it would be a great addition and value add for Flink's SQL
> > > > >> support and Table API.
> > > > >>
> > > > >> I read the proposal and think it is very good. We might need to
> add
> > a
> > > > bit
> > > > >> more details, esp. when planning the concrete steps of the
> > > > implementation.
> > > > >>
> > > > >> A few comments to the proposal:
> > > > >> - IMO, the development should start focusing on SQL and its
> > semantics.
> > > > >> Pattern support for the Table API should be added later. We
> followed
> > > > that
> > > > >> approach for the OVER windows and I think it worked quiet well.
> > > > >> - We probably want to reuse as much as possible from the CEP
> > library.
> > > > >> That means we need to check if the semantics of the CEP library
> and
> > > > >> Oracle's PATTERN syntax are aligned (or how we can express the
> > PATTERN
> > > > >> semantics with the CEP library). This should be one of the first
> > > steps,
> > > > IMO.
> > > > >> - I would make the orderBy() clause required. In regular SQL rows
> > have
> > > > no
> > > > >> order, so we need to make that explicit (this would also be
> > consistent
> > > > with
> > > > >> the OVER windows).
> > > > >>
> > > > >> Let me know what you think.
> > > > >>
> > > > >> Best, Fabian
> > > > >>
> > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > [hidden email]
> > > > >:
> > > > >>
> > > > >>> Thanks a lot for opening the discussion!
> > > > >>>
> > > > >>> This is a really interesting idea that has been in our heads
> > > > >>> since the first implementation of the CEP library.
> > > > >>>
> > > > >>> A big +1 for moving forward with this.
> > > > >>>
> > > > >>> And as for the design document, I will definitely have a look
> > > > >>> and comment there.
> > > > >>>
> > > > >>> Kostas
> > > > >>>
> > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
> > > > >>>
> > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > > > >>>
> > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > >>>
> > > > >>>> Hi devs,
> > > > >>>>
> > > > >>>> Dian and me and our teammates have investigated this for a long
> > > time.
> > > > >>>> We think consolidating Flink SQL and CEP is an exciting thing
> for
> > > > Flink.
> > > > >>>> It'll make SQL more powerful and give users the ability to
> easily
> > > and
> > > > >>>> quickly build CEP applications.  And I find Flink community has
> > also
> > > > talked
> > > > >>>> about this idea before, such as the mailing list [1] and [2] and
> > > > Fabian &
> > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > >>>>
> > > > >>>> I think THIS IS THE POINT to bring up this topic again. Because
> we
> > > > >>>> already have pattern matching foundation in Flink CEP library,
> and
> > > > Stream
> > > > >>>> SQL is ready now and Calcite has partially supported pattern
> > > matching
> > > > >>>> syntax!  We also drafted a design doc about how to integrate SQL
> > and
> > > > CEP,
> > > > >>>> and how to support CEP on Table API.
> > https://docs.google.com/docume
> > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=
> > sharing
> > > > >>>>
> > > > >>>>
> > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> discussion,
> > > it
> > > > >>>> would be great to hear your response.
> > > > >>>>
> > > > >>>>
> > > > >>>> What do others think?
> > > > >>>>
> > > > >>>>
> > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.c
> > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > >>>>
> > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> nabble.c
> > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > > >>>>
> > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> analytics-
> > > > >>>> cep-two-sides-of-the-same-coin
> > > > >>>>
> > > > >>>>
> > > > >>>> Regards,
> > > > >>>>
> > > > >>>> Jark Wu
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > > >>>>
> > > > >>>>> Hi everyone,
> > > > >>>>>
> > > > >>>>> Flink's CEP library is a great library for complex event
> > > processing,
> > > > >>>>> more
> > > > >>>>> and more customers are expressing their interests in it. But it
> > > also
> > > > >>>>> has
> > > > >>>>> some limitations that users usually have to write a lot of code
> > > even
> > > > >>>>> for a
> > > > >>>>> very simple pattern match use case as it currently only
> supports
> > > the
> > > > >>>>> Java
> > > > >>>>> API.
> > > > >>>>> We have investigated some popular CEP products such as esper
> [1]
> > > and
> > > > >>>>> siddhi
> > > > >>>>> [2] and found that most of these CEP products support SQL-like
> > > > >>>>> expressions
> > > > >>>>> such as EPL to describe the match pattern. But these solutions
> > also
> > > > >>>>> have
> > > > >>>>> the drawbacks that the pattern match languages are not standard
> > > SQL,
> > > > >>>>> the
> > > > >>>>> learn curve is steep for users and it's impossible to integrate
> > > them
> > > > >>>>> into
> > > > >>>>> the Flink Table API & SQL.
> > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new
> pattern
> > > > >>>>> recognition clause match_recognize which is a pattern
> recognition
> > > > >>>>> clause
> > > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes
> to
> > > > define
> > > > >>>>> match pattern in sql expression. Calcite already supports part
> of
> > > > this
> > > > >>>>> standard [5].  I think it will be of great value to support
> > > > expressing
> > > > >>>>> pattern recognition clause with match_recognize clause by
> > > integrating
> > > > >>>>> it
> > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> > thoughts?
> > > > >>>>>
> > > > >>>>> [1] http://www.espertech.com
> > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > >>>>> [3]
> > > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> > CQLLR1531
> > > > >>>>> [4]
> > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> > > > >>>>> tern-recogniton-11.pdf
> > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > >>>>>
> > > > >>>>> Best Regards,
> > > > >>>>> Dian
> > > > >>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Dawid Wysakowicz
Hi all,

Integrating SQL with CEP seems like a really nice idea. Unfortunately I had
time just for a brief look at the design doc, but it looks really cool and
thorough. Also will have a second run tomorrow and will try to provide more
comments. Anyway will be glad to help pushing the initiative forward.

Z pozdrowieniami! / Cheers!

Dawid Wysakowicz

*Data/Software Engineer*

Skype: dawid_wys | Twitter: @OneMoreCoder

<http://getindata.com/>

2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:

> Hi Dian & Jark,
>
> I checked out your prototype code, but it didn't pass the CEPITCase test in
> the flink-table component.
> It turns out that in the `MatchCodeGenerator.scala` file, line 74 should
> use `${classOf[IterativeCondition.Context[_]].getCanonicalName}` instead
> of
> `${classOf[IterativeCondition.Context[_]]}`.
>
> I've also read your desgin document and it looks fine to me. Actually, I am
> working on the same thing recently, I think maybe we can work together to
> push this forward.
>
> Thanks,
> Yueting Chen
>
> On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]> wrote:
>
> > Hi Fabian,
> >
> > We have evaluated the missing features of Flink CEP roughly, it should
> not
> > be quite difficult to support them. Kostas, Dawid, what's your thought?
> >
> > For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs
> and
> > start to work right now or we should wait until the release of calcite
> > 1.13?
> >
> > Btw, could you help to add me(dian.fu) to the contributor list, then I
> can
> > assign the JIRAs to myself? Thanks a lot.
> >
> > Best regards,
> > Dian
> >
> > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi Jark,
> > >
> > > Thanks for updating the design doc and sharing your prototype!
> > > I didn't look at the code in detail, but the fact that it is less than
> 1k
> > > LOC is very promising. It seems that most of the complex CEP logic can
> be
> > > reused :-)
> > > Adding a dependency on flink-cep should be fine, IMO. It is a very slim
> > > library with almost none external dependencies.
> > >
> > > Regarding the missing features of Flink CEP that you listed in the
> design
> > > doc, it would be good to get some in put from Kostas and Dawid which
> are
> > > the main contributors to CEP.
> > > Do you have already plans regarding some of the missing features or can
> > you
> > > assess how hard it would be to integrate them?
> > >
> > > Cheers, Fabian
> > >
> > > Btw. The Calcite community started a discussion about releasing Calcite
> > > 1.13. So, the missing features might soon be available.
> > >
> > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > >
> > > > Hi guys,
> > > >
> > > > Good news! We have made a prototype for integrating CEP and SQL. See
> > this
> > > > link
> > > > https://github.com/apache/flink/compare/master...
> > > > wuchong:cep-on-sql?expand=1
> > > >
> > > >
> > > > You can check CepITCase to try the simple CQL example.
> > > >
> > > > Meanwhile, we updated our design doc with additional implementation
> > > detail,
> > > > including how
> > > > to translate MATCH_RECOGNIZE into CEP API, and the features needed to
> > add
> > > > to Flink CEP,
> > > > and the implementation plan. See the document
> > > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > >
> > > > In the prototype, we make flink-table dependency on flink-cep. Do you
> > > think
> > > > that is fine?
> > > >
> > > > What do you think about the prototype and the design doc ?
> > > >
> > > > Any feedbacks are welcome!
> > > >
> > > > Cheers,
> > > > Jark Wu
> > > >
> > > >
> > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
> > > >
> > > > > Thanks for sharing your ideas with the community. I really like the
> > > > design
> > > > > document and think that it's a good approach to follow Oracle's SQL
> > > > > extension for pattern matching. Looking forward to having support
> for
> > > SQL
> > > > > with CEP capabilities :-)
> > > > >
> > > > > Cheers,
> > > > > Till
> > > > >
> > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
> > > > >
> > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > >
> > > > > > @Fabian, I totally agree with you that we should focus on SQL
> > first.
> > > > > Let's
> > > > > > keep Table API in mind and discuss that later.
> > > > > >
> > > > > > Regarding to the orderBy() clause, I'm not sure about that. I
> think
> > > it
> > > > > > makes sense to make it required in streaming mode(either order by
> > > > rowtime
> > > > > > or order by proctime). But CEP also works in batch mode, and not
> > > > > necessary
> > > > > > to order by some column. Nevertheless, we can support CEP on
> batch
> > > SQL
> > > > > > later.
> > > > > >
> > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP
> library
> > > > (with
> > > > > > NFA, CEP operator). And we will output a detailed doc and a
> > prototype
> > > > in
> > > > > > the next days.
> > > > > >
> > > > > > Regards,
> > > > > > Jark Wu
> > > > > >
> > > > > >
> > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> > > > > >
> > > > > >> Thanks Dian and Jark for this proposal!
> > > > > >>
> > > > > >> As you wrote, Till and I (and Kostas) have been thinking about
> > this
> > > > for
> > > > > >> some time but haven't had time to work on this feature.
> > > > > >> I think it would be a great addition and value add for Flink's
> SQL
> > > > > >> support and Table API.
> > > > > >>
> > > > > >> I read the proposal and think it is very good. We might need to
> > add
> > > a
> > > > > bit
> > > > > >> more details, esp. when planning the concrete steps of the
> > > > > implementation.
> > > > > >>
> > > > > >> A few comments to the proposal:
> > > > > >> - IMO, the development should start focusing on SQL and its
> > > semantics.
> > > > > >> Pattern support for the Table API should be added later. We
> > followed
> > > > > that
> > > > > >> approach for the OVER windows and I think it worked quiet well.
> > > > > >> - We probably want to reuse as much as possible from the CEP
> > > library.
> > > > > >> That means we need to check if the semantics of the CEP library
> > and
> > > > > >> Oracle's PATTERN syntax are aligned (or how we can express the
> > > PATTERN
> > > > > >> semantics with the CEP library). This should be one of the first
> > > > steps,
> > > > > IMO.
> > > > > >> - I would make the orderBy() clause required. In regular SQL
> rows
> > > have
> > > > > no
> > > > > >> order, so we need to make that explicit (this would also be
> > > consistent
> > > > > with
> > > > > >> the OVER windows).
> > > > > >>
> > > > > >> Let me know what you think.
> > > > > >>
> > > > > >> Best, Fabian
> > > > > >>
> > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > [hidden email]
> > > > > >:
> > > > > >>
> > > > > >>> Thanks a lot for opening the discussion!
> > > > > >>>
> > > > > >>> This is a really interesting idea that has been in our heads
> > > > > >>> since the first implementation of the CEP library.
> > > > > >>>
> > > > > >>> A big +1 for moving forward with this.
> > > > > >>>
> > > > > >>> And as for the design document, I will definitely have a look
> > > > > >>> and comment there.
> > > > > >>>
> > > > > >>> Kostas
> > > > > >>>
> > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]> wrote:
> > > > > >>>
> > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > > > > >>>
> > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > >>>
> > > > > >>>> Hi devs,
> > > > > >>>>
> > > > > >>>> Dian and me and our teammates have investigated this for a
> long
> > > > time.
> > > > > >>>> We think consolidating Flink SQL and CEP is an exciting thing
> > for
> > > > > Flink.
> > > > > >>>> It'll make SQL more powerful and give users the ability to
> > easily
> > > > and
> > > > > >>>> quickly build CEP applications.  And I find Flink community
> has
> > > also
> > > > > talked
> > > > > >>>> about this idea before, such as the mailing list [1] and [2]
> and
> > > > > Fabian &
> > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > >>>>
> > > > > >>>> I think THIS IS THE POINT to bring up this topic again.
> Because
> > we
> > > > > >>>> already have pattern matching foundation in Flink CEP library,
> > and
> > > > > Stream
> > > > > >>>> SQL is ready now and Calcite has partially supported pattern
> > > > matching
> > > > > >>>> syntax!  We also drafted a design doc about how to integrate
> SQL
> > > and
> > > > > CEP,
> > > > > >>>> and how to support CEP on Table API.
> > > https://docs.google.com/docume
> > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=
> > > sharing
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> > discussion,
> > > > it
> > > > > >>>> would be great to hear your response.
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> What do others think?
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> > nabble.c
> > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > >>>>
> > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> > nabble.c
> > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > > > >>>>
> > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> > analytics-
> > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> Regards,
> > > > > >>>>
> > > > > >>>> Jark Wu
> > > > > >>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > > > >>>>
> > > > > >>>>> Hi everyone,
> > > > > >>>>>
> > > > > >>>>> Flink's CEP library is a great library for complex event
> > > > processing,
> > > > > >>>>> more
> > > > > >>>>> and more customers are expressing their interests in it. But
> it
> > > > also
> > > > > >>>>> has
> > > > > >>>>> some limitations that users usually have to write a lot of
> code
> > > > even
> > > > > >>>>> for a
> > > > > >>>>> very simple pattern match use case as it currently only
> > supports
> > > > the
> > > > > >>>>> Java
> > > > > >>>>> API.
> > > > > >>>>> We have investigated some popular CEP products such as esper
> > [1]
> > > > and
> > > > > >>>>> siddhi
> > > > > >>>>> [2] and found that most of these CEP products support
> SQL-like
> > > > > >>>>> expressions
> > > > > >>>>> such as EPL to describe the match pattern. But these
> solutions
> > > also
> > > > > >>>>> have
> > > > > >>>>> the drawbacks that the pattern match languages are not
> standard
> > > > SQL,
> > > > > >>>>> the
> > > > > >>>>> learn curve is steep for users and it's impossible to
> integrate
> > > > them
> > > > > >>>>> into
> > > > > >>>>> the Flink Table API & SQL.
> > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new
> > pattern
> > > > > >>>>> recognition clause match_recognize which is a pattern
> > recognition
> > > > > >>>>> clause
> > > > > >>>>> proposed in this paper [4]. It proposes a set of new syntaxes
> > to
> > > > > define
> > > > > >>>>> match pattern in sql expression. Calcite already supports
> part
> > of
> > > > > this
> > > > > >>>>> standard [5].  I think it will be of great value to support
> > > > > expressing
> > > > > >>>>> pattern recognition clause with match_recognize clause by
> > > > integrating
> > > > > >>>>> it
> > > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> > > thoughts?
> > > > > >>>>>
> > > > > >>>>> [1] http://www.espertech.com
> > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > >>>>> [3]
> > > > > >>>>> https://docs.oracle.com/middleware/1213/eventprocessing/cql-
> > > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> > > CQLLR1531
> > > > > >>>>> [4]
> > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/row-pat
> > > > > >>>>> tern-recogniton-11.pdf
> > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > > >>>>>
> > > > > >>>>> Best Regards,
> > > > > >>>>> Dian
> > > > > >>>>>
> > > > > >>>>
> > > > > >>>>
> > > > > >>>
> > > > > >>>
> > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Dian Fu-2
Hi Yueting & Dawid Wysakowicz,

Very glad that you're interested in this feature and you're definitely
welcome to join this work and also anyone else.:)

Best regards,
Dian

On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
[hidden email]> wrote:

> Hi all,
>
> Integrating SQL with CEP seems like a really nice idea. Unfortunately I had
> time just for a brief look at the design doc, but it looks really cool and
> thorough. Also will have a second run tomorrow and will try to provide more
> comments. Anyway will be glad to help pushing the initiative forward.
>
> Z pozdrowieniami! / Cheers!
>
> Dawid Wysakowicz
>
> *Data/Software Engineer*
>
> Skype: dawid_wys | Twitter: @OneMoreCoder
>
> <http://getindata.com/>
>
> 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
>
> > Hi Dian & Jark,
> >
> > I checked out your prototype code, but it didn't pass the CEPITCase test
> in
> > the flink-table component.
> > It turns out that in the `MatchCodeGenerator.scala` file, line 74 should
> > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}` instead
> > of
> > `${classOf[IterativeCondition.Context[_]]}`.
> >
> > I've also read your desgin document and it looks fine to me. Actually, I
> am
> > working on the same thing recently, I think maybe we can work together to
> > push this forward.
> >
> > Thanks,
> > Yueting Chen
> >
> > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]> wrote:
> >
> > > Hi Fabian,
> > >
> > > We have evaluated the missing features of Flink CEP roughly, it should
> > not
> > > be quite difficult to support them. Kostas, Dawid, what's your thought?
> > >
> > > For supporting MATCH_RECOGNIZE, do you think we could create the JIRAs
> > and
> > > start to work right now or we should wait until the release of calcite
> > > 1.13?
> > >
> > > Btw, could you help to add me(dian.fu) to the contributor list, then I
> > can
> > > assign the JIRAs to myself? Thanks a lot.
> > >
> > > Best regards,
> > > Dian
> > >
> > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]>
> > wrote:
> > >
> > > > Hi Jark,
> > > >
> > > > Thanks for updating the design doc and sharing your prototype!
> > > > I didn't look at the code in detail, but the fact that it is less
> than
> > 1k
> > > > LOC is very promising. It seems that most of the complex CEP logic
> can
> > be
> > > > reused :-)
> > > > Adding a dependency on flink-cep should be fine, IMO. It is a very
> slim
> > > > library with almost none external dependencies.
> > > >
> > > > Regarding the missing features of Flink CEP that you listed in the
> > design
> > > > doc, it would be good to get some in put from Kostas and Dawid which
> > are
> > > > the main contributors to CEP.
> > > > Do you have already plans regarding some of the missing features or
> can
> > > you
> > > > assess how hard it would be to integrate them?
> > > >
> > > > Cheers, Fabian
> > > >
> > > > Btw. The Calcite community started a discussion about releasing
> Calcite
> > > > 1.13. So, the missing features might soon be available.
> > > >
> > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > >
> > > > > Hi guys,
> > > > >
> > > > > Good news! We have made a prototype for integrating CEP and SQL.
> See
> > > this
> > > > > link
> > > > > https://github.com/apache/flink/compare/master...
> > > > > wuchong:cep-on-sql?expand=1
> > > > >
> > > > >
> > > > > You can check CepITCase to try the simple CQL example.
> > > > >
> > > > > Meanwhile, we updated our design doc with additional implementation
> > > > detail,
> > > > > including how
> > > > > to translate MATCH_RECOGNIZE into CEP API, and the features needed
> to
> > > add
> > > > > to Flink CEP,
> > > > > and the implementation plan. See the document
> > > > > https://docs.google.com/document/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > >
> > > > > In the prototype, we make flink-table dependency on flink-cep. Do
> you
> > > > think
> > > > > that is fine?
> > > > >
> > > > > What do you think about the prototype and the design doc ?
> > > > >
> > > > > Any feedbacks are welcome!
> > > > >
> > > > > Cheers,
> > > > > Jark Wu
> > > > >
> > > > >
> > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
> > > > >
> > > > > > Thanks for sharing your ideas with the community. I really like
> the
> > > > > design
> > > > > > document and think that it's a good approach to follow Oracle's
> SQL
> > > > > > extension for pattern matching. Looking forward to having support
> > for
> > > > SQL
> > > > > > with CEP capabilities :-)
> > > > > >
> > > > > > Cheers,
> > > > > > Till
> > > > > >
> > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]> wrote:
> > > > > >
> > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > >
> > > > > > > @Fabian, I totally agree with you that we should focus on SQL
> > > first.
> > > > > > Let's
> > > > > > > keep Table API in mind and discuss that later.
> > > > > > >
> > > > > > > Regarding to the orderBy() clause, I'm not sure about that. I
> > think
> > > > it
> > > > > > > makes sense to make it required in streaming mode(either order
> by
> > > > > rowtime
> > > > > > > or order by proctime). But CEP also works in batch mode, and
> not
> > > > > > necessary
> > > > > > > to order by some column. Nevertheless, we can support CEP on
> > batch
> > > > SQL
> > > > > > > later.
> > > > > > >
> > > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP
> > library
> > > > > (with
> > > > > > > NFA, CEP operator). And we will output a detailed doc and a
> > > prototype
> > > > > in
> > > > > > > the next days.
> > > > > > >
> > > > > > > Regards,
> > > > > > > Jark Wu
> > > > > > >
> > > > > > >
> > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]>:
> > > > > > >
> > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > >>
> > > > > > >> As you wrote, Till and I (and Kostas) have been thinking about
> > > this
> > > > > for
> > > > > > >> some time but haven't had time to work on this feature.
> > > > > > >> I think it would be a great addition and value add for Flink's
> > SQL
> > > > > > >> support and Table API.
> > > > > > >>
> > > > > > >> I read the proposal and think it is very good. We might need
> to
> > > add
> > > > a
> > > > > > bit
> > > > > > >> more details, esp. when planning the concrete steps of the
> > > > > > implementation.
> > > > > > >>
> > > > > > >> A few comments to the proposal:
> > > > > > >> - IMO, the development should start focusing on SQL and its
> > > > semantics.
> > > > > > >> Pattern support for the Table API should be added later. We
> > > followed
> > > > > > that
> > > > > > >> approach for the OVER windows and I think it worked quiet
> well.
> > > > > > >> - We probably want to reuse as much as possible from the CEP
> > > > library.
> > > > > > >> That means we need to check if the semantics of the CEP
> library
> > > and
> > > > > > >> Oracle's PATTERN syntax are aligned (or how we can express the
> > > > PATTERN
> > > > > > >> semantics with the CEP library). This should be one of the
> first
> > > > > steps,
> > > > > > IMO.
> > > > > > >> - I would make the orderBy() clause required. In regular SQL
> > rows
> > > > have
> > > > > > no
> > > > > > >> order, so we need to make that explicit (this would also be
> > > > consistent
> > > > > > with
> > > > > > >> the OVER windows).
> > > > > > >>
> > > > > > >> Let me know what you think.
> > > > > > >>
> > > > > > >> Best, Fabian
> > > > > > >>
> > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > [hidden email]
> > > > > > >:
> > > > > > >>
> > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > >>>
> > > > > > >>> This is a really interesting idea that has been in our heads
> > > > > > >>> since the first implementation of the CEP library.
> > > > > > >>>
> > > > > > >>> A big +1 for moving forward with this.
> > > > > > >>>
> > > > > > >>> And as for the design document, I will definitely have a look
> > > > > > >>> and comment there.
> > > > > > >>>
> > > > > > >>> Kostas
> > > > > > >>>
> > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]>
> wrote:
> > > > > > >>>
> > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till, @Kostas
> > > > > > >>>
> > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > >>>
> > > > > > >>>> Hi devs,
> > > > > > >>>>
> > > > > > >>>> Dian and me and our teammates have investigated this for a
> > long
> > > > > time.
> > > > > > >>>> We think consolidating Flink SQL and CEP is an exciting
> thing
> > > for
> > > > > > Flink.
> > > > > > >>>> It'll make SQL more powerful and give users the ability to
> > > easily
> > > > > and
> > > > > > >>>> quickly build CEP applications.  And I find Flink community
> > has
> > > > also
> > > > > > talked
> > > > > > >>>> about this idea before, such as the mailing list [1] and [2]
> > and
> > > > > > Fabian &
> > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > >>>>
> > > > > > >>>> I think THIS IS THE POINT to bring up this topic again.
> > Because
> > > we
> > > > > > >>>> already have pattern matching foundation in Flink CEP
> library,
> > > and
> > > > > > Stream
> > > > > > >>>> SQL is ready now and Calcite has partially supported pattern
> > > > > matching
> > > > > > >>>> syntax!  We also drafted a design doc about how to integrate
> > SQL
> > > > and
> > > > > > CEP,
> > > > > > >>>> and how to support CEP on Table API.
> > > > https://docs.google.com/docume
> > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7iK15H0YbniTnN30/edit?usp=
> > > > sharing
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> > > discussion,
> > > > > it
> > > > > > >>>> would be great to hear your response.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> What do others think?
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> > > nabble.c
> > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > >>>>
> > > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> > > nabble.c
> > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > > > > >>>>
> > > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> > > analytics-
> > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Regards,
> > > > > > >>>>
> > > > > > >>>> Jark Wu
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > > > > >>>>
> > > > > > >>>>> Hi everyone,
> > > > > > >>>>>
> > > > > > >>>>> Flink's CEP library is a great library for complex event
> > > > > processing,
> > > > > > >>>>> more
> > > > > > >>>>> and more customers are expressing their interests in it.
> But
> > it
> > > > > also
> > > > > > >>>>> has
> > > > > > >>>>> some limitations that users usually have to write a lot of
> > code
> > > > > even
> > > > > > >>>>> for a
> > > > > > >>>>> very simple pattern match use case as it currently only
> > > supports
> > > > > the
> > > > > > >>>>> Java
> > > > > > >>>>> API.
> > > > > > >>>>> We have investigated some popular CEP products such as
> esper
> > > [1]
> > > > > and
> > > > > > >>>>> siddhi
> > > > > > >>>>> [2] and found that most of these CEP products support
> > SQL-like
> > > > > > >>>>> expressions
> > > > > > >>>>> such as EPL to describe the match pattern. But these
> > solutions
> > > > also
> > > > > > >>>>> have
> > > > > > >>>>> the drawbacks that the pattern match languages are not
> > standard
> > > > > SQL,
> > > > > > >>>>> the
> > > > > > >>>>> learn curve is steep for users and it's impossible to
> > integrate
> > > > > them
> > > > > > >>>>> into
> > > > > > >>>>> the Flink Table API & SQL.
> > > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new
> > > pattern
> > > > > > >>>>> recognition clause match_recognize which is a pattern
> > > recognition
> > > > > > >>>>> clause
> > > > > > >>>>> proposed in this paper [4]. It proposes a set of new
> syntaxes
> > > to
> > > > > > define
> > > > > > >>>>> match pattern in sql expression. Calcite already supports
> > part
> > > of
> > > > > > this
> > > > > > >>>>> standard [5].  I think it will be of great value to support
> > > > > > expressing
> > > > > > >>>>> pattern recognition clause with match_recognize clause by
> > > > > integrating
> > > > > > >>>>> it
> > > > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> > > > thoughts?
> > > > > > >>>>>
> > > > > > >>>>> [1] http://www.espertech.com
> > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > >>>>> [3]
> > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> eventprocessing/cql-
> > > > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> > > > CQLLR1531
> > > > > > >>>>> [4]
> > > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/
> row-pat
> > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > > > >>>>>
> > > > > > >>>>> Best Regards,
> > > > > > >>>>> Dian
> > > > > > >>>>>
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Liangfei Su
+1 for the feature. Myself was a user of Siddhi, this is pretty user
friendly feature to provide to user.

On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]> wrote:

> Hi Yueting & Dawid Wysakowicz,
>
> Very glad that you're interested in this feature and you're definitely
> welcome to join this work and also anyone else.:)
>
> Best regards,
> Dian
>
> On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> [hidden email]> wrote:
>
> > Hi all,
> >
> > Integrating SQL with CEP seems like a really nice idea. Unfortunately I
> had
> > time just for a brief look at the design doc, but it looks really cool
> and
> > thorough. Also will have a second run tomorrow and will try to provide
> more
> > comments. Anyway will be glad to help pushing the initiative forward.
> >
> > Z pozdrowieniami! / Cheers!
> >
> > Dawid Wysakowicz
> >
> > *Data/Software Engineer*
> >
> > Skype: dawid_wys | Twitter: @OneMoreCoder
> >
> > <http://getindata.com/>
> >
> > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> >
> > > Hi Dian & Jark,
> > >
> > > I checked out your prototype code, but it didn't pass the CEPITCase
> test
> > in
> > > the flink-table component.
> > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> should
> > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> instead
> > > of
> > > `${classOf[IterativeCondition.Context[_]]}`.
> > >
> > > I've also read your desgin document and it looks fine to me. Actually,
> I
> > am
> > > working on the same thing recently, I think maybe we can work together
> to
> > > push this forward.
> > >
> > > Thanks,
> > > Yueting Chen
> > >
> > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]> wrote:
> > >
> > > > Hi Fabian,
> > > >
> > > > We have evaluated the missing features of Flink CEP roughly, it
> should
> > > not
> > > > be quite difficult to support them. Kostas, Dawid, what's your
> thought?
> > > >
> > > > For supporting MATCH_RECOGNIZE, do you think we could create the
> JIRAs
> > > and
> > > > start to work right now or we should wait until the release of
> calcite
> > > > 1.13?
> > > >
> > > > Btw, could you help to add me(dian.fu) to the contributor list, then
> I
> > > can
> > > > assign the JIRAs to myself? Thanks a lot.
> > > >
> > > > Best regards,
> > > > Dian
> > > >
> > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]>
> > > wrote:
> > > >
> > > > > Hi Jark,
> > > > >
> > > > > Thanks for updating the design doc and sharing your prototype!
> > > > > I didn't look at the code in detail, but the fact that it is less
> > than
> > > 1k
> > > > > LOC is very promising. It seems that most of the complex CEP logic
> > can
> > > be
> > > > > reused :-)
> > > > > Adding a dependency on flink-cep should be fine, IMO. It is a very
> > slim
> > > > > library with almost none external dependencies.
> > > > >
> > > > > Regarding the missing features of Flink CEP that you listed in the
> > > design
> > > > > doc, it would be good to get some in put from Kostas and Dawid
> which
> > > are
> > > > > the main contributors to CEP.
> > > > > Do you have already plans regarding some of the missing features or
> > can
> > > > you
> > > > > assess how hard it would be to integrate them?
> > > > >
> > > > > Cheers, Fabian
> > > > >
> > > > > Btw. The Calcite community started a discussion about releasing
> > Calcite
> > > > > 1.13. So, the missing features might soon be available.
> > > > >
> > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > >
> > > > > > Hi guys,
> > > > > >
> > > > > > Good news! We have made a prototype for integrating CEP and SQL.
> > See
> > > > this
> > > > > > link
> > > > > > https://github.com/apache/flink/compare/master...
> > > > > > wuchong:cep-on-sql?expand=1
> > > > > >
> > > > > >
> > > > > > You can check CepITCase to try the simple CQL example.
> > > > > >
> > > > > > Meanwhile, we updated our design doc with additional
> implementation
> > > > > detail,
> > > > > > including how
> > > > > > to translate MATCH_RECOGNIZE into CEP API, and the features
> needed
> > to
> > > > add
> > > > > > to Flink CEP,
> > > > > > and the implementation plan. See the document
> > > > > > https://docs.google.com/document/d/
> 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > >
> > > > > > In the prototype, we make flink-table dependency on flink-cep. Do
> > you
> > > > > think
> > > > > > that is fine?
> > > > > >
> > > > > > What do you think about the prototype and the design doc ?
> > > > > >
> > > > > > Any feedbacks are welcome!
> > > > > >
> > > > > > Cheers,
> > > > > > Jark Wu
> > > > > >
> > > > > >
> > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]>:
> > > > > >
> > > > > > > Thanks for sharing your ideas with the community. I really like
> > the
> > > > > > design
> > > > > > > document and think that it's a good approach to follow Oracle's
> > SQL
> > > > > > > extension for pattern matching. Looking forward to having
> support
> > > for
> > > > > SQL
> > > > > > > with CEP capabilities :-)
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Till
> > > > > > >
> > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]>
> wrote:
> > > > > > >
> > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > >
> > > > > > > > @Fabian, I totally agree with you that we should focus on SQL
> > > > first.
> > > > > > > Let's
> > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > >
> > > > > > > > Regarding to the orderBy() clause, I'm not sure about that. I
> > > think
> > > > > it
> > > > > > > > makes sense to make it required in streaming mode(either
> order
> > by
> > > > > > rowtime
> > > > > > > > or order by proctime). But CEP also works in batch mode, and
> > not
> > > > > > > necessary
> > > > > > > > to order by some column. Nevertheless, we can support CEP on
> > > batch
> > > > > SQL
> > > > > > > > later.
> > > > > > > >
> > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP
> > > library
> > > > > > (with
> > > > > > > > NFA, CEP operator). And we will output a detailed doc and a
> > > > prototype
> > > > > > in
> > > > > > > > the next days.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Jark Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <[hidden email]
> >:
> > > > > > > >
> > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > >>
> > > > > > > >> As you wrote, Till and I (and Kostas) have been thinking
> about
> > > > this
> > > > > > for
> > > > > > > >> some time but haven't had time to work on this feature.
> > > > > > > >> I think it would be a great addition and value add for
> Flink's
> > > SQL
> > > > > > > >> support and Table API.
> > > > > > > >>
> > > > > > > >> I read the proposal and think it is very good. We might need
> > to
> > > > add
> > > > > a
> > > > > > > bit
> > > > > > > >> more details, esp. when planning the concrete steps of the
> > > > > > > implementation.
> > > > > > > >>
> > > > > > > >> A few comments to the proposal:
> > > > > > > >> - IMO, the development should start focusing on SQL and its
> > > > > semantics.
> > > > > > > >> Pattern support for the Table API should be added later. We
> > > > followed
> > > > > > > that
> > > > > > > >> approach for the OVER windows and I think it worked quiet
> > well.
> > > > > > > >> - We probably want to reuse as much as possible from the CEP
> > > > > library.
> > > > > > > >> That means we need to check if the semantics of the CEP
> > library
> > > > and
> > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can express
> the
> > > > > PATTERN
> > > > > > > >> semantics with the CEP library). This should be one of the
> > first
> > > > > > steps,
> > > > > > > IMO.
> > > > > > > >> - I would make the orderBy() clause required. In regular SQL
> > > rows
> > > > > have
> > > > > > > no
> > > > > > > >> order, so we need to make that explicit (this would also be
> > > > > consistent
> > > > > > > with
> > > > > > > >> the OVER windows).
> > > > > > > >>
> > > > > > > >> Let me know what you think.
> > > > > > > >>
> > > > > > > >> Best, Fabian
> > > > > > > >>
> > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > > >>
> > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > >>>
> > > > > > > >>> This is a really interesting idea that has been in our
> heads
> > > > > > > >>> since the first implementation of the CEP library.
> > > > > > > >>>
> > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > >>>
> > > > > > > >>> And as for the design document, I will definitely have a
> look
> > > > > > > >>> and comment there.
> > > > > > > >>>
> > > > > > > >>> Kostas
> > > > > > > >>>
> > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]>
> > wrote:
> > > > > > > >>>
> > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till,
> @Kostas
> > > > > > > >>>
> > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > > >>>
> > > > > > > >>>> Hi devs,
> > > > > > > >>>>
> > > > > > > >>>> Dian and me and our teammates have investigated this for a
> > > long
> > > > > > time.
> > > > > > > >>>> We think consolidating Flink SQL and CEP is an exciting
> > thing
> > > > for
> > > > > > > Flink.
> > > > > > > >>>> It'll make SQL more powerful and give users the ability to
> > > > easily
> > > > > > and
> > > > > > > >>>> quickly build CEP applications.  And I find Flink
> community
> > > has
> > > > > also
> > > > > > > talked
> > > > > > > >>>> about this idea before, such as the mailing list [1] and
> [2]
> > > and
> > > > > > > Fabian &
> > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > >>>>
> > > > > > > >>>> I think THIS IS THE POINT to bring up this topic again.
> > > Because
> > > > we
> > > > > > > >>>> already have pattern matching foundation in Flink CEP
> > library,
> > > > and
> > > > > > > Stream
> > > > > > > >>>> SQL is ready now and Calcite has partially supported
> pattern
> > > > > > matching
> > > > > > > >>>> syntax!  We also drafted a design doc about how to
> integrate
> > > SQL
> > > > > and
> > > > > > > CEP,
> > > > > > > >>>> and how to support CEP on Table API.
> > > > > https://docs.google.com/docume
> > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> K15H0YbniTnN30/edit?usp=
> > > > > sharing
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> > > > discussion,
> > > > > > it
> > > > > > > >>>> would be great to hear your response.
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> What do others think?
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3.
> > > > nabble.c
> > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > >>>>
> > > > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3.
> > > > nabble.c
> > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.html#a9790
> > > > > > > >>>>
> > > > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> > > > analytics-
> > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> Regards,
> > > > > > > >>>>
> > > > > > > >>>> Jark Wu
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > > > > > >>>>
> > > > > > > >>>>> Hi everyone,
> > > > > > > >>>>>
> > > > > > > >>>>> Flink's CEP library is a great library for complex event
> > > > > > processing,
> > > > > > > >>>>> more
> > > > > > > >>>>> and more customers are expressing their interests in it.
> > But
> > > it
> > > > > > also
> > > > > > > >>>>> has
> > > > > > > >>>>> some limitations that users usually have to write a lot
> of
> > > code
> > > > > > even
> > > > > > > >>>>> for a
> > > > > > > >>>>> very simple pattern match use case as it currently only
> > > > supports
> > > > > > the
> > > > > > > >>>>> Java
> > > > > > > >>>>> API.
> > > > > > > >>>>> We have investigated some popular CEP products such as
> > esper
> > > > [1]
> > > > > > and
> > > > > > > >>>>> siddhi
> > > > > > > >>>>> [2] and found that most of these CEP products support
> > > SQL-like
> > > > > > > >>>>> expressions
> > > > > > > >>>>> such as EPL to describe the match pattern. But these
> > > solutions
> > > > > also
> > > > > > > >>>>> have
> > > > > > > >>>>> the drawbacks that the pattern match languages are not
> > > standard
> > > > > > SQL,
> > > > > > > >>>>> the
> > > > > > > >>>>> learn curve is steep for users and it's impossible to
> > > integrate
> > > > > > them
> > > > > > > >>>>> into
> > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a new
> > > > pattern
> > > > > > > >>>>> recognition clause match_recognize which is a pattern
> > > > recognition
> > > > > > > >>>>> clause
> > > > > > > >>>>> proposed in this paper [4]. It proposes a set of new
> > syntaxes
> > > > to
> > > > > > > define
> > > > > > > >>>>> match pattern in sql expression. Calcite already supports
> > > part
> > > > of
> > > > > > > this
> > > > > > > >>>>> standard [5].  I think it will be of great value to
> support
> > > > > > > expressing
> > > > > > > >>>>> pattern recognition clause with match_recognize clause by
> > > > > > integrating
> > > > > > > >>>>> it
> > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP library. Any
> > > > > thoughts?
> > > > > > > >>>>>
> > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > >>>>> [3]
> > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > eventprocessing/cql-
> > > > > > > >>>>> reference/GUID-34D4968E-C55A-4BC7-B1CE-C84B202217BD.htm#
> > > > > CQLLR1531
> > > > > > > >>>>> [4]
> > > > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/
> > row-pat
> > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > > > > >>>>>
> > > > > > > >>>>> Best Regards,
> > > > > > > >>>>> Dian
> > > > > > > >>>>>
> > > > > > > >>>>
> > > > > > > >>>>
> > > > > > > >>>
> > > > > > > >>>
> > > > > > > >>
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark Wu
Thank you Yueting for pointing out the mistake in the prototype. I
accidentally introduced it when merge code.

I'm so glad to see so many people are interested in the feature. Let's work
out together to push it forward!

Cheers,
Jark Wu


2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:

> +1 for the feature. Myself was a user of Siddhi, this is pretty user
> friendly feature to provide to user.
>
> On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]> wrote:
>
> > Hi Yueting & Dawid Wysakowicz,
> >
> > Very glad that you're interested in this feature and you're definitely
> > welcome to join this work and also anyone else.:)
> >
> > Best regards,
> > Dian
> >
> > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > [hidden email]> wrote:
> >
> > > Hi all,
> > >
> > > Integrating SQL with CEP seems like a really nice idea. Unfortunately I
> > had
> > > time just for a brief look at the design doc, but it looks really cool
> > and
> > > thorough. Also will have a second run tomorrow and will try to provide
> > more
> > > comments. Anyway will be glad to help pushing the initiative forward.
> > >
> > > Z pozdrowieniami! / Cheers!
> > >
> > > Dawid Wysakowicz
> > >
> > > *Data/Software Engineer*
> > >
> > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > >
> > > <http://getindata.com/>
> > >
> > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> > >
> > > > Hi Dian & Jark,
> > > >
> > > > I checked out your prototype code, but it didn't pass the CEPITCase
> > test
> > > in
> > > > the flink-table component.
> > > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> > should
> > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > instead
> > > > of
> > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > >
> > > > I've also read your desgin document and it looks fine to me.
> Actually,
> > I
> > > am
> > > > working on the same thing recently, I think maybe we can work
> together
> > to
> > > > push this forward.
> > > >
> > > > Thanks,
> > > > Yueting Chen
> > > >
> > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]> wrote:
> > > >
> > > > > Hi Fabian,
> > > > >
> > > > > We have evaluated the missing features of Flink CEP roughly, it
> > should
> > > > not
> > > > > be quite difficult to support them. Kostas, Dawid, what's your
> > thought?
> > > > >
> > > > > For supporting MATCH_RECOGNIZE, do you think we could create the
> > JIRAs
> > > > and
> > > > > start to work right now or we should wait until the release of
> > calcite
> > > > > 1.13?
> > > > >
> > > > > Btw, could you help to add me(dian.fu) to the contributor list,
> then
> > I
> > > > can
> > > > > assign the JIRAs to myself? Thanks a lot.
> > > > >
> > > > > Best regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <[hidden email]>
> > > > wrote:
> > > > >
> > > > > > Hi Jark,
> > > > > >
> > > > > > Thanks for updating the design doc and sharing your prototype!
> > > > > > I didn't look at the code in detail, but the fact that it is less
> > > than
> > > > 1k
> > > > > > LOC is very promising. It seems that most of the complex CEP
> logic
> > > can
> > > > be
> > > > > > reused :-)
> > > > > > Adding a dependency on flink-cep should be fine, IMO. It is a
> very
> > > slim
> > > > > > library with almost none external dependencies.
> > > > > >
> > > > > > Regarding the missing features of Flink CEP that you listed in
> the
> > > > design
> > > > > > doc, it would be good to get some in put from Kostas and Dawid
> > which
> > > > are
> > > > > > the main contributors to CEP.
> > > > > > Do you have already plans regarding some of the missing features
> or
> > > can
> > > > > you
> > > > > > assess how hard it would be to integrate them?
> > > > > >
> > > > > > Cheers, Fabian
> > > > > >
> > > > > > Btw. The Calcite community started a discussion about releasing
> > > Calcite
> > > > > > 1.13. So, the missing features might soon be available.
> > > > > >
> > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > >
> > > > > > > Hi guys,
> > > > > > >
> > > > > > > Good news! We have made a prototype for integrating CEP and
> SQL.
> > > See
> > > > > this
> > > > > > > link
> > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > >
> > > > > > >
> > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > >
> > > > > > > Meanwhile, we updated our design doc with additional
> > implementation
> > > > > > detail,
> > > > > > > including how
> > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the features
> > needed
> > > to
> > > > > add
> > > > > > > to Flink CEP,
> > > > > > > and the implementation plan. See the document
> > > > > > > https://docs.google.com/document/d/
> > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > >
> > > > > > > In the prototype, we make flink-table dependency on flink-cep.
> Do
> > > you
> > > > > > think
> > > > > > > that is fine?
> > > > > > >
> > > > > > > What do you think about the prototype and the design doc ?
> > > > > > >
> > > > > > > Any feedbacks are welcome!
> > > > > > >
> > > > > > > Cheers,
> > > > > > > Jark Wu
> > > > > > >
> > > > > > >
> > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <[hidden email]
> >:
> > > > > > >
> > > > > > > > Thanks for sharing your ideas with the community. I really
> like
> > > the
> > > > > > > design
> > > > > > > > document and think that it's a good approach to follow
> Oracle's
> > > SQL
> > > > > > > > extension for pattern matching. Looking forward to having
> > support
> > > > for
> > > > > > SQL
> > > > > > > > with CEP capabilities :-)
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Till
> > > > > > > >
> > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]>
> > wrote:
> > > > > > > >
> > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > >
> > > > > > > > > @Fabian, I totally agree with you that we should focus on
> SQL
> > > > > first.
> > > > > > > > Let's
> > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > >
> > > > > > > > > Regarding to the orderBy() clause, I'm not sure about
> that. I
> > > > think
> > > > > > it
> > > > > > > > > makes sense to make it required in streaming mode(either
> > order
> > > by
> > > > > > > rowtime
> > > > > > > > > or order by proctime). But CEP also works in batch mode,
> and
> > > not
> > > > > > > > necessary
> > > > > > > > > to order by some column. Nevertheless, we can support CEP
> on
> > > > batch
> > > > > > SQL
> > > > > > > > > later.
> > > > > > > > >
> > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with CEP
> > > > library
> > > > > > > (with
> > > > > > > > > NFA, CEP operator). And we will output a detailed doc and a
> > > > > prototype
> > > > > > > in
> > > > > > > > > the next days.
> > > > > > > > >
> > > > > > > > > Regards,
> > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> [hidden email]
> > >:
> > > > > > > > >
> > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > >>
> > > > > > > > >> As you wrote, Till and I (and Kostas) have been thinking
> > about
> > > > > this
> > > > > > > for
> > > > > > > > >> some time but haven't had time to work on this feature.
> > > > > > > > >> I think it would be a great addition and value add for
> > Flink's
> > > > SQL
> > > > > > > > >> support and Table API.
> > > > > > > > >>
> > > > > > > > >> I read the proposal and think it is very good. We might
> need
> > > to
> > > > > add
> > > > > > a
> > > > > > > > bit
> > > > > > > > >> more details, esp. when planning the concrete steps of the
> > > > > > > > implementation.
> > > > > > > > >>
> > > > > > > > >> A few comments to the proposal:
> > > > > > > > >> - IMO, the development should start focusing on SQL and
> its
> > > > > > semantics.
> > > > > > > > >> Pattern support for the Table API should be added later.
> We
> > > > > followed
> > > > > > > > that
> > > > > > > > >> approach for the OVER windows and I think it worked quiet
> > > well.
> > > > > > > > >> - We probably want to reuse as much as possible from the
> CEP
> > > > > > library.
> > > > > > > > >> That means we need to check if the semantics of the CEP
> > > library
> > > > > and
> > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can express
> > the
> > > > > > PATTERN
> > > > > > > > >> semantics with the CEP library). This should be one of the
> > > first
> > > > > > > steps,
> > > > > > > > IMO.
> > > > > > > > >> - I would make the orderBy() clause required. In regular
> SQL
> > > > rows
> > > > > > have
> > > > > > > > no
> > > > > > > > >> order, so we need to make that explicit (this would also
> be
> > > > > > consistent
> > > > > > > > with
> > > > > > > > >> the OVER windows).
> > > > > > > > >>
> > > > > > > > >> Let me know what you think.
> > > > > > > > >>
> > > > > > > > >> Best, Fabian
> > > > > > > > >>
> > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > [hidden email]
> > > > > > > > >:
> > > > > > > > >>
> > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > >>>
> > > > > > > > >>> This is a really interesting idea that has been in our
> > heads
> > > > > > > > >>> since the first implementation of the CEP library.
> > > > > > > > >>>
> > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > >>>
> > > > > > > > >>> And as for the design document, I will definitely have a
> > look
> > > > > > > > >>> and comment there.
> > > > > > > > >>>
> > > > > > > > >>> Kostas
> > > > > > > > >>>
> > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]>
> > > wrote:
> > > > > > > > >>>
> > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till,
> > @Kostas
> > > > > > > > >>>
> > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > > > >>>
> > > > > > > > >>>> Hi devs,
> > > > > > > > >>>>
> > > > > > > > >>>> Dian and me and our teammates have investigated this
> for a
> > > > long
> > > > > > > time.
> > > > > > > > >>>> We think consolidating Flink SQL and CEP is an exciting
> > > thing
> > > > > for
> > > > > > > > Flink.
> > > > > > > > >>>> It'll make SQL more powerful and give users the ability
> to
> > > > > easily
> > > > > > > and
> > > > > > > > >>>> quickly build CEP applications.  And I find Flink
> > community
> > > > has
> > > > > > also
> > > > > > > > talked
> > > > > > > > >>>> about this idea before, such as the mailing list [1] and
> > [2]
> > > > and
> > > > > > > > Fabian &
> > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > >>>>
> > > > > > > > >>>> I think THIS IS THE POINT to bring up this topic again.
> > > > Because
> > > > > we
> > > > > > > > >>>> already have pattern matching foundation in Flink CEP
> > > library,
> > > > > and
> > > > > > > > Stream
> > > > > > > > >>>> SQL is ready now and Calcite has partially supported
> > pattern
> > > > > > > matching
> > > > > > > > >>>> syntax!  We also drafted a design doc about how to
> > integrate
> > > > SQL
> > > > > > and
> > > > > > > > CEP,
> > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > https://docs.google.com/docume
> > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > K15H0YbniTnN30/edit?usp=
> > > > > > sharing
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> > > > > discussion,
> > > > > > > it
> > > > > > > > >>>> would be great to hear your response.
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> What do others think?
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> [1] http://apache-flink-mailing-list-archive.1008284.n3
> .
> > > > > nabble.c
> > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > >>>>
> > > > > > > > >>>> [2] http://apache-flink-mailing-list-archive.1008284.n3
> .
> > > > > nabble.c
> > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> html#a9790
> > > > > > > > >>>>
> > > > > > > > >>>> [3] https://www.slideshare.net/tillrohrmann/streaming-
> > > > > analytics-
> > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> Regards,
> > > > > > > > >>>>
> > > > > > > > >>>> Jark Wu
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]>:
> > > > > > > > >>>>
> > > > > > > > >>>>> Hi everyone,
> > > > > > > > >>>>>
> > > > > > > > >>>>> Flink's CEP library is a great library for complex
> event
> > > > > > > processing,
> > > > > > > > >>>>> more
> > > > > > > > >>>>> and more customers are expressing their interests in
> it.
> > > But
> > > > it
> > > > > > > also
> > > > > > > > >>>>> has
> > > > > > > > >>>>> some limitations that users usually have to write a lot
> > of
> > > > code
> > > > > > > even
> > > > > > > > >>>>> for a
> > > > > > > > >>>>> very simple pattern match use case as it currently only
> > > > > supports
> > > > > > > the
> > > > > > > > >>>>> Java
> > > > > > > > >>>>> API.
> > > > > > > > >>>>> We have investigated some popular CEP products such as
> > > esper
> > > > > [1]
> > > > > > > and
> > > > > > > > >>>>> siddhi
> > > > > > > > >>>>> [2] and found that most of these CEP products support
> > > > SQL-like
> > > > > > > > >>>>> expressions
> > > > > > > > >>>>> such as EPL to describe the match pattern. But these
> > > > solutions
> > > > > > also
> > > > > > > > >>>>> have
> > > > > > > > >>>>> the drawbacks that the pattern match languages are not
> > > > standard
> > > > > > > SQL,
> > > > > > > > >>>>> the
> > > > > > > > >>>>> learn curve is steep for users and it's impossible to
> > > > integrate
> > > > > > > them
> > > > > > > > >>>>> into
> > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a
> new
> > > > > pattern
> > > > > > > > >>>>> recognition clause match_recognize which is a pattern
> > > > > recognition
> > > > > > > > >>>>> clause
> > > > > > > > >>>>> proposed in this paper [4]. It proposes a set of new
> > > syntaxes
> > > > > to
> > > > > > > > define
> > > > > > > > >>>>> match pattern in sql expression. Calcite already
> supports
> > > > part
> > > > > of
> > > > > > > > this
> > > > > > > > >>>>> standard [5].  I think it will be of great value to
> > support
> > > > > > > > expressing
> > > > > > > > >>>>> pattern recognition clause with match_recognize clause
> by
> > > > > > > integrating
> > > > > > > > >>>>> it
> > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP library.
> Any
> > > > > > thoughts?
> > > > > > > > >>>>>
> > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > >>>>> [3]
> > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > eventprocessing/cql-
> > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> 4BC7-B1CE-C84B202217BD.htm#
> > > > > > CQLLR1531
> > > > > > > > >>>>> [4]
> > > > > > > > >>>>> http://web.cs.ucla.edu/classes/winter17/cs240B/notes/
> > > row-pat
> > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > >>>>> [5] https://issues.apache.org/jira/browse/CALCITE-1570
> > > > > > > > >>>>>
> > > > > > > > >>>>> Best Regards,
> > > > > > > > >>>>> Dian
> > > > > > > > >>>>>
> > > > > > > > >>>>
> > > > > > > > >>>>
> > > > > > > > >>>
> > > > > > > > >>>
> > > > > > > > >>
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

jincheng sun
Hi Jark, Dian,

Thanks for bring up this discuss and share the prototype.

+1 to push this great feature forward!

Cheers,
SunJincheng

2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:

> Thank you Yueting for pointing out the mistake in the prototype. I
> accidentally introduced it when merge code.
>
> I'm so glad to see so many people are interested in the feature. Let's work
> out together to push it forward!
>
> Cheers,
> Jark Wu
>
>
> 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
>
> > +1 for the feature. Myself was a user of Siddhi, this is pretty user
> > friendly feature to provide to user.
> >
> > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]> wrote:
> >
> > > Hi Yueting & Dawid Wysakowicz,
> > >
> > > Very glad that you're interested in this feature and you're definitely
> > > welcome to join this work and also anyone else.:)
> > >
> > > Best regards,
> > > Dian
> > >
> > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > [hidden email]> wrote:
> > >
> > > > Hi all,
> > > >
> > > > Integrating SQL with CEP seems like a really nice idea.
> Unfortunately I
> > > had
> > > > time just for a brief look at the design doc, but it looks really
> cool
> > > and
> > > > thorough. Also will have a second run tomorrow and will try to
> provide
> > > more
> > > > comments. Anyway will be glad to help pushing the initiative forward.
> > > >
> > > > Z pozdrowieniami! / Cheers!
> > > >
> > > > Dawid Wysakowicz
> > > >
> > > > *Data/Software Engineer*
> > > >
> > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > >
> > > > <http://getindata.com/>
> > > >
> > > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> > > >
> > > > > Hi Dian & Jark,
> > > > >
> > > > > I checked out your prototype code, but it didn't pass the CEPITCase
> > > test
> > > > in
> > > > > the flink-table component.
> > > > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> > > should
> > > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > > instead
> > > > > of
> > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > >
> > > > > I've also read your desgin document and it looks fine to me.
> > Actually,
> > > I
> > > > am
> > > > > working on the same thing recently, I think maybe we can work
> > together
> > > to
> > > > > push this forward.
> > > > >
> > > > > Thanks,
> > > > > Yueting Chen
> > > > >
> > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]>
> wrote:
> > > > >
> > > > > > Hi Fabian,
> > > > > >
> > > > > > We have evaluated the missing features of Flink CEP roughly, it
> > > should
> > > > > not
> > > > > > be quite difficult to support them. Kostas, Dawid, what's your
> > > thought?
> > > > > >
> > > > > > For supporting MATCH_RECOGNIZE, do you think we could create the
> > > JIRAs
> > > > > and
> > > > > > start to work right now or we should wait until the release of
> > > calcite
> > > > > > 1.13?
> > > > > >
> > > > > > Btw, could you help to add me(dian.fu) to the contributor list,
> > then
> > > I
> > > > > can
> > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > >
> > > > > > Best regards,
> > > > > > Dian
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> [hidden email]>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Jark,
> > > > > > >
> > > > > > > Thanks for updating the design doc and sharing your prototype!
> > > > > > > I didn't look at the code in detail, but the fact that it is
> less
> > > > than
> > > > > 1k
> > > > > > > LOC is very promising. It seems that most of the complex CEP
> > logic
> > > > can
> > > > > be
> > > > > > > reused :-)
> > > > > > > Adding a dependency on flink-cep should be fine, IMO. It is a
> > very
> > > > slim
> > > > > > > library with almost none external dependencies.
> > > > > > >
> > > > > > > Regarding the missing features of Flink CEP that you listed in
> > the
> > > > > design
> > > > > > > doc, it would be good to get some in put from Kostas and Dawid
> > > which
> > > > > are
> > > > > > > the main contributors to CEP.
> > > > > > > Do you have already plans regarding some of the missing
> features
> > or
> > > > can
> > > > > > you
> > > > > > > assess how hard it would be to integrate them?
> > > > > > >
> > > > > > > Cheers, Fabian
> > > > > > >
> > > > > > > Btw. The Calcite community started a discussion about releasing
> > > > Calcite
> > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > >
> > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > > >
> > > > > > > > Hi guys,
> > > > > > > >
> > > > > > > > Good news! We have made a prototype for integrating CEP and
> > SQL.
> > > > See
> > > > > > this
> > > > > > > > link
> > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > >
> > > > > > > >
> > > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > > >
> > > > > > > > Meanwhile, we updated our design doc with additional
> > > implementation
> > > > > > > detail,
> > > > > > > > including how
> > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the features
> > > needed
> > > > to
> > > > > > add
> > > > > > > > to Flink CEP,
> > > > > > > > and the implementation plan. See the document
> > > > > > > > https://docs.google.com/document/d/
> > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > >
> > > > > > > > In the prototype, we make flink-table dependency on
> flink-cep.
> > Do
> > > > you
> > > > > > > think
> > > > > > > > that is fine?
> > > > > > > >
> > > > > > > > What do you think about the prototype and the design doc ?
> > > > > > > >
> > > > > > > > Any feedbacks are welcome!
> > > > > > > >
> > > > > > > > Cheers,
> > > > > > > > Jark Wu
> > > > > > > >
> > > > > > > >
> > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> [hidden email]
> > >:
> > > > > > > >
> > > > > > > > > Thanks for sharing your ideas with the community. I really
> > like
> > > > the
> > > > > > > > design
> > > > > > > > > document and think that it's a good approach to follow
> > Oracle's
> > > > SQL
> > > > > > > > > extension for pattern matching. Looking forward to having
> > > support
> > > > > for
> > > > > > > SQL
> > > > > > > > > with CEP capabilities :-)
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Till
> > > > > > > > >
> > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]>
> > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > > >
> > > > > > > > > > @Fabian, I totally agree with you that we should focus on
> > SQL
> > > > > > first.
> > > > > > > > > Let's
> > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > >
> > > > > > > > > > Regarding to the orderBy() clause, I'm not sure about
> > that. I
> > > > > think
> > > > > > > it
> > > > > > > > > > makes sense to make it required in streaming mode(either
> > > order
> > > > by
> > > > > > > > rowtime
> > > > > > > > > > or order by proctime). But CEP also works in batch mode,
> > and
> > > > not
> > > > > > > > > necessary
> > > > > > > > > > to order by some column. Nevertheless, we can support CEP
> > on
> > > > > batch
> > > > > > > SQL
> > > > > > > > > > later.
> > > > > > > > > >
> > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with
> CEP
> > > > > library
> > > > > > > > (with
> > > > > > > > > > NFA, CEP operator). And we will output a detailed doc
> and a
> > > > > > prototype
> > > > > > > > in
> > > > > > > > > > the next days.
> > > > > > > > > >
> > > > > > > > > > Regards,
> > > > > > > > > > Jark Wu
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > [hidden email]
> > > >:
> > > > > > > > > >
> > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > >>
> > > > > > > > > >> As you wrote, Till and I (and Kostas) have been thinking
> > > about
> > > > > > this
> > > > > > > > for
> > > > > > > > > >> some time but haven't had time to work on this feature.
> > > > > > > > > >> I think it would be a great addition and value add for
> > > Flink's
> > > > > SQL
> > > > > > > > > >> support and Table API.
> > > > > > > > > >>
> > > > > > > > > >> I read the proposal and think it is very good. We might
> > need
> > > > to
> > > > > > add
> > > > > > > a
> > > > > > > > > bit
> > > > > > > > > >> more details, esp. when planning the concrete steps of
> the
> > > > > > > > > implementation.
> > > > > > > > > >>
> > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > >> - IMO, the development should start focusing on SQL and
> > its
> > > > > > > semantics.
> > > > > > > > > >> Pattern support for the Table API should be added later.
> > We
> > > > > > followed
> > > > > > > > > that
> > > > > > > > > >> approach for the OVER windows and I think it worked
> quiet
> > > > well.
> > > > > > > > > >> - We probably want to reuse as much as possible from the
> > CEP
> > > > > > > library.
> > > > > > > > > >> That means we need to check if the semantics of the CEP
> > > > library
> > > > > > and
> > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can
> express
> > > the
> > > > > > > PATTERN
> > > > > > > > > >> semantics with the CEP library). This should be one of
> the
> > > > first
> > > > > > > > steps,
> > > > > > > > > IMO.
> > > > > > > > > >> - I would make the orderBy() clause required. In regular
> > SQL
> > > > > rows
> > > > > > > have
> > > > > > > > > no
> > > > > > > > > >> order, so we need to make that explicit (this would also
> > be
> > > > > > > consistent
> > > > > > > > > with
> > > > > > > > > >> the OVER windows).
> > > > > > > > > >>
> > > > > > > > > >> Let me know what you think.
> > > > > > > > > >>
> > > > > > > > > >> Best, Fabian
> > > > > > > > > >>
> > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > [hidden email]
> > > > > > > > > >:
> > > > > > > > > >>
> > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > >>>
> > > > > > > > > >>> This is a really interesting idea that has been in our
> > > heads
> > > > > > > > > >>> since the first implementation of the CEP library.
> > > > > > > > > >>>
> > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > >>>
> > > > > > > > > >>> And as for the design document, I will definitely have
> a
> > > look
> > > > > > > > > >>> and comment there.
> > > > > > > > > >>>
> > > > > > > > > >>> Kostas
> > > > > > > > > >>>
> > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <[hidden email]>
> > > > wrote:
> > > > > > > > > >>>
> > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till,
> > > @Kostas
> > > > > > > > > >>>
> > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]>:
> > > > > > > > > >>>
> > > > > > > > > >>>> Hi devs,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Dian and me and our teammates have investigated this
> > for a
> > > > > long
> > > > > > > > time.
> > > > > > > > > >>>> We think consolidating Flink SQL and CEP is an
> exciting
> > > > thing
> > > > > > for
> > > > > > > > > Flink.
> > > > > > > > > >>>> It'll make SQL more powerful and give users the
> ability
> > to
> > > > > > easily
> > > > > > > > and
> > > > > > > > > >>>> quickly build CEP applications.  And I find Flink
> > > community
> > > > > has
> > > > > > > also
> > > > > > > > > talked
> > > > > > > > > >>>> about this idea before, such as the mailing list [1]
> and
> > > [2]
> > > > > and
> > > > > > > > > Fabian &
> > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > >>>>
> > > > > > > > > >>>> I think THIS IS THE POINT to bring up this topic
> again.
> > > > > Because
> > > > > > we
> > > > > > > > > >>>> already have pattern matching foundation in Flink CEP
> > > > library,
> > > > > > and
> > > > > > > > > Stream
> > > > > > > > > >>>> SQL is ready now and Calcite has partially supported
> > > pattern
> > > > > > > > matching
> > > > > > > > > >>>> syntax!  We also drafted a design doc about how to
> > > integrate
> > > > > SQL
> > > > > > > and
> > > > > > > > > CEP,
> > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > https://docs.google.com/docume
> > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > K15H0YbniTnN30/edit?usp=
> > > > > > > sharing
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into this
> > > > > > discussion,
> > > > > > > > it
> > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> What do others think?
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> [1] http://apache-flink-mailing-
> list-archive.1008284.n3
> > .
> > > > > > nabble.c
> > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > >>>>
> > > > > > > > > >>>> [2] http://apache-flink-mailing-
> list-archive.1008284.n3
> > .
> > > > > > nabble.c
> > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > html#a9790
> > > > > > > > > >>>>
> > > > > > > > > >>>> [3] https://www.slideshare.net/
> tillrohrmann/streaming-
> > > > > > analytics-
> > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> Regards,
> > > > > > > > > >>>>
> > > > > > > > > >>>> Jark Wu
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <[hidden email]
> >:
> > > > > > > > > >>>>
> > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Flink's CEP library is a great library for complex
> > event
> > > > > > > > processing,
> > > > > > > > > >>>>> more
> > > > > > > > > >>>>> and more customers are expressing their interests in
> > it.
> > > > But
> > > > > it
> > > > > > > > also
> > > > > > > > > >>>>> has
> > > > > > > > > >>>>> some limitations that users usually have to write a
> lot
> > > of
> > > > > code
> > > > > > > > even
> > > > > > > > > >>>>> for a
> > > > > > > > > >>>>> very simple pattern match use case as it currently
> only
> > > > > > supports
> > > > > > > > the
> > > > > > > > > >>>>> Java
> > > > > > > > > >>>>> API.
> > > > > > > > > >>>>> We have investigated some popular CEP products such
> as
> > > > esper
> > > > > > [1]
> > > > > > > > and
> > > > > > > > > >>>>> siddhi
> > > > > > > > > >>>>> [2] and found that most of these CEP products support
> > > > > SQL-like
> > > > > > > > > >>>>> expressions
> > > > > > > > > >>>>> such as EPL to describe the match pattern. But these
> > > > > solutions
> > > > > > > also
> > > > > > > > > >>>>> have
> > > > > > > > > >>>>> the drawbacks that the pattern match languages are
> not
> > > > > standard
> > > > > > > > SQL,
> > > > > > > > > >>>>> the
> > > > > > > > > >>>>> learn curve is steep for users and it's impossible to
> > > > > integrate
> > > > > > > > them
> > > > > > > > > >>>>> into
> > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3] supports a
> > new
> > > > > > pattern
> > > > > > > > > >>>>> recognition clause match_recognize which is a pattern
> > > > > > recognition
> > > > > > > > > >>>>> clause
> > > > > > > > > >>>>> proposed in this paper [4]. It proposes a set of new
> > > > syntaxes
> > > > > > to
> > > > > > > > > define
> > > > > > > > > >>>>> match pattern in sql expression. Calcite already
> > supports
> > > > > part
> > > > > > of
> > > > > > > > > this
> > > > > > > > > >>>>> standard [5].  I think it will be of great value to
> > > support
> > > > > > > > > expressing
> > > > > > > > > >>>>> pattern recognition clause with match_recognize
> clause
> > by
> > > > > > > > integrating
> > > > > > > > > >>>>> it
> > > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP library.
> > Any
> > > > > > > thoughts?
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > >>>>> [3]
> > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > eventprocessing/cql-
> > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > CQLLR1531
> > > > > > > > > >>>>> [4]
> > > > > > > > > >>>>> http://web.cs.ucla.edu/
> classes/winter17/cs240B/notes/
> > > > row-pat
> > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > >>>>> [5] https://issues.apache.org/
> jira/browse/CALCITE-1570
> > > > > > > > > >>>>>
> > > > > > > > > >>>>> Best Regards,
> > > > > > > > > >>>>> Dian
> > > > > > > > > >>>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>>
> > > > > > > > > >>>
> > > > > > > > > >>>
> > > > > > > > > >>
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Fabian Hueske-2
Hi,

@Dian Fu: I gave you contributor permissions. :-)

I don't think we have to wait for Calcite 1.13 to start working on the
missing features of the CEP library and extending the prototype.
We might want to wait with the integration into flink-table until Calcite
1.13 is out and we updated the dependency though.

Best, Fabian

2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:

> Hi Jark, Dian,
>
> Thanks for bring up this discuss and share the prototype.
>
> +1 to push this great feature forward!
>
> Cheers,
> SunJincheng
>
> 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
>
> > Thank you Yueting for pointing out the mistake in the prototype. I
> > accidentally introduced it when merge code.
> >
> > I'm so glad to see so many people are interested in the feature. Let's
> work
> > out together to push it forward!
> >
> > Cheers,
> > Jark Wu
> >
> >
> > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> >
> > > +1 for the feature. Myself was a user of Siddhi, this is pretty user
> > > friendly feature to provide to user.
> > >
> > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]>
> wrote:
> > >
> > > > Hi Yueting & Dawid Wysakowicz,
> > > >
> > > > Very glad that you're interested in this feature and you're
> definitely
> > > > welcome to join this work and also anyone else.:)
> > > >
> > > > Best regards,
> > > > Dian
> > > >
> > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > [hidden email]> wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > Integrating SQL with CEP seems like a really nice idea.
> > Unfortunately I
> > > > had
> > > > > time just for a brief look at the design doc, but it looks really
> > cool
> > > > and
> > > > > thorough. Also will have a second run tomorrow and will try to
> > provide
> > > > more
> > > > > comments. Anyway will be glad to help pushing the initiative
> forward.
> > > > >
> > > > > Z pozdrowieniami! / Cheers!
> > > > >
> > > > > Dawid Wysakowicz
> > > > >
> > > > > *Data/Software Engineer*
> > > > >
> > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > >
> > > > > <http://getindata.com/>
> > > > >
> > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> > > > >
> > > > > > Hi Dian & Jark,
> > > > > >
> > > > > > I checked out your prototype code, but it didn't pass the
> CEPITCase
> > > > test
> > > > > in
> > > > > > the flink-table component.
> > > > > > It turns out that in the `MatchCodeGenerator.scala` file, line 74
> > > > should
> > > > > > use `${classOf[IterativeCondition.Context[_]].getCanonicalName}`
> > > > instead
> > > > > > of
> > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > >
> > > > > > I've also read your desgin document and it looks fine to me.
> > > Actually,
> > > > I
> > > > > am
> > > > > > working on the same thing recently, I think maybe we can work
> > > together
> > > > to
> > > > > > push this forward.
> > > > > >
> > > > > > Thanks,
> > > > > > Yueting Chen
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]>
> > wrote:
> > > > > >
> > > > > > > Hi Fabian,
> > > > > > >
> > > > > > > We have evaluated the missing features of Flink CEP roughly, it
> > > > should
> > > > > > not
> > > > > > > be quite difficult to support them. Kostas, Dawid, what's your
> > > > thought?
> > > > > > >
> > > > > > > For supporting MATCH_RECOGNIZE, do you think we could create
> the
> > > > JIRAs
> > > > > > and
> > > > > > > start to work right now or we should wait until the release of
> > > > calcite
> > > > > > > 1.13?
> > > > > > >
> > > > > > > Btw, could you help to add me(dian.fu) to the contributor list,
> > > then
> > > > I
> > > > > > can
> > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Dian
> > > > > > >
> > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > [hidden email]>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Jark,
> > > > > > > >
> > > > > > > > Thanks for updating the design doc and sharing your
> prototype!
> > > > > > > > I didn't look at the code in detail, but the fact that it is
> > less
> > > > > than
> > > > > > 1k
> > > > > > > > LOC is very promising. It seems that most of the complex CEP
> > > logic
> > > > > can
> > > > > > be
> > > > > > > > reused :-)
> > > > > > > > Adding a dependency on flink-cep should be fine, IMO. It is a
> > > very
> > > > > slim
> > > > > > > > library with almost none external dependencies.
> > > > > > > >
> > > > > > > > Regarding the missing features of Flink CEP that you listed
> in
> > > the
> > > > > > design
> > > > > > > > doc, it would be good to get some in put from Kostas and
> Dawid
> > > > which
> > > > > > are
> > > > > > > > the main contributors to CEP.
> > > > > > > > Do you have already plans regarding some of the missing
> > features
> > > or
> > > > > can
> > > > > > > you
> > > > > > > > assess how hard it would be to integrate them?
> > > > > > > >
> > > > > > > > Cheers, Fabian
> > > > > > > >
> > > > > > > > Btw. The Calcite community started a discussion about
> releasing
> > > > > Calcite
> > > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > > >
> > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > > > >
> > > > > > > > > Hi guys,
> > > > > > > > >
> > > > > > > > > Good news! We have made a prototype for integrating CEP and
> > > SQL.
> > > > > See
> > > > > > > this
> > > > > > > > > link
> > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > > > >
> > > > > > > > > Meanwhile, we updated our design doc with additional
> > > > implementation
> > > > > > > > detail,
> > > > > > > > > including how
> > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the features
> > > > needed
> > > > > to
> > > > > > > add
> > > > > > > > > to Flink CEP,
> > > > > > > > > and the implementation plan. See the document
> > > > > > > > > https://docs.google.com/document/d/
> > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > >
> > > > > > > > > In the prototype, we make flink-table dependency on
> > flink-cep.
> > > Do
> > > > > you
> > > > > > > > think
> > > > > > > > > that is fine?
> > > > > > > > >
> > > > > > > > > What do you think about the prototype and the design doc ?
> > > > > > > > >
> > > > > > > > > Any feedbacks are welcome!
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > > Jark Wu
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > [hidden email]
> > > >:
> > > > > > > > >
> > > > > > > > > > Thanks for sharing your ideas with the community. I
> really
> > > like
> > > > > the
> > > > > > > > > design
> > > > > > > > > > document and think that it's a good approach to follow
> > > Oracle's
> > > > > SQL
> > > > > > > > > > extension for pattern matching. Looking forward to having
> > > > support
> > > > > > for
> > > > > > > > SQL
> > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Till
> > > > > > > > > >
> > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <[hidden email]
> >
> > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > > > >
> > > > > > > > > > > @Fabian, I totally agree with you that we should focus
> on
> > > SQL
> > > > > > > first.
> > > > > > > > > > Let's
> > > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > > >
> > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure about
> > > that. I
> > > > > > think
> > > > > > > > it
> > > > > > > > > > > makes sense to make it required in streaming
> mode(either
> > > > order
> > > > > by
> > > > > > > > > rowtime
> > > > > > > > > > > or order by proctime). But CEP also works in batch
> mode,
> > > and
> > > > > not
> > > > > > > > > > necessary
> > > > > > > > > > > to order by some column. Nevertheless, we can support
> CEP
> > > on
> > > > > > batch
> > > > > > > > SQL
> > > > > > > > > > > later.
> > > > > > > > > > >
> > > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE with
> > CEP
> > > > > > library
> > > > > > > > > (with
> > > > > > > > > > > NFA, CEP operator). And we will output a detailed doc
> > and a
> > > > > > > prototype
> > > > > > > > > in
> > > > > > > > > > > the next days.
> > > > > > > > > > >
> > > > > > > > > > > Regards,
> > > > > > > > > > > Jark Wu
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > [hidden email]
> > > > >:
> > > > > > > > > > >
> > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > >>
> > > > > > > > > > >> As you wrote, Till and I (and Kostas) have been
> thinking
> > > > about
> > > > > > > this
> > > > > > > > > for
> > > > > > > > > > >> some time but haven't had time to work on this
> feature.
> > > > > > > > > > >> I think it would be a great addition and value add for
> > > > Flink's
> > > > > > SQL
> > > > > > > > > > >> support and Table API.
> > > > > > > > > > >>
> > > > > > > > > > >> I read the proposal and think it is very good. We
> might
> > > need
> > > > > to
> > > > > > > add
> > > > > > > > a
> > > > > > > > > > bit
> > > > > > > > > > >> more details, esp. when planning the concrete steps of
> > the
> > > > > > > > > > implementation.
> > > > > > > > > > >>
> > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > >> - IMO, the development should start focusing on SQL
> and
> > > its
> > > > > > > > semantics.
> > > > > > > > > > >> Pattern support for the Table API should be added
> later.
> > > We
> > > > > > > followed
> > > > > > > > > > that
> > > > > > > > > > >> approach for the OVER windows and I think it worked
> > quiet
> > > > > well.
> > > > > > > > > > >> - We probably want to reuse as much as possible from
> the
> > > CEP
> > > > > > > > library.
> > > > > > > > > > >> That means we need to check if the semantics of the
> CEP
> > > > > library
> > > > > > > and
> > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can
> > express
> > > > the
> > > > > > > > PATTERN
> > > > > > > > > > >> semantics with the CEP library). This should be one of
> > the
> > > > > first
> > > > > > > > > steps,
> > > > > > > > > > IMO.
> > > > > > > > > > >> - I would make the orderBy() clause required. In
> regular
> > > SQL
> > > > > > rows
> > > > > > > > have
> > > > > > > > > > no
> > > > > > > > > > >> order, so we need to make that explicit (this would
> also
> > > be
> > > > > > > > consistent
> > > > > > > > > > with
> > > > > > > > > > >> the OVER windows).
> > > > > > > > > > >>
> > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > >>
> > > > > > > > > > >> Best, Fabian
> > > > > > > > > > >>
> > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > [hidden email]
> > > > > > > > > > >:
> > > > > > > > > > >>
> > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > >>>
> > > > > > > > > > >>> This is a really interesting idea that has been in
> our
> > > > heads
> > > > > > > > > > >>> since the first implementation of the CEP library.
> > > > > > > > > > >>>
> > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > >>>
> > > > > > > > > > >>> And as for the design document, I will definitely
> have
> > a
> > > > look
> > > > > > > > > > >>> and comment there.
> > > > > > > > > > >>>
> > > > > > > > > > >>> Kostas
> > > > > > > > > > >>>
> > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> [hidden email]>
> > > > > wrote:
> > > > > > > > > > >>>
> > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo, @Till,
> > > > @Kostas
> > > > > > > > > > >>>
> > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <[hidden email]
> >:
> > > > > > > > > > >>>
> > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Dian and me and our teammates have investigated this
> > > for a
> > > > > > long
> > > > > > > > > time.
> > > > > > > > > > >>>> We think consolidating Flink SQL and CEP is an
> > exciting
> > > > > thing
> > > > > > > for
> > > > > > > > > > Flink.
> > > > > > > > > > >>>> It'll make SQL more powerful and give users the
> > ability
> > > to
> > > > > > > easily
> > > > > > > > > and
> > > > > > > > > > >>>> quickly build CEP applications.  And I find Flink
> > > > community
> > > > > > has
> > > > > > > > also
> > > > > > > > > > talked
> > > > > > > > > > >>>> about this idea before, such as the mailing list [1]
> > and
> > > > [2]
> > > > > > and
> > > > > > > > > > Fabian &
> > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this topic
> > again.
> > > > > > Because
> > > > > > > we
> > > > > > > > > > >>>> already have pattern matching foundation in Flink
> CEP
> > > > > library,
> > > > > > > and
> > > > > > > > > > Stream
> > > > > > > > > > >>>> SQL is ready now and Calcite has partially supported
> > > > pattern
> > > > > > > > > matching
> > > > > > > > > > >>>> syntax!  We also drafted a design doc about how to
> > > > integrate
> > > > > > SQL
> > > > > > > > and
> > > > > > > > > > CEP,
> > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > https://docs.google.com/docume
> > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > sharing
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into
> this
> > > > > > > discussion,
> > > > > > > > > it
> > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> What do others think?
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > list-archive.1008284.n3
> > > .
> > > > > > > nabble.c
> > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > list-archive.1008284.n3
> > > .
> > > > > > > nabble.c
> > > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > > html#a9790
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > tillrohrmann/streaming-
> > > > > > > analytics-
> > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Regards,
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> [hidden email]
> > >:
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Flink's CEP library is a great library for complex
> > > event
> > > > > > > > > processing,
> > > > > > > > > > >>>>> more
> > > > > > > > > > >>>>> and more customers are expressing their interests
> in
> > > it.
> > > > > But
> > > > > > it
> > > > > > > > > also
> > > > > > > > > > >>>>> has
> > > > > > > > > > >>>>> some limitations that users usually have to write a
> > lot
> > > > of
> > > > > > code
> > > > > > > > > even
> > > > > > > > > > >>>>> for a
> > > > > > > > > > >>>>> very simple pattern match use case as it currently
> > only
> > > > > > > supports
> > > > > > > > > the
> > > > > > > > > > >>>>> Java
> > > > > > > > > > >>>>> API.
> > > > > > > > > > >>>>> We have investigated some popular CEP products such
> > as
> > > > > esper
> > > > > > > [1]
> > > > > > > > > and
> > > > > > > > > > >>>>> siddhi
> > > > > > > > > > >>>>> [2] and found that most of these CEP products
> support
> > > > > > SQL-like
> > > > > > > > > > >>>>> expressions
> > > > > > > > > > >>>>> such as EPL to describe the match pattern. But
> these
> > > > > > solutions
> > > > > > > > also
> > > > > > > > > > >>>>> have
> > > > > > > > > > >>>>> the drawbacks that the pattern match languages are
> > not
> > > > > > standard
> > > > > > > > > SQL,
> > > > > > > > > > >>>>> the
> > > > > > > > > > >>>>> learn curve is steep for users and it's impossible
> to
> > > > > > integrate
> > > > > > > > > them
> > > > > > > > > > >>>>> into
> > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3]
> supports a
> > > new
> > > > > > > pattern
> > > > > > > > > > >>>>> recognition clause match_recognize which is a
> pattern
> > > > > > > recognition
> > > > > > > > > > >>>>> clause
> > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a set of
> new
> > > > > syntaxes
> > > > > > > to
> > > > > > > > > > define
> > > > > > > > > > >>>>> match pattern in sql expression. Calcite already
> > > supports
> > > > > > part
> > > > > > > of
> > > > > > > > > > this
> > > > > > > > > > >>>>> standard [5].  I think it will be of great value to
> > > > support
> > > > > > > > > > expressing
> > > > > > > > > > >>>>> pattern recognition clause with match_recognize
> > clause
> > > by
> > > > > > > > > integrating
> > > > > > > > > > >>>>> it
> > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP
> library.
> > > Any
> > > > > > > > thoughts?
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > >>>>> [3]
> > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > eventprocessing/cql-
> > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > CQLLR1531
> > > > > > > > > > >>>>> [4]
> > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > classes/winter17/cs240B/notes/
> > > > > row-pat
> > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > jira/browse/CALCITE-1570
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > >>>>> Dian
> > > > > > > > > > >>>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>>
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> > > > > > > > > > >>
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Dian Fu
Hi Fabian,

Thanks a lot. Agree that we can start working by adding the missing
features of the CEP library.

Best regards,
Dian

On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]> wrote:

> Hi,
>
> @Dian Fu: I gave you contributor permissions. :-)
>
> I don't think we have to wait for Calcite 1.13 to start working on the
> missing features of the CEP library and extending the prototype.
> We might want to wait with the integration into flink-table until Calcite
> 1.13 is out and we updated the dependency though.
>
> Best, Fabian
>
> 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:
>
> > Hi Jark, Dian,
> >
> > Thanks for bring up this discuss and share the prototype.
> >
> > +1 to push this great feature forward!
> >
> > Cheers,
> > SunJincheng
> >
> > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> >
> > > Thank you Yueting for pointing out the mistake in the prototype. I
> > > accidentally introduced it when merge code.
> > >
> > > I'm so glad to see so many people are interested in the feature. Let's
> > work
> > > out together to push it forward!
> > >
> > > Cheers,
> > > Jark Wu
> > >
> > >
> > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> > >
> > > > +1 for the feature. Myself was a user of Siddhi, this is pretty user
> > > > friendly feature to provide to user.
> > > >
> > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]>
> > wrote:
> > > >
> > > > > Hi Yueting & Dawid Wysakowicz,
> > > > >
> > > > > Very glad that you're interested in this feature and you're
> > definitely
> > > > > welcome to join this work and also anyone else.:)
> > > > >
> > > > > Best regards,
> > > > > Dian
> > > > >
> > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > [hidden email]> wrote:
> > > > >
> > > > > > Hi all,
> > > > > >
> > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > Unfortunately I
> > > > > had
> > > > > > time just for a brief look at the design doc, but it looks really
> > > cool
> > > > > and
> > > > > > thorough. Also will have a second run tomorrow and will try to
> > > provide
> > > > > more
> > > > > > comments. Anyway will be glad to help pushing the initiative
> > forward.
> > > > > >
> > > > > > Z pozdrowieniami! / Cheers!
> > > > > >
> > > > > > Dawid Wysakowicz
> > > > > >
> > > > > > *Data/Software Engineer*
> > > > > >
> > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > >
> > > > > > <http://getindata.com/>
> > > > > >
> > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> > > > > >
> > > > > > > Hi Dian & Jark,
> > > > > > >
> > > > > > > I checked out your prototype code, but it didn't pass the
> > CEPITCase
> > > > > test
> > > > > > in
> > > > > > > the flink-table component.
> > > > > > > It turns out that in the `MatchCodeGenerator.scala` file, line
> 74
> > > > > should
> > > > > > > use `${classOf[IterativeCondition.
> Context[_]].getCanonicalName}`
> > > > > instead
> > > > > > > of
> > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > >
> > > > > > > I've also read your desgin document and it looks fine to me.
> > > > Actually,
> > > > > I
> > > > > > am
> > > > > > > working on the same thing recently, I think maybe we can work
> > > > together
> > > > > to
> > > > > > > push this forward.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Yueting Chen
> > > > > > >
> > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]>
> > > wrote:
> > > > > > >
> > > > > > > > Hi Fabian,
> > > > > > > >
> > > > > > > > We have evaluated the missing features of Flink CEP roughly,
> it
> > > > > should
> > > > > > > not
> > > > > > > > be quite difficult to support them. Kostas, Dawid, what's
> your
> > > > > thought?
> > > > > > > >
> > > > > > > > For supporting MATCH_RECOGNIZE, do you think we could create
> > the
> > > > > JIRAs
> > > > > > > and
> > > > > > > > start to work right now or we should wait until the release
> of
> > > > > calcite
> > > > > > > > 1.13?
> > > > > > > >
> > > > > > > > Btw, could you help to add me(dian.fu) to the contributor
> list,
> > > > then
> > > > > I
> > > > > > > can
> > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > >
> > > > > > > > Best regards,
> > > > > > > > Dian
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Jark,
> > > > > > > > >
> > > > > > > > > Thanks for updating the design doc and sharing your
> > prototype!
> > > > > > > > > I didn't look at the code in detail, but the fact that it
> is
> > > less
> > > > > > than
> > > > > > > 1k
> > > > > > > > > LOC is very promising. It seems that most of the complex
> CEP
> > > > logic
> > > > > > can
> > > > > > > be
> > > > > > > > > reused :-)
> > > > > > > > > Adding a dependency on flink-cep should be fine, IMO. It
> is a
> > > > very
> > > > > > slim
> > > > > > > > > library with almost none external dependencies.
> > > > > > > > >
> > > > > > > > > Regarding the missing features of Flink CEP that you listed
> > in
> > > > the
> > > > > > > design
> > > > > > > > > doc, it would be good to get some in put from Kostas and
> > Dawid
> > > > > which
> > > > > > > are
> > > > > > > > > the main contributors to CEP.
> > > > > > > > > Do you have already plans regarding some of the missing
> > > features
> > > > or
> > > > > > can
> > > > > > > > you
> > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > >
> > > > > > > > > Cheers, Fabian
> > > > > > > > >
> > > > > > > > > Btw. The Calcite community started a discussion about
> > releasing
> > > > > > Calcite
> > > > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > > > >
> > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > > > > >
> > > > > > > > > > Hi guys,
> > > > > > > > > >
> > > > > > > > > > Good news! We have made a prototype for integrating CEP
> and
> > > > SQL.
> > > > > > See
> > > > > > > > this
> > > > > > > > > > link
> > > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > > > > >
> > > > > > > > > > Meanwhile, we updated our design doc with additional
> > > > > implementation
> > > > > > > > > detail,
> > > > > > > > > > including how
> > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the
> features
> > > > > needed
> > > > > > to
> > > > > > > > add
> > > > > > > > > > to Flink CEP,
> > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > https://docs.google.com/document/d/
> > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > >
> > > > > > > > > > In the prototype, we make flink-table dependency on
> > > flink-cep.
> > > > Do
> > > > > > you
> > > > > > > > > think
> > > > > > > > > > that is fine?
> > > > > > > > > >
> > > > > > > > > > What do you think about the prototype and the design doc
> ?
> > > > > > > > > >
> > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > >
> > > > > > > > > > Cheers,
> > > > > > > > > > Jark Wu
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > [hidden email]
> > > > >:
> > > > > > > > > >
> > > > > > > > > > > Thanks for sharing your ideas with the community. I
> > really
> > > > like
> > > > > > the
> > > > > > > > > > design
> > > > > > > > > > > document and think that it's a good approach to follow
> > > > Oracle's
> > > > > > SQL
> > > > > > > > > > > extension for pattern matching. Looking forward to
> having
> > > > > support
> > > > > > > for
> > > > > > > > > SQL
> > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Till
> > > > > > > > > > >
> > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > > > > >
> > > > > > > > > > > > @Fabian, I totally agree with you that we should
> focus
> > on
> > > > SQL
> > > > > > > > first.
> > > > > > > > > > > Let's
> > > > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > > > >
> > > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure about
> > > > that. I
> > > > > > > think
> > > > > > > > > it
> > > > > > > > > > > > makes sense to make it required in streaming
> > mode(either
> > > > > order
> > > > > > by
> > > > > > > > > > rowtime
> > > > > > > > > > > > or order by proctime). But CEP also works in batch
> > mode,
> > > > and
> > > > > > not
> > > > > > > > > > > necessary
> > > > > > > > > > > > to order by some column. Nevertheless, we can support
> > CEP
> > > > on
> > > > > > > batch
> > > > > > > > > SQL
> > > > > > > > > > > > later.
> > > > > > > > > > > >
> > > > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE
> with
> > > CEP
> > > > > > > library
> > > > > > > > > > (with
> > > > > > > > > > > > NFA, CEP operator). And we will output a detailed doc
> > > and a
> > > > > > > > prototype
> > > > > > > > > > in
> > > > > > > > > > > > the next days.
> > > > > > > > > > > >
> > > > > > > > > > > > Regards,
> > > > > > > > > > > > Jark Wu
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > [hidden email]
> > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > >>
> > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have been
> > thinking
> > > > > about
> > > > > > > > this
> > > > > > > > > > for
> > > > > > > > > > > >> some time but haven't had time to work on this
> > feature.
> > > > > > > > > > > >> I think it would be a great addition and value add
> for
> > > > > Flink's
> > > > > > > SQL
> > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > >>
> > > > > > > > > > > >> I read the proposal and think it is very good. We
> > might
> > > > need
> > > > > > to
> > > > > > > > add
> > > > > > > > > a
> > > > > > > > > > > bit
> > > > > > > > > > > >> more details, esp. when planning the concrete steps
> of
> > > the
> > > > > > > > > > > implementation.
> > > > > > > > > > > >>
> > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > >> - IMO, the development should start focusing on SQL
> > and
> > > > its
> > > > > > > > > semantics.
> > > > > > > > > > > >> Pattern support for the Table API should be added
> > later.
> > > > We
> > > > > > > > followed
> > > > > > > > > > > that
> > > > > > > > > > > >> approach for the OVER windows and I think it worked
> > > quiet
> > > > > > well.
> > > > > > > > > > > >> - We probably want to reuse as much as possible from
> > the
> > > > CEP
> > > > > > > > > library.
> > > > > > > > > > > >> That means we need to check if the semantics of the
> > CEP
> > > > > > library
> > > > > > > > and
> > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can
> > > express
> > > > > the
> > > > > > > > > PATTERN
> > > > > > > > > > > >> semantics with the CEP library). This should be one
> of
> > > the
> > > > > > first
> > > > > > > > > > steps,
> > > > > > > > > > > IMO.
> > > > > > > > > > > >> - I would make the orderBy() clause required. In
> > regular
> > > > SQL
> > > > > > > rows
> > > > > > > > > have
> > > > > > > > > > > no
> > > > > > > > > > > >> order, so we need to make that explicit (this would
> > also
> > > > be
> > > > > > > > > consistent
> > > > > > > > > > > with
> > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > >>
> > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > >>
> > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > >>
> > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > > [hidden email]
> > > > > > > > > > > >:
> > > > > > > > > > > >>
> > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> This is a really interesting idea that has been in
> > our
> > > > > heads
> > > > > > > > > > > >>> since the first implementation of the CEP library.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> And as for the design document, I will definitely
> > have
> > > a
> > > > > look
> > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Kostas
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > [hidden email]>
> > > > > > wrote:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo,
> @Till,
> > > > > @Kostas
> > > > > > > > > > > >>>
> > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> [hidden email]
> > >:
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Dian and me and our teammates have investigated
> this
> > > > for a
> > > > > > > long
> > > > > > > > > > time.
> > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP is an
> > > exciting
> > > > > > thing
> > > > > > > > for
> > > > > > > > > > > Flink.
> > > > > > > > > > > >>>> It'll make SQL more powerful and give users the
> > > ability
> > > > to
> > > > > > > > easily
> > > > > > > > > > and
> > > > > > > > > > > >>>> quickly build CEP applications.  And I find Flink
> > > > > community
> > > > > > > has
> > > > > > > > > also
> > > > > > > > > > > talked
> > > > > > > > > > > >>>> about this idea before, such as the mailing list
> [1]
> > > and
> > > > > [2]
> > > > > > > and
> > > > > > > > > > > Fabian &
> > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this topic
> > > again.
> > > > > > > Because
> > > > > > > > we
> > > > > > > > > > > >>>> already have pattern matching foundation in Flink
> > CEP
> > > > > > library,
> > > > > > > > and
> > > > > > > > > > > Stream
> > > > > > > > > > > >>>> SQL is ready now and Calcite has partially
> supported
> > > > > pattern
> > > > > > > > > > matching
> > > > > > > > > > > >>>> syntax!  We also drafted a design doc about how to
> > > > > integrate
> > > > > > > SQL
> > > > > > > > > and
> > > > > > > > > > > CEP,
> > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > sharing
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you into
> > this
> > > > > > > > discussion,
> > > > > > > > > > it
> > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > list-archive.1008284.n3
> > > > .
> > > > > > > > nabble.c
> > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > list-archive.1008284.n3
> > > > .
> > > > > > > > nabble.c
> > > > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > > > html#a9790
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > tillrohrmann/streaming-
> > > > > > > > analytics-
> > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > [hidden email]
> > > >:
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Flink's CEP library is a great library for
> complex
> > > > event
> > > > > > > > > > processing,
> > > > > > > > > > > >>>>> more
> > > > > > > > > > > >>>>> and more customers are expressing their interests
> > in
> > > > it.
> > > > > > But
> > > > > > > it
> > > > > > > > > > also
> > > > > > > > > > > >>>>> has
> > > > > > > > > > > >>>>> some limitations that users usually have to
> write a
> > > lot
> > > > > of
> > > > > > > code
> > > > > > > > > > even
> > > > > > > > > > > >>>>> for a
> > > > > > > > > > > >>>>> very simple pattern match use case as it
> currently
> > > only
> > > > > > > > supports
> > > > > > > > > > the
> > > > > > > > > > > >>>>> Java
> > > > > > > > > > > >>>>> API.
> > > > > > > > > > > >>>>> We have investigated some popular CEP products
> such
> > > as
> > > > > > esper
> > > > > > > > [1]
> > > > > > > > > > and
> > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > >>>>> [2] and found that most of these CEP products
> > support
> > > > > > > SQL-like
> > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > >>>>> such as EPL to describe the match pattern. But
> > these
> > > > > > > solutions
> > > > > > > > > also
> > > > > > > > > > > >>>>> have
> > > > > > > > > > > >>>>> the drawbacks that the pattern match languages
> are
> > > not
> > > > > > > standard
> > > > > > > > > > SQL,
> > > > > > > > > > > >>>>> the
> > > > > > > > > > > >>>>> learn curve is steep for users and it's
> impossible
> > to
> > > > > > > integrate
> > > > > > > > > > them
> > > > > > > > > > > >>>>> into
> > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3]
> > supports a
> > > > new
> > > > > > > > pattern
> > > > > > > > > > > >>>>> recognition clause match_recognize which is a
> > pattern
> > > > > > > > recognition
> > > > > > > > > > > >>>>> clause
> > > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a set of
> > new
> > > > > > syntaxes
> > > > > > > > to
> > > > > > > > > > > define
> > > > > > > > > > > >>>>> match pattern in sql expression. Calcite already
> > > > supports
> > > > > > > part
> > > > > > > > of
> > > > > > > > > > > this
> > > > > > > > > > > >>>>> standard [5].  I think it will be of great value
> to
> > > > > support
> > > > > > > > > > > expressing
> > > > > > > > > > > >>>>> pattern recognition clause with match_recognize
> > > clause
> > > > by
> > > > > > > > > > integrating
> > > > > > > > > > > >>>>> it
> > > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP
> > library.
> > > > Any
> > > > > > > > > thoughts?
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > > eventprocessing/cql-
> > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > CQLLR1531
> > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > classes/winter17/cs240B/notes/
> > > > > > row-pat
> > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > jira/browse/CALCITE-1570
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > >>>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>>
> > > > > > > > > > > >>
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Jark
Hi,

Do you think whether we should create a FLIP for this proposal to track
progress?

Regards,
Jark

2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:

> Hi Fabian,
>
> Thanks a lot. Agree that we can start working by adding the missing
> features of the CEP library.
>
> Best regards,
> Dian
>
> On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]> wrote:
>
> > Hi,
> >
> > @Dian Fu: I gave you contributor permissions. :-)
> >
> > I don't think we have to wait for Calcite 1.13 to start working on the
> > missing features of the CEP library and extending the prototype.
> > We might want to wait with the integration into flink-table until Calcite
> > 1.13 is out and we updated the dependency though.
> >
> > Best, Fabian
> >
> > 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:
> >
> > > Hi Jark, Dian,
> > >
> > > Thanks for bring up this discuss and share the prototype.
> > >
> > > +1 to push this great feature forward!
> > >
> > > Cheers,
> > > SunJincheng
> > >
> > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > >
> > > > Thank you Yueting for pointing out the mistake in the prototype. I
> > > > accidentally introduced it when merge code.
> > > >
> > > > I'm so glad to see so many people are interested in the feature.
> Let's
> > > work
> > > > out together to push it forward!
> > > >
> > > > Cheers,
> > > > Jark Wu
> > > >
> > > >
> > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> > > >
> > > > > +1 for the feature. Myself was a user of Siddhi, this is pretty
> user
> > > > > friendly feature to provide to user.
> > > > >
> > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]>
> > > wrote:
> > > > >
> > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > >
> > > > > > Very glad that you're interested in this feature and you're
> > > definitely
> > > > > > welcome to join this work and also anyone else.:)
> > > > > >
> > > > > > Best regards,
> > > > > > Dian
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > [hidden email]> wrote:
> > > > > >
> > > > > > > Hi all,
> > > > > > >
> > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > Unfortunately I
> > > > > > had
> > > > > > > time just for a brief look at the design doc, but it looks
> really
> > > > cool
> > > > > > and
> > > > > > > thorough. Also will have a second run tomorrow and will try to
> > > > provide
> > > > > > more
> > > > > > > comments. Anyway will be glad to help pushing the initiative
> > > forward.
> > > > > > >
> > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > >
> > > > > > > Dawid Wysakowicz
> > > > > > >
> > > > > > > *Data/Software Engineer*
> > > > > > >
> > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > >
> > > > > > > <http://getindata.com/>
> > > > > > >
> > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]>:
> > > > > > >
> > > > > > > > Hi Dian & Jark,
> > > > > > > >
> > > > > > > > I checked out your prototype code, but it didn't pass the
> > > CEPITCase
> > > > > > test
> > > > > > > in
> > > > > > > > the flink-table component.
> > > > > > > > It turns out that in the `MatchCodeGenerator.scala` file,
> line
> > 74
> > > > > > should
> > > > > > > > use `${classOf[IterativeCondition.
> > Context[_]].getCanonicalName}`
> > > > > > instead
> > > > > > > > of
> > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > >
> > > > > > > > I've also read your desgin document and it looks fine to me.
> > > > > Actually,
> > > > > > I
> > > > > > > am
> > > > > > > > working on the same thing recently, I think maybe we can work
> > > > > together
> > > > > > to
> > > > > > > > push this forward.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Yueting Chen
> > > > > > > >
> > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <[hidden email]
> >
> > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Fabian,
> > > > > > > > >
> > > > > > > > > We have evaluated the missing features of Flink CEP
> roughly,
> > it
> > > > > > should
> > > > > > > > not
> > > > > > > > > be quite difficult to support them. Kostas, Dawid, what's
> > your
> > > > > > thought?
> > > > > > > > >
> > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we could
> create
> > > the
> > > > > > JIRAs
> > > > > > > > and
> > > > > > > > > start to work right now or we should wait until the release
> > of
> > > > > > calcite
> > > > > > > > > 1.13?
> > > > > > > > >
> > > > > > > > > Btw, could you help to add me(dian.fu) to the contributor
> > list,
> > > > > then
> > > > > > I
> > > > > > > > can
> > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > >
> > > > > > > > > Best regards,
> > > > > > > > > Dian
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Jark,
> > > > > > > > > >
> > > > > > > > > > Thanks for updating the design doc and sharing your
> > > prototype!
> > > > > > > > > > I didn't look at the code in detail, but the fact that it
> > is
> > > > less
> > > > > > > than
> > > > > > > > 1k
> > > > > > > > > > LOC is very promising. It seems that most of the complex
> > CEP
> > > > > logic
> > > > > > > can
> > > > > > > > be
> > > > > > > > > > reused :-)
> > > > > > > > > > Adding a dependency on flink-cep should be fine, IMO. It
> > is a
> > > > > very
> > > > > > > slim
> > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > >
> > > > > > > > > > Regarding the missing features of Flink CEP that you
> listed
> > > in
> > > > > the
> > > > > > > > design
> > > > > > > > > > doc, it would be good to get some in put from Kostas and
> > > Dawid
> > > > > > which
> > > > > > > > are
> > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > Do you have already plans regarding some of the missing
> > > > features
> > > > > or
> > > > > > > can
> > > > > > > > > you
> > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > >
> > > > > > > > > > Cheers, Fabian
> > > > > > > > > >
> > > > > > > > > > Btw. The Calcite community started a discussion about
> > > releasing
> > > > > > > Calcite
> > > > > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > > > > >
> > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > > > > > >
> > > > > > > > > > > Hi guys,
> > > > > > > > > > >
> > > > > > > > > > > Good news! We have made a prototype for integrating CEP
> > and
> > > > > SQL.
> > > > > > > See
> > > > > > > > > this
> > > > > > > > > > > link
> > > > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > You can check CepITCase to try the simple CQL example.
> > > > > > > > > > >
> > > > > > > > > > > Meanwhile, we updated our design doc with additional
> > > > > > implementation
> > > > > > > > > > detail,
> > > > > > > > > > > including how
> > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the
> > features
> > > > > > needed
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > >
> > > > > > > > > > > In the prototype, we make flink-table dependency on
> > > > flink-cep.
> > > > > Do
> > > > > > > you
> > > > > > > > > > think
> > > > > > > > > > > that is fine?
> > > > > > > > > > >
> > > > > > > > > > > What do you think about the prototype and the design
> doc
> > ?
> > > > > > > > > > >
> > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > >
> > > > > > > > > > > Cheers,
> > > > > > > > > > > Jark Wu
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > [hidden email]
> > > > > >:
> > > > > > > > > > >
> > > > > > > > > > > > Thanks for sharing your ideas with the community. I
> > > really
> > > > > like
> > > > > > > the
> > > > > > > > > > > design
> > > > > > > > > > > > document and think that it's a good approach to
> follow
> > > > > Oracle's
> > > > > > > SQL
> > > > > > > > > > > > extension for pattern matching. Looking forward to
> > having
> > > > > > support
> > > > > > > > for
> > > > > > > > > > SQL
> > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Till
> > > > > > > > > > > >
> > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > [hidden email]
> > > >
> > > > > > wrote:
> > > > > > > > > > > >
> > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > > > > > >
> > > > > > > > > > > > > @Fabian, I totally agree with you that we should
> > focus
> > > on
> > > > > SQL
> > > > > > > > > first.
> > > > > > > > > > > > Let's
> > > > > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure
> about
> > > > > that. I
> > > > > > > > think
> > > > > > > > > > it
> > > > > > > > > > > > > makes sense to make it required in streaming
> > > mode(either
> > > > > > order
> > > > > > > by
> > > > > > > > > > > rowtime
> > > > > > > > > > > > > or order by proctime). But CEP also works in batch
> > > mode,
> > > > > and
> > > > > > > not
> > > > > > > > > > > > necessary
> > > > > > > > > > > > > to order by some column. Nevertheless, we can
> support
> > > CEP
> > > > > on
> > > > > > > > batch
> > > > > > > > > > SQL
> > > > > > > > > > > > > later.
> > > > > > > > > > > > >
> > > > > > > > > > > > > We are estimating how to implement MATCH_RECOGNIZE
> > with
> > > > CEP
> > > > > > > > library
> > > > > > > > > > > (with
> > > > > > > > > > > > > NFA, CEP operator). And we will output a detailed
> doc
> > > > and a
> > > > > > > > > prototype
> > > > > > > > > > > in
> > > > > > > > > > > > > the next days.
> > > > > > > > > > > > >
> > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > > > > >
> > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have been
> > > thinking
> > > > > > about
> > > > > > > > > this
> > > > > > > > > > > for
> > > > > > > > > > > > >> some time but haven't had time to work on this
> > > feature.
> > > > > > > > > > > > >> I think it would be a great addition and value add
> > for
> > > > > > Flink's
> > > > > > > > SQL
> > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> I read the proposal and think it is very good. We
> > > might
> > > > > need
> > > > > > > to
> > > > > > > > > add
> > > > > > > > > > a
> > > > > > > > > > > > bit
> > > > > > > > > > > > >> more details, esp. when planning the concrete
> steps
> > of
> > > > the
> > > > > > > > > > > > implementation.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > >> - IMO, the development should start focusing on
> SQL
> > > and
> > > > > its
> > > > > > > > > > semantics.
> > > > > > > > > > > > >> Pattern support for the Table API should be added
> > > later.
> > > > > We
> > > > > > > > > followed
> > > > > > > > > > > > that
> > > > > > > > > > > > >> approach for the OVER windows and I think it
> worked
> > > > quiet
> > > > > > > well.
> > > > > > > > > > > > >> - We probably want to reuse as much as possible
> from
> > > the
> > > > > CEP
> > > > > > > > > > library.
> > > > > > > > > > > > >> That means we need to check if the semantics of
> the
> > > CEP
> > > > > > > library
> > > > > > > > > and
> > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we can
> > > > express
> > > > > > the
> > > > > > > > > > PATTERN
> > > > > > > > > > > > >> semantics with the CEP library). This should be
> one
> > of
> > > > the
> > > > > > > first
> > > > > > > > > > > steps,
> > > > > > > > > > > > IMO.
> > > > > > > > > > > > >> - I would make the orderBy() clause required. In
> > > regular
> > > > > SQL
> > > > > > > > rows
> > > > > > > > > > have
> > > > > > > > > > > > no
> > > > > > > > > > > > >> order, so we need to make that explicit (this
> would
> > > also
> > > > > be
> > > > > > > > > > consistent
> > > > > > > > > > > > with
> > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > >>
> > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > > > [hidden email]
> > > > > > > > > > > > >:
> > > > > > > > > > > > >>
> > > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> This is a really interesting idea that has been
> in
> > > our
> > > > > > heads
> > > > > > > > > > > > >>> since the first implementation of the CEP
> library.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> And as for the design document, I will definitely
> > > have
> > > > a
> > > > > > look
> > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > [hidden email]>
> > > > > > > wrote:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo,
> > @Till,
> > > > > > @Kostas
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > [hidden email]
> > > >:
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Dian and me and our teammates have investigated
> > this
> > > > > for a
> > > > > > > > long
> > > > > > > > > > > time.
> > > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP is an
> > > > exciting
> > > > > > > thing
> > > > > > > > > for
> > > > > > > > > > > > Flink.
> > > > > > > > > > > > >>>> It'll make SQL more powerful and give users the
> > > > ability
> > > > > to
> > > > > > > > > easily
> > > > > > > > > > > and
> > > > > > > > > > > > >>>> quickly build CEP applications.  And I find
> Flink
> > > > > > community
> > > > > > > > has
> > > > > > > > > > also
> > > > > > > > > > > > talked
> > > > > > > > > > > > >>>> about this idea before, such as the mailing list
> > [1]
> > > > and
> > > > > > [2]
> > > > > > > > and
> > > > > > > > > > > > Fabian &
> > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this topic
> > > > again.
> > > > > > > > Because
> > > > > > > > > we
> > > > > > > > > > > > >>>> already have pattern matching foundation in
> Flink
> > > CEP
> > > > > > > library,
> > > > > > > > > and
> > > > > > > > > > > > Stream
> > > > > > > > > > > > >>>> SQL is ready now and Calcite has partially
> > supported
> > > > > > pattern
> > > > > > > > > > > matching
> > > > > > > > > > > > >>>> syntax!  We also drafted a design doc about how
> to
> > > > > > integrate
> > > > > > > > SQL
> > > > > > > > > > and
> > > > > > > > > > > > CEP,
> > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > sharing
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you
> into
> > > this
> > > > > > > > > discussion,
> > > > > > > > > > > it
> > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > list-archive.1008284.n3
> > > > > .
> > > > > > > > > nabble.c
> > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > list-archive.1008284.n3
> > > > > .
> > > > > > > > > nabble.c
> > > > > > > > > > > > >>>> om/Effort-to-add-SQL-StreamSQL-to-Flink-td9727.
> > > > > html#a9790
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > tillrohrmann/streaming-
> > > > > > > > > analytics-
> > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > [hidden email]
> > > > >:
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Flink's CEP library is a great library for
> > complex
> > > > > event
> > > > > > > > > > > processing,
> > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > >>>>> and more customers are expressing their
> interests
> > > in
> > > > > it.
> > > > > > > But
> > > > > > > > it
> > > > > > > > > > > also
> > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > >>>>> some limitations that users usually have to
> > write a
> > > > lot
> > > > > > of
> > > > > > > > code
> > > > > > > > > > > even
> > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > >>>>> very simple pattern match use case as it
> > currently
> > > > only
> > > > > > > > > supports
> > > > > > > > > > > the
> > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > >>>>> We have investigated some popular CEP products
> > such
> > > > as
> > > > > > > esper
> > > > > > > > > [1]
> > > > > > > > > > > and
> > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > >>>>> [2] and found that most of these CEP products
> > > support
> > > > > > > > SQL-like
> > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > >>>>> such as EPL to describe the match pattern. But
> > > these
> > > > > > > > solutions
> > > > > > > > > > also
> > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > >>>>> the drawbacks that the pattern match languages
> > are
> > > > not
> > > > > > > > standard
> > > > > > > > > > > SQL,
> > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > >>>>> learn curve is steep for users and it's
> > impossible
> > > to
> > > > > > > > integrate
> > > > > > > > > > > them
> > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3]
> > > supports a
> > > > > new
> > > > > > > > > pattern
> > > > > > > > > > > > >>>>> recognition clause match_recognize which is a
> > > pattern
> > > > > > > > > recognition
> > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a set
> of
> > > new
> > > > > > > syntaxes
> > > > > > > > > to
> > > > > > > > > > > > define
> > > > > > > > > > > > >>>>> match pattern in sql expression. Calcite
> already
> > > > > supports
> > > > > > > > part
> > > > > > > > > of
> > > > > > > > > > > > this
> > > > > > > > > > > > >>>>> standard [5].  I think it will be of great
> value
> > to
> > > > > > support
> > > > > > > > > > > > expressing
> > > > > > > > > > > > >>>>> pattern recognition clause with match_recognize
> > > > clause
> > > > > by
> > > > > > > > > > > integrating
> > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP
> > > library.
> > > > > Any
> > > > > > > > > > thoughts?
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > > > eventprocessing/cql-
> > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > CQLLR1531
> > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > classes/winter17/cs240B/notes/
> > > > > > > row-pat
> > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > >>>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>>
> > > > > > > > > > > > >>
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [DISCUSS]: Integrating Flink Table API & SQL with CEP

Till Rohrmann
I think that the integration of SQL and CEP would make a good FLIP.

Cheers,
Till

On Wed, Jun 14, 2017 at 2:40 PM, Jark Wu <[hidden email]> wrote:

> Hi,
>
> Do you think whether we should create a FLIP for this proposal to track
> progress?
>
> Regards,
> Jark
>
> 2017-06-13 16:59 GMT+08:00 Dian Fu <[hidden email]>:
>
> > Hi Fabian,
> >
> > Thanks a lot. Agree that we can start working by adding the missing
> > features of the CEP library.
> >
> > Best regards,
> > Dian
> >
> > On Tue, Jun 13, 2017 at 4:26 PM, Fabian Hueske <[hidden email]>
> wrote:
> >
> > > Hi,
> > >
> > > @Dian Fu: I gave you contributor permissions. :-)
> > >
> > > I don't think we have to wait for Calcite 1.13 to start working on the
> > > missing features of the CEP library and extending the prototype.
> > > We might want to wait with the integration into flink-table until
> Calcite
> > > 1.13 is out and we updated the dependency though.
> > >
> > > Best, Fabian
> > >
> > > 2017-06-13 9:45 GMT+02:00 jincheng sun <[hidden email]>:
> > >
> > > > Hi Jark, Dian,
> > > >
> > > > Thanks for bring up this discuss and share the prototype.
> > > >
> > > > +1 to push this great feature forward!
> > > >
> > > > Cheers,
> > > > SunJincheng
> > > >
> > > > 2017-06-13 15:34 GMT+08:00 Jark Wu <[hidden email]>:
> > > >
> > > > > Thank you Yueting for pointing out the mistake in the prototype. I
> > > > > accidentally introduced it when merge code.
> > > > >
> > > > > I'm so glad to see so many people are interested in the feature.
> > Let's
> > > > work
> > > > > out together to push it forward!
> > > > >
> > > > > Cheers,
> > > > > Jark Wu
> > > > >
> > > > >
> > > > > 2017-06-13 15:27 GMT+08:00 Liangfei Su <[hidden email]>:
> > > > >
> > > > > > +1 for the feature. Myself was a user of Siddhi, this is pretty
> > user
> > > > > > friendly feature to provide to user.
> > > > > >
> > > > > > On Tue, Jun 13, 2017 at 3:09 PM, Dian Fu <[hidden email]>
> > > > wrote:
> > > > > >
> > > > > > > Hi Yueting & Dawid Wysakowicz,
> > > > > > >
> > > > > > > Very glad that you're interested in this feature and you're
> > > > definitely
> > > > > > > welcome to join this work and also anyone else.:)
> > > > > > >
> > > > > > > Best regards,
> > > > > > > Dian
> > > > > > >
> > > > > > > On Tue, Jun 13, 2017 at 2:35 PM, Dawid Wysakowicz <
> > > > > > > [hidden email]> wrote:
> > > > > > >
> > > > > > > > Hi all,
> > > > > > > >
> > > > > > > > Integrating SQL with CEP seems like a really nice idea.
> > > > > Unfortunately I
> > > > > > > had
> > > > > > > > time just for a brief look at the design doc, but it looks
> > really
> > > > > cool
> > > > > > > and
> > > > > > > > thorough. Also will have a second run tomorrow and will try
> to
> > > > > provide
> > > > > > > more
> > > > > > > > comments. Anyway will be glad to help pushing the initiative
> > > > forward.
> > > > > > > >
> > > > > > > > Z pozdrowieniami! / Cheers!
> > > > > > > >
> > > > > > > > Dawid Wysakowicz
> > > > > > > >
> > > > > > > > *Data/Software Engineer*
> > > > > > > >
> > > > > > > > Skype: dawid_wys | Twitter: @OneMoreCoder
> > > > > > > >
> > > > > > > > <http://getindata.com/>
> > > > > > > >
> > > > > > > > 2017-06-13 8:19 GMT+02:00 yueting chen <[hidden email]
> >:
> > > > > > > >
> > > > > > > > > Hi Dian & Jark,
> > > > > > > > >
> > > > > > > > > I checked out your prototype code, but it didn't pass the
> > > > CEPITCase
> > > > > > > test
> > > > > > > > in
> > > > > > > > > the flink-table component.
> > > > > > > > > It turns out that in the `MatchCodeGenerator.scala` file,
> > line
> > > 74
> > > > > > > should
> > > > > > > > > use `${classOf[IterativeCondition.
> > > Context[_]].getCanonicalName}`
> > > > > > > instead
> > > > > > > > > of
> > > > > > > > > `${classOf[IterativeCondition.Context[_]]}`.
> > > > > > > > >
> > > > > > > > > I've also read your desgin document and it looks fine to
> me.
> > > > > > Actually,
> > > > > > > I
> > > > > > > > am
> > > > > > > > > working on the same thing recently, I think maybe we can
> work
> > > > > > together
> > > > > > > to
> > > > > > > > > push this forward.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Yueting Chen
> > > > > > > > >
> > > > > > > > > On Tue, Jun 13, 2017 at 10:44 AM, Dian Fu <
> [hidden email]
> > >
> > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Fabian,
> > > > > > > > > >
> > > > > > > > > > We have evaluated the missing features of Flink CEP
> > roughly,
> > > it
> > > > > > > should
> > > > > > > > > not
> > > > > > > > > > be quite difficult to support them. Kostas, Dawid, what's
> > > your
> > > > > > > thought?
> > > > > > > > > >
> > > > > > > > > > For supporting MATCH_RECOGNIZE, do you think we could
> > create
> > > > the
> > > > > > > JIRAs
> > > > > > > > > and
> > > > > > > > > > start to work right now or we should wait until the
> release
> > > of
> > > > > > > calcite
> > > > > > > > > > 1.13?
> > > > > > > > > >
> > > > > > > > > > Btw, could you help to add me(dian.fu) to the contributor
> > > list,
> > > > > > then
> > > > > > > I
> > > > > > > > > can
> > > > > > > > > > assign the JIRAs to myself? Thanks a lot.
> > > > > > > > > >
> > > > > > > > > > Best regards,
> > > > > > > > > > Dian
> > > > > > > > > >
> > > > > > > > > > On Tue, Jun 13, 2017 at 3:59 AM, Fabian Hueske <
> > > > > [hidden email]>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > > > Hi Jark,
> > > > > > > > > > >
> > > > > > > > > > > Thanks for updating the design doc and sharing your
> > > > prototype!
> > > > > > > > > > > I didn't look at the code in detail, but the fact that
> it
> > > is
> > > > > less
> > > > > > > > than
> > > > > > > > > 1k
> > > > > > > > > > > LOC is very promising. It seems that most of the
> complex
> > > CEP
> > > > > > logic
> > > > > > > > can
> > > > > > > > > be
> > > > > > > > > > > reused :-)
> > > > > > > > > > > Adding a dependency on flink-cep should be fine, IMO.
> It
> > > is a
> > > > > > very
> > > > > > > > slim
> > > > > > > > > > > library with almost none external dependencies.
> > > > > > > > > > >
> > > > > > > > > > > Regarding the missing features of Flink CEP that you
> > listed
> > > > in
> > > > > > the
> > > > > > > > > design
> > > > > > > > > > > doc, it would be good to get some in put from Kostas
> and
> > > > Dawid
> > > > > > > which
> > > > > > > > > are
> > > > > > > > > > > the main contributors to CEP.
> > > > > > > > > > > Do you have already plans regarding some of the missing
> > > > > features
> > > > > > or
> > > > > > > > can
> > > > > > > > > > you
> > > > > > > > > > > assess how hard it would be to integrate them?
> > > > > > > > > > >
> > > > > > > > > > > Cheers, Fabian
> > > > > > > > > > >
> > > > > > > > > > > Btw. The Calcite community started a discussion about
> > > > releasing
> > > > > > > > Calcite
> > > > > > > > > > > 1.13. So, the missing features might soon be available.
> > > > > > > > > > >
> > > > > > > > > > > 2017-06-12 14:25 GMT+02:00 Jark Wu <[hidden email]>:
> > > > > > > > > > >
> > > > > > > > > > > > Hi guys,
> > > > > > > > > > > >
> > > > > > > > > > > > Good news! We have made a prototype for integrating
> CEP
> > > and
> > > > > > SQL.
> > > > > > > > See
> > > > > > > > > > this
> > > > > > > > > > > > link
> > > > > > > > > > > > https://github.com/apache/flink/compare/master...
> > > > > > > > > > > > wuchong:cep-on-sql?expand=1
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > You can check CepITCase to try the simple CQL
> example.
> > > > > > > > > > > >
> > > > > > > > > > > > Meanwhile, we updated our design doc with additional
> > > > > > > implementation
> > > > > > > > > > > detail,
> > > > > > > > > > > > including how
> > > > > > > > > > > > to translate MATCH_RECOGNIZE into CEP API, and the
> > > features
> > > > > > > needed
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > > to Flink CEP,
> > > > > > > > > > > > and the implementation plan. See the document
> > > > > > > > > > > > https://docs.google.com/document/d/
> > > > > > > 1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > > > > > > K15H0YbniTnN30/edit#heading=h.4oas4koy8qu3
> > > > > > > > > > > >
> > > > > > > > > > > > In the prototype, we make flink-table dependency on
> > > > > flink-cep.
> > > > > > Do
> > > > > > > > you
> > > > > > > > > > > think
> > > > > > > > > > > > that is fine?
> > > > > > > > > > > >
> > > > > > > > > > > > What do you think about the prototype and the design
> > doc
> > > ?
> > > > > > > > > > > >
> > > > > > > > > > > > Any feedbacks are welcome!
> > > > > > > > > > > >
> > > > > > > > > > > > Cheers,
> > > > > > > > > > > > Jark Wu
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > 2017-06-08 17:54 GMT+08:00 Till Rohrmann <
> > > > > [hidden email]
> > > > > > >:
> > > > > > > > > > > >
> > > > > > > > > > > > > Thanks for sharing your ideas with the community. I
> > > > really
> > > > > > like
> > > > > > > > the
> > > > > > > > > > > > design
> > > > > > > > > > > > > document and think that it's a good approach to
> > follow
> > > > > > Oracle's
> > > > > > > > SQL
> > > > > > > > > > > > > extension for pattern matching. Looking forward to
> > > having
> > > > > > > support
> > > > > > > > > for
> > > > > > > > > > > SQL
> > > > > > > > > > > > > with CEP capabilities :-)
> > > > > > > > > > > > >
> > > > > > > > > > > > > Cheers,
> > > > > > > > > > > > > Till
> > > > > > > > > > > > >
> > > > > > > > > > > > > On Thu, Jun 8, 2017 at 8:57 AM, Jark Wu <
> > > [hidden email]
> > > > >
> > > > > > > wrote:
> > > > > > > > > > > > >
> > > > > > > > > > > > > > Hi  @Kostas, @Fabian, thank you for your support.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > @Fabian, I totally agree with you that we should
> > > focus
> > > > on
> > > > > > SQL
> > > > > > > > > > first.
> > > > > > > > > > > > > Let's
> > > > > > > > > > > > > > keep Table API in mind and discuss that later.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regarding to the orderBy() clause, I'm not sure
> > about
> > > > > > that. I
> > > > > > > > > think
> > > > > > > > > > > it
> > > > > > > > > > > > > > makes sense to make it required in streaming
> > > > mode(either
> > > > > > > order
> > > > > > > > by
> > > > > > > > > > > > rowtime
> > > > > > > > > > > > > > or order by proctime). But CEP also works in
> batch
> > > > mode,
> > > > > > and
> > > > > > > > not
> > > > > > > > > > > > > necessary
> > > > > > > > > > > > > > to order by some column. Nevertheless, we can
> > support
> > > > CEP
> > > > > > on
> > > > > > > > > batch
> > > > > > > > > > > SQL
> > > > > > > > > > > > > > later.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > We are estimating how to implement
> MATCH_RECOGNIZE
> > > with
> > > > > CEP
> > > > > > > > > library
> > > > > > > > > > > > (with
> > > > > > > > > > > > > > NFA, CEP operator). And we will output a detailed
> > doc
> > > > > and a
> > > > > > > > > > prototype
> > > > > > > > > > > > in
> > > > > > > > > > > > > > the next days.
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > Regards,
> > > > > > > > > > > > > > Jark Wu
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >
> > > > > > > > > > > > > > 2017-06-07 21:40 GMT+08:00 Fabian Hueske <
> > > > > > [hidden email]
> > > > > > > >:
> > > > > > > > > > > > > >
> > > > > > > > > > > > > >> Thanks Dian and Jark for this proposal!
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> As you wrote, Till and I (and Kostas) have been
> > > > thinking
> > > > > > > about
> > > > > > > > > > this
> > > > > > > > > > > > for
> > > > > > > > > > > > > >> some time but haven't had time to work on this
> > > > feature.
> > > > > > > > > > > > > >> I think it would be a great addition and value
> add
> > > for
> > > > > > > Flink's
> > > > > > > > > SQL
> > > > > > > > > > > > > >> support and Table API.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> I read the proposal and think it is very good.
> We
> > > > might
> > > > > > need
> > > > > > > > to
> > > > > > > > > > add
> > > > > > > > > > > a
> > > > > > > > > > > > > bit
> > > > > > > > > > > > > >> more details, esp. when planning the concrete
> > steps
> > > of
> > > > > the
> > > > > > > > > > > > > implementation.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> A few comments to the proposal:
> > > > > > > > > > > > > >> - IMO, the development should start focusing on
> > SQL
> > > > and
> > > > > > its
> > > > > > > > > > > semantics.
> > > > > > > > > > > > > >> Pattern support for the Table API should be
> added
> > > > later.
> > > > > > We
> > > > > > > > > > followed
> > > > > > > > > > > > > that
> > > > > > > > > > > > > >> approach for the OVER windows and I think it
> > worked
> > > > > quiet
> > > > > > > > well.
> > > > > > > > > > > > > >> - We probably want to reuse as much as possible
> > from
> > > > the
> > > > > > CEP
> > > > > > > > > > > library.
> > > > > > > > > > > > > >> That means we need to check if the semantics of
> > the
> > > > CEP
> > > > > > > > library
> > > > > > > > > > and
> > > > > > > > > > > > > >> Oracle's PATTERN syntax are aligned (or how we
> can
> > > > > express
> > > > > > > the
> > > > > > > > > > > PATTERN
> > > > > > > > > > > > > >> semantics with the CEP library). This should be
> > one
> > > of
> > > > > the
> > > > > > > > first
> > > > > > > > > > > > steps,
> > > > > > > > > > > > > IMO.
> > > > > > > > > > > > > >> - I would make the orderBy() clause required. In
> > > > regular
> > > > > > SQL
> > > > > > > > > rows
> > > > > > > > > > > have
> > > > > > > > > > > > > no
> > > > > > > > > > > > > >> order, so we need to make that explicit (this
> > would
> > > > also
> > > > > > be
> > > > > > > > > > > consistent
> > > > > > > > > > > > > with
> > > > > > > > > > > > > >> the OVER windows).
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Let me know what you think.
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> Best, Fabian
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >> 2017-06-07 11:41 GMT+02:00 Kostas Kloudas <
> > > > > > > > > > > > [hidden email]
> > > > > > > > > > > > > >:
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >>> Thanks a lot for opening the discussion!
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> This is a really interesting idea that has been
> > in
> > > > our
> > > > > > > heads
> > > > > > > > > > > > > >>> since the first implementation of the CEP
> > library.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> A big +1 for moving forward with this.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> And as for the design document, I will
> definitely
> > > > have
> > > > > a
> > > > > > > look
> > > > > > > > > > > > > >>> and comment there.
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Kostas
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> On Jun 7, 2017, at 10:05 AM, Jark Wu <
> > > > [hidden email]>
> > > > > > > > wrote:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> Sorry, I forgot to cc you guys @Fabian, @Timo,
> > > @Till,
> > > > > > > @Kostas
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>> 2017-06-07 15:42 GMT+08:00 Jark Wu <
> > > [hidden email]
> > > > >:
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>> Hi devs,
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Dian and me and our teammates have
> investigated
> > > this
> > > > > > for a
> > > > > > > > > long
> > > > > > > > > > > > time.
> > > > > > > > > > > > > >>>> We think consolidating Flink SQL and CEP is an
> > > > > exciting
> > > > > > > > thing
> > > > > > > > > > for
> > > > > > > > > > > > > Flink.
> > > > > > > > > > > > > >>>> It'll make SQL more powerful and give users
> the
> > > > > ability
> > > > > > to
> > > > > > > > > > easily
> > > > > > > > > > > > and
> > > > > > > > > > > > > >>>> quickly build CEP applications.  And I find
> > Flink
> > > > > > > community
> > > > > > > > > has
> > > > > > > > > > > also
> > > > > > > > > > > > > talked
> > > > > > > > > > > > > >>>> about this idea before, such as the mailing
> list
> > > [1]
> > > > > and
> > > > > > > [2]
> > > > > > > > > and
> > > > > > > > > > > > > Fabian &
> > > > > > > > > > > > > >>>> Till's talk in Flink Forward 2016 [3].
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> I think THIS IS THE POINT to bring up this
> topic
> > > > > again.
> > > > > > > > > Because
> > > > > > > > > > we
> > > > > > > > > > > > > >>>> already have pattern matching foundation in
> > Flink
> > > > CEP
> > > > > > > > library,
> > > > > > > > > > and
> > > > > > > > > > > > > Stream
> > > > > > > > > > > > > >>>> SQL is ready now and Calcite has partially
> > > supported
> > > > > > > pattern
> > > > > > > > > > > > matching
> > > > > > > > > > > > > >>>> syntax!  We also drafted a design doc about
> how
> > to
> > > > > > > integrate
> > > > > > > > > SQL
> > > > > > > > > > > and
> > > > > > > > > > > > > CEP,
> > > > > > > > > > > > > >>>> and how to support CEP on Table API.
> > > > > > > > > > > https://docs.google.com/docume
> > > > > > > > > > > > > >>>> nt/d/1HaaO5eYI1VZjyhtVPZOi3jVzikU7i
> > > > > > > K15H0YbniTnN30/edit?usp=
> > > > > > > > > > > sharing
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> @Fabian, @Timo, @Till, @Kostas I include you
> > into
> > > > this
> > > > > > > > > > discussion,
> > > > > > > > > > > > it
> > > > > > > > > > > > > >>>> would be great to hear your response.
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> What do others think?
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> [1] http://apache-flink-mailing-
> > > > > list-archive.1008284.n3
> > > > > > .
> > > > > > > > > > nabble.c
> > > > > > > > > > > > > >>>> om/Add-CEP-library-to-Flink-td9743.html#a9787
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> [2] http://apache-flink-mailing-
> > > > > list-archive.1008284.n3
> > > > > > .
> > > > > > > > > > nabble.c
> > > > > > > > > > > > > >>>> om/Effort-to-add-SQL-
> StreamSQL-to-Flink-td9727.
> > > > > > html#a9790
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> [3] https://www.slideshare.net/
> > > > > tillrohrmann/streaming-
> > > > > > > > > > analytics-
> > > > > > > > > > > > > >>>> cep-two-sides-of-the-same-coin
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Regards,
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> Jark Wu
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>> 2017-06-07 13:50 GMT+08:00 Dian Fu <
> > > > [hidden email]
> > > > > >:
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>> Hi everyone,
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Flink's CEP library is a great library for
> > > complex
> > > > > > event
> > > > > > > > > > > > processing,
> > > > > > > > > > > > > >>>>> more
> > > > > > > > > > > > > >>>>> and more customers are expressing their
> > interests
> > > > in
> > > > > > it.
> > > > > > > > But
> > > > > > > > > it
> > > > > > > > > > > > also
> > > > > > > > > > > > > >>>>> has
> > > > > > > > > > > > > >>>>> some limitations that users usually have to
> > > write a
> > > > > lot
> > > > > > > of
> > > > > > > > > code
> > > > > > > > > > > > even
> > > > > > > > > > > > > >>>>> for a
> > > > > > > > > > > > > >>>>> very simple pattern match use case as it
> > > currently
> > > > > only
> > > > > > > > > > supports
> > > > > > > > > > > > the
> > > > > > > > > > > > > >>>>> Java
> > > > > > > > > > > > > >>>>> API.
> > > > > > > > > > > > > >>>>> We have investigated some popular CEP
> products
> > > such
> > > > > as
> > > > > > > > esper
> > > > > > > > > > [1]
> > > > > > > > > > > > and
> > > > > > > > > > > > > >>>>> siddhi
> > > > > > > > > > > > > >>>>> [2] and found that most of these CEP products
> > > > support
> > > > > > > > > SQL-like
> > > > > > > > > > > > > >>>>> expressions
> > > > > > > > > > > > > >>>>> such as EPL to describe the match pattern.
> But
> > > > these
> > > > > > > > > solutions
> > > > > > > > > > > also
> > > > > > > > > > > > > >>>>> have
> > > > > > > > > > > > > >>>>> the drawbacks that the pattern match
> languages
> > > are
> > > > > not
> > > > > > > > > standard
> > > > > > > > > > > > SQL,
> > > > > > > > > > > > > >>>>> the
> > > > > > > > > > > > > >>>>> learn curve is steep for users and it's
> > > impossible
> > > > to
> > > > > > > > > integrate
> > > > > > > > > > > > them
> > > > > > > > > > > > > >>>>> into
> > > > > > > > > > > > > >>>>> the Flink Table API & SQL.
> > > > > > > > > > > > > >>>>> We find that Oracle's CEP solution CQL [3]
> > > > supports a
> > > > > > new
> > > > > > > > > > pattern
> > > > > > > > > > > > > >>>>> recognition clause match_recognize which is a
> > > > pattern
> > > > > > > > > > recognition
> > > > > > > > > > > > > >>>>> clause
> > > > > > > > > > > > > >>>>> proposed in this paper [4]. It proposes a set
> > of
> > > > new
> > > > > > > > syntaxes
> > > > > > > > > > to
> > > > > > > > > > > > > define
> > > > > > > > > > > > > >>>>> match pattern in sql expression. Calcite
> > already
> > > > > > supports
> > > > > > > > > part
> > > > > > > > > > of
> > > > > > > > > > > > > this
> > > > > > > > > > > > > >>>>> standard [5].  I think it will be of great
> > value
> > > to
> > > > > > > support
> > > > > > > > > > > > > expressing
> > > > > > > > > > > > > >>>>> pattern recognition clause with
> match_recognize
> > > > > clause
> > > > > > by
> > > > > > > > > > > > integrating
> > > > > > > > > > > > > >>>>> it
> > > > > > > > > > > > > >>>>> with Flink Table API & SQL and the Flink CEP
> > > > library.
> > > > > > Any
> > > > > > > > > > > thoughts?
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> [1] http://www.espertech.com
> > > > > > > > > > > > > >>>>> [2] https://github.com/wso2/siddhi
> > > > > > > > > > > > > >>>>> [3]
> > > > > > > > > > > > > >>>>> https://docs.oracle.com/middleware/1213/
> > > > > > > > eventprocessing/cql-
> > > > > > > > > > > > > >>>>> reference/GUID-34D4968E-C55A-
> > > > > > 4BC7-B1CE-C84B202217BD.htm#
> > > > > > > > > > > CQLLR1531
> > > > > > > > > > > > > >>>>> [4]
> > > > > > > > > > > > > >>>>> http://web.cs.ucla.edu/
> > > > > classes/winter17/cs240B/notes/
> > > > > > > > row-pat
> > > > > > > > > > > > > >>>>> tern-recogniton-11.pdf
> > > > > > > > > > > > > >>>>> [5] https://issues.apache.org/
> > > > > jira/browse/CALCITE-1570
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>> Best Regards,
> > > > > > > > > > > > > >>>>> Dian
> > > > > > > > > > > > > >>>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>>
> > > > > > > > > > > > > >>
> > > > > > > > > > > > > >
> > > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
12
Loading...