[DISCUSS] FLIP-32: Restructure flink-table for future contributions

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

[DISCUSS] FLIP-32: Restructure flink-table for future contributions

Timo Walther-2
Hi everyone,

as Stephan already announced on the mailing list [1], the Flink
community will receive a big code contribution from Alibaba. The
flink-table module is one of the biggest parts that will receive many
new features and major architectural improvements. Instead of waiting
until the next major version of Flink or introducing big API-breaking
changes, we would like to gradually build up the Blink-based planner and
runtime while keeping the Table & SQL API mostly stable. Users will be
able to play around with the current merge status of the new planner or
fall back to the old planner until the new one is stable.

We have prepared a design document that discusses a restructuring of the
flink-table module and suggests a rough implementation plan:

https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing

I will briefly summarize the steps we would like to do:

- Split the flink-table module similar to the proposal of FLIP-28 [3]
which is outdated. This is a preparation to separate API from core
(targeted for Flink 1.8).
- Perform minor API changes to separate API from actual implementation
(targeted for Flink 1.8).
- Merge a MVP Blink SQL planner given that necessary Flink core/runtime
changes have been completed.
   The merging will happen in stages (e.g. basic planner framework, then
operator by operator). The exact merging plan still needs to be determined.
- Rework the type system in order to unblock work on unified table
environments, UDFs, sources/sinks, and catalog.
- Enable full end-to-end batch and stream execution features.

Our mid-term goal:

Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
only support ingesting data coming from the DataStream API. Once we
reworked the sources/sink interfaces, we will target full end-to-end
TPC-DS query execution with table connectors.

A rough task dependency graph is illustrated in the design document. A
more detailed task dependency structure will be added to JIRA after we
agreed on this FLIP.

Looking forward to any feedback.

Thanks,
Timo

[1]
https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
[2]
https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
[3]
https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

jincheng sun
Hi Timo,

Thanks a lot for bring up the FLIP-32 discussion and the very detailed
implementation plan document !

Restructure `flink-table` is an important part of flink merge blink,
looking forward to the JIRAs which will be opened !

Cheers,
Jincheng


Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:

> Hi everyone,
>
> as Stephan already announced on the mailing list [1], the Flink
> community will receive a big code contribution from Alibaba. The
> flink-table module is one of the biggest parts that will receive many
> new features and major architectural improvements. Instead of waiting
> until the next major version of Flink or introducing big API-breaking
> changes, we would like to gradually build up the Blink-based planner and
> runtime while keeping the Table & SQL API mostly stable. Users will be
> able to play around with the current merge status of the new planner or
> fall back to the old planner until the new one is stable.
>
> We have prepared a design document that discusses a restructuring of the
> flink-table module and suggests a rough implementation plan:
>
>
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>
> I will briefly summarize the steps we would like to do:
>
> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> which is outdated. This is a preparation to separate API from core
> (targeted for Flink 1.8).
> - Perform minor API changes to separate API from actual implementation
> (targeted for Flink 1.8).
> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
> changes have been completed.
>    The merging will happen in stages (e.g. basic planner framework, then
> operator by operator). The exact merging plan still needs to be determined.
> - Rework the type system in order to unblock work on unified table
> environments, UDFs, sources/sinks, and catalog.
> - Enable full end-to-end batch and stream execution features.
>
> Our mid-term goal:
>
> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
> only support ingesting data coming from the DataStream API. Once we
> reworked the sources/sink interfaces, we will target full end-to-end
> TPC-DS query execution with table connectors.
>
> A rough task dependency graph is illustrated in the design document. A
> more detailed task dependency structure will be added to JIRA after we
> agreed on this FLIP.
>
> Looking forward to any feedback.
>
> Thanks,
> Timo
>
> [1]
>
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> [2]
>
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> [3]
>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

Piotr Nowojski-2
Hi,

+1 from my side for this plan. The proposed decoupling TypeSystem and most interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of merging MVP Blink's runtime gives us best shot of handling Blink merging as fluently and as painless as possible. I’m also looking forward to the follow up discussions that will need to take place in order to achieve the final goal :)

Piotrek

> On 25 Jan 2019, at 07:53, jincheng sun <[hidden email]> wrote:
>
> Hi Timo,
>
> Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> implementation plan document !
>
> Restructure `flink-table` is an important part of flink merge blink,
> looking forward to the JIRAs which will be opened !
>
> Cheers,
> Jincheng
>
>
> Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:
>
>> Hi everyone,
>>
>> as Stephan already announced on the mailing list [1], the Flink
>> community will receive a big code contribution from Alibaba. The
>> flink-table module is one of the biggest parts that will receive many
>> new features and major architectural improvements. Instead of waiting
>> until the next major version of Flink or introducing big API-breaking
>> changes, we would like to gradually build up the Blink-based planner and
>> runtime while keeping the Table & SQL API mostly stable. Users will be
>> able to play around with the current merge status of the new planner or
>> fall back to the old planner until the new one is stable.
>>
>> We have prepared a design document that discusses a restructuring of the
>> flink-table module and suggests a rough implementation plan:
>>
>>
>> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>>
>> I will briefly summarize the steps we would like to do:
>>
>> - Split the flink-table module similar to the proposal of FLIP-28 [3]
>> which is outdated. This is a preparation to separate API from core
>> (targeted for Flink 1.8).
>> - Perform minor API changes to separate API from actual implementation
>> (targeted for Flink 1.8).
>> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
>> changes have been completed.
>>   The merging will happen in stages (e.g. basic planner framework, then
>> operator by operator). The exact merging plan still needs to be determined.
>> - Rework the type system in order to unblock work on unified table
>> environments, UDFs, sources/sinks, and catalog.
>> - Enable full end-to-end batch and stream execution features.
>>
>> Our mid-term goal:
>>
>> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
>> only support ingesting data coming from the DataStream API. Once we
>> reworked the sources/sink interfaces, we will target full end-to-end
>> TPC-DS query execution with table connectors.
>>
>> A rough task dependency graph is illustrated in the design document. A
>> more detailed task dependency structure will be added to JIRA after we
>> agreed on this FLIP.
>>
>> Looking forward to any feedback.
>>
>> Thanks,
>> Timo
>>
>> [1]
>>
>> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
>> [2]
>>
>> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
>> [3]
>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>>
>>

Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

Hequn Cheng
Hi Timo,

+1 for the FLIP!

Great work and thanks a lot for the detailed document! The task dependency
graph is very helpful for teasing out relationships between tasks.
Looking forward to the JIRAs and hoping to contribute to it!

Best,
Hequn


On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski <[hidden email]>
wrote:

> Hi,
>
> +1 from my side for this plan. The proposed decoupling TypeSystem and most
> interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
> merging MVP Blink's runtime gives us best shot of handling Blink merging as
> fluently and as painless as possible. I’m also looking forward to the
> follow up discussions that will need to take place in order to achieve the
> final goal :)
>
> Piotrek
>
> > On 25 Jan 2019, at 07:53, jincheng sun <[hidden email]> wrote:
> >
> > Hi Timo,
> >
> > Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> > implementation plan document !
> >
> > Restructure `flink-table` is an important part of flink merge blink,
> > looking forward to the JIRAs which will be opened !
> >
> > Cheers,
> > Jincheng
> >
> >
> > Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:
> >
> >> Hi everyone,
> >>
> >> as Stephan already announced on the mailing list [1], the Flink
> >> community will receive a big code contribution from Alibaba. The
> >> flink-table module is one of the biggest parts that will receive many
> >> new features and major architectural improvements. Instead of waiting
> >> until the next major version of Flink or introducing big API-breaking
> >> changes, we would like to gradually build up the Blink-based planner and
> >> runtime while keeping the Table & SQL API mostly stable. Users will be
> >> able to play around with the current merge status of the new planner or
> >> fall back to the old planner until the new one is stable.
> >>
> >> We have prepared a design document that discusses a restructuring of the
> >> flink-table module and suggests a rough implementation plan:
> >>
> >>
> >>
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
> >>
> >> I will briefly summarize the steps we would like to do:
> >>
> >> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> >> which is outdated. This is a preparation to separate API from core
> >> (targeted for Flink 1.8).
> >> - Perform minor API changes to separate API from actual implementation
> >> (targeted for Flink 1.8).
> >> - Merge a MVP Blink SQL planner given that necessary Flink core/runtime
> >> changes have been completed.
> >>   The merging will happen in stages (e.g. basic planner framework, then
> >> operator by operator). The exact merging plan still needs to be
> determined.
> >> - Rework the type system in order to unblock work on unified table
> >> environments, UDFs, sources/sinks, and catalog.
> >> - Enable full end-to-end batch and stream execution features.
> >>
> >> Our mid-term goal:
> >>
> >> Run full TPC-DS on a unified batch/streaming runtime. Initially, we will
> >> only support ingesting data coming from the DataStream API. Once we
> >> reworked the sources/sink interfaces, we will target full end-to-end
> >> TPC-DS query execution with table connectors.
> >>
> >> A rough task dependency graph is illustrated in the design document. A
> >> more detailed task dependency structure will be added to JIRA after we
> >> agreed on this FLIP.
> >>
> >> Looking forward to any feedback.
> >>
> >> Thanks,
> >> Timo
> >>
> >> [1]
> >>
> >>
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> >> [2]
> >>
> >>
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> >> [3]
> >>
> >>
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
> >>
> >>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

Jark
Hi Timo,

Thanks for the detailed design. +1 for the FLIP.

It's very nice to have Blink planner and runtime as a plugin in the early
stages. This will keep flink-table stable as much as possible.

Best,
Jark


On Fri, 25 Jan 2019 at 22:26, Hequn Cheng <[hidden email]> wrote:

> Hi Timo,
>
> +1 for the FLIP!
>
> Great work and thanks a lot for the detailed document! The task dependency
> graph is very helpful for teasing out relationships between tasks.
> Looking forward to the JIRAs and hoping to contribute to it!
>
> Best,
> Hequn
>
>
> On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski <[hidden email]>
> wrote:
>
> > Hi,
> >
> > +1 from my side for this plan. The proposed decoupling TypeSystem and
> most
> > interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
> > merging MVP Blink's runtime gives us best shot of handling Blink merging
> as
> > fluently and as painless as possible. I’m also looking forward to the
> > follow up discussions that will need to take place in order to achieve
> the
> > final goal :)
> >
> > Piotrek
> >
> > > On 25 Jan 2019, at 07:53, jincheng sun <[hidden email]>
> wrote:
> > >
> > > Hi Timo,
> > >
> > > Thanks a lot for bring up the FLIP-32 discussion and the very detailed
> > > implementation plan document !
> > >
> > > Restructure `flink-table` is an important part of flink merge blink,
> > > looking forward to the JIRAs which will be opened !
> > >
> > > Cheers,
> > > Jincheng
> > >
> > >
> > > Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:
> > >
> > >> Hi everyone,
> > >>
> > >> as Stephan already announced on the mailing list [1], the Flink
> > >> community will receive a big code contribution from Alibaba. The
> > >> flink-table module is one of the biggest parts that will receive many
> > >> new features and major architectural improvements. Instead of waiting
> > >> until the next major version of Flink or introducing big API-breaking
> > >> changes, we would like to gradually build up the Blink-based planner
> and
> > >> runtime while keeping the Table & SQL API mostly stable. Users will be
> > >> able to play around with the current merge status of the new planner
> or
> > >> fall back to the old planner until the new one is stable.
> > >>
> > >> We have prepared a design document that discusses a restructuring of
> the
> > >> flink-table module and suggests a rough implementation plan:
> > >>
> > >>
> > >>
> >
> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
> > >>
> > >> I will briefly summarize the steps we would like to do:
> > >>
> > >> - Split the flink-table module similar to the proposal of FLIP-28 [3]
> > >> which is outdated. This is a preparation to separate API from core
> > >> (targeted for Flink 1.8).
> > >> - Perform minor API changes to separate API from actual implementation
> > >> (targeted for Flink 1.8).
> > >> - Merge a MVP Blink SQL planner given that necessary Flink
> core/runtime
> > >> changes have been completed.
> > >>   The merging will happen in stages (e.g. basic planner framework,
> then
> > >> operator by operator). The exact merging plan still needs to be
> > determined.
> > >> - Rework the type system in order to unblock work on unified table
> > >> environments, UDFs, sources/sinks, and catalog.
> > >> - Enable full end-to-end batch and stream execution features.
> > >>
> > >> Our mid-term goal:
> > >>
> > >> Run full TPC-DS on a unified batch/streaming runtime. Initially, we
> will
> > >> only support ingesting data coming from the DataStream API. Once we
> > >> reworked the sources/sink interfaces, we will target full end-to-end
> > >> TPC-DS query execution with table connectors.
> > >>
> > >> A rough task dependency graph is illustrated in the design document. A
> > >> more detailed task dependency structure will be added to JIRA after we
> > >> agreed on this FLIP.
> > >>
> > >> Looking forward to any feedback.
> > >>
> > >> Thanks,
> > >> Timo
> > >>
> > >> [1]
> > >>
> > >>
> >
> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
> > >> [2]
> > >>
> > >>
> >
> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
> > >> [3]
> > >>
> > >>
> >
> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
> > >>
> > >>
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

fudian.fd
Hi Timo,

Thanks a lot for bringing up the discussion and the detailed implementation plan. The implementation plan makes much sense to me. +1 to the FLIP.

Regards,
Dian


> 在 2019年1月25日,下午11:00,Jark Wu <[hidden email]> 写道:
>
> Hi Timo,
>
> Thanks for the detailed design. +1 for the FLIP.
>
> It's very nice to have Blink planner and runtime as a plugin in the early
> stages. This will keep flink-table stable as much as possible.
>
> Best,
> Jark
>
>
> On Fri, 25 Jan 2019 at 22:26, Hequn Cheng <[hidden email]> wrote:
>
>> Hi Timo,
>>
>> +1 for the FLIP!
>>
>> Great work and thanks a lot for the detailed document! The task dependency
>> graph is very helpful for teasing out relationships between tasks.
>> Looking forward to the JIRAs and hoping to contribute to it!
>>
>> Best,
>> Hequn
>>
>>
>> On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski <[hidden email]>
>> wrote:
>>
>>> Hi,
>>>
>>> +1 from my side for this plan. The proposed decoupling TypeSystem and
>> most
>>> interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
>>> merging MVP Blink's runtime gives us best shot of handling Blink merging
>> as
>>> fluently and as painless as possible. I’m also looking forward to the
>>> follow up discussions that will need to take place in order to achieve
>> the
>>> final goal :)
>>>
>>> Piotrek
>>>
>>>> On 25 Jan 2019, at 07:53, jincheng sun <[hidden email]>
>> wrote:
>>>>
>>>> Hi Timo,
>>>>
>>>> Thanks a lot for bring up the FLIP-32 discussion and the very detailed
>>>> implementation plan document !
>>>>
>>>> Restructure `flink-table` is an important part of flink merge blink,
>>>> looking forward to the JIRAs which will be opened !
>>>>
>>>> Cheers,
>>>> Jincheng
>>>>
>>>>
>>>> Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> as Stephan already announced on the mailing list [1], the Flink
>>>>> community will receive a big code contribution from Alibaba. The
>>>>> flink-table module is one of the biggest parts that will receive many
>>>>> new features and major architectural improvements. Instead of waiting
>>>>> until the next major version of Flink or introducing big API-breaking
>>>>> changes, we would like to gradually build up the Blink-based planner
>> and
>>>>> runtime while keeping the Table & SQL API mostly stable. Users will be
>>>>> able to play around with the current merge status of the new planner
>> or
>>>>> fall back to the old planner until the new one is stable.
>>>>>
>>>>> We have prepared a design document that discusses a restructuring of
>> the
>>>>> flink-table module and suggests a rough implementation plan:
>>>>>
>>>>>
>>>>>
>>>
>> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>>>>>
>>>>> I will briefly summarize the steps we would like to do:
>>>>>
>>>>> - Split the flink-table module similar to the proposal of FLIP-28 [3]
>>>>> which is outdated. This is a preparation to separate API from core
>>>>> (targeted for Flink 1.8).
>>>>> - Perform minor API changes to separate API from actual implementation
>>>>> (targeted for Flink 1.8).
>>>>> - Merge a MVP Blink SQL planner given that necessary Flink
>> core/runtime
>>>>> changes have been completed.
>>>>> The merging will happen in stages (e.g. basic planner framework,
>> then
>>>>> operator by operator). The exact merging plan still needs to be
>>> determined.
>>>>> - Rework the type system in order to unblock work on unified table
>>>>> environments, UDFs, sources/sinks, and catalog.
>>>>> - Enable full end-to-end batch and stream execution features.
>>>>>
>>>>> Our mid-term goal:
>>>>>
>>>>> Run full TPC-DS on a unified batch/streaming runtime. Initially, we
>> will
>>>>> only support ingesting data coming from the DataStream API. Once we
>>>>> reworked the sources/sink interfaces, we will target full end-to-end
>>>>> TPC-DS query execution with table connectors.
>>>>>
>>>>> A rough task dependency graph is illustrated in the design document. A
>>>>> more detailed task dependency structure will be added to JIRA after we
>>>>> agreed on this FLIP.
>>>>>
>>>>> Looking forward to any feedback.
>>>>>
>>>>> Thanks,
>>>>> Timo
>>>>>
>>>>> [1]
>>>>>
>>>>>
>>>
>> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
>>>>> [2]
>>>>>
>>>>>
>>>
>> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
>>>>> [3]
>>>>>
>>>>>
>>>
>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>>>>>
>>>>>
>>>
>>>
>>


smime.p7s (3K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [DISCUSS] FLIP-32: Restructure flink-table for future contributions

Timo Walther-2
Hi everyone,

thanks for the positive feedback we received so far. I converted the
design document into a actual FLIP in the wiki. You can find it here:

https://cwiki.apache.org/confluence/display/FLINK/FLIP-32%3A+Restructure+flink-table+for+future+contributions

I'm in the process of creating the first JIRA issues to kick off the
preparation efforts. Most of the issues will be first or second level
children of FLINK-11439:

https://issues.apache.org/jira/browse/FLINK-11439

Feel free to further join the discussions and provide feedback. I'm sure
there will be a lot of stuff to discuss :-)

Thanks,
Timo


Am 25.01.19 um 16:43 schrieb fudian.fd:

> Hi Timo,
>
> Thanks a lot for bringing up the discussion and the detailed implementation plan. The implementation plan makes much sense to me. +1 to the FLIP.
>
> Regards,
> Dian
>
>
>> 在 2019年1月25日,下午11:00,Jark Wu <[hidden email]> 写道:
>>
>> Hi Timo,
>>
>> Thanks for the detailed design. +1 for the FLIP.
>>
>> It's very nice to have Blink planner and runtime as a plugin in the early
>> stages. This will keep flink-table stable as much as possible.
>>
>> Best,
>> Jark
>>
>>
>> On Fri, 25 Jan 2019 at 22:26, Hequn Cheng <[hidden email]> wrote:
>>
>>> Hi Timo,
>>>
>>> +1 for the FLIP!
>>>
>>> Great work and thanks a lot for the detailed document! The task dependency
>>> graph is very helpful for teasing out relationships between tasks.
>>> Looking forward to the JIRAs and hoping to contribute to it!
>>>
>>> Best,
>>> Hequn
>>>
>>>
>>> On Fri, Jan 25, 2019 at 7:54 PM Piotr Nowojski <[hidden email]>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> +1 from my side for this plan. The proposed decoupling TypeSystem and
>>> most
>>>> interface (TableSource/Sinks, UDFs, Catalogs, …) discussions from the of
>>>> merging MVP Blink's runtime gives us best shot of handling Blink merging
>>> as
>>>> fluently and as painless as possible. I’m also looking forward to the
>>>> follow up discussions that will need to take place in order to achieve
>>> the
>>>> final goal :)
>>>>
>>>> Piotrek
>>>>
>>>>> On 25 Jan 2019, at 07:53, jincheng sun <[hidden email]>
>>> wrote:
>>>>> Hi Timo,
>>>>>
>>>>> Thanks a lot for bring up the FLIP-32 discussion and the very detailed
>>>>> implementation plan document !
>>>>>
>>>>> Restructure `flink-table` is an important part of flink merge blink,
>>>>> looking forward to the JIRAs which will be opened !
>>>>>
>>>>> Cheers,
>>>>> Jincheng
>>>>>
>>>>>
>>>>> Timo Walther <[hidden email]> 于2019年1月24日周四 下午9:06写道:
>>>>>
>>>>>> Hi everyone,
>>>>>>
>>>>>> as Stephan already announced on the mailing list [1], the Flink
>>>>>> community will receive a big code contribution from Alibaba. The
>>>>>> flink-table module is one of the biggest parts that will receive many
>>>>>> new features and major architectural improvements. Instead of waiting
>>>>>> until the next major version of Flink or introducing big API-breaking
>>>>>> changes, we would like to gradually build up the Blink-based planner
>>> and
>>>>>> runtime while keeping the Table & SQL API mostly stable. Users will be
>>>>>> able to play around with the current merge status of the new planner
>>> or
>>>>>> fall back to the old planner until the new one is stable.
>>>>>>
>>>>>> We have prepared a design document that discusses a restructuring of
>>> the
>>>>>> flink-table module and suggests a rough implementation plan:
>>>>>>
>>>>>>
>>>>>>
>>> https://docs.google.com/document/d/1Tfl2dBqBV3qSBy7oV3qLYvRRDbUOasvA1lhvYWWljQw/edit?usp=sharing
>>>>>> I will briefly summarize the steps we would like to do:
>>>>>>
>>>>>> - Split the flink-table module similar to the proposal of FLIP-28 [3]
>>>>>> which is outdated. This is a preparation to separate API from core
>>>>>> (targeted for Flink 1.8).
>>>>>> - Perform minor API changes to separate API from actual implementation
>>>>>> (targeted for Flink 1.8).
>>>>>> - Merge a MVP Blink SQL planner given that necessary Flink
>>> core/runtime
>>>>>> changes have been completed.
>>>>>> The merging will happen in stages (e.g. basic planner framework,
>>> then
>>>>>> operator by operator). The exact merging plan still needs to be
>>>> determined.
>>>>>> - Rework the type system in order to unblock work on unified table
>>>>>> environments, UDFs, sources/sinks, and catalog.
>>>>>> - Enable full end-to-end batch and stream execution features.
>>>>>>
>>>>>> Our mid-term goal:
>>>>>>
>>>>>> Run full TPC-DS on a unified batch/streaming runtime. Initially, we
>>> will
>>>>>> only support ingesting data coming from the DataStream API. Once we
>>>>>> reworked the sources/sink interfaces, we will target full end-to-end
>>>>>> TPC-DS query execution with table connectors.
>>>>>>
>>>>>> A rough task dependency graph is illustrated in the design document. A
>>>>>> more detailed task dependency structure will be added to JIRA after we
>>>>>> agreed on this FLIP.
>>>>>>
>>>>>> Looking forward to any feedback.
>>>>>>
>>>>>> Thanks,
>>>>>> Timo
>>>>>>
>>>>>> [1]
>>>>>>
>>>>>>
>>> https://lists.apache.org/thread.html/2f7330e85d702a53b4a2b361149930b50f2e89d8e8a572f8ee2a0e6d@%3Cdev.flink.apache.org%3E
>>>>>> [2]
>>>>>>
>>>>>>
>>> https://lists.apache.org/thread.html/6066abd0f09fc1c41190afad67770ede8efd0bebc36f00938eecc118@%3Cdev.flink.apache.org%3E
>>>>>> [3]
>>>>>>
>>>>>>
>>> https://cwiki.apache.org/confluence/display/FLINK/FLIP-28%3A+Long-term+goal+of+making+flink-table+Scala-free
>>>>>>
>>>>