[DISCUSS] Integrate Flink SQL well with Hive ecosystem

classic Classic list List threaded Threaded
46 messages Options
[DISCUSS] Integrate Flink SQL well with Hive ecosystem – Hi all, Along with the community's effort, inside Alibaba we have explored Flink's potential as an execution engine not just for stream processin...
Hi Xuefu, Welcome to the Flink community and thanks for starting this discussion! Better Hive integration would be really great! Can you go i...
Hi Xuefu, Appreciate this proposal, and like Fabian, it would look better if you can give more details of the plan. Thanks, vino. Fabian...
Hi Fabian/Vno, Thank you very much for your encouragement inquiry. Sorry that I didn't see Fabian's email until I read Vino's response just now. ...
Welcome to the community and thanks for the great proposal, Xuefu! I think the proposal can be divided into 2 stages: making Flink to support Hiv...
Hi Shuyi, Thank you for your input. Yes, I agreed with a phased approach and like to move forward fast. :) We did some work internally on DDL uti...
Hi all, To wrap up the discussion, I have attached a PDF describing the proposal, which is also attached to FLINK-10556 [1]. Please feel free to ...
Hi all, I have also shared a design doc on Hive metastore integration that is attached here and also to FLINK-10556[1]. Please kindly review and ...
Hi Xuefu, Thanks a lot for driving this big effort. I would suggest convert your proposal and design doc into a google doc, and share it on th...
Hi Shuiyi, Good idea. Actually the PDF was converted from a google doc. Here is its link: https://docs.google.com/document/d/1SkppRD_rE3uOKSN-LuZ...
Hi all, As Xuefu has published the design doc on google, I agree with Shuyi's suggestion that we probably should start a new email thread like...
After taking a look at how other discussion threads work, I think it's actually fine just keep our discussion here. It's up to you, Xuefu. The...
Hi Xuefu and all, Thanks for sharing this design document! I'm very much in favor of restructuring / reworking the catalog handling in Flink ...
Hi, Maybe we should split this topic (and the design doc) into couple of smaller ones, hopefully independent. The questions that you have asked...
Hi Piotr, That seems to be good idea! Since the google doc for the design is currently under extensive review, I will leave it as it is for now....
Hi, Yes, it seems like the best solution. Maybe someone else can also suggests if we can split it further? Maybe changes in the interface in on...
Hi Piotr, That sounds good to me. Let's close all the open questions ((there are a couple of them)) in the Google doc and I should be able to qui...
Hi Piotr I have extracted the API portion of the design and the google doc is here. Please review and provide your feedback. Thanks, Xuefu --...
Hi Xuefu, Currently the new design doc <https://docs.google.com/document/d/1Y9it78yaUvbv4g572ZK_lZnZaAGjqwM_EhjdOv4yJtw/edit> is on “vi...
Thanks, Bowen, for catching the error. I have granted comment permission with the link. I also updated the doc with the latest class definitions....
Thanks for keeping on improving the overall design, Xuefu! It looks quite good to me now. Would be nice that cc-ed Flink committers can help t...
Thanks Xuefu for the detailed design doc! One question on the properties associated with the catalog objects. Are we going to leave them complete...
Hi Xiaowei, Thanks for bringing up the question. In the current design, the properties for meta objects are meant to cover anything that's specif...
Hi folks, there's clearly some incremental steps to be taken to introduce catalog support to SQL Client, complementary to what is proposed in the...
Hi Eron, Happy New Year! Thank you very much for your contribution, especially during the holidays. Wile I'm encouraged by your work, I'd also l...
I propose that the community review and merge the PRs that I posted, and then evolve the design thru 1.8 and beyond. I think having a basic in...
Would a couple folks raise their hand to make a review pass thru the 6 PRs listed above? It is a lovely stack of PRs that is 'all green' at the ...
Thanks Timo for merging a couple of the PRs. Are you also able to review the others that I mentioned? Xuefu I would like to incorporate your ...
Hi Eron, thank you very much for the contributions. I merged the first little bug fixes. For the remaining PRs I think we can review and merg...
Hi everyone, Xuefu and I had multiple iterations over the catalog design document [1]. I believe that it is in a good shape now to be convert...
Thanks, Timo! I have started put the content from the google doc to FLIP-30 [1]. However, please still keep the discussion along this thread. Th...
Thank you, Xuefu and Timo, for putting together the FLIP! I like that both its scope and implementation plan are clear. Look forward to feedbacks...
Hi Bowen, thanks for your feedback. We should not change the Google doc anymore but apply additional comments in the wiki page. I will also a...
Would it maybe make sense to provide Flink as an engine on Hive („flink-on-Hive“)? Eg to address 4,5,6,8,9,10. this could be more loosely coupled ...
Hi Jörn, Thanks for your feedback. Yes, I think Hive on Flink makes sense and in fact it is one of the two approaches that I named in the beginni...
Thank you very nice , I fully agree with that. > Am 11.10.2018 um 19:31 schrieb Zhang, Xuefu <xuefu.z@...>: > > Hi Jörn, &g...
Thank you Xuefu, for bringing up this awesome, detailed proposal! It will resolve lots of existing pain for users like me. In general, I totall...
Hi Bowen, Thank you for your feedback and interest in the project. Your contribution is certainly welcome. Per your suggestion, I have created an...
Hi Xuefu, thanks for your proposal, it is a nice summary. Here are my thoughts to your list: 1. I think this is also on our current mid-te...
Hi Xuefu, Thanks for putting together the overview. I would like to add some more on top of Timo's comments. 1,2. I agree with Timo that a pr...
Hi Rong, Thanks for your feedback. Some of my earlier comments might have addressed some of your points, so here I'd like to cover some specifics...
I think integrating Flink with Hive would be an amazing option and also to get Flink's SQL up to pace would be amazing. Current Flink Sql synt...
One other thought on the same lines was to use hive tables to store kafka information to process streaming tables. Something like "create...
Hi Taher, Thank you for your input. I think you emphasized two important points: 1. Hive metastore could be used for storing Flink metadata 2. T...
Sounds smashing; I think the initial integration will help 60% or so flink sql users and a lot other use cases will emerge when we solve the firs...
Hi Timo, Thank you for your input. It's exciting to see that the community has already initiated some of the topics. We'd certainly like to lever...