Update Netty version

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Update Netty version

Alexey Demin
Hi

For now we use very old netty version.

Netty 4.0.27.Final released on 02-Apr-15

<!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
contain an improvement by Netty, which slices a Netty buffer
instead of doing a memory copy [1] in the
LengthFieldBasedFrameDecoder. In some situations, this
interacts badly with our Netty pipeline leading to OutOfMemory
errors.

[1] https://github.com/netty/netty/issues/3704 -->

If we so worry about slice in LengthFieldBasedFrameDecoder we can add custom
LengthFieldBasedCopyFrameDecoder which extend original
LengthFieldBasedFrameDecoder and override extractFrame for keep current
behavior.

With this small changes we can update for last 4.0.x.

For now LengthFieldBasedFrameDecoder also used in KvStateClient and
KvStateServer.
Can we keep use original LengthFieldBasedFrameDecoder or must also change
on LengthFieldBasedCopyFrameDecoder?

If we want we can migrate on 4.1.
I already did tests and all work correctly, small changes in
NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
method which added to interface).


Thanks
Alexey Diomin
Reply | Threaded
Open this post in threaded view
|

Re: Update Netty version

Till Rohrmann
Hi Alexey,

thanks for looking into it. Are we currently facing any problems with Netty
4.0.27 (bugs or performance)? I agree that in general we should try to use
the latest bug fix release. However, in the past we have seen that they
might entail some slight behaviour changes which breaks things on the Flink
side. Since Netty is quite crucial for Flink, I would be extra careful here
when bumping versions, especially if there is no strong need for it.

However, if we do the change at the beginning of the next release cycle, we
might have enough exposure time to verify whether things work or not.

Cheers,
Till

On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <[hidden email]> wrote:

> Hi
>
> For now we use very old netty version.
>
> Netty 4.0.27.Final released on 02-Apr-15
>
> <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> contain an improvement by Netty, which slices a Netty buffer
> instead of doing a memory copy [1] in the
> LengthFieldBasedFrameDecoder. In some situations, this
> interacts badly with our Netty pipeline leading to OutOfMemory
> errors.
>
> [1] https://github.com/netty/netty/issues/3704 -->
>
> If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> custom
> LengthFieldBasedCopyFrameDecoder which extend original
> LengthFieldBasedFrameDecoder and override extractFrame for keep current
> behavior.
>
> With this small changes we can update for last 4.0.x.
>
> For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> KvStateServer.
> Can we keep use original LengthFieldBasedFrameDecoder or must also change
> on LengthFieldBasedCopyFrameDecoder?
>
> If we want we can migrate on 4.1.
> I already did tests and all work correctly, small changes in
> NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> method which added to interface).
>
>
> Thanks
> Alexey Diomin
>
Reply | Threaded
Open this post in threaded view
|

Re: Update Netty version

Alexey Demin
Hi

Problem not directly in flink, but it you use flink with beam then in
classpath you have original netty 4.0.27 from flink and netty 4.1.x from
beam (grpc use netty 4.1.x for communication).

Other interest (specific for me now): netty have custom wrapper for openssl
library which have more productivity versus default jdk version, when you
have enabled ssl cluster communication it's will be very usefull give users
select implementation of ssl (default,openssl,boringssl).

But netty have a lot of fixes for openssl/boringssl in latest versions and
more preferable do update netty as first step and enable select sslengine
as second step, not all in one step.

> However, if we do the change at the beginning of the next release cycle,
we
> might have enough exposure time to verify whether things work or not.

We just start 1.4 iteration and have time for testing.

Thank,
Alexey Demin


2017-05-18 11:48 GMT+04:00 Till Rohrmann <[hidden email]>:

> Hi Alexey,
>
> thanks for looking into it. Are we currently facing any problems with Netty
> 4.0.27 (bugs or performance)? I agree that in general we should try to use
> the latest bug fix release. However, in the past we have seen that they
> might entail some slight behaviour changes which breaks things on the Flink
> side. Since Netty is quite crucial for Flink, I would be extra careful here
> when bumping versions, especially if there is no strong need for it.
>
> However, if we do the change at the beginning of the next release cycle, we
> might have enough exposure time to verify whether things work or not.
>
> Cheers,
> Till
>
> On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <[hidden email]> wrote:
>
> > Hi
> >
> > For now we use very old netty version.
> >
> > Netty 4.0.27.Final released on 02-Apr-15
> >
> > <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> > contain an improvement by Netty, which slices a Netty buffer
> > instead of doing a memory copy [1] in the
> > LengthFieldBasedFrameDecoder. In some situations, this
> > interacts badly with our Netty pipeline leading to OutOfMemory
> > errors.
> >
> > [1] https://github.com/netty/netty/issues/3704 -->
> >
> > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > custom
> > LengthFieldBasedCopyFrameDecoder which extend original
> > LengthFieldBasedFrameDecoder and override extractFrame for keep current
> > behavior.
> >
> > With this small changes we can update for last 4.0.x.
> >
> > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > KvStateServer.
> > Can we keep use original LengthFieldBasedFrameDecoder or must also change
> > on LengthFieldBasedCopyFrameDecoder?
> >
> > If we want we can migrate on 4.1.
> > I already did tests and all work correctly, small changes in
> > NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> > method which added to interface).
> >
> >
> > Thanks
> > Alexey Diomin
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Update Netty version

Greg Hogan
Hi Alexey,

Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?

Greg

On Thu, May 18, 2017 at 4:41 AM, Alexey Demin <[hidden email]> wrote:

> Hi
>
> Problem not directly in flink, but it you use flink with beam then in
> classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> beam (grpc use netty 4.1.x for communication).
>
> Other interest (specific for me now): netty have custom wrapper for openssl
> library which have more productivity versus default jdk version, when you
> have enabled ssl cluster communication it's will be very usefull give users
> select implementation of ssl (default,openssl,boringssl).
>
> But netty have a lot of fixes for openssl/boringssl in latest versions and
> more preferable do update netty as first step and enable select sslengine
> as second step, not all in one step.
>
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
>
> We just start 1.4 iteration and have time for testing.
>
> Thank,
> Alexey Demin
>
>
> 2017-05-18 11:48 GMT+04:00 Till Rohrmann <[hidden email]>:
>
> > Hi Alexey,
> >
> > thanks for looking into it. Are we currently facing any problems with
> Netty
> > 4.0.27 (bugs or performance)? I agree that in general we should try to
> use
> > the latest bug fix release. However, in the past we have seen that they
> > might entail some slight behaviour changes which breaks things on the
> Flink
> > side. Since Netty is quite crucial for Flink, I would be extra careful
> here
> > when bumping versions, especially if there is no strong need for it.
> >
> > However, if we do the change at the beginning of the next release cycle,
> we
> > might have enough exposure time to verify whether things work or not.
> >
> > Cheers,
> > Till
> >
> > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <[hidden email]>
> wrote:
> >
> > > Hi
> > >
> > > For now we use very old netty version.
> > >
> > > Netty 4.0.27.Final released on 02-Apr-15
> > >
> > > <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> > > contain an improvement by Netty, which slices a Netty buffer
> > > instead of doing a memory copy [1] in the
> > > LengthFieldBasedFrameDecoder. In some situations, this
> > > interacts badly with our Netty pipeline leading to OutOfMemory
> > > errors.
> > >
> > > [1] https://github.com/netty/netty/issues/3704 -->
> > >
> > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > custom
> > > LengthFieldBasedCopyFrameDecoder which extend original
> > > LengthFieldBasedFrameDecoder and override extractFrame for keep current
> > > behavior.
> > >
> > > With this small changes we can update for last 4.0.x.
> > >
> > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > KvStateServer.
> > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> change
> > > on LengthFieldBasedCopyFrameDecoder?
> > >
> > > If we want we can migrate on 4.1.
> > > I already did tests and all work correctly, small changes in
> > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement new
> > > method which added to interface).
> > >
> > >
> > > Thanks
> > > Alexey Diomin
> > >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Update Netty version

Alexey Demin
Hi

I found ticket in jira: https://issues.apache.org/jira/browse/FLINK-3952

For update to latest 4.0 I can create separate ticket and pull request in
nearest 2 day. (I test in my env and all work correctly)

Update on 4.1 not so easy:
1) we use tv.cntt:netty-router:jar:1.10 for rest endpoints
2) netty-router mostly stoped development and have support only netty 4.0
3) latest versions say then they support 4.1 netty, but from my opinion
they under heavy refactoring and I can't recommend replace netty-router
1.10 on this version

we have 2 ways to resolve this problem:
1) remove dependency on netty-router
2) rebuild and repack netty-router 1.10 with new version on netty 4.1 (as
PoC I already did this work, and all work correctly, only small changes
required)

We already have custom akka build and build also netty-router under
discussion.

Thanks
Alexey Diomin



2017-06-08 0:13 GMT+04:00 Greg Hogan <[hidden email]>:

> Hi Alexey,
>
> Are you looking to create pull requests for upgrading Netty 4.0 and/or 4.1?
>
> Greg
>
> On Thu, May 18, 2017 at 4:41 AM, Alexey Demin <[hidden email]> wrote:
>
> > Hi
> >
> > Problem not directly in flink, but it you use flink with beam then in
> > classpath you have original netty 4.0.27 from flink and netty 4.1.x from
> > beam (grpc use netty 4.1.x for communication).
> >
> > Other interest (specific for me now): netty have custom wrapper for
> openssl
> > library which have more productivity versus default jdk version, when you
> > have enabled ssl cluster communication it's will be very usefull give
> users
> > select implementation of ssl (default,openssl,boringssl).
> >
> > But netty have a lot of fixes for openssl/boringssl in latest versions
> and
> > more preferable do update netty as first step and enable select sslengine
> > as second step, not all in one step.
> >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> >
> > We just start 1.4 iteration and have time for testing.
> >
> > Thank,
> > Alexey Demin
> >
> >
> > 2017-05-18 11:48 GMT+04:00 Till Rohrmann <[hidden email]>:
> >
> > > Hi Alexey,
> > >
> > > thanks for looking into it. Are we currently facing any problems with
> > Netty
> > > 4.0.27 (bugs or performance)? I agree that in general we should try to
> > use
> > > the latest bug fix release. However, in the past we have seen that they
> > > might entail some slight behaviour changes which breaks things on the
> > Flink
> > > side. Since Netty is quite crucial for Flink, I would be extra careful
> > here
> > > when bumping versions, especially if there is no strong need for it.
> > >
> > > However, if we do the change at the beginning of the next release
> cycle,
> > we
> > > might have enough exposure time to verify whether things work or not.
> > >
> > > Cheers,
> > > Till
> > >
> > > On Thu, May 18, 2017 at 8:51 AM, Alexey Demin <[hidden email]>
> > wrote:
> > >
> > > > Hi
> > > >
> > > > For now we use very old netty version.
> > > >
> > > > Netty 4.0.27.Final released on 02-Apr-15
> > > >
> > > > <!-- Don't upgrade for now. Netty versions >= 4.0.28.Final
> > > > contain an improvement by Netty, which slices a Netty buffer
> > > > instead of doing a memory copy [1] in the
> > > > LengthFieldBasedFrameDecoder. In some situations, this
> > > > interacts badly with our Netty pipeline leading to OutOfMemory
> > > > errors.
> > > >
> > > > [1] https://github.com/netty/netty/issues/3704 -->
> > > >
> > > > If we so worry about slice in LengthFieldBasedFrameDecoder we can add
> > > > custom
> > > > LengthFieldBasedCopyFrameDecoder which extend original
> > > > LengthFieldBasedFrameDecoder and override extractFrame for keep
> current
> > > > behavior.
> > > >
> > > > With this small changes we can update for last 4.0.x.
> > > >
> > > > For now LengthFieldBasedFrameDecoder also used in KvStateClient and
> > > > KvStateServer.
> > > > Can we keep use original LengthFieldBasedFrameDecoder or must also
> > change
> > > > on LengthFieldBasedCopyFrameDecoder?
> > > >
> > > > If we want we can migrate on 4.1.
> > > > I already did tests and all work correctly, small changes in
> > > > NettyBufferPool.java and ChunkedByteBuf.java is required (implement
> new
> > > > method which added to interface).
> > > >
> > > >
> > > > Thanks
> > > > Alexey Diomin
> > > >
> > >
> >
>