Why documentation always say checkpoint does not support Flink specific features like rescaling.

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Why documentation always say checkpoint does not support Flink specific features like rescaling.

Yun Tang
Hi, all

From the latest 1.6 (or all the previous verions) documentation https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html#difference-to-savepoints, it said "Checkpoints have a few differences from savepoints, they do not support Flink specific features like rescaling."

However, if you just take a look at RocksDB-StateBackend's code, you could see clearly that RocksDBStateBackend support rescaling from previous checkpoint. Could anyone give an explanation?

Best
Yun
Reply | Threaded
Open this post in threaded view
|

Re: Why documentation always say checkpoint does not support Flink specific features like rescaling.

Chesnay Schepler-3
Well that's a single StateBackend. In order to say that "Checkpoints
support rescaling" this must apply to all StateBackends, which AFAIK is
not the case.

On 27.08.2018 10:52, Yun Tang wrote:
> Hi, all
>
> >From the latest 1.6 (or all the previous verions) documentation https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html#difference-to-savepoints, it said "Checkpoints have a few differences from savepoints, they do not support Flink specific features like rescaling."
>
> However, if you just take a look at RocksDB-StateBackend's code, you could see clearly that RocksDBStateBackend support rescaling from previous checkpoint. Could anyone give an explanation?
>
> Best
> Yun
>

Reply | Threaded
Open this post in threaded view
|

Re: Why documentation always say checkpoint does not support Flink specific features like rescaling.

Andrey Zagrebin
As I understand, this is more about the documented semantics of checkpoint feature that Flink community agreed to guarantee for users so far in the last releases. Although technically the rescaling is currently possible for full and incremental checkpoints, it can change in future if checkpoints would need breaking optimisations or on contrary it might be fixed in the docs as supported from some release and on. In general, checkpoint API is currently designed for failure recovery of running job, not as savepoints for manual restart including rescaling.

> On 27 Aug 2018, at 11:23, Chesnay Schepler <[hidden email]> wrote:
>
> Well that's a single StateBackend. In order to say that "Checkpoints support rescaling" this must apply to all StateBackends, which AFAIK is not the case.
>
> On 27.08.2018 10:52, Yun Tang wrote:
>> Hi, all
>>
>> >From the latest 1.6 (or all the previous verions) documentation https://ci.apache.org/projects/flink/flink-docs-release-1.6/ops/state/checkpoints.html#difference-to-savepoints, it said "Checkpoints have a few differences from savepoints, they do not support Flink specific features like rescaling."
>>
>> However, if you just take a look at RocksDB-StateBackend's code, you could see clearly that RocksDBStateBackend support rescaling from previous checkpoint. Could anyone give an explanation?
>>
>> Best
>> Yun
>>
>