Visualizing topologies

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

Visualizing topologies

Ken Krugler
Hi list,

I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)

So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file

The results look like this...

http://scaleunlimited.com/downloads/CrawlTopology.png

If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.

— Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr



Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Pattarawat Chormai
Hi Ken,

Maybe you can look into this one : http://flink.apache.org/visualizer/.

- Pat

> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote:
>
> Hi list,
>
> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>
> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>
> The results look like this...
>
> http://scaleunlimited.com/downloads/CrawlTopology.png
>
> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>
> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>

Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Ufuk Celebi-2
Hey Ken!

This looks really good. +1 to make this available publicly.

We can link it from the Flink website and the viz tool Pat linked to.
The vizualizer has currently some open issues, it is not up to date
with the one that is part of the Flink web UI.

– Ufuk


On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
<[hidden email]> wrote:

> Hi Ken,
>
> Maybe you can look into this one : http://flink.apache.org/visualizer/.
>
> - Pat
>
>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote:
>>
>> Hi list,
>>
>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>
>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>
>> The results look like this...
>>
>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>
>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>
>> — Ken
>>
>> --------------------------
>> Ken Krugler
>> +1 530-210-6378
>> http://www.scaleunlimited.com
>> custom big data solutions & training
>> Hadoop, Cascading, Cassandra & Solr
>>
>>
>>
>
Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Ken Krugler
In reply to this post by Pattarawat Chormai
Hi Pat,

> On Feb 21, 2017, at 6:01pm, Pattarawat Chormai <[hidden email]> wrote:
>
> Hi Ken,
>
> Maybe you can look into this one : http://flink.apache.org/visualizer/.

Thanks, that’s interesting and convenient.

Though I’d probably keep using OmniGraffle with a dot file as that gives me the ability to edit/annotate as needed.

Thoughts on having the iteration sink connect to the iteration source? I found that helpful when working on my topology.

Regards,

— Ken



>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote:
>>
>> Hi list,
>>
>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>
>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>
>> The results look like this...
>>
>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>
>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>
>> — Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr



Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Ken Krugler
In reply to this post by Ufuk Celebi-2
Hi Ufuk,

> On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote:
>
> Hey Ken!
>
> This looks really good. +1 to make this available publicly.
>
> We can link it from the Flink website and the viz tool Pat linked to.
> The vizualizer has currently some open issues, it is not up to date
> with the one that is part of the Flink web UI.

I wasn’t planning on running a web interface to the tool.

My thought was to just add a writeDotFile(“path to file") method to the StreamExecutionEnvironment…this is similar to what Cascading supports.

If that’s reasonable, I’ll open an issue and attach the code.

— Ken


> On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> <[hidden email]> wrote:
>> Hi Ken,
>>
>> Maybe you can look into this one : http://flink.apache.org/visualizer/.
>>
>> - Pat
>>
>>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]> wrote:
>>>
>>> Hi list,
>>>
>>> I poked around a bit and didn’t see a way to easily visualize workflows in the same way I was used to with Cascading (generate a blah.dot file, open with OminGraffle or other graph tools)
>>>
>>> So I wrote a bit of code that takes the JSON output from StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>>
>>> The results look like this...
>>>
>>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>>
>>> If this is useful (and doesn’t already exist) then I can easily drop it in somewhere.
>>>
>>> — Ken

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr



Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Fabian Hueske-2
Hi Ken,

I think this would be an interesting feature!
I'd suggest to open a JIRA for it.

When extending the API of core classes such as ExecutionEnvironment, there
is often some discussion whether the feature is important enough or whether
it should be rather added to some external util class (which makes it
obviously less visible).
I think this feature would be valuable enough to be added directly into the
ExecutionEnvironment. DOT is a common format for graph and flow
visualizations.

Let's see what others think.

Best, Fabian


2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>:

> Hi Ufuk,
>
> > On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote:
> >
> > Hey Ken!
> >
> > This looks really good. +1 to make this available publicly.
> >
> > We can link it from the Flink website and the viz tool Pat linked to.
> > The vizualizer has currently some open issues, it is not up to date
> > with the one that is part of the Flink web UI.
>
> I wasn’t planning on running a web interface to the tool.
>
> My thought was to just add a writeDotFile(“path to file") method to the
> StreamExecutionEnvironment…this is similar to what Cascading supports.
>
> If that’s reasonable, I’ll open an issue and attach the code.
>
> — Ken
>
>
> > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> > <[hidden email]> wrote:
> >> Hi Ken,
> >>
> >> Maybe you can look into this one : http://flink.apache.org/visualizer/.
> >>
> >> - Pat
> >>
> >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <[hidden email]>
> wrote:
> >>>
> >>> Hi list,
> >>>
> >>> I poked around a bit and didn’t see a way to easily visualize
> workflows in the same way I was used to with Cascading (generate a blah.dot
> file, open with OminGraffle or other graph tools)
> >>>
> >>> So I wrote a bit of code that takes the JSON output from
> StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
> >>>
> >>> The results look like this...
> >>>
> >>> http://scaleunlimited.com/downloads/CrawlTopology.png
> >>>
> >>> If this is useful (and doesn’t already exist) then I can easily drop
> it in somewhere.
> >>>
> >>> — Ken
>
> --------------------------
> Ken Krugler
> +1 530-210-6378
> http://www.scaleunlimited.com
> custom big data solutions & training
> Hadoop, Cascading, Cassandra & Solr
>
>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Greg Hogan
Ken and Fabian,

Is the use case to generate and act on the dot file from within the user
program? Would it be more maintainable to make the plan JSON more
accessible (through the CLI and web interface) which users could then pipe
through a converter script?

Greg

On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <[hidden email]> wrote:

> Hi Ken,
>
> I think this would be an interesting feature!
> I'd suggest to open a JIRA for it.
>
> When extending the API of core classes such as ExecutionEnvironment, there
> is often some discussion whether the feature is important enough or whether
> it should be rather added to some external util class (which makes it
> obviously less visible).
> I think this feature would be valuable enough to be added directly into the
> ExecutionEnvironment. DOT is a common format for graph and flow
> visualizations.
>
> Let's see what others think.
>
> Best, Fabian
>
>
> 2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>:
>
> > Hi Ufuk,
> >
> > > On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote:
> > >
> > > Hey Ken!
> > >
> > > This looks really good. +1 to make this available publicly.
> > >
> > > We can link it from the Flink website and the viz tool Pat linked to.
> > > The vizualizer has currently some open issues, it is not up to date
> > > with the one that is part of the Flink web UI.
> >
> > I wasn’t planning on running a web interface to the tool.
> >
> > My thought was to just add a writeDotFile(“path to file") method to the
> > StreamExecutionEnvironment…this is similar to what Cascading supports.
> >
> > If that’s reasonable, I’ll open an issue and attach the code.
> >
> > — Ken
> >
> >
> > > On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
> > > <[hidden email]> wrote:
> > >> Hi Ken,
> > >>
> > >> Maybe you can look into this one : http://flink.apache.org/
> visualizer/.
> > >>
> > >> - Pat
> > >>
> > >>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <
> [hidden email]>
> > wrote:
> > >>>
> > >>> Hi list,
> > >>>
> > >>> I poked around a bit and didn’t see a way to easily visualize
> > workflows in the same way I was used to with Cascading (generate a
> blah.dot
> > file, open with OminGraffle or other graph tools)
> > >>>
> > >>> So I wrote a bit of code that takes the JSON output from
> > StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
> > >>>
> > >>> The results look like this...
> > >>>
> > >>> http://scaleunlimited.com/downloads/CrawlTopology.png
> > >>>
> > >>> If this is useful (and doesn’t already exist) then I can easily drop
> > it in somewhere.
> > >>>
> > >>> — Ken
> >
> > --------------------------
> > Ken Krugler
> > +1 530-210-6378
> > http://www.scaleunlimited.com
> > custom big data solutions & training
> > Hadoop, Cascading, Cassandra & Solr
> >
> >
> >
> >
>
Reply | Threaded
Open this post in threaded view
|

Re: Visualizing topologies

Ken Krugler
Hi Greg,

The use case is to create a visualization of the topology.

So I don’t think there’s any reason to “act on the dot file from within the user program”

Regards,

— Ken

> On Feb 24, 2017, at 7:51am, Greg Hogan <[hidden email]> wrote:
>
> Ken and Fabian,
>
> Is the use case to generate and act on the dot file from within the user
> program? Would it be more maintainable to make the plan JSON more
> accessible (through the CLI and web interface) which users could then pipe
> through a converter script?
>
> Greg
>
> On Fri, Feb 24, 2017 at 4:55 AM, Fabian Hueske <[hidden email]> wrote:
>
>> Hi Ken,
>>
>> I think this would be an interesting feature!
>> I'd suggest to open a JIRA for it.
>>
>> When extending the API of core classes such as ExecutionEnvironment, there
>> is often some discussion whether the feature is important enough or whether
>> it should be rather added to some external util class (which makes it
>> obviously less visible).
>> I think this feature would be valuable enough to be added directly into the
>> ExecutionEnvironment. DOT is a common format for graph and flow
>> visualizations.
>>
>> Let's see what others think.
>>
>> Best, Fabian
>>
>>
>> 2017-02-24 0:15 GMT+01:00 Ken Krugler <[hidden email]>:
>>
>>> Hi Ufuk,
>>>
>>>> On Feb 22, 2017, at 2:18am, Ufuk Celebi <[hidden email]> wrote:
>>>>
>>>> Hey Ken!
>>>>
>>>> This looks really good. +1 to make this available publicly.
>>>>
>>>> We can link it from the Flink website and the viz tool Pat linked to.
>>>> The vizualizer has currently some open issues, it is not up to date
>>>> with the one that is part of the Flink web UI.
>>>
>>> I wasn’t planning on running a web interface to the tool.
>>>
>>> My thought was to just add a writeDotFile(“path to file") method to the
>>> StreamExecutionEnvironment…this is similar to what Cascading supports.
>>>
>>> If that’s reasonable, I’ll open an issue and attach the code.
>>>
>>> — Ken
>>>
>>>
>>>> On Wed, Feb 22, 2017 at 3:01 AM, Pattarawat Chormai
>>>> <[hidden email]> wrote:
>>>>> Hi Ken,
>>>>>
>>>>> Maybe you can look into this one : http://flink.apache.org/
>> visualizer/.
>>>>>
>>>>> - Pat
>>>>>
>>>>>> On Feb 21, 2017, at 11:59 PM, Ken Krugler <
>> [hidden email]>
>>> wrote:
>>>>>>
>>>>>> Hi list,
>>>>>>
>>>>>> I poked around a bit and didn’t see a way to easily visualize
>>> workflows in the same way I was used to with Cascading (generate a
>> blah.dot
>>> file, open with OminGraffle or other graph tools)
>>>>>>
>>>>>> So I wrote a bit of code that takes the JSON output from
>>> StreamExecutionEnvironment#getExecutionPlan and generates a .dot file
>>>>>>
>>>>>> The results look like this...
>>>>>>
>>>>>> http://scaleunlimited.com/downloads/CrawlTopology.png
>>>>>>
>>>>>> If this is useful (and doesn’t already exist) then I can easily drop
>>> it in somewhere.
>>>>>>
>>>>>> — Ken
>>>
>>> --------------------------
>>> Ken Krugler
>>> +1 530-210-6378
>>> http://www.scaleunlimited.com
>>> custom big data solutions & training
>>> Hadoop, Cascading, Cassandra & Solr
>>>
>>>
>>>
>>>
>>

--------------------------
Ken Krugler
+1 530-210-6378
http://www.scaleunlimited.com
custom big data solutions & training
Hadoop, Cascading, Cassandra & Solr