Data Mining and Data Visualization Tools
- Thread starter name
- Start date
name
Jedi Master
@Guardian
Knime current (V2.5.4) looks quite OK, certainly by far not as buggy as Gephi. I use it for small load jobs and to evaluate data from different sources.
@Megan
I made the Fucilla Graph by simply scraping the data about persons and organizations by hand from the first page of the thread and saving it as CSV, then doing the same for the edges, followed by some adjustment by hand. I am looking forward to see what you are doing.
I do SQL, PL/SQL and data design (Postgres/Oracle), and I do not yet understand NoSQL/graph-oriented databases.
@Dant
The bugs of gephi are reported.
Hint: For all the java tools (Gephi, yED, knime, Rapidminer), check that the allowed memory settings for Java are sufficient for what you intend to do.
Look for the conf file or startup script of the tool and set -Xmx to something sensible. From experience I set it to half the physical memory, so, if you have 4GB, set -Xmx2048m if you have lots of data.
Knime current (V2.5.4) looks quite OK, certainly by far not as buggy as Gephi. I use it for small load jobs and to evaluate data from different sources.
@Megan
I made the Fucilla Graph by simply scraping the data about persons and organizations by hand from the first page of the thread and saving it as CSV, then doing the same for the edges, followed by some adjustment by hand. I am looking forward to see what you are doing.
I do SQL, PL/SQL and data design (Postgres/Oracle), and I do not yet understand NoSQL/graph-oriented databases.
@Dant
The bugs of gephi are reported.
Hint: For all the java tools (Gephi, yED, knime, Rapidminer), check that the allowed memory settings for Java are sufficient for what you intend to do.
Look for the conf file or startup script of the tool and set -Xmx to something sensible. From experience I set it to half the physical memory, so, if you have 4GB, set -Xmx2048m if you have lots of data.
mcb
The Living Force
name said:
I am using MS SQL Server 2012 to build a database of nodes and connections. An SSIS package runs once an hour and queries the latest forum posts using a script component written in C#, and then adds any new or changed messages to the database.
A second SSIS package runs on demand (it could be schedule to run every day or so) and applies the Term Extraction component to the message body of each new or changed message. This information is used to build a "term" table containing significant nouns and noun phrases for each message. This represents the graph that I am trying to build, and the message table (maintained by the first SSIS package) represents attributes associated with each node.
After updating the term table the package runs one additional Term Extraction across the entire message table, calculating a TF-IDF score for each noun or noun phrase and storing the values in the term table. I don't know if this will be useful or not, but it was easy to do and is easy to remove.
The next step is to query the graph out as XML, perhaps in GEXF format or in a format that can transform to GEXF or other formats that Gephi or other tools recognize.
The code that extracts messages from the forum needs quite a bit of work. Right now the message bodies are being stored as HTML, and I provide an exception list to the Term Extraction component that causes it to remove HTML tag and style names. This part could be done a lot better.
foofighter
Jedi Council Member
(Disclaimer: I work for Neo)
For the types of data and queries mentioned here I think the graph data model, in general is the ideal solution, and Neo4j would be a good fit technology wise for the storage. I've been considering doing a forum scrape project myself, whereby one would first make a list of people to be tracked manually (like Fucilla et. al), and the extractor would then simply note what people are mentioned in the same posts, as a way to infer relationships. This would not give high quality data, but would probably produce lots of it, given the amount of forum posts. One option is to use this as a first step to produce suggested relationships, that are then manually confirmed.
When it comes to visualization I know my coworkers and the community have played with a bunch of really nifty JavaScript frameworks. I don't have any experience with it myself, but could easily find out what works and how to do it, if necessary.
I should also say that the new query language for Neo opens some interesting possibilities. Asking things like "Is mr X related to mr Y, and through whom?" is a one liner, literally.
To me the big problem seems to be finding out a good way to get the data in. A Semantic Wiki is one option, which then could be fed to Neo4j or any other visualization tools.
For the types of data and queries mentioned here I think the graph data model, in general is the ideal solution, and Neo4j would be a good fit technology wise for the storage. I've been considering doing a forum scrape project myself, whereby one would first make a list of people to be tracked manually (like Fucilla et. al), and the extractor would then simply note what people are mentioned in the same posts, as a way to infer relationships. This would not give high quality data, but would probably produce lots of it, given the amount of forum posts. One option is to use this as a first step to produce suggested relationships, that are then manually confirmed.
When it comes to visualization I know my coworkers and the community have played with a bunch of really nifty JavaScript frameworks. I don't have any experience with it myself, but could easily find out what works and how to do it, if necessary.
I should also say that the new query language for Neo opens some interesting possibilities. Asking things like "Is mr X related to mr Y, and through whom?" is a one liner, literally.
To me the big problem seems to be finding out a good way to get the data in. A Semantic Wiki is one option, which then could be fed to Neo4j or any other visualization tools.
mcb
The Living Force
foofighter said:
No kidding. I have been working on cleaning the body text by removing HTML tags. All of SMF's XML formats (XML, Atom, RSS) return the message body as HTML. I am trying to treat the HTML as XML and extract just the text and CDATA portions. It doesn't matter what it looks like because it is only seen by the term extraction component. Once I solve that problem I will go back to producing a standard XML graph output format.
The term table is looking good, if a bit dull. The top words are rather ordinary, and I am adding some of them to the exclusion table. I am trying to capture all forum messages, not just select ones. It should be possible to filter the result to remove unwanted topics. The more message attributes I can make available, the more filtering options there will be.
foofighter
Jedi Council Member
For converting messy HTML to XML, TagSoup is your friend: http://ccil.org/~cowan/XML/tagsoup/ It's awesome for this kind of stuff!Megan said:
What about what I suggested, i.e. the other way round where you have a manual list of words you are looking for, such as names of individuals. If using Neo, you could then tag those differently from, for example, organizations.
mcb
The Living Force
If you capture everything, you can search for what you are looking for within it. If you only capture what you are looking for, you will miss anything you didn't think to look for.
I did quite a bit of cleanup on the message bodies just using an XmlTextReader. It's very fast -- much faster than the term extraction process, and it works because SMF is emitting valid XHTML. If I start to read from other sources, TagSoup could come in handy -- thanks!
Here is a sample from the term table:
When a word repeats, that means it appears in more than one post. The numbers are the TF-IDF scores.
I did quite a bit of cleanup on the message bodies just using an XmlTextReader. It's very fast -- much faster than the term extraction process, and it works because SMF is emitting valid XHTML. If I start to read from other sources, TagSoup could come in handy -- thanks!
Here is a sample from the term table:
When a word repeats, that means it appears in more than one post. The numbers are the TF-IDF scores.
foofighter
Jedi Council Member
Good point! So, basically you could first create a "raw" data capture, then investigate it manually and mark the things you look for (in a particular instance), and then do further analysis based on that subset.Megan said:
Nice!
Let me know if you need any suggestions on how to use Neo4j or visualizations related to it. I can easily find out from the team, if need be.
mcb
The Living Force
foofighter said:
I was planning to make the data available for download. I am not sure what to do with data from boards that are not public, however. I could suppress that data altogether (or control access to it), or I could limit the attributes included in the graph. The message bodies will not be part of the graph, but exposing the term list could provide a clue as to what was being discussed privately. I guess the best approach for now would be to suppress the private and access-limited boards.
My collector accesses the forum directly without using a web browser, and I did not attempt to pass my logon information, so I assume that the present database contains only public posts. Hopefully that means I can ignore the issue for now (after verifying my assumption), although I will certainly want to address it later on. What I might do is collect all public boards but only specific private or limited access boards. Detecting topics and replies that are moved from a public board to a private one could be a little tricky but I think I can do it.
So while collecting the data and building graphs are things that relate to my specialties, I would be more than happy to share figuring out the NoSQL and graph visualization parts of it, where I am starting out from scratch.
mcb
The Living Force
A little more progress -- I build a nodelist:
I spent most of my time tonight figuring out the best way to write the data out, leaving little time to work on the actual data. The next steps are to add attributes to each node, build the edge list, wrap it all with a few more elements, and try to feed it into Gephi. The export process currently writes to a file, but it could potentially upload to my website, automatically.
I spent most of my time tonight figuring out the best way to write the data out, leaving little time to work on the actual data. The next steps are to add attributes to each node, build the edge list, wrap it all with a few more elements, and try to feed it into Gephi. The export process currently writes to a file, but it could potentially upload to my website, automatically.
mcb
The Living Force
name said:
I am able to write out a GEXF file now and read it into Gephi. I need to shape the data differently, however, and I need to learn how to use Gephi.
The simple node lists I am creating now are too simple. They contain "parallel edges" that Gephi can't process. I can solve that by adding hierarchy -- instead of having a flat list of message nodes the graph can contain each "term" as a child of each message (forum post) in which it appears. The edges would then connect terms instead of messages. While I am at it, I should include the board & topic hierarchy above the messages.
Would that be a useful graph? Now that I have a basic set of components for building graphs, I can construct them any way that would work for visualization, but what would work well?
Presently, the only edges in the graph are from a "repeats" relationship, where an edge represents a word in a message that is a repeat of that same word in some earlier message. I can add another set of edges for a "quotes" relationship, where an edge connects an earlier message with a later one that quotes from it. I also have data for each message about who posted it (name and forum profile link) and who started the topic (name and forum profile link) that contains it.
paralleloscope
The Living Force
Most of the tech jargon and acronyms here is over my head, but find it an interesting topic and aim to learn more about it (recently did my first mysql).
Is it possible to do a layered graph, so that one could choose what level of complexity to view or combine. Like board/topic data on one layer, names in network on another, additional related data on a third layer and so forth ? That way one could make presentations match the presentation context by adding/subtracting layers.
Megan said:
Is it possible to do a layered graph, so that one could choose what level of complexity to view or combine. Like board/topic data on one layer, names in network on another, additional related data on a third layer and so forth ? That way one could make presentations match the presentation context by adding/subtracting layers.
name
Jedi Master
Cool! Congratulaitons for your advances and thanks for your interest in this and the great work!Megan said:
I don't know. I suppose that it depends on what one wants to understand with such a graph.Megan said:
My original idea was to use graphs to help understand the relationship of people, organizations... mentioned in threads such as the Fucilla one, or the longer Rense thread (for example), where people discuss a subject and apport information about people, entities, events ... over time. I am currently trying to make knime get the entities (see http://en.wikipedia.org/wiki/Named_entity_recognition) from a text, and in next step I'll look if can make it find relations between these entities and if there is a way to tag them in a sensible way. What I want is to eventually get a graph which more-or-less resembles something I'd also do by hand.
The thing with the "parallel edges" to which you allude is the weight of the relation between those particular two nodes. You must use the "weight" field of the edge so that gephi can process it and count up each time you find another relation between these two nodes - see the GEXF draft primer at "2.3.3 Declaring an Edge" on page 7.
You may also want to look at http://gexf.net if you haven't discovered it yet.
mcb
The Living Force
parallel said:
It wants to be over my head too, but I keep batting it back down. When I started in this field, a long time ago, it was actually possible to understand what you were doing, and I understood what was going on right down to the circuit level. Now you learn things at a certain high level and don't worry about the rest unless it quits working -- there are too many levels. How far down you go depends on how fascinated you are with with it. I am not fascinated with it much at all; I just want to be able to do something with it.
Part of the challenge is that "lazy" system 2. It doesn't want to deal with so much complexity, and you have to keep telling it that it isn't that bad and to hang in there. I learned long ago to divide problems into "chunks" of limited size, each of which (hopefully) is not overwhelming. The chunks might be nested one within another, or they might be parts of a system flow where one chunk does its work and hands off the results to the next chunk.
I think that is what making the node list hierarchical is going to do. I don't know Gephi well enough yet to predict how changes to the input file are going to manifest in the visualizations. Once I have a good set of input data I will gain experience with Gephi.
mcb
The Living Force
name said:
Thanks. I'll digest that when I have some free time. :)
Gephi just flat ignored the parallel edges; said it wasn't implemented yet. The problem should go away once I implement a board/topic/message/term hierarchy in the node list.
Apparently I can also model changes in the graph over time. That sounds like loads of fun!
