Cassiopaean timeline
- Thread starter shijing
- Start date
allen
Jedi Master
Galahad said:
Well, the way the admin is being made at the moment, it would be one entry at a time. Could do bulk upload, but where would we get the source files? One way or another, those would have to be put into some sort of machine readable form one by one. Can you give an example of the sort of file you are talking about?
henry
The Cosmic Force
I have a number of timeline files in different formats depending upon the source. There are probably many more available on the web that could be downloaded or gotten through screen scraping. If there was a file format for reading files into the db, then it might be possible to write some small text manipulation programmes that could put the files into the correct format. We would just need to know what are the fields in the db and their order.
Some examples:
This goes up to the present.
Again, this one goes to the present.
Some examples:
Sunspots said:
This goes up to the present.
Wars said:
Again, this one goes to the present.
foofighter
Jedi Council Member
One by one is probably not going to be scalable at all, and doesn't work very well when the data is coming from another source that can be scraped. One way that I have done this in the past in other projects is to allow OpenOffice documents to be uploaded (OpenOffice since the internal format is XML, which is easy to work with). The column headers would be well-defined, like "Description", "Date", "Source", etc., and then each line below would have the actual data. A submitted document can then be parsed and the data extracted and put into the database. Not only does this allow lots of data to be input at once, but it also helps scraping scenarios (e.g. scrape a webpage and generate the OpenOffice document), and it also makes it possible to do the data entering "offline", which can then be uploaded in bulk.Burma Jones said:
allen
Jedi Master
foofighter said:
Excellent points. So, the minimum data required for each point is a title (the text that appears on the timeline) and a start date/time (at least the year, but will take all date/time info including day-of-week).
Optional information:
description (can be html, will default to the title if not included)
end date/time (if the event is a date span like "Crimean War")
icon
event type (Cs, historical document, archeological evidence, mythology, religious document) - probably the best way to handle this with bulk upload is to create separate XML docs for each type
links
What we have in the db schema that is implemented on the timeline yet but will be very soon:
tags (for filtering)
people involved (this will give an option of either getting more info on the person or filtering the timeline to show events involving this person)
So, would be cool is some of you could start collecting the above information and putting it in any machine readable format, meaning that there is a pattern to where each data point is so that I can easily parse it. Can be XML, spreadsheet, comma delimited file, whatever. While you do that, I'll get tags and people implemented along with the several thousand other things I've got to get done. :o)
foofighter
Jedi Council Member
And you could add source (e.g. book, article), location (Country/City/Region), author (who put it in, without which you're going to have security issues), when it was put in, etc. etc. One problem with all this, and with using a relational database in the first place, is that we are talking about a sparse data model, where each node could have one or a hundred possible attributes. Relational databases are really really bad at handling those kind of datasets, and are also really lousy at being able to do interesting things with it once you have the data. It's simply not what they were made for (they rock at keeping bank accounts though). If it was me I'd opt for using either an RDF database instead (which the Simile widgets prefer also, and then you can do faceted browsing much easier), or a graph database (like Neo4j, which would allow you to do some really cool analysis algorithms on the data). Anything would be better than a relational database, really, for this type of data.Burma Jones said:
For instance, if you work with importing datasets from Excel spreadsheets, how do you handle updates of the data? Let's say I enter 100 rows first and upload it. Then I add 100 more, and change 10 of the old ones, and remove 3 that I found were wrong, and then want to upload that new version. How do you deal with that? How do you figure out what has changed? In an RDF database that's trivial. With a relational database it's a bit more tricky.
I know you are eager to rush forward with this (I would be too), but IMHO you should stop and consider what your options are first, and what the consequences are with going different routes, both for the data management, input handling, and visualization.
If you want people to start working on this Right Now, I would strongly suggest that you make an example spreadsheet with given row headers, as that is going to make it a heck of a lot easier to parse it later on. Don't forget to specify date formats and any other kind of formats for fields as well. I would write todays date as "2009-10-16", but other people might not. Having a consistent format from the beginning helps a lot.
foofighter
Jedi Council Member
If you want to look at a technology that really would fit quite well with the intent of this project, look at freebase.com. They have an open database where anyone can add more data, and then there's tons of tools to import/export/visualize it, and use it in your own website. Read their About page for more info on why a relational database just doesn't work all that well for this kind of semi-structured data.foofighter said:
In a perfect world, you'd simply use that instead of making your own tool for this. The problem, I think, is that the terms of use states that it is not allowed to upload material that "is illegal, harmful, threatening, abusive, harassing, defamatory, obscene, offensive, invades another’s privacy, or promotes bigotry, racism, or hatred for or harm against any individual or group". Pointing out historical errors could probably be at least "harmful", "threatening", "offensive" and sometimes even "illegal", if seen through pathological lenses. It's a pity there's no way to run your own "freebase", because then it'd be easy to just use that software as-is.
allen
Jedi Master
foofighter said:
Already have source (didn't list that, sorry) and location (lat and long). Proli should also put in text of the location. Author, for security reasons, is always a good idea, yeah.
foofighter said:
This isn't rocket science. We have a timeline from open source code with a predefined set of attributes. We also have a database that already can hold said data, deliver it to the timeline and display it. I don't see how simile timeline prefers RDF. What I feed it is a json encoded array of objects, which it likes just fine. Creating that array from my relational database is simple as pie.
As for your comments on relational databases pretty much only being good for keeping bank accounts....uhm, what?!
foofighter said:
Actually, you don't know anything of the sort. You have no idea how busy I am right now. For the past few nights I haven't been able to get to bed until about 3am, working solid through the day and night. What I am eager to do is get some sleep! However, I know from a whole lot of real world experience that a project as simple as this doesn't require a lot of design (see above comment on rocket science). What it does require is a continual movement of energy so that it doesn't stall.
The most difficult part of it is the data collection. Once we have the data, we can do anything we like with it. I really don't care what form it comes in or what headings are used. The only thing I care about is that it is complete and in some sort of structured format that can be parsed. As for date formats, if someone is going to collect 1,000 data points from various sources that use various date formats, they would have to go through all of those to make sure the date formatting is correct. At that point, they might as well just enter the data into a web form. I'd rather just write a parser that can handle any date format and make things as easy as possible on those collecting the data.
dant
The Living Force
The "upside" to Webhosting (co-hosting) is
low or no maintenance (with exception to
co-hosting), and high bandwidth connection,
redundancy, and perhaps with a few "bells and
whistles" thrown in such as "software: website,
database, etc...", but with less control?
The "downside" of private hosting is costly
software(linux is "free"), server-maintenance,
no-redundancy, and bandwidth, but with a higher
degree of control?
As for policies, is this what it breaks down to:
(1) (Integrated) Web hosted site: Policy
[which also includes 2(a)(b)(c) below]
(2) Private, but public site:
(a) Domain name provider: Policy
(b) ISP provider: Policy
(c) Phone company (wireless/landline): Policy
FWIW,
Dan
low or no maintenance (with exception to
co-hosting), and high bandwidth connection,
redundancy, and perhaps with a few "bells and
whistles" thrown in such as "software: website,
database, etc...", but with less control?
The "downside" of private hosting is costly
software(linux is "free"), server-maintenance,
no-redundancy, and bandwidth, but with a higher
degree of control?
As for policies, is this what it breaks down to:
(1) (Integrated) Web hosted site: Policy
[which also includes 2(a)(b)(c) below]
(2) Private, but public site:
(a) Domain name provider: Policy
(b) ISP provider: Policy
(c) Phone company (wireless/landline): Policy
FWIW,
Dan
dant
The Living Force
Burma Jones said:
It is not so much as server/space availability
but that of accessibility and preservation of
the data itself.
The question is, is the site you are working on,
trustable, and the work will not be co-opted?
The data/work ought to be backed up off-site,
often, in order preserve and restore the data to
another site if the need arises?
FWIW,
Dan
