IMPORTANT: Regarding What Makes the Desert Beautiful...

[0creativity.livejournal.com]

As I'm sure most of you know, there's been a message on the front page of WMTDB for a long, long time now explaining why the site is only a skeleton of its former self.

For the two and a half years that Kelly and I have been working with the site, it has always been a struggle to keep up with its amazing growth. The hardest part has been finding the right balance between allowing user-submitted content and the time required for us to actually get that content on the site. After a couple of failed experiments, the last one ending when spambots flooded our server/database with tons of crap, we've decided to try again.

I've written a custom application that allows the posting/sharing of fic, icons, videos, and site announcements. This ties every aspect of the site together under one script instead of having a separate script each for fic, image galleries, etc. The problem is this custom script doesn't really mesh with the current eFiction script we're using, so there's no easy way for us to transfer all of that fic, user info, etc. over. Before you ask why we're attempting to fix something that's not really broken, here's the deal: For what we're currently getting in terms of bandwidth and space for the amount of money being spent, we could be doing a lot better. We're currently working on moving to a new hosting provider that will 1) run the custom script I've written and 2) give us about ten times the amount of storage and bandwidth we currently receive (10,000 MB storage and 600GB transfer/month). This means that we can continue to host videos, which are our huge bandwidth and storage drain, for a long time and the site can continue to grow without us having to worry about storage and bandwidth limitations.

It also means that now I'll actually be familiar with the system running the site and can make changes and updates to the system much more easily. This way we can continue to add new features and functionality as the site expands and grows.

This is the important part, though: We're currently hosting just shy of 1700 fics, not mention all of the videos and graphics. There's just no feasible way Kelly and I can transfer all of that content over. This is what we're asking: Please, start saving ANY fic you want to keep NOW. We think we'll be able to get all of the videos and graphics by ourselves, but the fic is another story. We simply cannot promise that everything will be transferred. So if there's a story you'd like to make sure you'll always have, save it now. We will be sending out a mass email and posting an announcement on the site soon asking authors to save their content and upload it to the new system (when it goes online), but there are many people who aren't active in the fandom anymore. Of course Kelly and I have our favorites that we'll make sure get carried over regardless, but everyone likes different things so our favorites probably aren't your favorites.

We are currently looking at following these phases:
1) Let still-active authors save and transfer their own work
2) Kelly and I will then look at what hasn't been transferred and do as much as we can
3) Let other users save the work and then add it to the new system (the original author will definitely still receive credit for their work)

We realize this is not an ideal situation. We're not at all happy about the idea of leaving things behind. However, it's been clear for some time that unless we somehow integrate all functions of the site into one system, we're going to be struggling trying to keep the site updated.

You can check out a test version of the new site at this link. Please note that I'm still working on the style of the pages and I haven't even started in some areas, so things will look a little odd. There are two test fics currently uploaded and I will be adding some test pictures and videos later tonight so you can have a feel for how things will work. There is not currently a way to leave comments/reviews but that will be one of the first additions I make after we get things (mostly) transferred. Feel free to register an account and look around and even add some content, if you want (the video component seems a little unstable at the moment so you might have some trouble adding video). Please remember, though, that anything you add to this test site will not be transferred over to the final new site.

I'm sure I'm leaving a lot of information out, so if you have any questions or comments about what's going on, just leave a comment here and I'll do my best to answer them. Also, if you have an suggestions or concerns about the new system, please also leave those here and I'll do my best to respond to them, too.

Currently, our present hosting expires at the end of the month. Kelly is looking to see if we can purchase an additional month to give us more time to transfer things. But in case that doesn't work out, please do not waste any time before saving fic you'd like to keep.

Thanks,
matt

Comments

For Saving Fics

[laureen.livejournal.com]

Because I go in and out of the city--where I have high speed there, but dialup here--I tend to save/download a lot of fic to read on my laptop. I've found that Offline Explorer (http://www.metaproducts.com/) is a great program to use for downloading fic because then you don't have to go through and save page by page--it does it for you. You could even, in theory (and if you have the hard drive space and the time to let it run) allow it to save everything on the site. Just as a suggestion for people who will probably be scrambling to get the fics they want. It's not a free program (unfortunately) but I believe it does have a trial-period of one month, and the fic is saved to your computer regardless. It's just not as easy to find the links without it.

I know I'll probably be one of those people who are scrambling to download the fics I want to read!

[adnamamai.livejournal.com]

I have fics there, but my last Final is Wensday. After that I can looking into moving it.

I also have a lot of favorites.

How will we know if a fic has been saved already so not everyone is saving the same fics?

[0creativity.livejournal.com]

How will we know if a fic has been saved already so not everyone is saving the same fics?
Good question. Now that I think about it, I might create a sort of "request" system where people can request the fic they definitely want to be transferred over (that isn't their own, of course). Then Kelly and I can work on transferring all of that over first and then get done what we can on the rest of it. I might also be able to rig it to where authors can "respond" and say whether or not they will be transferring their own fic over.

Just to be safe, though, I'd still save whatever you absolutely don't want to lose. I hate the thought of someone no longer being able to read an old favorite of their's...

[mjmcca.livejournal.com]

It was easy for me since I only have one fic there with only 2 chapters. I posted the first one to the new archive and will post the second one tommorrow. Thanks for the heads up

[idrool4greg.livejournal.com]

Thank-you so much for the heads up. I will be saving like there is no tomorrow.

One question, I tried to access from my cell phone... and no dice. Is this an ability that will be lost?

Love the existing site, I am sure I will love the new one.

[0creativity.livejournal.com]

I've never specifically done anything to make the old site work on a cell phone so I'm not sure what exactly is keeping it from working on the new site. I don't have much expertise in the area of designing for cell phones, so I can't promise that this will still be available. I will keep it in mind though and see if I can't find out something, but this is probably going to be low priority, while things like adding a comment system would be high priority.

[boonesfarmgirl.livejournal.com]

Been to the new site. Signed up. Not an author but a voracious reader. Not computer savvy though.

Chelle

[beelikej.livejournal.com]

Hey Matt!

Good to hear from you. I'm so glad you managed to save the fiction part all this time and I always suspected you were working hard to create a better Desert. *applauds you*

I'd love to help out in a more sytemetical manner. Maybe there are others too. I think the favorite-thing is very arbitrary, no matter how many people are saving theirs.
A system I can think of is to give those willing to download and save until transfertime a list of authors by alphabet, so everyone can save a portion of the site.
Is there a particular format that's best for transfering fic (Word? HTML?)

Let me know what I can do; should I start with the A's? I am ready to go as soon as I get home from work;) I will check out the new stuff then as well.

beelikej[at]gmail.com

[bflyw.livejournal.com]

I was thinking the same thing! I am ready to do my part! I'll be happy to take B (or whatever needed)

Matt - I know you have been thinking about it - but I'll have to ask! The fics are saved in a database, right? Isn't there any way you can download the database and import that data into the next database (via excel or something if you need to change the field names....)

Tell me if you need more help!

dagdrommer[at]gmail.com

[0creativity.livejournal.com]

The actual fic is stored in a regular old text file under a folder of the author's name. The title, rating, comments, etc. are stored in the database. I'm really not sure how feasible it is to write a script to transfer the database data over to the new one for a couple of different reasons. It'll be a pain but one way or another we'll get most of it!

[bflyw.livejournal.com]

Okay. So downloading the actual text files won't be a problem, because that's just dl all the folders through FTP.

My guess (after reading your answer to BeeLikeJ and looking at WMTDB again) is that in the individual folders (take BflyW for instance) there is txt files witch are called a something like 4385.txt, and in the database there is a field called sid, and in one of the records there is a sid=4385. So when I click on a link to read my story "Som man reder så ligger man" then I am directed to viewstory.php and the parameter sid=4385 is attached. So then the script look up the text file called 4385.txt under the folder BflyW. Correct?

The problem is of course that in the new system you probably want different names on the txt files, if you still want to use txt files?

How are you going to store the fics in the new system?

Please tell me if you need brainstorming or something! I am not very good with PHP, I am a ASP girl.... but the basics are the same, so I can probably try to help find some ideas to how to use the info in the database (username, and title), and maybe use that in a script to automatically change the names of the txt files, to whatever you need it to be! If you want to do that, that is.... You have of course given this a lot of thought, and I have full confidence in you. It is just an offer to help, not an attempt to take over! :-)

[0creativity.livejournal.com]

I appreciate any help that would make this go smoother! You are exactly right about how eFiction stores and retrieves the files.

The new system works in a similar manner. Instead of storing the fic in a .txt file, it stores it a .xml file. Also, with eFiction each chapter is stored in a separate .txt file and I believe gets its own record in the database. With the new system, there is only one record in the database (for the first chapter), and every chapter is stored in the same .xml file.

The process is basically the same. When a user clicks submit, a record is written to the database, which uses an integer field (I believe, it's been a while since I set up the database) that autoincrements by one as the primary key. Then, that new primary key field is pulled and that number is used as the filename for the .xml file that is written.

Another difference is that eFiction is using MySQL and the new system is using Microsoft SQL Server.

I'm sure that there's some way to more quickly transfer all that data over, but I'm not really sure what it is. If you have any ideas I'd love to hear them!

[bflyw.livejournal.com]

Okay. I'll give this some thought, I might have some ideas :-)
Write me an email at dagdrommer[at]gmail.com, because I have a few more questions. It'll be too much to take here I think.

[tanisafan.livejournal.com]

I'll take on the C's ;)!

[0creativity.livejournal.com]

That's a good idea, too, but I'm just not sure how many people would have time to do that. I've just looked at the way the fic is saved on the server and I could download all of it fairly easy (they're just regular text files stored under a folder with the author's name). The only problem is that these files don't have the title of the fic. I'll see if I can get something to work from that end so people don't have to scramble to save.

[beelikej.livejournal.com]

Hey, I'm willing to do the whole alphabet if needed ;) but it sounds like you already figured out an easier way, you total WebMaster, you.

(Here if you need me:)

[beelikej.livejournal.com]

Matt! I think I have found a way to collect the whole archive in one go as seperate html-files. This includes reviews! (each subject (user/story/review) is saved by the numbers bflyw mentioned)

I used the free Backstreet Browser, which can be downloaded here: http://www.spadixbd.com/backstreet/index.htm

I'm still checking, but I might have managed to save all the fics on my system.

[0creativity.livejournal.com]

Thanks for the link. I'll check it out!

[welleg-fic.livejournal.com]

How exactly do you go about saving the fics? I'm not the most technically literate of people.

And in keeping with the alphabet theme - I can take the Ds once I figure out how to save.

[0creativity.livejournal.com]

I think the easiest way would just be to act like you're going to read a fic, and then when it loads, copy and paste into a word document or something.

[welleg-fic.livejournal.com]

I'll get the D's for now and can grab more later if need be.

[myserenade.livejournal.com]

I'll support the suggestion to save all the fics off site first using some sort of program.
How big is the size of the fics?
If a zip file of each author(those who didn't save their fics themselves) can be created and upload/host it to a file hosting site (eg MediaFire).
You can choose to make the file public/private/password-protected so that the files can be used later for the new site.

Hope everything works out okay!

[mangojunkie.livejournal.com]

Wow that sounds like a lot of work, not that I woudl have any clue of it since this is like a whole new language to me, but hopefully everything will run as smoothly as possible.

This question might sound really sefish and really self centered but when we transfer our own fics over to that ew site, I guess the comments to that fics are lost forever than, aren't they?

ANd one las question, hve you any plans of ever bringing someting like the beta service back? Because I really liked that feature.


Oh ad I sign up for the E's and F's. Just let me know until when they should be saved.
I hope you guys work things out and if you need any help *not that I am really a big help because I usualy know nothing* let me know...

[0creativity.livejournal.com]

It is looking like comments might be lost. I think they are stored directly in the database as opposed to in external files like the fic. This is another one of those things that will depend on whether or not we find a way to transfer the stuff in the database or not.

We'd love to implement a beta system again. That's one of the other plusses of moving to this new system. I wrote it so I know how every part of it works and can more easily add new features and fix bugs.

[wojo62.livejournal.com]

You and Kelly have done an awesome job on the site. My question is this: When the new site is up will we have to re-register?

[0creativity.livejournal.com]

Most likely. We're still trying to figure out if there's an easy way to transfer stuff in the database. If we can find a way to do that, then it won't be necessary. But right now it is looking like you'll probably have to reregister.

[beelikej.livejournal.com]

Me again. I checked out the new site and I love the feel of it! I registered and uploaded pics and that was both very easy and convenient. I also got stuck in one of your stories, which was lovely (The college not-quite AU verse;)

A suggestion: it would be convenient if the chapterselection could -also- be situated at the bottom of the story-page, so you don't have to scroll up when you reach the cliff-hanger;)

A question: I noticed the icons in the gallery are a lower quality compared to the original picture? Is this because of available space/bandwidth? I have come across people using low resolution editions of my icons and I'd rather they are not able to accidently spread those around.

[0creativity.livejournal.com]

Yeah, the chapter selection definitely needs to be at the bottom, too. I'll keep that in mind.

Did the icons look lower quality even when you clicked view original image? What happens is that it's grabbing the pictures and automatically resizing them to 100x100 for the thumbnail view-- even if the original image is 100x100 already. So, the thumbnail view will probably look lower quality. If it still doesn't look right when you click "View Original Image" I'm not sure what the deal is, because all it's doing is uploading them to the server and renaming them.

[beelikej.livejournal.com]

The icons look fine once you view the original. But some people might simply right-click and save the thumbnail, since it already looks a suitable icon-size. A way around that would be to make the thumbnails smaller I suppose, but that will also make them harder to appraise. I just don't like the idea of people not getting the best quality available;)

[nicky69.livejournal.com]

I DL'd the entire site using this free software.

http://pagenest.com/

I've used this software before and it's an excellent free resource and totally safe and free of spyware and other nasties.

If there is enough demand for it, I can upload the archive(all 343MB, 23,294 files) to Megaupload for anyone who wants it.

Lou:)

Wow

[(Anonymous)]

For that feat you are all kinds of awesome. If you do upload it to Megaupload (don't have to for me but if you do) I might actually DL it. That would fit onto a DVD easy.

[kaijinscendre.livejournal.com]

If there is anyway you can put this on Megaupload I would greatly appreciate. I DL'd the page nest software but for some reason cannot get it to work.

[acroarcs.livejournal.com]

This post is getting to be quite old, but hopefully you will receive the email for it. Anyway. The test site that you have up there exposes email addresses for users, and I (as one of those exposed users) would appreciate having the site taken down. It's already been google-indexed. Thanks for whatever you can do.

[0creativity.livejournal.com]

The offending page has been removed.