Welcome, Guest. Please Login
 
  HomeHelpSearchLogin FAQ Radified Ghost.Classic Ghost.New Bootable CD Blog  
 
Pages: 1 2 
Send Topic Print
Trouble Backing-up Individual Forum Threads + Problems with YaBB (Read 48444 times)
Rad.Test
Technoluster
***
Offline


Rad's non-Admin test-profile
in Firefox

Posts: 108


Back to top
Trouble Backing-up Individual Forum Threads + Problems with YaBB
Feb 10th, 2008 at 9:48am
 
I'm having a little trouble backing-up the thread. Think I found the problem .. that either my FTP client (have tried both Cute.Pro & FileZilla so far) .. are TRUNCATING my list of displayed entries to 2000 .. but we have ~3500 threads. Here's code from FileZila:

Quote:
Response:      257 "/public_html/cgi-bin/yabb2/Messages" is your current location
Command:      PASV
Response:      227 Entering Passive Mode (216,227,209,174,99,121)
Command:      LIST
Response:      150 Accepted data connection
Response:      226-ASCII
Response:      226-Options: -a -l
Response:      226 Output truncated to 2000 matches
Status:      Directory listing successful

More info via Gooogle search:

http://www.smartftp.com/support/kb/226-output-truncated-f197.html

Quote:
Some servers are truncating the output of the directory listing to a certain number of lines (files / folders). The server cannot display more than 2000 items in file listings. The reason is to prevent a DoS attack where a user could just issue a few "ls" commands to the FTP server and tie up huge amounts of bandwidth and CPU resources.
A good advise is to increase the depth of the directory structure by creating additional sub folders and moving the files from the main folder into the newly created sub folders.

This is a problem, since I can't move these files.

http://help.globalscape.com/help/cuteftppro7/truncated_lists.htm

Quote:
If you connect to a server and you see 266-Options: -1 or 226 Output truncated to 2000 matches, it means the server cannot display more than 2000 items in file listings. This is a limitation of the server and not CuteFTP.

Lotsa talk about this on the Web. Right now, I see no option but to back-up and download THE ENTIRE SITE (>1 gig), which will take a few hours.
 
 
IP Logged
 

MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: (In?)Security of Linux
Reply #1 - Feb 10th, 2008 at 3:14pm
 
I split the topic since this this could easily morph into a whole different discussion.  I just realized what you've been saying this whole time - that each thread is contained in a text file.  I didn't realize that before.  That is not the way most of the forum software I have worked with operates...

I'm used to the software storing the threads in a SQL database, usually MySQL.  Backing it up is then dead simple - run a database dump to a single file and then stick the file someplace safe.  Besides backing up, I wonder if storing the threads in flat text files has any effect on CPU resources...  I mean, if that was a good way to do things, then why did they invent databases...

At a minimum, I would think the database could take advantage of some caching.
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: (In?)Security of Linux
Reply #2 - Feb 10th, 2008 at 3:28pm
 
As I suspected, I'm finding some web hosting providers won't allow YaBB specifically because of its use of flat text files.  They suggest a database driven forum instead:

Flat File Database Driven Forums
Due to their use of flat files instead of databases, Premier Website Solutions does not allow the use of UBB or YABB bulletin boards. Use of these boards for large or very active forums, results in system performance degradation, and cannot be allowed in our shared server environment. Premier Website Solutions suggests using VBulletin or phpBB as a more efficient, database-driven alternative to these bulletin boards.

http://www.premierwebsitesolutions.com/TOS/resourceabuse.html
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: (In?)Security of Linux
Reply #3 - Feb 10th, 2008 at 3:36pm
 
YaBB and Ultimate Bulletin Board Forums tend to require significant system resources and therefore cannot be permitted. WebIntellects recommends vBulletin or phpBB.

http://www.webintellects.com/about/termsandconditionsofuse.htm

Due to their use of flat files instead of databases, Warez-Server does not allow the use of UBB or YABB bulletin boards.
Use of these boards for large or very active forums, results in system performance degradation, and cannot be allowed in our shared hosting environment.
We suggest using VBulletin or phpBB or Invision Power Board as a more efficient, database-driven alternative to these bulletin boards.

http://warez-host.info/forum/showthread.php?t=3

 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: (In?)Security of Linux
Reply #4 - Feb 10th, 2008 at 3:50pm
 
After some non-scientific googling, it looks like vBullitin is the favorite forum of the web, but it costs $$.  phpBB is popular and efficent on resources, but very insecure.  SMF is fast, customizable, and free.  Some references seem to indicate that it is based on YaBB but rewritten to use PHP and MySQL.  There is a converter to go from YaBB 2.1 to SMF.  I don't know if it works for 2.2.
 
WWW  
IP Logged
 
Rad.Test
Technoluster
***
Offline


Rad's non-Admin test-profile
in Firefox

Posts: 108


Back to top
Re: (In?)Security of Linux
Reply #5 - Feb 10th, 2008 at 5:50pm
 
Thanks. Nice info.

Suks cuz I just got this version dialed in.
 
 
IP Logged
 

Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: (In?)Security of Linux
Reply #6 - Feb 10th, 2008 at 8:36pm
 
Okay, been researching YaBB->to->SimpleMachinesForum convertors.

http://www.simplemachines.org/community/index.php?topic=214612.0

They have been tweaking the "YaBB v2.1-to-SMF 1.1.4" convertor to get it to work with YaBB v2.2 (which we use).

Some report success, tho seems there are still problems with converting polls .. altho we don't use many polls, and those we have used would not represent a great loss.

Actually, my biggest balk comes from trying to make it look *pretty* as this forum. I mean, I really like the way this forum LOOKs now.

I have CSS design skillz now, so I guess I could muscle my way thru it.

Also SMF 2.0 beta2 was just released Jan 20:

http://www.simplemachines.org/community/index.php?topic=217789.0

SMF 1.1.4 is the current stable version. It was released back in Sept:

http://www.simplemachines.org/community/index.php?topic=196380.0

Maybe SMF 2.0 might be the time to convert/install. Looking fwd, it would be easier to upgrade to SMF 2.01 from SMF 2.0 .. than from SMF 1.1.4 to SMF 2.0

Or maybe we archive YaBB v2.2 and do a clean/fresh install of SMF 2.0??
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: (In?)Security of Linux
Reply #7 - Feb 10th, 2008 at 11:55pm
 
I don't think archiving it would be the best route.  Then it will still be there using up resources.  You have several years of quality discussions that your lurkers are going to be digging through to try to glean advice.  Converting all of that to something database driven is what is going to give you relief from high resource usage.

I don't think making it look good will be a huge problem.  There are lots of themes.  It would probably be better to build a theme than to muscle through the CSS manually, since it is easy to revert to the basic theme if your custom one gets broken by an upgrade (which unfortunately seems to be common with SMF.)
 
WWW  
IP Logged
 
Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: Trouble Backing-up Individual Forum Threads
Reply #8 - Feb 11th, 2008 at 2:34pm
 
My thought regarding "archiving" Y2.2 (in other words, leaving it up, but eliminating the ability to register or post new messages) .. comes from the fact that many current threads/posts reference (contain links to) *other* threads/posts.

If I converted the Y2.2 forum to another program (PHP/MySQL-based), all these links would no longer be valid (dead) .. which would suk.

Leaving Y2.2 up as an "archive forum" (I don't know what other term to use) would solve this (linkage) problem.

We already have some posts that reference threads started in Y1.4, which are no longer valid. (All links to Y1.4 takes readers back to the Y2.2 forum home page).

The last thing I want to do invalid all the links in the forums, which point to other links .. since I just did that last March, when we upgraded to Y2.1. (There was no way to avoid it.)

Eventually the archive forum would begin to lose relevance. I didn't feel so bad about doing away with the Y1.4 forum, when it had been archived for 9 months. But to eliminate it right off the bat would made it more difficult to find valuable info.
 
WWW  
IP Logged
 
Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #9 - Feb 11th, 2008 at 2:39pm
 
I asked about YaBB being banned by some web-hosts over at the YaBB forums. The lead developer (Cory) responded (altho I can no longer find the thread visible on the board .. had to *search* for it):

http://www.yabbforum.com/community/YaBB.pl?num=1202570554/6#6

Quote:
Most web hosts blindly "ban" YaBB and similar programs for three reasons:

1. Bad rep for Perl programs
2. Bad rep for flat file databases and
3. Bad rep because of some of the YaBB 1 releases.

Any host who has to limit your alloted CPU usage and memory usage is not a good host. They should properly manage the number of accounts hosted on one server based on the server capabilities and the projected traffic of each site.  If their server is overloaded due to abuse by a certain again, deal with that account, but in most cases you just get an additional server or beef up the existing one.

Indeed YaBB can be server resource intensive.  However, YaBB 2 is much better than it's predecessor version (YaBB 1) at using FAR less memory and even CPU.  When YaBB executes, it can spike in memory usage and load for certain actions in the forum, but most typical actions are pretty quick.  Yes, YaBB 2 still has its issues because of the flat file database.

When YaBB really appears to be at its worst is when the server is overloaded - this is typical of ANY Perl or PHP application, but especially those not using an optimized database system.  I see this here whenever other websites are getting really high web (httpd) traffic, and the server is overloaded.  The first thing you'll see is a lot of YaBB processes near the top, followed by the httpd traffic.

This is only because YaBB is then given second priority on the CPU and takes longer for it to process because there is so little room for it left.  Everyone automatically assumes YaBB is the culprit, but it's often not.  You'll also see a lot of spiking PHP and MySQL processes during this same period for the same reasons.

However, where everyone gets it wrong is total resource usage.  They assume PHP uses less.  Where they are wrong is that MySQL (being the typical database used by PHP programs) has a huge memory usage as a separate process.

Nobody ever considers the usage of PHP + MySQL as a combined memory and CPU usage for that program.  The same will be seen when we go to YaBB 3, which will support any database type (MySQL being default).

For more information, see this:  http://www.yabbisback.com/media/YaBB_Versus.pdf

Also, a user of YaBB is trying to kick start an optimization project for YaBB 2:
http://www.yabbforum.com/community/YaBB.pl?board=yabb2optimize

 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #10 - Feb 11th, 2008 at 4:45pm
 
I understand your thinking on archiving the forum.  I actually kind of agree.  After I made that post yesterday I started thinking that coverting might not go too well and it would be far better to have YaBB archived than to loose all that info.  Also, most of the activity is in more recent posts, so the new forum should bear the brunt, which is what you want.  Plus you would have plenty of time to install and dial in the new forum before moving everyone over.  My only concern is search engines spidering through the old forum.

The response from the YaBB developer is interesting.  I understand his frustration with hosts that overload their shared servers rather than growing with customer demand.  I also agree with his point that YaBB performance takes a bigger hit as the server gets overlaoded.  On the other hand, the hosting provider can't allow someone on the cheapest plans to take up more than a fair share of the server.  They are just making sure you and everyone else on your server gets what they pay for, which seems fair to me.  I think that LunarPages specifically has done a good job of maintianing a high level of customer service throughout this resource usage issue.  Some hosts would have cut you off or demanded you either git rid of YaBB or upgrade your plan, but they simply moved your site to a different server without interrupting your traffic and then gave you the information you needed to deal with it however you see fit.

I disagree with his thoughts on PHP/MySQL.  PHP runs as an Apache module and only uses system resources when it is called.  MySQL is likely already running anyway because many other users on your shared server are using it, so the additional memory it would consume due to your forum wouldn't be nearly as much as if you had to load the SQL server just for yourself.  Even then, I've found that MySQL is pretty frugal with its resource use - I run it all the time on old dusty computers at home with only a few MB of ram.

He seems to agree (without outright saying so) that flat text files use far more CPU resources than a database would, and the use of flat text files instead of a database is the stated reason many hosts don't allow YaBB. 

I really think that the small increase in memory usage would be more than worth the dramatic decrease in CPU usage.  I think that you may be able to hold off on moving to VPS if you switch your forum, but would recommend the change even if you do upgrade to VPS.

On the other hand, if you want to stick with YaBB I can certianly make it work on a VPS server.  I could set up some graphs to keep an eye on resource usage and write some scripts to make sure it gets backed up properly.
 
WWW  
IP Logged
 

NightOwl
Radministrator
*****
Offline


"I tought I saw a puddy
tat..."

Posts: 5826
Olympia, WA--Puget Sound--USA


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #11 - Feb 12th, 2008 at 9:43am
 
Rad

(Perhaps *off topic*--split this out MrMagoo, if you wish.)

Regarding you post #8 above:

Quote:
If I converted the Y2.2 forum to another program (PHP/MySQL-based), all these links would no longer be valid (dead) .. which would suk.

Leaving Y2.2 up as an "archive forum" (I don't know what other term to use) would solve this (linkage) problem.

We already have some posts that reference threads started in Y1.4, which are no longer valid. (All links to Y1.4 takes readers back to the Y2.2 forum home page).

The last thing I want to do invalid all the links in the forums, which point to other links .. since I just did that last March, when we upgraded to Y2.1. (There was no way to avoid it.)

I think you might be able to *fix* this, if you want!  In one of the other threads about the new (2.2) forum software features, I asked if the YaBB folks had an upgrade converter for when folks went from version 1.x to 2.x because of the link structure change used by the newer software--that question, I think, got lost!

But, now that you have discovered that all the threads are *txt* file based, it occurred to me that you should be able to do a *search and replace* file search for multiple files--I think you could very quickly upgrade all the *old* style forum links to the *new* style!

Did a Google search:  search and replace in multiple files.

There appears to be multiple trial shareware and some freeware that would probably work!

I keep my frequently used links in a WordPad file (text file), and when you upgraded last year from 1.4 to 2.1--I used WordPad's menu item *Edit > Replace* function to update all the links in that file:

Find what:  YaBB/YaBB.cgi?board=general;action=display;num=

Replace with:  yabb2/YaBB.pl?num=

You would probably have to double check my search and replace strings above to make sure they work as advertised on the forum thread text files--but it should work pretty fast--if it works!  (As always--I'm sure you will do a *backup* first  Wink !)  Might have to do a download of the files, first, to get them on a local computer, do the search and replace, and then upload the updated files--may have to put the forum in *maintenance* mode while doing it--don't know the details!!!

All the *old* threads are still present in the *new* forum--just the link structure is wrong!

You would not need to keep around the old v1.4 forum as you did in the past--and perhaps that would set everything up for a transition to alternate forum software, if they have a conversion tool for upgrading the links to the new software!
 

____________________________________________________________________________________________

No question is stupid ... but, possibly the answers are Wink !
 
IP Logged
 
Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #12 - Feb 12th, 2008 at 4:12pm
 
Well, that would be some serious searching and replacing .. given we have more than 4K threads .. with ~33K posts.

Seems I'd have to download all 4K threads, open each .. run search & replace, then re-up them all. No?
 
WWW  
IP Logged
 
Rad
Radministrator
*****
Offline


Sufferin' succotash

Posts: 4090
Newport Beach, California


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #13 - Feb 12th, 2008 at 4:18pm
 
Something else to consider > YaBB v3.0 .. which is currently in development .. will support databases. I'm guessing they would have a convertor .. that would change all the flat *.txt files into database entries. Here's some screenshots & discussion:

http://www.yabbforum.com/community/YaBB.pl?num=1154855907

I'm admittedly weak on databases. I merely know how to create one (in cPanel), give it a name, apply a user name & password. The Movable Type blog uses a database (MySQL). That much I find easy. And I've set them up using phpMyAdmin in WAMP.
 
WWW  
IP Logged
 
MrMagoo
Übermensch
*****
Offline


Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)


Back to top
Re: Trouble Backing-up Individual Forum Threads + Problems with YaBB
Reply #14 - Feb 12th, 2008 at 6:50pm
 
I saw that.  I just don't know if they have any kind of release date for it yet, especially given that they are still working out the bugs on 2.2.  I was under the impression that you could only spend a limited amount of time on the "stabilization server."
 
WWW  
IP Logged
 
Pages: 1 2 
Send Topic Print