Radified Community Forums - Search function disabled

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Search function disabled
Feb 12^th, 2008 at 9:46am

Hi.

I'm getting nastigrams from my webhost (Lunarpages) about excessive server usage. Specifically, they claim CPU usage has skyrocketed (to 11%) in the last few days.

Quote:

radified.com

CPU usage: 11.24
Memory usage: 1.53
MySQL usage: 0.0

Unfortunately, I don't have access to their reporting module, so I cannot verify for myself. Yet nothing I can see here suggests such a dramatic increase.

Naturally, they want me to upgrade to a pricier plan. $$$ .. which might be motivation to pad the numbers.

I have temporarily disabled the search function, to see what effect that might have. Also I lowered the "Max messages returned on recent posts by user.." to 10 (originally 30).

I apologize for any inconvenience. Hopefully we'll be able to resolve this quickly. I'm considering moving to a new host.

Rad

IP Logged

NightOwl

Radministrator

Offline

"I tought I saw a puddy
tat..."

Posts: 5826
Olympia, WA--Puget Sound--USA

Back to top

Re: Search function disabled
Reply #1 - Feb 12^th, 2008 at 9:56am

Rad

The forum has operated for a long time without such problems being reported, either by Lunarpages or your previous host--it's hard to imagine why this has suddenly now become a *problem*--Lunarpages should be able to give you more *specific* information on where the usage is occurring--seems like--no?

The number of guests and users has remained fairly constant--the only obvious change I've seen is the listing of the *Search Engines* on the main page--you could disable that function to see if it has any effect--maybe monitoring their presents is spiking the forum software!?

Have you looked at your forum statistics--has there been a big change in the number of site *hits* in the last 3-6 months?

Can you disable *statistics* gathering/reporting by the forum software?

____________________________________________________________________________________________

No question is stupid ... but, possibly the answers are Wink

!

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #2 - Feb 12^th, 2008 at 10:09am

That's my whole point. I see nothing to confirm their numbers.

We upgraded to Y2.2 *after* they moved us to the stabilization server.

Before Y2.2, search engines were part of "guests". (Remember the high number of "guests" we had back in June? .. when we discovered they were from Yahoo search?) Now search engines are separated out in the new reporting.

Reporting requires minimal resources. You know from using your own home machine that searches are what consume resources.

IP Logged

John.

Übermensch

Offline

Posts: 2072

Back to top

Re: Search function disabled
Reply #3 - Feb 12^th, 2008 at 10:28am

I've noticed in the last couple weeks that response time, just going from board to board, often is 4 or 5 seconds. It used to be instantaneous (from here in Southern California). I haven't run pings or tracert's to confirm all this; just my unscientific observation.

I knew you were tweaking the new software, but I sort of wondered if it was a server issue. I also didn't want to just "complain" about response time. The new yabb looks nice!

Were all the crawlers always there? Or just that their presence is shown now?

Ghost4me Ghost 9, 10, 12, 14, 15. Windows XP, Vista, Windows 7

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #4 - Feb 12^th, 2008 at 12:05pm

Yeah, they were here all along, as our research back in June confirms, when we identified many of the "guests" as being from Yahoo search.

The new YaBB v2.2 software now identifies search engines, and separates them out from "guests".

We have been moved to a "stabilization server" which is supposedly less robust than a "production server". (I only know what they tell me,and have no way to verify any of this for myself.)

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #5 - Feb 12^th, 2008 at 12:10pm

The only scenario I can imagine, which might explain recent server spikes is .. search engine queries (from Search engines like Google, not our YaBB search function) return results to the OLD (YaBB v1.4) forum, and that would take searchers to the home page for THESE forums.

(All old Y1.4 URLS are now *forwarded* to the home page of this forum. The old forum, and its threads, have been deleted.)

Searchers might then SEARCH for related terms using THIS forum. Indeed I have done this myself when searching for data with Googe, and then being forwarded to the home page a new forum (not an individual thread) > I then *search* the new forum for the title of the old thread.

Searches are obviously resource intensive. Lunarpages however is citing the YaBB.pl script as the culprit. I am going to ask (over at the YaBB forums) if the search functions runs as a separate script (or if it's cobbled into the YaBB.pl script.

This is also the reason I've disabled the search function here .. to see if my theories hold any water.

But, as you, know, Y1.4 has been "archived" (available, but no longer active) for nearly a YEAR .. so this seems unlikely. The new forums have been up and active since last March.

Because I am seeing nothing here to support their claims of a 10-fold increase is CPU usage. And the fact that money is involved makes me suspicious.

What I'd *really* like to do is to enable the search function only for registered members. But I don't think that's possible. (I will ask.)

IP Logged

John.

Übermensch

Offline

Posts: 2072

Back to top

Re: Search function disabled
Reply #6 - Feb 13^th, 2008 at 12:21pm

Rad wrote on Feb 12^th, 2008 at 12:10pm:

What I'd *really* like to do is to enable the search function only for registered members. But I don't think that's possible. (I will ask.)

I thought that you MUST be logged in to use the search function (of course I can't verify that now because it isn't there, but that is what I recall.)

My unscientific observation is that there has been a noticeable improvement in last two days.

I don't know anything about the YABB data structure but is there a reorg or optimize function for the data? Most databases have something similar because they become rather inefficient over time with adds/deletes.

Ghost4me Ghost 9, 10, 12, 14, 15. Windows XP, Vista, Windows 7

IP Logged

NightOwl

Radministrator

Offline

"I tought I saw a puddy
tat..."

Posts: 5826
Olympia, WA--Puget Sound--USA

Back to top

Re: Search function disabled
Reply #7 - Feb 13^th, 2008 at 2:25pm

Ghost4me

Quote:

I thought that you MUST be logged in to use the search function

Now that you have said that--I think you are correct! If you visit as a *guest*, I think I also remember now that you can not use the *Search* function--must be logged on!

That would also probably remove the *Search Engines* from using CPU cycles for searching--they must open each thread as a *guest*, read and index the key words and phrases, and then move on to the next thread--because they can not be *logged on* to the forum!

____________________________________________________________________________________________

No question is stupid ... but, possibly the answers are Wink

!

IP Logged

John.

Übermensch

Offline

Posts: 2072

Back to top

	Re: Search function disabled Reply #8 - Feb 13^th, 2008 at 2:47pm OK. So then why has the response time improved in the last couple days? Or a better question would be, has the yabb resources and cpu time gone down on the server or some other explanation/change?
	Ghost4me Ghost 9, 10, 12, 14, 15. Windows XP, Vista, Windows 7 IP Logged

MrMagoo

Übermensch

Offline

Resident Linux Guru

Posts: 1026
Phoenix, AZ (USA)

Back to top

Re: Search function disabled
Reply #9 - Feb 13^th, 2008 at 3:45pm

John. wrote on Feb 13^th, 2008 at 12:21pm:

I don't know anything about the YABB data structure but is there a reorg or optimize function for the data? Most databases have something similar because they become rather inefficient over time with adds/deletes.

YaBB uses flat text files (which may explain its intensive CPU use) instead of a database. I don't think there is a way to accomplish the same result as a database optimization. If anything, the data might need to be defraged, but that should have been accomplished with the import/upgrade to 2.2.

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #10 - Feb 14^th, 2008 at 8:11pm

I re-enabled search .. to deliver 10 results. Here are server stats from 3AM today:

Quote:

Hello, Your usage levels is reduced but still high to put back onto a production server. Here is your resource usage for today.

radified.com
CPU usage: 3.66
Memory usage: 1.45
MySQL usage: 0.0

Top Process %CPU 48.0 [YaBB.pl]
Top Process %CPU 22.0 /usr/bin/peArl -- YaBB.pl
Top Process %CPU 21.0 /usr/bin/peArl -- YaBB.pl

According to your account usage your account is not suitable for shared hosting. Please upgrade your account to our dedicated server. Dedicated detail can be found at http://lpdedicated.com/.

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #11 - Feb 14^th, 2008 at 8:50pm

NightOwl wrote on Feb 13^th, 2008 at 2:25pm:

Ghost4me

Quote:

I thought that you MUST be logged in to use the search function

Now that you have said that--I think you are correct! If you visit as a *guest*, I think I also remember now that you can not use the *Search* function--must be logged on!

That would also probably remove the *Search Engines* from using CPU cycles for searching--they must open each thread as a *guest*, read and index the key words and phrases, and then move on to the next thread--because they can not be *logged on* to the forum!

Click the log-out button and you'll see the search function is still available. I have tried to figure out how to limit it to registered members only (logged in), but this doesn't seem possible.

http://www.yabbforum.com/community/YaBB.pl?num=1202841022

IP Logged

NightOwl

Radministrator

Offline

"I tought I saw a puddy
tat..."

Posts: 5826
Olympia, WA--Puget Sound--USA

Back to top

	Re: Search function disabled Reply #12 - Feb 14^th, 2008 at 9:33pm Rad Quote: Click the log-out button and you'll see the search function is still available. Just did! Yup--as I usually do every month or two--I stand corrected !
	____________________________________________________________________________________________ No question is stupid ... but, possibly the answers are ! IP Logged

NightOwl

Radministrator

Offline

"I tought I saw a puddy
tat..."

Posts: 5826
Olympia, WA--Puget Sound--USA

Back to top

Re: Search function disabled
Reply #13 - Feb 14^th, 2008 at 9:49pm

Rad

Okay, so the *Search Engines* may be using the *Search* function since they don't have to be *Registered Users*!

I was going to mention this yesterday, but stopped because I thought the non-registered status would keep the *Search Engines* from having much impact--not sure you noticed this, but MrMagoo mentioned there is a way to keep the *Search Engines* out--you could limit them for a day or two to see if they are the source of high usage--see here: http://radified.com/cgi-bin/yabb2/YaBB.pl?num=1201321547/15#21

Quote:

I know of one more trick that might help - not sure if you already considered it. You could add a robots.txt file to tell the search engines to stay away from the forum. Obviously, it would keep the forum from getting indexed, which I'm sure you want, but you could see if it has any effect on resource usage.

Probably not what you want to do *permanently*--but, maybe just long enough to see if that's the issue!

____________________________________________________________________________________________

No question is stupid ... but, possibly the answers are Wink

!

IP Logged

Rad

Radministrator

Offline

Sufferin' succotash

Posts: 4090
Newport Beach, California

Back to top

Re: Search function disabled
Reply #14 - Feb 14^th, 2008 at 10:19pm

Yeah, but I just wanted to see if it had any effect on server resources. I mean, traffic via search engine queries is the life blood of *any* site. If we're going to throttle search engine access we might as well shut down the site. People pay big bucks to SEO firms (Search Engine Optimization) to *increase* their standing with search engines.

I will move the site before I'd kill search engine access. We have lots of high-quality content here, which people using a search engine query deserve access to.

I killed the search function for a day or two to see what happened. and LP teld me our usage is down, but not enuf for us to stay shared. In fact, they *still* want me to go dedicated. So I have my answer.

I'm pretty sure (not positive) we're going VPS. Altho LP says I don't qualify for their VPS, so we'll find someone else. There's no way we need a dedicated server .. much as my ego would like to believe that.

Here's a good question:

http://www.yabbforum.com/community/YaBB.pl?num=1203049807/0

IP Logged

	Welcome, Guest. Please Login