Hi.

I'm getting nastigrams from my webhost (Lunarpages) about excessive server usage. Specifically, they claim CPU usage has skyrocketed (to 11%) in the last few days.


Quote:

radified.com CPU usage: 11.24 Memory usage: 1.53 MySQL usage: 0.0

Unfortunately, I don't have access to their reporting module, so I cannot verify for myself. Yet nothing I can see here suggests such a dramatic increase.

Naturally, they want me to upgrade to a pricier plan. $$$ .. which might be motivation to pad the numbers.

I have temporarily disabled the search function, to see what effect that might have. Also I lowered the "Max messages returned on recent posts by user.." to 10 (originally 30).

I apologize for any inconvenience. Hopefully we'll be able to resolve this quickly. I'm considering moving to a new host.

Rad

Rad

The forum has operated for a long time without such problems being reported, either by Lunarpages or your previous host--it's hard to imagine why this has suddenly now become a *problem*--Lunarpages should be able to give you more *specific* information on where the usage is occurring--seems like--no?

The number of guests and users has remained fairly constant--the only obvious change I've seen is the listing of the *Search Engines* on the main page--you could disable that function to see if it has any effect--maybe monitoring their presents is spiking the forum software!?

Have you looked at your forum statistics--has there been a big change in the number of site *hits* in the last 3-6 months?

Can you disable *statistics* gathering/reporting by the forum software?

That's my whole point. I see nothing to confirm their numbers.

We upgraded to Y2.2 *after* they moved us to the stabilization server.

Before Y2.2, search engines were part of "guests". (Remember the high number of "guests" we had back in June? .. when we discovered they were from Yahoo search?) Now search engines are separated out in the new reporting.

Reporting requires minimal resources. You know from using your own home machine that searches are what consume resources.

I've noticed in the last couple weeks that response time, just going from board to board, often is 4 or 5 seconds.  It used to be instantaneous (from here in Southern California).  I haven't run pings or tracert's to confirm all this; just my unscientific observation.

I knew you were tweaking the new software, but I sort of wondered if it was a server issue.  I also didn't want to just "complain" about response time.  The new yabb looks nice!   :)

Were all the crawlers always there?  Or just that their presence is shown now?

Yeah, they were here all along, as our research back in June confirms, when we identified many of the "guests" as being from Yahoo search.

The new YaBB v2.2 software now identifies search engines, and separates them out from "guests".

We have been moved to a "stabilization server" which is supposedly less robust than a "production server". (I only know what they tell me,and have no way to verify any of this for myself.)

The only scenario I can imagine, which might explain recent server spikes is .. search engine queries (from Search engines like Google, not our YaBB search function) return results to the OLD (YaBB v1.4) forum, and that would take searchers to the home page for THESE forums.

(All old Y1.4 URLS are now *forwarded* to the home page of this forum. The old forum, and its threads, have been deleted.)

Searchers might then SEARCH for related terms using THIS forum. Indeed I have done this myself when searching for data with Googe, and then being forwarded to the home page a new forum (not an individual thread) > I then *search* the new forum for the title of the old thread.

Searches are obviously resource intensive. Lunarpages however is citing the YaBB.pl script as the culprit. I am going to ask (over at the YaBB forums) if the search functions runs as a separate script (or if it's cobbled into the YaBB.pl script.

This is also the reason I've disabled the search function here .. to see if my theories hold any water.

But, as you, know, Y1.4 has been "archived" (available, but no longer active) for nearly a YEAR .. so this seems unlikely. The new forums have been up and active since last March.

Because I am seeing nothing here to support their claims of a 10-fold increase is CPU usage. And the fact that money is involved makes me suspicious.

What I'd *really* like to do is to enable the search function only for registered members. But I don't think that's possible. (I will ask.)

Rad wrote on Feb 12^th, 2008 at 12:10pm:

What I'd *really* like to do is to enable the search function only for registered members. But I don't think that's possible. (I will ask.)

I thought that you MUST be logged in to use the search function (of course I can't verify that now because it isn't there, but that is what I recall.)

My unscientific observation is that there has been a noticeable improvement in last two days.
 
I don't know anything about the YABB data structure but is there a reorg or optimize function for the data?  Most databases have something similar because they become rather inefficient over time with adds/deletes.

Ghost4me


Quote:

I thought that you MUST be logged in to use the search function

Now that you have said that--I think you are correct!  If you visit as a *guest*, I think I also remember now that you can not use the *Search* function--must be logged on!

That would also probably remove the *Search Engines* from using CPU cycles for searching--they must open each thread as a *guest*, read and index the key words and phrases, and then move on to the next thread--because they can not be *logged on* to the forum!

OK.  So then why has the response time improved in the last couple days?  Or a better question would be, has the yabb resources and cpu time gone down on the server or some other explanation/change?

John. wrote on Feb 13^th, 2008 at 12:21pm:

I don't know anything about the YABB data structure but is there a reorg or optimize function for the data?  Most databases have something similar because they become rather inefficient over time with adds/deletes.

YaBB uses flat text files (which may explain its intensive CPU use) instead of a database.  I don't think there is a way to accomplish the same result as a database optimization.  If anything, the data might need to be defraged, but that should have been accomplished with the import/upgrade to 2.2.

I re-enabled search .. to deliver 10 results. Here are server stats from 3AM today:


Quote:

Hello, Your usage levels is reduced but still high to put back onto a production server. Here is your resource usage for today. radified.com CPU usage: 3.66 Memory usage: 1.45 MySQL usage: 0.0 Top Process %CPU 48.0 [YaBB.pl] Top Process %CPU 22.0 /usr/bin/peArl -- YaBB.pl Top Process %CPU 21.0 /usr/bin/peArl -- YaBB.pl According to your account usage your account is not suitable for shared hosting. Please upgrade your account to our dedicated server. Dedicated detail can be found at http://lpdedicated.com/.

NightOwl wrote on Feb 13^th, 2008 at 2:25pm:

Ghost4me Quote: I thought that you MUST be logged in to use the search function Now that you have said that--I think you are correct!  If you visit as a *guest*, I think I also remember now that you can not use the *Search* function--must be logged on! That would also probably remove the *Search Engines* from using CPU cycles for searching--they must open each thread as a *guest*, read and index the key words and phrases, and then move on to the next thread--because they can not be *logged on* to the forum!

Click the log-out button and you'll see the search function is still available. I have tried to figure out how to limit it to registered members only (logged in), but this doesn't seem possible.

http://www.yabbforum.com/community/YaBB.pl?num=1202841022

Rad


Quote:

Click the log-out button and you'll see the search function is still available.

Just did!  Yup--as I usually do every month or two--*I stand corrected*  ;) !

Rad

Okay, so the *Search Engines* may be using the *Search* function since they don't have to be *Registered Users*!

I was going to mention this yesterday, but stopped because I thought the non-registered status would keep the *Search Engines* from having much impact--not sure you noticed this, but MrMagoo mentioned there is a way to keep the *Search Engines* out--you could limit them for a day or two to see if they are the source of high usage--see here:  http://radified.com/cgi-bin/yabb2/YaBB.pl?num=1201321547/15#21


Quote:

I know of one more trick that might help - not sure if you already considered it.  You could add a robots.txt file to tell the search engines to stay away from the forum.  Obviously, it would keep the forum from getting indexed, which I'm sure you want, but you could see if it has any effect on resource usage.

Probably not what you want to do *permanently*--but, maybe just long enough to see if that's the issue!

Yeah, but I just wanted to see if it had any effect on server resources. I mean, traffic via search engine queries is the life blood of *any* site. If we're going to throttle search engine access we might as well shut down the site. People pay big bucks to SEO firms (Search Engine Optimization) to *increase* their standing with search engines.

I will move the site before I'd kill search engine access. We have lots of high-quality content here, which people using a search engine query deserve access to.

I killed the search function for a day or two to see what happened. and LP teld me our usage is down, but not enuf for us to stay shared. In fact, they *still* want me to go dedicated. So I have my answer.

I'm pretty sure (not positive) we're going VPS. Altho LP says I don't qualify for their VPS, so we'll find someone else. There's no way we need a dedicated server .. much as my ego would like to believe that.

Here's a good question:

http://www.yabbforum.com/community/YaBB.pl?num=1203049807/0

Rad

Hmmm...just thought things through a little bit more--I guess if the *Search* function was off all together--then the *Search Engines* couldn't be doing any searching either--but they were still here the whole time--so I guess it doesn't make any sense to try and restrict them--they appear *not* to be using the *Search* function--actually!

Ya, but the fact remains that there are still more search engines using the forum than guests, so I'm sure they are having an impact.  Limiting them might be a good experiment to help us understand the resource usage.  Granted, it solves nothing in the end, since Rad has said he would rather move the site then limit the search engines for any extended period.

There are usually more guests than SE's. Only rarely do # of SE's exceed guests. During normal business hours, # of guests seems highest.

Right now (Friday 9:30 PM) there are 33 guests. 9 SEs.

The most defining result would be > our usage drops to acceptable (for Shared) if we bar SE's. But I would still move the site .. than leave SE's banned.

That would be a nice data-point to have. But I don't want to exclude SE's and then have them mark our forums as "off-limits" .. not knowing when/if they'd return.

Just noticed this eve that Yahoo has scaled way back. They are usually here like rabbits. I'll ask LP for a usage number for today .. to see if (naturally) reduced SE's here are having a noticeable effect (even *with* the Search function re-enabled).

Rad

Tonight at approx. 10:45 pm--Yahoo SE's are gone--and only 13 total SE's visiting the forum (I've been amazed--Google has just one--Yahoo has been up to 25-30 and Microsoft up to 15-20--how can Google use just one and the other guys have to use so many?!--how's your server usage now?!

It's 11:30 PM. I see 4 Yahoo Crawlers. Sent Lunarpages a query for server stats 2 hours ago. Still haven't heard back.

Quote:

CPU usage: 2.25 Memory usage: 1.63 MySQL usage: 0.0

7am saturday. no yahoo crawlers. that's a first.

As I understand it from reading this thread, YABB structure is a large flat file, not a hierarchical or  indexed database.  So when a guest (or user) searches for an item, and selects "all posts", the search function must read ALL threads (the entire history of Rad forum) until it satisfies the 10 results or until it reaches the beginning of time date/thread (about 35,000 posts).  That's a lot of i/o and memory and cpu, not even counting how lengthy each post might be.

That's not a problem if you spell your search term correctly, but if you don't or you have a couple AND operatives, you easily cause yabb to read through everything.

Is this possibly a contributing factor?

Ghost4me

I'm sure it is...but completely disabling the Search function only helped, but did not resolve the usage problems--so there must be something else going on--as well!

NightOwl wrote on Feb 16^th, 2008 at 10:06am:

I'm sure it is...but completely disabling the Search function only helped, but did not resolve the usage problems--so there must be something else going on--as well!

Other than relying on LP to provide their summary statistics, it will probably take some kind of internal YABB trace or YABB internal statistics to determine what is going on.  Maybe some YABB support or expert can help.  I doubt that this board is the first or only one to question how resources are being consumed.

Interesting problem; sorry I don't have any solutions, just theories.  I looked this morning and there were 35 guests, so I just assumed they were all searching and reading, since they obviously weren't posting.

Then there's the possibility that they are feeding me padded numbers .. since money is at stake. (Higher resource usage leads to pricier hosting plans.)

Note that I don't have access to their reporting module. I only get what they give me. So the process is far from transparent and not without incentive for them to inflate the numbers.

It also raised my eyebrows when one of their tech sppt ppl started offering me pricing deals, telling me what it would cost to go different paths .. when I had already had these discussions with their Sales people .. and the sppt guy was giving me way-costlier numbers than I got from the Sales people.

And they've given me some strange results .. such as one report claiming PHP was one of my top processes. When I questioned it, they later claimed the report was inaccurate. (Guess so, since I don't use PHP.)

.. and another claiming the *.ico (tiny icon graphic) was also a top process. I twice asked them about that, but they still have never responded to my queries.. made like they never heard my question.

Just last night they sent me a report with another site's domain name ( bernos.org ) .. only to say later that the stats were accurate, but the domain name was not.

So the process has been less-than-perfect, .. and note that we received no usage complaints back in June .. when we had record traffic here at the forums (nearly 300).

When I recently began to discuss the process to upgrade to a VPS acct, LP told me our usage exceeds their guidelines for VPS, and that I needed a dedicated server. So something about this fails the smell test.

Well, lots of "facts" being thrown out there.  As you correctly said, "Who to believe?".  It may not have anything to do at all with search and can easily be some other completely unrelated problem--even a problem on the SERVER end , or how user clients are handled, and nothing to do with the rad forum.  Or some quirky yabb setting.

It's easier to tell a customer "just get a faster pc" or "just upgrade to xyz..." than to delve into the real problem.

Be vigilant.  Trust no one.   :)

Yahoo! crawlers are back (noon). I see 27 .. out of 44 total SE's.

Here's another oddity:


Quote:

---- TECH RESPONSE STARTS HERE ---- Dear Customer, Since the stats numbers are averages for the day, it is better to wait 24 hours in order to see if the changes reduced the load. Please reply to this ticket in 24hrs so we can check your account usage. If you have any questions we will gladly assist you. Thank you for the cooperation. If you have any further questions, please feel free to contact us.

It's like they don't want you to have too much information. Weird, huh? Certainly more data points can't *hurt*. (Or can they?)

Is it possible we are blaming search engines where the problem is somewhere else?

Very quiet this morning and yet I saw 5 second delay just getting from Home to Ghost 10 board; of course yabb that would have to display the 20 thread titles, but that's all.  There are few users/guests and even fewer search engines using forum:

Members: 3
Amish., Rad.Test, NightOwl
Guests: 21
Search Engines: 10
Yahoo! Crawler (1), Microsoft Live Search (9)

LP has said the stabilization server is not up to the standards of a "production server". While all I know is what they tell me, and they could be telling me anything.

It's always good to run a quick trace when you see such long delays. Sometimes, when I thought the problem was with the site, it was actually with my ISP.

Right now, for example, I am pinging 17ms to the Rad server, which is beautiful, The highest is actually with my ISP.


Quote:

NeoTrace Trace  Version 3.25  Results Target: radified.com Date: 2/17/2008 (Sunday), 7:31:27 AM Nodes: 10 Node Data Node Net Reg IP Address      Location            Node Name   1   -   -                             Anaheim   2   1   -                            Unknown               3   2   1                            Irvine   4   3   1 67.114.50.2     Irvine              dist4-vlan50.irvnca.sbcglobal.net   5   4   1 151.164.42.7    Kansas City         bb1-p2-1.ksc2mo.sbcglobal.net   6   4   1 151.164.191.225 Los Angeles         ex1-p14-0.eqlaca.sbcglobal.net   7   5   2 206.223.123.62  Los Angeles         eqix-peer.lax01.mzima.net   8   6   2 64.235.224.182  Los Angeles         xe1-0.cr01.lax02.mzima.net   9   7   2 216.193.192.66  Los Angeles         ge1-lunarpages.cust.lax02.mzima.net  10   8   3 216.227.209.174 Fullerton           arneb.lunarpages.com Packet Data Node High Low  Avg  Tot  Lost   1    0    0    0    1    0   2    3    3    3    1    0   3   15   15   15    1    0   4   17   17   17    1    0   5   67   67   67    1    0   6   17   17   17    1    0   7   31   31   31    1    0   8   23   23   23    1    0   9   22   22   22    1    0  10   17   17   17    1    0 NeoTrace Copyright ©1997-2001 NeoWorx Inc

Something is weird with the network when I'm being sent to (way out to) Kansas City to get to LA. But I see no delays. Seems pretty zippy from here (Newport Beach).

That said, I would imagine LP doesn't want our experience on the stabilization server to be very pleasant, or we'd have no incentive to ever move off.

Rad wrote on Feb 17^th, 2008 at 9:29am:

LP has said the stabilization server is not up to the standards of a "production server". While all I know is what they tell me, and they could be telling me anything. It's always good to run a quick trace when you see such long delays. Sometimes, when I thought the problem was with the site, it was actually with my ISP. Right now, for example, I am pinging 17ms to the Rad server, which is beautiful, The highest is actually with my ISP. Something is weird with the network when I'm being sent to (way out to) Kansas City to get to LA. But I see no delays. Seems pretty zippy from here (Newport Beach). That said, I would imagine LP doesn't want our experience on the stabilization server to be very pleasant, or we'd have no incentive to ever move off.

I'm not seeing long ping times either.  I wouldn't assume that the pauses are network issues.  Just as likely (maybe more than likely) that the LP server is temporarily busy doing something else.  It could be THEIR problem, not a Radified problem.

I guess Radified forum is guilty unless proven innocent!   >:(

I don't think it is a network problem (just ran this one):


Quote:

Microsoft Windows [Version 6.0.6000] Copyright (c) 2006 Microsoft Corporation.  All rights reserved. C:\Users\John>ping radified.com Pinging radified.com [216.227.209.174] with 32 bytes of data: Reply from 216.227.209.174: bytes=32 time=15ms TTL=55 Reply from 216.227.209.174: bytes=32 time=13ms TTL=55 Reply from 216.227.209.174: bytes=32 time=19ms TTL=55 Reply from 216.227.209.174: bytes=32 time=15ms TTL=55 Ping statistics for 216.227.209.174:    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:    Minimum = 13ms, Maximum = 19ms, Average = 15ms C:\Users\John>tracert radified.com Tracing route to radified.com [216.227.209.174] over a maximum of 30 hops:  1    <1 ms    <1 ms    <1 ms    2    14 ms     8 ms     *     ....cox.net  3     9 ms     6 ms     *     68.4.15.65  4    16 ms     9 ms     *     wsip-64-58-133-105.oc.oc.cox.net [64.58.133.105]  5    12 ms    10 ms     *     ip68-4-14-121.oc.oc.cox.net [68.4.14.121]  6    12 ms    11 ms     *     rsmtdsrj02-ge600.0.rd.oc.cox.net [68.4.14.213]  7    13 ms    13 ms     *     langbbrj01-ae2.0.r2.la.cox.net [68.1.0.107]  8    20 ms    17 ms     *     xe1-0.cr01.lax02.mzima.net [64.235.224.182]  9    25 ms    18 ms     *     ge2-lunarpages.cust.lax02.mzima.net [72.37.172.26] 10    15 ms    12 ms     *     arneb.lunarpages.com [216.227.209.174] 11    13 ms    13 ms     *     arneb.lunarpages.com [216.227.209.174] 12    13 ms    13 ms     *     arneb.lunarpages.com [216.227.209.174] 13    12 ms    13 ms     *     arneb.lunarpages.com [216.227.209.174] 14    12 ms    13 ms     *     arneb.lunarpages.com [216.227.209.174] 15    14 ms    13 ms     *     arneb.lunarpages.com [216.227.209.174]

13ms .. you must live next door to the Lunarpages data center.

Rad.Test wrote on Feb 17^th, 2008 at 6:08pm:

13ms .. you must live next door to the Lunarpages data center.

I have a high speed fiber connection to LP:   ;D

www.hathawaypr.com/images/tinCan.jpg

I am going to crank up search functions returns .. in a few days or so .. just wanna get a handle on our server resources first .. so I know where I baseline lies.

Cranked up our search results to 30, which I thing is the default.

WiredTree says we're using plenty of RAM, but that our CPU usage is "minimal".

I just chanked up to 30 (from 15): "Max messages returned on recent posts by user"