After getting booted off the "Producton server," and trading countless emails with Lunarpages support over the last few months about our resource usage there, I am understandably sensitive to this topic.

So I've been researching our resource usage at the new VPS server. They tell me our CPU usage (which was our #1 problem at Lunarpages) is insignificant. See here:


Quote:

Your CPU usage is minimal. Typing top in a SSH prompt will give you exact usage stats.

Rather, they say MEMORY is the problem most accounts run into on a VPS server.

If you recall, when I signed up with WiredTree (our new host), I was able to find (via Google query) a COUPON which got us NOT ONLY a 10% discount (or $5/month):

http://www.retailmenot.com/view/wiredtree.com

.. but MORE IMPORTANTLY (I'm learning) a free 128-MB RAM upgrade.

I say "more importantly," cuz it appears we have ALREADY EXCEEDED the standard allotment of 256-MB (RAM) which comes standard with our hosting package:

http://www.wiredtree.com/managedvps/index.php

In other words, if we didn't have the free 128-MB memory upgrade, we would *already* be over our limit.

I repeatedly asked pointed questions: "Are you *sure* I've been assigned 384-MB RAM? .. and not (the standard) 256?"

In all case both techs I've been dealing with *assure* me we've been asigned 384. The following emails are both from from WT sppt:


Quote:

Checking your config file you have 384Mb of RAM. If you didn't your server would be going over it's memory limit and that would cause issues. The system would start becoming unstable if you go over your RAM limit and start killing off processes.

Quote:

The most common resource a VPS runs out of first is RAM. Below is a reading of your current RAM usage: -bash-3.1# free -m             total       used       free     shared    buffers     cached Mem:     1024        270        753          0          0          0 -/+ buffers/cache:  270        753 Swap:         0          0          0 Look at the used section which shows that you are using 270Mb out of your 384Mb. So with that said you have about 114Mb of free RAM which is a good thing. You can get your RAM usage at anytime by logging into your server via SSH as the root user and typing: free -m

Quote:

There really isn't a way to check to see how much guaranteed RAM you have since when you check your memory, it will show you all memory including burstable. The limits should have been set correctly last night when I setup your server. However, just for good measure, I set them again. Let me know if you need anything else.

Apparently, the "total" figure comes from our "burstable" limit of 1-GB (or 1024-MB).

They also informed me that, if we were to run out of memory, we'd start getting errors, due to processes shutting down, and that we could upgrade our memory by increments of 64-MB at a rate of $7.50 per 64-MB.

Sooo .. the free 128-MB RAM upgrade we got (via coupon) equates in value to $15 .. or *triple* the 10% - off discount (which only equals $5). So it seems like that coupon was worth $20/month.

I asked why my old host didn't have any problems with my memory usage. Their response:


Quote:

Don't know what kind of setup they were running. VPS providers configure their servers differently or run totally different virtualization software which calculates RAM usage differently.

I asked them about installing and playing with Drupal .. if that would put us over 384-MB RAM. He didn't think so. See here:


Quote:

I wouldn't think that installing Drupal would push you over. I can't tell you if it would or not since there are so many factors involved when you run PHP driven site.

I would've hated to upgrade to a VPS .. only to be told we were *already* exceeding our resource allotment.

You could easily write a scheduled task to email you the amount of used memory and any other statistics you are interested in.  Let me know if you are interested and I'll show you where to look to learn how to do it.  Wouldn't take long to learn.  Maybe an hour.

Yes, that would be cool. And this today from Lunarpages support.


Quote:

Hello, We have moved your account back to the Production server. I sent the welcome e-mail. Please login and verify your site and account features function properly so we can terminate your old account. If you have any further question related to this issue, feel free to contact us. We are committed to making your hosting experiences pleasant and fulfilling.

So the easiest way to get an email with your performance stats is to write a BASH script and execute it with crontab.  A BASH script is little more than a series of shell commands saved in a text document and executed in order.  So, the first step is to determine what commands  you want to run.

'free -m' would be a good one to keep an eye on your memory.  Another one might be 'top -bn 1'.  That one will give you a snapshot of your top processes, cpu usage, load average, and memory usage.  There are a lot more options to top, so read the man page if you'd like to tweak it.

Add both of these commands (and/or any others you want) to a text file (each on a separate line.)  You can use the vi editor to create and edit the text file.  Vi takes some getting used to, so here's a helpful link:

http://www.wam.umd.edu/~petersd/vi.html

Once the text file is made, edit crontab to tell cron to run the script at whatever time you want.  The man pages will be your best friend here, but an example crontab entry might look like this:

0 0,12 * * * sh textfilename > DailyLog.txt 2>&1
1 0,12 * * * mail -s ServerStats youremail@yourdomain < DailyLog.txt

The first entry runs the script you wrote and puts the output into a text file.  The second entry mails that text file to youremail.  The first entry will run every day at midnight and noon.  The second entry runs one minute later.  The second command assumes that the sendmail service is running, which is should be on CentOS 5 by default.  The crontab man pages should explain how to run it at any time a day you want, as often as you want (up to once a minute.)

I'd be happy to help if you have any questions.

MrMagoo wrote on Feb 22^nd, 2008 at 3:43pm:

You can use the vi editor to create and edit the text file.  Vi takes some getting used to, so here's a helpful link:

You don't say! vi is certainly an acquired taste and I managed to avoid having to learn it in my years of unix. emacs was installed on most systems, and if it wasn't I persuaded the admin to install it.

emacs is also available in compiled form for windows - I use it as my default text editor in vista.

I've used both, and I didn't want my post to turn into a long explanation about the pluses and minuses of each, since volumes exist on the subject on other sites.  The bare basics of the version of vi that Cent5 includes by default aren't hard to learn - and it's friendlier than older Unix versions, so I'm guessing spending a few minutes learning vi will be easier at this point than learning to install emacs through 'yum' or 'RPM'.  "Persuading the admin to install it" isn't an option, since Rad has taken on that role himself.

That's a good point about working with what is already there.

Good to hear that vi is friendlier than in my unix days. Back then vi users tended to be hard-core computer science types who appeared to me to get perverse pleasure out of using it. Lesser (saner?) mortals would quickly use something more intuitive.

I find these discussions interesting, tho I do not fully understand.

You guys seem to say vi comes with CentOS 5 by default, tho is more difficult to learn than another, similar text editor called emacs, which I would have to install (or let Magoo install).

I have not yet played with vi, but ask myself, "How hard can it be to learn a text editing prgm?"

I was focused, until late last night (midnight), trying to get the Movable Type blogging software working. For some strange reason, the script (mt.cgi) prompted a *download* .. instead of running the program:

http://forums.sixapart.com/index.php?showtopic=65176

Found a fix here:

http://orangescale.net/blognotes/movable-type/movable-type-troubleshooting-base-class-package-dataobjectdriverbaseobject-is-empty/

.. which involves tellling the program exactly where it resides via an absolute path > /home etc. ..using a "strict path"  Still have to apply the fix to older MT 3.35 install. (MTOS v4.1 now works.)

Now I'm trying to connect via SFTP/SSH2. Cute Pro is timing out on me. WiredTree sppt recommends this client:

http://winscp.net/eng/index.php

So I am still in the process of dialing in the new account and learning VPS skills. (I've never used SFTP before.)

WiredTree hasn't been steering you wrong.  WinSCP is a good program.  Since it works through scp, which is a file transfer protocol known as rcp that has been updated to use secure ssh tunneling, I have always found it much faster than FTP (and it is far more secure.)

Vi was designed to be much more than a simple text editor.  It has advanced search and replace functions, scripting functions, the ability to cut or copy any 32 abritrary strings and then paste any one of them back in later in the file, ect.  Since vi was written to be used from the command line, drop down menus weren't available to access these advanced features.  So you end up with the command to do a search and replace looking something like :%s/OLD/NEW/g.

The good news is, if you just want to edit the text, it isn't very complicated.  

1. From the command line, type 'vi textfile' to open or create a text file in vi
2. Press 'i' to get into insert mode.
3. Edit the file to your hearts content
4. Press 'ESC' to get out of insert mode
5. Press ':wq' to save the file and exit the editor

BONUS:
I just logged into one of my Cent5 servers and checked.  It seems the nano text editor is also installed by default.  Nano is dead simple to use (even more so than emacs.)  So, if you don't want to learn vi, use nano.  'nano textfile' to open the text file.  Edit normally.  'Ctlr-x' to save the file.

As a side note, I just noticed a folder on the root called "emacs". Not sure what this means, but it would seem the program is available, if not already installed. Data-point.

I still haven't gotten around to connecting via SFTP/SSH2.

Are these two terms synonymous? SFTP and SSH2?

Are you saying the extra security/encryption of SFTP does nothing to slow down the transfers/protocol, but rather SFTP is actually *faster* than FTP?

Rad - at the command line, type:


Quote:

which emacs

if you get this response:


Quote:

emacs: Command not found.

then is isn't installed. However, if it is, you'll get back the absolute path to the emacs executable. You can to this with any program you are interested in to see if it is installed on your system and in your current $PATH.

Rad.Test wrote on Feb 23^rd, 2008 at 3:15pm:

Are these two terms synonymous? SFTP and SSH2?

Not synonymous, but related.  SSH is Secure SHell.  SSH2 is just an updated version of SSH.  Most people used them interchangeably.  It is a secure protocol for opening a connection to a server and logging in to give the server commands.  I hope that is how you are connecting to the server to get to the command line.  Putty and SecureCRT are popular SSH clients.  SSH was designed to replace telnet, which is the protocol programs like HyperTerminal uses.  Telnet is very insecure, sending all your commands across the connection in clear text for all to see.

SFTP is a file transfer protocol designed to tunnel file transfers across a secure connection.  Security is provided by the SSH protocol.  It shouldn't be any harder to use than downloading WinSCP and connecting to your server.  After that, it's pretty much drag and drop.


Quote:

Are you saying the extra security/encryption of SFTP does nothing to slow down the transfers/protocol, but rather SFTP is actually *faster* than FTP?

FTP is very old.  It was one of the first networking protocols developed, back in the days when security wasn't a concern.  SFTP is not directly related to FTP.  It is a much newer protocol, and therefor much faster and more reliable.  The security provided by SSH that SFTP uses has a negligible impact on the speed of the transfers (but a big impact on my piece of mind.)

Thanks.

I'm a little frustrated, cuz I just want to get the command line to check free -m

I can now connect via SFTP drag-n-drop files, just like regular FTP, but when I tried PuTTY to get command line, it told me "Shell access is not enabled on your account."

So I'm finding out how to enable shell access now. I've been at this for days it seems.

I will x-fer via WinSCP if you say that's the way to go.

Here are some stats I found in a different part of WebHost Mgr:


Quote:

Current Memory Usage             total           used         free Mem:     1048576     303952     744624

.. which would indicate we're using ~300MB, with 75MB remaining free.

Okay, I finally got shell access. Woohoo!

free -m


Quote:

total       used       free Mem:      1024        270        753

Here's graph from WiredTree's monitoring program interface, which they say has same accuracy as shell access:

http://radified.com/gfx3/vpsMemoryUsage.gif

They said the "Allocated" number comes from fact that:


Quote:

Programs often allocate more memory to be used than they are actual using.

Here's another for Server Load:

http://radified.com/gfx3/vpsLoad.gif

Server load can be understood here:

http://en.wikipedia.org/wiki/Load_%28computing%29

From the WiredTree VPS page:


Quote:

Our VPS servers use only the highest quality server-grade components. Every VPS server uses dual-core Intel Xeon Clovertown processors (8 CPUs). Combined with 12GB+ Fully Buffered DDR2 memory

http://www.wiredtree.com/managedvps/

Here's another view of our load. WiredTree sppt tells me these stats represent load generated only by my account (Radified.com):

http://radified.com/gfx3/vpsLoadDaily.gif

So, I asked how I could tell if *other* accounts, who are assigned to the same server, were overloading it. Their response:


Quote:

You won't be able to tell, since Virtuozzo has a pretty good grip on other people's accounts to ensure this doesn't happen.

http://www.parallels.com/en/products/virtuozzo/

So I asked what are my *limits* are .. regarding server LOAD (since that's the stat they give me). Their response:


Quote:

A load of 1 to 2.5 is minimal for the kind of hardware (8 CPUs) that your VPS is running on. There is no specific limit on server load for your account. Your account limits are a combo of CPU time and RAM usage.

So I asked at what point they would start to get concerned about server "load". Their response:


Quote:

On a VPS account, anything over 10 would be an issue. You would most likely run out of memory prior to that point.

Along these lines of resource usage & management, I found this site:

http://www.gplhost.com/hosting-vps.html

.. which offers Xen, which I read a little about previously. From this page comes this interesting paragraph:


Quote:

Few words about Xen virtualisation software: We use Xen virtualization system because it's simply the state of the art in virtual server hosting (VPS hosting). With Xen, end users enjoy short cpu, hard drive and network latencies thanks to accurate scheduling (note: this is totaly different from other virtualization technologies like UML, vServer or Virtuozzo). Memory is not shared and not burstable among users, the physical memory is simply split a slice that we rent to you, the same way we do with LVM HDD slices.

They definitely got a cool name (Zen). Anybody familiar with Xen? Comments?

I like the phrase "state-of-the-art" used to describe it. Wikipedia has a » page on Xen. When asked, Joe at WiredTree support said this:


Quote:

I've never personally administrated a Xen VPS, so my comments about it are limited. A few of my friends that do system admin have worked on Xen-based servers and their comments for the most part have been positive. In the end, both Xen and Virtuozzo basically do the same thing.

Just pinged the site. 73ms is my lowest yet (normally 80-ish)


Quote:

NeoTrace Trace  Version 3.25  Results Target: radified.com Date: 2/24/2008 (Sunday), 5:57:03 PM Nodes: 14 Node Data Node Net Reg IP Address      Location            Node Name   1   -   -                        Anaheim   2   1   -                      Unknown               3   2   1                     Irvine   4   3   1 67.114.50.2     Irvine              dist4-vlan50.irvnca.sbcglobal.net   5   4   1 151.164.92.196  Irvine              bb2-g9-0.irvnca.sbcglobal.net   6   4   - 151.164.93.167  Little Rock           7   4   1 151.164.89.2    38.983N,  77.378W   asn3561-savvis2.eqlaca.sbcglobal.net   8   5   2 204.70.203.50   Los Angeles         cr2-pos-0-0-3-0.lay.savvis.net   9   -   - 0.0.0.0         Unknown             No Response  10   5   2 208.172.3.54    Chicago             acr1-so-0-0-0.chicago.savvis.net  11   6   - 216.90.240.162  Chicago              12   7   - 208.79.234.8    Chicago              13   7   - 208.79.238.73   Chicago              14   7   3 208.79.239.88   Chicago             host.radified.com Packet Data Node High Low  Avg  Tot  Lost   1    0    0    0     1    0   2    8    8    8     1    0   3   15   15   15    1    0   4   15   15   15    1    0   5   20   20   20    1    0   6   16   16   16    1    0   7   15   15   15    1    0   8   26   26   26    2    1   9   --- ---  ---    2    2  10   76   76   76    1    0  11   78   78   78    1    0  12   75   75   75    1    0  13   75   75   75    1    0  14   73   73   73    1    0 Network Data OrgName:    Cogswell Enterprises Inc. OrgID:      COGSW Address:    53 W Jackson Blvd. Address:    Suite 635 City:       Chicago StateProv:  IL PostalCode: 60604 Country:    US _____ NeoTrace Copyright ©1997-2001 NeoWorx Inc

I think "Cogswell Enterprises" is Zac Cogswell, who I traded a bunch of emails with last night: http://www.wiredtree.com/aboutus/managementprofiles.php

2/24/08 11:50pm PT from Southern California


Quote:

ping radified.com Pinging radified.com [208.79.239.88] with 32 bytes of data: Reply from 208.79.239.88: bytes=32 time=66ms TTL=53 Reply from 208.79.239.88: bytes=32 time=65ms TTL=53 Reply from 208.79.239.88: bytes=32 time=65ms TTL=53 Reply from 208.79.239.88: bytes=32 time=63ms TTL=53 Ping statistics for 208.79.239.88:    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss), Approximate round trip times in milli-seconds:    Minimum = 63ms, Maximum = 66ms, Average = 64ms

64 ms. That's good, especially considering you're not much closer to Chicago than I am. You're 9ms faster. That's what? .. about 12% faster?

Here's latest LOAD stat, taken 8PM Monday night (2-day graph):

http://www.radified.com/gfx3/vpsLoad25Feb.gif

I wonder what makes those spikes to 2.5? Looks like about 8 to 10 AM. I wonder if that's local (Chicago) time. That would be what here on the West coast? 6-8AM? Looks like the same time every morning. Even the shape of the spike is the same. Kinda weird that it even happens on Sunday? Think it might be search engine bots?

UPDATE - Yeah, it's a little after 9PM here and the daily graph is giving me stats for 11PM. So Chicago must be 2 hours ahead.

http://www.radified.com/gfx3/vpsLoad25FebDaily.gif

From the daily graph, it looks like the graphs starts to climb shortly after 8:30 Chicago time, which would be about 6:40 AM here. It peaks around 9:15, which would be 7:15 here. By 10 o'clock (or 8AM here) things are back to normal. Even on Sunday.

You think it's a "wake-up-and-check-Radified" thing? That's what I do, but I don't know about others.

Rad - you are still advertising LP in the animated gif at the bottom of the forum pages.

Sun pm

Yeah, I still have the account. Paid a year in advance. Tho not using it any longer. Plus LP has an affiliate prgm, which pays $65 for every person who signs up using one of those links. WiredTree does not. So removing those links is not a top priority right now, especially when I have so many other things to do. But it will be good to move on, and do away with all things Lunar.

Along another line, here's a graph of server load taken this morning at 0730 AM PST. First the DAILY view (which is the most accurate), then the WEEKLY. More load today. Almost up to 4.0 on the lighter-shade 5-minute load. Looks like it came earlier, too. Wonder what is causing those spikes?

http://www.radified.com/gfx3/vpsLoad26Feb.gif

http://www.radified.com/gfx3/vpsLoad26FebWeekly.gif

The peak looks like it came around 4:30 AM Chicago time, which would be 0230 here on the West coast. That sounds like search engine bots, no? (Which never sleep.)

Do you have any crong jobs running on the server?  Disk checks?  Log rotate?  Anything in the logs (/var/log)?

Uh, I don't know what those are (cron jobs). The only thing I know about is a daily back-up.

I will ask WT sppt. I just like to know what's going on with the server.

Maybe the backup runs right at that time...?

I just asked.


Quote:

This is probably from the rkhunter cronjob. Basically, rkhunter does md5sums on many files, which is both disk and processor intensive. Because of this, we have a cron on the root node that inspects when rkhunter is run in each server, and spreads the times out across all the vps's on the server. What happened is that the cron was not running properly, so rkhunter was running on all vps's on the server at the same time. So you would see more of a spike than normal. We actually caught this problem this morning, and have fixed the cron. You will still see a bit of a spike when rkhunter runs, but it should be much less than what you were seeing previously.

Do you understand what he is saying?

I understand what he is saying, but I'm not positive why they do it.  Creating an MD5 sum on a file allows you to uniquely identify that version of that file.  I'm guessing they use the MD5 sums to determine which files need to be backed up in an incremental backup.

Sounds like it was running too often on too many of the virtual accounts on this host, creating extra load, but they corrected it before you emailed them.

rkhunter is a tool that looks for rootkits; the MD5sum will be being done to verify that files haven't been changed inappropriately (i.e., been subverted by a rootkit).

Load, taken at noon, 2-day view:

http://www.radified.com/gfx3/vpsLoad27Feb.gif

Here's our memory usage, taken at noon,2-day view. This looks much weirder than other views, which were much more constant.

http://www.radified.com/gfx3/vpsMemory27Feb.gif

wrote on Feb 27^th, 2008 at 5:39am:

rkhunter is a tool that looks for rootkits

Neat.  Learn stuff every day.

Rad

Approx. 3:25 pm west coast time--4 Users, 90 Guests, 0 Search Engines:

5-10 second delays waiting for server to respond to click on a thread link or forum index link--server apparently not quite up to the demand!

4:30 now.  No delays for me.  6 members, 45 guests, 20 search engines.  Musta been a short-live issue.

I received this email (auto-generated):

httpd failed @ Wed Feb 27 18:15:46 2008. A restart was attempted automagically.

Seems like Apache crashed and was automatically restarted. How do the times jive? This would be Chicago time, which I believe is 2 hours ahead.

There is also a 1-hour chunk of monitoring missing. I'll post a screen shot for you to peruse. Here it is:

http://www.radified.com/gfx3/vpsMem27FebGap.gif

Notice that memory use dropped significantly. from 275-ish before to 220-ish now.

Run a trace when you see delays to see if you can identify the source of the lag. Response from WT sppt:


Quote:

I wouldn't be terribly worried with those messages unless you're getting them all the time. From time to time, services will go unresponsive. cPanel's chkservd service monitors services and will restart them if they aren't responsive. This in turn will email you to let you know the service was restarted. This is what is referred to as self-healing. We experienced some issues with the caching resolvers on our network causing the resource graphing to fail momentarily. This has since been resolved. When apache restarts, your memory use will drop significantly. This is due to the way that apache uses memory. To reduce load times and load on the server, apache will cache information in memory as it has faster response times than reading from the hard disk.

Also notice short-term load spike around 6:15 PM Chicago time, which would be 4:15 here on the West coast.

http://www.radified.com/gfx3/vpsLoadSpike27Feb.gif

Late-night load surge:

http://www.radified.com/gfx3/load27Feb.gif

NightOwl wrote on Feb 27^th, 2008 at 5:29pm:

5-10 second delays waiting for server to respond to click on a thread link or forum index link--server apparently not quite up to the demand!

Same experience here yesterday - lightening fast right now though  :)

I was watching Tigger today. Bouncey.

If you guys could grab tracert's and post when you notice delays.

Lightning. Yeah, nice.

Port speeds. Do you guys know about port speeds? (I don't.)

I see where I can request either an upgrade .. o a downgrade .. with my port speeds.

Does upgrade = faster?

I like faster.

Not sure what that is refering to... Maybe your switch-port...?  Like 100Mbit, 1000Mbit, etc...  Don't think you'd need to upgrade.  100Mbit is plenty fast for a web site.  Unless they are talking about something else.

From WT sppt, regarding a query about port-speed upgrades (& downgrades).


Quote:

That is really a dedicated feature. It doesn't apply to VPS. Basically, it only affects the amount of maximum traffic you can push. Some people get a 10Mb port because it is cheaper. The the VPS parent nodes are on 100Mbs ports. We don't often put dedicated customers on 1000Mbs ports. It's generally done by special arrangement.

It's nice that this thread has gone quiet...

Ooops!

Rad

Approx. 11:15 PM, 50 Guests on-line, and 5-10 second delays for response from server--here's a PingPlot.  About 10 minutes later, and only 30 Guests on-line and the response time was much better!

---

I am certainly no networking expert, but it would seem the problem is coming from hops #4 (Comcast, 60% packet loss) and #9 (Savvis, 40% packet loss).

*Any* packet loss is bad, and a sign of a problem. But 60% is high.

Also #8 (AT&T, had one trace at 102, which is quadruple the average, tho no PL).

The last 3 hops (I believe) are WiredTree, and they look good.

Sometimes a simple refresh can find a different route, bypassing problematic hops. That's kinda how the web works.

Maybe Magoo can elaborate. That's his bailiwick.

By the way, you might be interested to know (I was) that the President of WiredTree (Zac) told me they purposefully block the names of our/their upstream nodes (notice blank dashes, instead of DNS names), cuz most people do not want others to know the identity of their providers.

Not sure if this is a security issue, or a business issue (or both).

Thanks for taking the time to run and post a tracert.

The traces to the destination look good.  Like you said, the last 3 hops look good.  Since the packet has to go through all of the routers in between to get there, all the hops along the way are probably ok.  Traceroute packets are a very low priority for a router, so busy routers sometimes drop them (creating apparent packet loss) while normal traffic gets through with no issues.  Those routers showing packet loss are probably busy, but not saturated.

I'd be interested in seeing another traceroute from a different starting point next time there are issues to compare the two graphs.  Server load graphs from the time in question would be fun to compare against too.

Everything is lightning quick now...

MrMagoo wrote on Mar 5^th, 2008 at 7:21pm:

Everything is lightning quick now...

Music to my ears.

Regarding load stats .. that's something I can do:

http://www.radified.com/gfx3/load05March.gif

Note there is a 2-hour diff between here and Chicago.

Hmm, nothing in the load graph around the time of the incident.  I'm seeing some big delays right now, but its that time of the night that the rkhunter cron job is probably running.  Network looks good; Here is a trace:

4  68.2.12.114 (68.2.12.114)  6.219 ms  6.539 ms  14.585 ms
5  68.2.12.26 (68.2.12.26)  8.110 ms  7.463 ms  7.730 ms
6  68.2.12.30 (68.2.12.30)  8.185 ms  10.507 ms  6.797 ms
7  chnddsrj01-ae1.0.rd.ph.cox.net (68.2.14.1)  10.426 ms  6.558 ms chnddsrj02-ae1.0.rd.ph.cox.net (68.2.14.13)  7.924 ms
8  langbbr01-ae0.r2.la.cox.net (68.1.0.232)  21.598 ms  20.870 ms  22.337 ms
9  vl3490.ccr02.lax01.atlas.cogentco.com (154.54.6.189)  23.484 ms vl3492.mpd01.lax01.atlas.cogentco.com (154.54.3.9)  19.688 ms vl3490.ccr02.lax01.atlas.cogentco.com (154.54.6.189)  23.260 ms
10  te4-2.ccr01.sna02.atlas.cogentco.com (154.54.3.37)  24.245 ms  23.259 ms te2-4.ccr02.iah01.atlas.cogentco.com (66.28.4.237)  46.853 ms
11  te2-1.ccr02.dfw01.atlas.cogentco.com (154.54.3.177)  45.213 ms te2-1.mpd01.dfw01.atlas.cogentco.com (154.54.5.133)  48.570 ms te8-2.mpd01.iah01.atlas.cogentco.com (154.54.1.66)  46.265 ms
12  te7-3.mpd01.mci01.atlas.cogentco.com (154.54.3.18)  53.138 ms te7-2.ccr02.mci01.atlas.cogentco.com (154.54.5.166)  56.350 ms te4-1.ccr02.ord01.atlas.cogentco.com (154.54.6.254)  66.454 ms
13  te2-3.mpd01.ord01.atlas.cogentco.com (154.54.7.137)  66.019 ms te4-1.ccr02.ord01.atlas.cogentco.com (154.54.6.254)  69.519 ms  66.591 ms
14  te4-1.ccr01.ord04.atlas.cogentco.com (154.54.24.178)  66.610 ms te4-3.ccr01.ord04.atlas.cogentco.com (154.54.24.182)  65.050 ms te7-4.mpd01.ord01.atlas.cogentco.com (154.54.2.190)  69.937 ms
15  38.104.106.78 (38.104.106.78)  65.099 ms  65.938 ms te4-3.ccr01.ord04.atlas.cogentco.com (154.54.24.182)  69.346 ms
16  208.79.234.8 (208.79.234.8)  64.720 ms  67.858 ms  66.646 ms
17  208.79.238.73 (208.79.238.73)  66.733 ms  68.824 ms  66.657 ms
18  host.radified.com (208.79.239.88)  69.973 ms  66.582 ms  66.701 ms

I removed the first 3 hops because they were just my internal network.

Rad

Around 7:05 am--53 Guests and 4 Users--Response to clicks asking for server response are a little sluggish--2-5 second delays--variable.

I'm doubting that the cross country signal is *slow*--just having to wait your turn for the system to respond--similar to when my system is doing some background task like a virus scan--my system puts certain keystrokes in a cue and there can be slight delays for opening another program or printing.

Rad

About 7:30 now, 73 Guests, 4 Users--no delays!!!

Delays may depend on what all those users are doing at any given moment!

Had a big load spike this morning (to 16-ish). Submitted sppt tik to look into it.

Quote:

There has been some high disk I/O activity on the main hardware node as a whole that most likely accounts for some of the sluggish responses that you are seeing. I think we found out the main reason why and if the trend continues action will be taken.

In an effort to keep thread file-sizes down, this thread is continued here:

http://radified.com/cgi-bin/yabb2/YaBB.pl?num=1204840577