I'm wondering if this is my computer or the new Rad server.  But lately (like this morning at 6:30 am Pacific time) there were lots of 5 to 10 second pauses or delays in getting responses from Radified webpages.  Again tonight at 9:00 pm Pacific time same problem.  In fact a couple "IE unable to display...web page" messges.

Being in Southern California, I would expect the response to be great.

Is the server in LA or Las Vegas?  I ran a tracert and traffic seems to be going to Las Vegas or maybe I am mis-reading it?

John. wrote on May 22^nd, 2007 at 11:12pm:

were lots of 5 to 10 second pauses or delays in getting responses from Radified webpages.  Again tonight at 9:00 pm Pacific time same problem.  In fact a couple "IE unable to display...web page" messges.

Same here.

Quote:

there were lots of 5 to 10 second pauses or delays

how about 1 minute or longer delays!

yeah, me too!

I've had several mornings starting on Friday of this last week where I get real slow loading of a web page--or a browser error say it can not load the page!

Rad--what's up--are you making setting changes?

Or has your host been adding and overloading your new server with other clients to the detriment of your domain?

Huh.  I haven't had any issues.  Are you guys seeing this on the site, the forums, or both?

MrMagoo


Quote:

Are you guys seeing this on the site, the forums, or both?

Well, I spend most of my time on the forum--but it has been effecting Rad's Homepage
as well!

Next time it happens (if it does), I'll check the *Guides* to see if they are effected equally.

But this morning--everything seems okay!

NightOwl wrote on May 23^rd, 2007 at 8:10am:

But this morning--everything seems okay!

About 10 minutes ago, I had one of those one MINUTE pauses.  Now it's back to normal.  Not sure if it is the network or server itself, although other websites refreshed fine while I was waiting for Radified.

The post above took OVER 4 minutes!  Now back to normal.

Here's a tracert I ran while I was waiting:



Tracing route to radified.com [74.50.3.195]
over a maximum of 30 hops:

 1    <1 ms    <1 ms    <1 ms  
 2     8 ms     7 ms     9 ms  ipxx-x-xx-xxx.net  
 3     8 ms     9 ms     9 ms  68.4.15.209
 4     8 ms     9 ms     9 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]
 5    12 ms     9 ms    11 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]
 6    10 ms    11 ms    11 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]
 7    19 ms    17 ms    18 ms  nwstdsrj01-ge710.rd.lv.cox.net [68.1.0.85]
 8     *        *        *     Request timed out.
 9     *        *       19 ms  24-234-6-66.ptp.lvcm.net [24.234.6.66]
10     *        *        *     Request timed out.
11     *       19 ms    19 ms  cust-lunarpages.marquisnet.com [208.65.158.130]
12    19 ms    18 ms    19 ms  hafgan.lunarservers.com [74.50.3.195]
Trace complete.

Here's one after things returned to normal response (6:3am from So. California):

Tracing route to radified.com [74.50.3.195]
over a maximum of 30 hops:

 1     1 ms    <1 ms    <1 ms  192.168.1.1
 2     8 ms    17 ms     9 ms  ipx-x-xx-x..cox.net [xx.xx.xx.xx]
 3    12 ms     7 ms     7 ms  68.4.15.209
 4    10 ms     9 ms     9 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]
 5    11 ms    15 ms     9 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]
 6    11 ms    11 ms     9 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]
 7     *        *        *     Request timed out.
 8    22 ms    25 ms     *     24-234-6-1.ptp.lvcm.net [24.234.6.1]
 9    18 ms    19 ms    19 ms  24-234-6-66.ptp.lvcm.net [24.234.6.66]
10    20 ms    18 ms    20 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]
11     *       18 ms    19 ms    cust-lunarpages.marquisnet.com [208.65.158.130]
12    18 ms    19 ms    19 ms  hafgan.lunarservers.com [74.50.3.195]
Trace complete.

I thought the server was in LA?  I assume that lv is Las Vegas?

Ghost4me

I had the same slowdown at the same time!

lvcm.net stands for Las Vegas Cable Modem.  It's part of the Cox backbone.  The weird thing is that response times at the end of the trace are still ok even when the hops in the middle are timing out.  It could be that those routers are too busy to reply to ICMP traffic (traceroute, for example), which is low priority, but are still passing traffic fine.

Next time it happens, also try some DNS queries on radified and some other sites:

Open a DOS prompt
type "nslookup"
type "radified.com"
try a few other sites

Note how long each query takes and post the output.  Also, try running a constant ping to both radified and google next time it happens and post some of that output ("ping -t radified.com").

When the page finally loads, does it load slow, piece by piece like you are on dial-up, or does it load all at once?

When the page finally loads, it's instaneous; like everything is back to unblocked and normal.

Here's a ping I just did (everything working fine):

Pinging radified.com [74.50.3.195] with 32 bytes of data:

Reply from 74.50.3.195: bytes=32 time=21ms TTL=54
Reply from 74.50.3.195: bytes=32 time=21ms TTL=54
Reply from 74.50.3.195: bytes=32 time=21ms TTL=54
Reply from 74.50.3.195: bytes=32 time=19ms TTL=54
Reply from 74.50.3.195: bytes=32 time=19ms TTL=54
Reply from 74.50.3.195: bytes=32 time=20ms TTL=54
Reply from 74.50.3.195: bytes=32 time=18ms TTL=54
Reply from 74.50.3.195: bytes=32 time=20ms TTL=54
Reply from 74.50.3.195: bytes=32 time=22ms TTL=54
Reply from 74.50.3.195: bytes=32 time=21ms TTL=54

Ping statistics for 74.50.3.195:
   Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
   Minimum = 18ms, Maximum = 22ms, Average = 20ms

=================

Pinging google.com [64.233.167.99] with 32 bytes of data:

Reply from 64.233.167.99: bytes=32 time=72ms TTL=244
Reply from 64.233.167.99: bytes=32 time=69ms TTL=244
Reply from 64.233.167.99: bytes=32 time=71ms TTL=244
Reply from 64.233.167.99: bytes=32 time=70ms TTL=244
Reply from 64.233.167.99: bytes=32 time=70ms TTL=244
Reply from 64.233.167.99: bytes=32 time=70ms TTL=244
Reply from 64.233.167.99: bytes=32 time=71ms TTL=244
Reply from 64.233.167.99: bytes=32 time=70ms TTL=244
Reply from 64.233.167.99: bytes=32 time=72ms TTL=244
Reply from 64.233.167.99: bytes=32 time=69ms TTL=244

Ping statistics for 64.233.167.99:
   Packets: Sent = 10, Received = 10, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:
   Minimum = 69ms, Maximum = 72ms, Average = 70ms

=============================

Right now radified.com pings look better than google.com!

I don't know. I'm not doing anything weird with the server.

There were some slowdowns I noticed back when we launched the new forum, but haven't noticed anything recently.

The server is physically located downtown Los Angeles.

I just pinged at an ave 77 (5 tries). That's slow, but not too bad, especially for this time of day. I contact LP and see what they say.

http://radified.com/gfx2/ping_23may.gif

Rad wrote on May 23^rd, 2007 at 5:56pm:

I don't know. I'm not doing anything weird with the server. There were some slowdowns I noticed back when we launched the new forum, but haven't noticed anything recently. The server is physically located downtown Los Angeles.

I think it is a server problem (not under your control), not a network problem.

The WHOIS for marquisnet.com is:
MarquisNet
7185 Pollock Drive
Las Vegas, Nevada 89119
http://marquisnet.com/

I think their server is co-located in Las Vegas, not LA.
Their offices may be in Los Angeles, but not the servers.  Why is the traffic going to Las Vegas where marquisnet is?

lunarservers.com
Web Hosting, Lunarpages  hostmaster@lunarpages.com
   100 E. La Habra Blvd.
   La Habra, Ca 90631
   US

hah! you guys are right.

the new server is physically located in las vegas.

tho they looked at the logs for past week and noted no latency probs.

they said they actively monitor server & network performance 24x7,and do load-balancing.

the *old* server was in la.

your guys are good.

Rad wrote on May 23^rd, 2007 at 10:07pm:

tho they looked at the logs afor the past week and noted no problems with latency.

Even though the sever is in Las Vegas, that doesn't explain why NightOwl, Brian, and I (all 3 different locations) noticed the same delays and problems.

This morning's problem was between 6:30 am and 7:00 am Pacific time.  I'll keep my eyes out for future blips.

Rad

If you want to *catch* the slow down--it has happened most consistently between 6:00 am and 7:00 am PST!

I agree, that's the time.  But it's not exactly a slowdown; it's more like a complete stop of traffic or response from the server.  Then a minute later (or more) the screen flashes with everything.

And it's not just a Los Angles to Las Vegas problem because I think all 3 (NightOwl, Brian, myself) are in different parts of the world.

if you can grab pings and post them here, i can show them hard data.

there are also times when search engines spider thru the site, and i think that will slow it down.

yeah, a minute or more .. that's unacceptable by ANYbody's standards.

here's their response to my email inquiry (earlier, I had *called* them to see where the server was physically located):

Ticket Subject : slow response times?

Reply:
Hello

Would it be possible to provide the following:
1.)  Your geographic location
2.)  Your ISP
3.)  Your IP Address (can be obtained at http://www.whatismyip.com/)

.:|Traceroute|:.
A traceroute doing the following:

Windows XP/2000
1. Go to Start > Run...
2. Enter cmd and click OK.
3. At the prompt, type tracert yourdomainname and hit enter.

Windows ME/98/98SE
1. Go to Start > Run...
2. Enter command and click OK.
3. At the prompt, type tracert yourdomainname and hit enter.

.:|NSlookup|:.
An NSlookup doing the following:

Windows XP/2000
1. Go to Start > Run...
2. Enter cmd and click OK.
3. At the prompt, type nslookup yourdomainname and hit enter.

We can then look into this issue further.

Thanks.  

--
If you have any more questions, please do not hesitate to contact the Lunarpages Help Desk again.

Best Regards

I basically posted all that info this morning (see posts 6, 7 etc.), but I can do it again tomorrow if I see it happen again.

Why don't you send them the tracert and ping info from this morning and see what they say.

I've just experienced about 2 minutes of "Unable to connect". During this time I ran several Ping Plotters and the round trip was 210 to 220 msec. It was the same during the "Unable to connect" as it is now.

It's happening again today (6:15 am to 6:30am Pacific Time from Los Angeles/Orange County), but shorter lockups/pauses, usually 10-20 seconds, some a few longer.  Several "IE cannot display the webpage" messages.  Tracert looks normal:

Tracing route to radified.com [74.50.3.195]
over a maximum of 30 hops:

 1    <1 ms    <1 ms    <1 ms  192.168.1.1
 2     9 ms     9 ms     7 ms  ip68-x-xx-xx.xx.oc.cox.net [xxxx]
 3     8 ms    10 ms     8 ms  68.4.15.209
 4     9 ms     9 ms     9 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]
 5    11 ms    10 ms     9 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]
 6    11 ms    11 ms    11 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]
 7    17 ms    19 ms    17 ms  nwstdsrj01-ge710.rd.lv.cox.net [68.1.0.85]
 8    29 ms    34 ms    27 ms  24-234-6-1.ptp.lvcm.net [24.234.6.1]
 9    17 ms    17 ms    17 ms  24-234-6-66.ptp.lvcm.net [24.234.6.66]
10    19 ms    19 ms    19 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]
11    18 ms    19 ms    17 ms  cust-lunarpages.marquisnet.com [208.65.158.130]
12    18 ms    17 ms    20 ms  hafgan.lunarservers.com [74.50.3.195]

Trace complete.

the above post took 3 "ie cannot display the webpages" messages.  I hit refresh after each one, finally got an invalid response.  Frustrating.  6:33am

38 guests, 4 members logged on.

test.

they are looking into it.

Rad wrote on May 24^th, 2007 at 8:42am:

test. they are looking into it.

Rad, are you seeing same problem from your location?

Ghost4me


Quote:

are you seeing same problem from your location?

Since 6:30 thru 7:30 this morning--page loading slower than *dial-up*--multiple time-out errors--yuk!

Rad

As of approx 7:45--speeds were back to *normal*!

my home page opens to the rad home page.

it came up 404 at 6:30 when i woke.

1st refresh brought it up.

i pinged, which came up normal.

go figure.

sent mail to lp.

It's looking pretty fast/normal now.  However, look at this tracert which shows a big lag at one router:

Tracing route to radified.com [74.50.3.195]
over a maximum of 30 hops:

 1     1 ms     1 ms     1 ms  192.168.1.1
 2     7 ms    10 ms     9 ms  ip.xxx.oc.cox.net [xx.xx.xx.xx]
 3    10 ms    11 ms     9 ms  68.4.15.209
 4    16 ms    20 ms    14 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]
 5    14 ms    12 ms    16 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]
 6    10 ms    11 ms    41 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]
 7    19 ms    18 ms    20 ms  nwstdsrj01-ge710.rd.lv.cox.net [68.1.0.85]
 8     *      250 ms   261 ms  24-234-6-1.ptp.lvcm.net [24.234.6.1]
 9    25 ms    28 ms    30 ms  24-234-6-66.ptp.lvcm.net [24.234.6.66]
10    27 ms    34 ms    31 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]
11    24 ms    27 ms    23 ms  cust-lunarpages.marquisnet.com [208.65.158.130]
12    33 ms    24 ms    26 ms  hafgan.lunarservers.com [74.50.3.195]

Trace complete.

Reply:
Hello,

At this time I am not seeing any issues with the server. Looking at the post that Ghost4me did on your forums (http://radified.com/cgi-bin/yabb2/YaBB.pl?num=1179893550)  I do see a few times out. This could cause slow load times and possible page errors.

Can you get a couple of your users to do traceroutes when they are experiencing issues, and then again when they are not having issues, and have them post them into the forums with the times the traceroutes were taken?

To answer a post in the forums, the reason more then one person in different geographic locations can both experience issues is at some point they both might run through the same internet router.

The more traceroutes with times that we can get, from varying locations, the better we can investigate where the issue may be.

Best regards

Noticed some nasty site responses > Thursday eve, 6:45 PM from Orange county. Pings follow:

http://radified.com/gfx2/ping_24may_1845.gif

Average ping 200ms from Orange county is bad.

Rad,

The trace you posted shows a network slowdown in a different place than the other guys.  They seem to be seeing a slow down on a Cox router (lvcm.com) while your first hop with high response time is AT&T.  I don't think we can assume they are part of the same issue.  We'll need to see more tracerts for the other guys, but it looks like Cox has some overloaded routers in Vegas.

If you guys can post a few more tracerts, we can contact Cox about our concerns.

Reply:
Hi.

I don't want to come off saying that it is your ISP but hops 6 and 9 show that the routing from your ISP is having latency.

6 shows 219ms at the  att.net router 9 shows 231ms which the router won't resolve the location.

Our network is cust-lunarpages.marquisnet.com and of course hafgan the server your account is on.

The culprit unfortunately is hop 6 this is where the packets get lost. I can not speak for your visitors since I am not aware of their routing or if indeed they are routed through ATT as well. .

I have a site that I frequent to check network routers which I can not vouch for their accuracy but it gives some indication of what may cause connection latencies on the internet.

http://www.dslreports.com/routerwatch

It is handy enough that your visitors may want to check to see if any of their ISP network are affected by these routers.
--
Please feel free to contact us with any concerns or questions.

Best Regards

Same problem again this morning at 6:28 am Pacific Time (from LA/Orange County area).

Got 3 IE cannot display this page messages so far.  Response time about 5-10 seconds for most screens.  During the day, response time is instantaneous.

I don't see any latency issues in this tracert (just taken during out of the "waiting for response" screens".  Cox is my ISP.

Tracing route to radified.com [74.50.3.195]
over a maximum of 30 hops:

 1     1 ms    <1 ms    <1 ms  192.168.1.1
 2    10 ms     9 ms     9 ms  ip68.xx.oc.cox.net [xxxxxx]
 3     9 ms     8 ms    11 ms  68.4.15.209
 4    10 ms     9 ms    10 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]
 5    12 ms     9 ms     9 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]
 6    13 ms    12 ms    10 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]
 7    28 ms    18 ms    19 ms  nwstdsrj01-ge710.rd.lv.cox.net [68.1.0.85]
 8    19 ms    41 ms    35 ms  24-234-6-1.ptp.lvcm.net [24.234.6.1]
 9    17 ms    17 ms    18 ms  24-234-6-66.ptp.lvcm.net [24.234.6.66]
10    20 ms    19 ms    19 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]
11    19 ms    21 ms    18 ms  cust-lunarpages.marquisnet.com [208.65.158.130]
12    20 ms    22 ms    19 ms  hafgan.lunarservers.com [74.50.3.195]

Trace complete.

my 1st page came up missing, as if the site were down.

1st refresh brought it up.

0630 from orange county.

sbc global.

Target Name: radified.com
        IP: 74.50.3.195
 Date/Time: 5/25/2007 6:45:33 AM

3   13 ms   13 ms   13 ms   13 ms   13 ms  dist4-vlan60.irvnca.sbcglobal.net [67.114.50.66]
4   65 ms   13 ms   42 ms   14 ms   13 ms  bb2-g2-0.irvnca.sbcglobal.net [151.164.41.239]
5   16 ms   15 ms   15 ms   16 ms   18 ms  ex2-p14-0.eqlaca.sbcglobal.net [151.164.191.75]
6   14 ms   16 ms   16 ms   16 ms   16 ms  gar6.la2ca.ip.att.net [12.122.79.109]
7   35 ms   33 ms   35 ms   35 ms   35 ms  [12.127.3.202]
8   33 ms   34 ms   33 ms   32 ms   32 ms  gar1.lv2nv.ip.att.net [12.123.199.245]
9   34 ms  146 ms   33 ms   34 ms   32 ms  [12.118.228.38]
10   33 ms  146 ms   34 ms   34 ms   32 ms  cust-lunarpages.marquisnet.com [208.67.189.246]
11   33 ms  116 ms   35 ms   32 ms   32 ms  hafgan.lunarservers.com [74.50.3.195]

Ping statistics for radified.com
Packets: Sent = 5, Received = 5, Lost = 0 (0.0%)
Round Trip Times: Minimum = 32ms, Maximum = 116ms, Average = 49ms

It looks like the problem hop is hop number 9 in  your latest trace.  Looking back at yesterday's trace, it looks like this same hop had and error.  Since its an AT&T IP address, I'm guessing its the customer side (Lunar Pages) of an Internet Circuit provided to Lunar Pages by AT&T.  If this circuit or the router it is on is maxing out in the mornings, that could be our culprit.  

Based on the traces provided by Ghost4me, his traffic takes a different route and doesn't go through AT&T.  Since he has the same issue at the same time, I would guess the circuit is not the issue, leaving me to suspect the router.

We could easily detect if the router is overutilized.  Next time this happens, we need to ping this router.  That is 24.234.18.130 for NightOwl and Ghost4me and 12.118.228.38 for Rad.  I'm assuming these two different circuits terminate on the same router, which is common practice.  Routers process ping replies as a very low priority, so pings to this router should be noticably higher than pings to the radified server if the router is overutilized.  If not, we'll have to go back to the drawing board.

Some but not as many pauses this morning (6:30am PDT).  Maybe everyone is on vacation or weekend.

Pinging 24.234.18.130 with 32 bytes of data:

Reply from 24.234.18.130: bytes=32 time=21ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247
Reply from 24.234.18.130: bytes=32 time=20ms TTL=247
Reply from 24.234.18.130: bytes=32 time=18ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247
Reply from 24.234.18.130: bytes=32 time=18ms TTL=247
Reply from 24.234.18.130: bytes=32 time=17ms TTL=247
Reply from 24.234.18.130: bytes=32 time=18ms TTL=247
Reply from 24.234.18.130: bytes=32 time=18ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247
Reply from 24.234.18.130: bytes=32 time=17ms TTL=247
Reply from 24.234.18.130: bytes=32 time=18ms TTL=247
Reply from 24.234.18.130: bytes=32 time=19ms TTL=247

Ping statistics for 24.234.18.130:
   Packets: Sent = 15, Received = 15, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:
   Minimum = 17ms, Maximum = 21ms, Average = 18ms

I spoke too soon.  Just got several repeated "IE unable to display this page"  Then error in the post from Rad board (like part of the packets got lost in the retries).

yesterday, i twice got "unable to display this page" errors. in the afternoon. could not to connect to *any* page on the site.

hit refresh like crazy .. no joy.

problem only lasted a few minutes. by the time in called the LP sppt number, the site was back.

duriung this time, i *could* ping the site (nice numbers), and could ALSO access both email abd the site via FTP. (It's like Apache was down intermittently.)

I sent email to LP. Their response:

Ticket Subject : slow response times?

Reply:
Hi.

I will need to escalate your ticket so that this can be investigated further. It appears at times that the issues lie soley with the routing and at other times it appears from the posts on your forum that the routing plays no part in it.

I have made notes to this ticket and hopefully when investigated further may prove to be helpful in providing a resolution.

Kind regards

[These guys are the experts.]

EDIT .. And posting this message gave me the same error. A retry worked fine. Never had any such problem at the old server (the one in LA).

here's a trace, done just now (26may 9:25 AM PST)

http://radified.com/gfx2/trace_26may.gif

Everything is normal right now (10:30am PDT).  However, look at this PingPlotter I just completed:

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter1.gif

It appears, as I read it, there was  33% packet loss at the router just before their server.  This is the one that MrMagoo suggested we look at.

Here's another 15 sample PingPlotter 30 minutes later than the first one.

14% packet loss over 15 samples.

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter2.gif

i see that.

otherwise, those numbers look good. thanks for helping to troubleshoot.

i wonder why my *gifs are 25 KB, and yours are 95.

today LP assigned my dedicated ip (finally, after a surprisingly long delay). that might help (tho probably won't).

i might add here, that my radified email-checking (i have a dozen accts i use, outlook express) is *blazing* fast. fastest i've ever seen. not sure what that might mean. but still a data-point to consider.

when i *do* get the site, it seems smokin' fast.

Rad wrote on May 26^th, 2007 at 1:40pm:

i see that. otherwise, those numbers look good. i wonder why my *gifs are 25 KB, and yours are 95.

I noticed the same thing when I posted the images.  The first one was a jpg, and I experimented with saving as a gif.  (I'm using PhotoShop Elements 5.0)  The image is 36K on my hard drive, so I don't know why it's 95K on the webpage. ??

i use photoshop cs3 > "save for web" *gif.

elements i believe is more for *pictures*. right?

your (gradient) xp-style title bar might be causing problems there. (i use "classic".)

Photoshop Elements is the poor-man's (about $100) version of Photoshop ($500+) but most of the editing features of Photoshop.

I still can't figure out why the image is 36K on my hard drive and 90K+ on the webpage.  Still researching, but give me any ideas if you have them.

It must be a problem with Photobucket, because I uploaded the same images to my Yahoo briefcase account, and the files are the same there (35-40K) as my hard drive.

Fixed.  The default option for Photobucket was 800 pixels wide, which is what caused the explosion.  I changed the default to larger pictures and now the file sizes are back to 35-40K.

I usually use jpg for pictures but noticed that Photoshop Elements help stated that for line art, gif is actually better and smaller.

Another PingPlotter from 4:00 pm PDT.  Note Packet losses at hop 9 and 10.

Here's a help entry on Packet Loss from their website:
http://www.nessoft.com/kb/24

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter4.gif

Wow, this is a tough issue.  It seems to morph daily.  At first, the network didn't seem to be involved.  Then it was network delays.  Now, we are seeing packet loss.  Each would be caused by a slightly different problem and would need a different solution.

Hopefully whoever they escalated your ticket to at LP has better visability into what might be going on and can tie it all together.

MrMagoo wrote on May 26^th, 2007 at 9:21pm:

Wow, this is a tough issue.  It seems to morph daily.  At first, the network didn't seem to be involved.  Then it was network delays.  Now, we are seeing packet loss.  Each would be caused by a slightly different problem and would need a different solution. Hopefully whoever they escalated your ticket to at LP has better visability into what might be going on and can tie it all together.

MrMagoo, Any idea about the packet loss?  The link seems to indicate it doesn't matter unless it's the final hop:

http://www.nessoft.com/kb/24

Quote:

There are numerous reasons why packet loss occurs: Insufficient bandwidth; network connection problems; hardware failure; routing problems; router configuration; and others. Let's talk a bit about what kind of packet loss is probably significant and what kind of packet loss might not be impacting your network experience. First, it's really important to understand that the only hop that matters is the final destination. If the final hop (your target) is showing 0% packet loss and acceptable latency, then all the hops before that can show all kinds of errors and it doesn't matter. As long as the final destination isn't affected, then all other latency and packet loss is an artifact of router configuration (or similar), and there is no problem. The only hop that matters is the final destination. If you're happy with the latency and packet loss being seen at the final destination, then none of the other hops matter.

I (mostly) agree with the quote you posted from the article.  A traceroute is really nothing more than a series of pings with an increasing Time To Live value, tricking each succesive hop to respond with an ICMP TTL expired packet.  From that packet, the tracert program can learn the time it takes each hop to respond and the IP address of each hop.  The problem is that a router will send ICMP packets at a very low priority in order to ensure that packets traveling through the router to their final destination are taken care of first.  This means that although this router may be close to maximum utilization, most of the traffic going through this router is getting to its destination with little or no delay.

Packet loss is usually a product of a circuit that has more traffic than it is designed to carry or a router whos CPU is maxed out.  In this case, I suspect the router CPU since we seem to see the same issue on the same hop (right before the cust.lunarpages.com) using different circuits.

I'm sure that LunarPages has graphs of the circuit utilization, router CPU utilization, and web server utilization over time.  I'm sure they also have software to check the response time of apache, SQL, and and other services running on this server.  If we could get one of their engineers to look into the issue right at the time we are having trouble, hopefully one of these graphs would show which one is the issue.  That is why I asked if it is happening in the forums, the main site, or both.  An issue in the forums but not the main site would point to delays on the SQL server.  Since you guys are reporting problems across the board, it seems we can rule out SQL.  

I was unable to access the site .. any page .. with any browser (tried IE, Firefox, SeaMonkey, Opera) .. this morning (Sunday) from ~8AM to ~ 9AM. Actually, the site came back ~8:50 AM.

I sent mail to LP sppt, and tried to call, but never got thru > was on hold the whole time (so I wasn't the only one calling).

Their 24x7 "Network status line" said all was fine.

I ran a bunch of pings, all of which were okay (nice, actually).

I could connect via FTP fine, and Radified email was blazing fast.

Only no web pages would display.

I had no problem connecting to any other page on the net.

It seems like Apache was down.

The site might have been down before 8AM, but I didn't try until then.

I was getting "IE unable to display webpage" over and over again from 8:30 to 8:55am today.  The total site seemed down.  Finally, now at 8:58am I got on.  Here's tracert and pingplotter taken during that time.

8:30am May 27, 2007 from So. California

Tracing route to radified.com [74.50.6.11]
over a maximum of 30 hops:
 
1      1 ms    <1 ms    <1 ms  192.168.1.1  
2     10 ms     *        9 ms  xxxx.cox.net   
3      9 ms     7 ms     7 ms  68.4.15.209  
4     10 ms     9 ms    11 ms  ip68-4-14-53.oc.oc.cox.net [68.4.14.53]  
5     10 ms    10 ms     9 ms  ip68-4-14-73.oc.oc.cox.net [68.4.14.73]  
6     14 ms     9 ms     9 ms  rsmtdsrj01-ge600.0.rd.oc.cox.net [68.4.14.253]  
7     21 ms    17 ms    17 ms  nwstdsrj01-ge710.rd.lv.cox.net [68.1.0.85]  
8     18 ms    19 ms    19 ms  24-234-6-9.ptp.lvcm.net [24.234.6.9]  
9     22 ms    20 ms    20 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]  
10    21 ms    19 ms    19 ms  24-234-18-130.ptp.lvcm.net [24.234.18.130]  
11    21 ms    40 ms    21 ms  cust-lunarpages.marquisnet.com [208.65.158.130]  
12    22 ms    20 ms    19 ms  hafgan.lunarservers.com [74.50.6.11]
Trace complete.

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter5.gif

same here. see above.

Note on my PingPlotter that 10% packet loss for marquisnet.com.  

That also corroborates and corresponds with the tracert hop 11 additional delay of 40 ms caused by marquisnet.com

saw that.

Of course, none of those symptoms match yesterday's symptoms exactly or the day before...

Rad, When you connected via FTP, did you attempt to transfer any files?  I'm starting to think that this is a problem with Apache specifically, and if file transfers worked normally, that would be good evidence the rest of the server and network are running fine.  You've previously said that email works fine when the site is down, which again points to apache.  Also, is your homepage static HTML, or is it tied to your blog?

Can someone try a static page like one of the guides next time we have trouble?  I'd like to know for sure that all HTML pages are affected, rather than just those that use server-side scripting and databases.

I have a few questions for LP support such as:

1. Is the web site data (html) stored on a local disk on your web server, or is it on some network share?
2. Is your dedicated IP tied directly to a specific server, or is it load balanced across a cluster of servers?
3. Is the SQL database on this same server?

Rad, maybe you can answer #3 for me - What is the address of you SQL server?  Is it "localhost", or something else?

At any rate, I think we've ruled out any issues with our individual ISP's and computers, so this issue is almost surly on the LP end, which means they need to get involved.  Even if we knew exactly what it was, they would need to be the one to fix it.

Rad, do you have a ticket open with support on this issue, or just a few emails back and forth so far?  You should probably call them and ask that they open a formal ticket to help track this issue.  Make them give you the ticket number so you can follow up.  Maybe then they can get enough data to get the right person to look into it.

I just had (6:20 am PDT Monday), two 20-second "pauses".  But they both recovered.  

Spoke too soon.  Now getting very erratic responses and lots of "IE Cannot display this webpage" errors.

John. wrote on May 28^th, 2007 at 8:42am:

Spoke too soon.  Now getting very erratic responses and lots of "IE Cannot display this webpage" errors.

And did you attempt to access the guide pages at all?  What was the result of that?  I'd like to narrow down if this is affecting all pages, or just dynamic ones.

MrMagoo wrote on May 28^th, 2007 at 3:14pm:

And did you attempt to access the guide pages at all?  What was the result of that?  I'd like to narrow down if this is affecting all pages, or just dynamic ones.

In one case, I was trying to click on the "Post" icon.
Then I couldn't get to the main page:  http://radified.com/cgi-bin/yabb2/YaBB.pl at all.

I'll save a shortcut to this one ( http://windows.radified.com/ ) to try next time.  I think that is a static webpage example you were looking for.

This is probably way off track, but I assume that the yabb server code maintains a long list of members to notify anytime you post to a thread.  Does anyone know if there is a "purge" of notifications for threads that are over for example 6 months old or 12 months old?  With a large database of topics and members, that could be a long list to search or update anytime someone replies.  (I just went through and pruned my notifications list myself.)

Just looking for ways to optimize the db code overhead or compact or reorganize the db.

I think that the list of people to notify is probably stored in the database per thread, so having an email notification to updates in one thread should not affect the performance of another thread.

I think we can narrow down what the problem is a little more, and then our ideas on how to help fix it can be more focused.

Personally, I think the problem is those slow, SCSI hard disk drives used by Lunapages Services.   ;D

(Humor intended, of course!)

MrMagoo wrote on May 28^th, 2007 at 3:14pm:

And did you attempt to access the guide pages at all?  What was the result of that?  I'd like to narrow down if this is affecting all pages, or just dynamic ones.

Just had another experience, occuring right now, at 9:40 pm PDT.  Very slow (about 30 seconds) for any page to display.  I then opened two new tabs (IE7) and tried loading these two static pages:

http://radified.com/Scans/image_12.htm
http://ghost.radified.com

NEITHER of them would load either.  Eventually they would display, but when I tried refreshing those pages or the Rad dynamic pages, ALL of them are very very slow.  (I hope I can get this posted now....)

Am I the only one (besides NightOwl and Rad) noticing these web response time problems?

Back to normal response time now.  9:50 pm PDT.

Quote:

Am I the only one (besides NightOwl and Rad) noticing these web response time problems?

It's the same here and Ping Plotter is no different during and after an outage.

Thanks Brian.  You probably have the record for being the furthest away from Las Vegas.

The "outage" tonight lasted only about 5 minutes from when I first checked it.  The ones on the mornings (6am Pacific Time) seem to last about 15 to 30 minutes.

So that confirms it is likely not the SQL server and has more to do with Apache itself.  I'd be interested in knowing if LP uses mounted network shares, or if this server is load balanced, or if all the content is server from a local disk on one server.

It was excellent until just a minute ago (6:48am PDT).  Now 5-10 second pauses.

Same problems on this simple small static webpage at the same times.

http://radified.com/Scans/image_12.htm

MrMagoo wrote on May 27^th, 2007 at 10:08pm:

Of course, none of those symptoms match yesterday's symptoms exactly or the day before... Rad, When you connected via FTP, did you attempt to transfer any files?

Not always, but yeah, when I did, FTP x-fers were fine. Fast as ever.


Quote:

I'm starting to think that this is a problem with Apache specifically,

me too. i've had apache problems at the last host, and the symtoms i'm seeing this past week look identical. Note that the version of Apache run on new server is SAME as that run on previous server (1.3.37).


Quote:

and if file transfers worked normally, that would be good evidence the rest of the server and network are running fine.  You've previously said that email works fine when the site is down, which again points to apache.  Also, is your homepage static HTML, or is it tied to your blog?

home page is static, plain html, not tied to the blog or any other script.


Quote:

Can someone try a static page like one of the guides next time we have trouble?  I'd like to know for sure that all HTML pages are affected, rather than just those that use server-side scripting and databases.

it's *all* pages. that's the first thing i tried .. were the (static) guide pages. unable to bring up *any* html pages (as if apache were down).


Quote:

I have a few questions for LP support such as: 1. Is the web site data (html) stored on a local disk on your web server, or is it on some network share? 2. Is your dedicated IP tied directly to a specific server, or is it load balanced across a cluster of servers? 3. Is the SQL database on this same server? Rad, maybe you can answer #3 for me - What is the address of you SQL server?  Is it "localhost", or something else?

yeah, mysql database is localhost.


Quote:

At any rate, I think we've ruled out any issues with our individual ISP's and computers, so this issue is almost surly on the LP end, which means they need to get involved.  Even if we knew exactly what it was, they would need to be the one to fix it. Rad, do you have a ticket open with support on this issue, or just a few emails back and forth so far?  You should probably call them and ask that they open a formal ticket to help track this issue.  Make them give you the ticket number so you can follow up.  Maybe then they can get enough data to get the right person to look into it.

i have an open ticket, which i've been informed has been "escalated" but haven't heard back since said escalation.

Pleonasm wrote on May 28^th, 2007 at 5:19pm:

Personally, I think the problem is those slow, SCSI hard disk drives used by Lunapages Services.   ;D (Humor intended, of course!)

Humor is good, especially when trying to troubleshoot these non-humorous issues.

Wow!  That was a 20 minute total outage.  From 11:30am PDT to 11:50, I was trying to open this small static webpage.  Nothing but repeated "IE cannot display this webpage" at which time I tried refreshing over and over again.

http://radified.com/Scans/image_12.htm

Complete outage for 20 minutes.   >:(  >:(

just got off the horn with LP supervisor.

he said i really have no option but to request a move to a new server. Sent this note just now:

hi.

Lunarpages moved my site to a new server on May 11th. Since then I've been having intermittent problems reaching the site.

It appears to be a problem with Apache, since I am always able to access both email & ftp.

The problem has been going on a couple of weeks now, and I'm getting desperate that a solution will never be found.

After speaking with one of your supervisors,he suggested I request to be moved to a new server (again).

Note that there was nothing wrong with my previous LP server (Callisto).

However, I need both PHP5 & MySQL 5.

So I am requesting that you again move my site to a new server (with both PHP 5 & MySQL 5).

I have backed-up both the site and its databases, so you are free to do this at your earliest convenience.

I also request that you move my site to a server physically located in Los Angeles, where the other trouble-free server was located (Callisto).

So, to recap, please move my site to a new server physically located in Los Angeles, supporting both PHP 5.x & MySQL 5.x, at your earliest convenience.
I *do* have a dedicated IP (that I pay extra for).

And I have one add-on domain (hardnuf.com).

I have just done this, so I'm familiar with the drill.

Rad, seems like you have done everything one could reasonably expect.

I thought that LP said they could/would monitor the problem if they knew when it occurred.  There should be plenty of times and traces on this thread to satisfy that.  I'm just surprised that they can't even guess or determine where the problem exists, especially since it happens every morning from 6:15 to 7:00 am PDT.  

To say the only resolution is to "move to a different server" is kind of like me telling a pc user, to just "buy a new computer" to solve any problem.

(just venting on them, not you.)

there are different groups within LP. Support and admins.

Supposedly, you can't actually *speak* to an admin .. only communicated via email.

And if you send emails, and get no response, what do you do?

Call support.

They can talk to admins, but you can't.

Maybe they are busy with bigger fish. I dunno.

In their defense, they (the admins) weren't notified of the problem until late 25th, early 26th, which includes a holiday.

If I could *talk* to whoever is working on my issue, and find out WHERE they are with it, and what they've discovered, I could make a better decision.

But not knowing ANYthing, leaves me little choice (like the LP supe said) but to request a move to a new server (which is never much fun).

It is most distressing for me to see the site unavailable.

Thanks for all your hard work and dedication to this!   :)

Comments from LP sppt:

Reply:
Hello.

I read the entire forum thread you posted.  At this point we cannot definitively say whether it is a network issue or a server issue.  There is nothing apparently wrong with the server - no load issues, no alarms ringing anywhere, nothing out of the ordinary, but it is clear from your users that there is a problem somewhere.

I saw in your latest forum posts that you requested a server move back to Los Angeles but I don't see that in this ticket.  Did you open a new ticket?  It would be easy enough to move the account back West if that's the path you want to take.

For the time being I will leave this ticket open while we dig around and we can hope to catch the problem while it's happening.  If it is a server problem it is likely affecting other people as well (although I have yet to see their tickets) so I would like to get it resolved.

I hope you understand the difficulty in troubleshooting an intermittent issue such as this, and I appreciate your patience in the matter.

--
Best Regards

[Rad note - yes, new ticket, and yes, I understand the difficulty of troubleshooting intermittent problems. But it's not just me, and it's not an uncommon problem. It seems Apache was down this morning. That's what the Supe told me. Why was Apache down? Why do I have to call to tell LP Apache (the site) is down? What if I wasn't around, or didn't call?]

Even when the site is down, I can still access email, FTP and pings look fine (most of the time). That seems like the characteristics of a problem with Apache, no?

We've been having problems since the 22nd, and it seems to be getting worse. At first the site was just laggy (10-60 secs). Now we're having outages that last upto an hour.

It's really unfortunate they can't locate the source of the problem.  Because response time is really excellent and fast most of the time.  It's just that when it's bad, it's bad.

yeah, i agree. it was fast most of the time. but when it wasn't, it was really distressing.

this is the new server.

whatdaya think?

any problems?

I think (knock on wood) it's better and (knock on wood) fast.  I'm amazed Rad at how quickly you got everything moved!

Here's a PingPlotter just taken (6:30am PDT) of the NEW NEW (Los Angeles) server.  Note the really fast round trip times., although in fairness Orange County to LAX is a shorter hop, than OC to Las Vegas.

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter6.gif

http://network-tools.com/default.asp?prog=network&host=209.200.242.173

209.200.242.173 is from United States(US) in region North America

whois query for 209.200.242.173...

Results returned from whois.arin.net:

OrgName:    ADDD2NET COM INC DBA LUNARPAGES
OrgID:      ACIDL
Address:    Add2Net, Inc.
Address:    Lunarpages Division
Address:    100 East La Habra Blvd.
City:       La Habra
StateProv:  CA
PostalCode: 90631
Country:    US

Rad

Looking good so far this a.m.--no slowdowns or delays noted!

Previous average pingplotter round trip was 60 ms--now it's 40 ms!!!

actually, lunarpages did everything. i just submitted a request & backed-up the site.

here's a ping taken 11:30 am today, probably the site's busiest time:

http://radified.com/gfx2/ping_kwan_server.gif

no outages observed yet.

the new server name is KWAN.

here's another taken at 11:45:

http://radified.com/gfx2/ping_kwan2.gif

Still looking great!  (2:15pm PDT)

http://i24.photobucket.com/albums/c5/ghostimage/PingPlotter7.gif

dang, john. that's smokin'.  i've seen network pings in the same house that weren't that fast.  :)

nice when things work the way they're supposed to. here a (nice) tracert i just took at 4:30 pm:

http://radified.com/gfx2/ping_kwan_30may.gif

Rad, I keep trying to find some anomaly to report, but every time I run PingPlotter at different times during the day and evening, I get the same 13 or 14 msec round trip.  

Everything looks and feels very consistent.

Seems like we can call this resolved with the server move.  The complaints stopped suddenly as soon as the move completed.

I agree.  Out of curiosity, Rad has Lunarpages acknowledged that there is/was a problem with the Apache software on their Las Vegas server?  I assume the problem is there lurking for their other customers using that server.

Unfortunately, I've seen a couple of "Page cannot be displayed". But it lasts less than 30 seconds. I was reluctant to report this but it hasn't happened with other sites.

here is their last response regarding the ticket:

Reply:
Hello

I read the entire forum thread you posted.  At this point we cannot
definitively say whether it is a network issue or a server issue.  There is
nothing apparently wrong with the server - no load issues, no alarms ringing
anywhere, nothing out of the ordinary, but it is clear from your users that
there is a problem somewhere.

I saw in your latest forum posts that you requested a server move back to Los
Angeles but I don't see that in this ticket.  Did you open a new ticket?  It
would be easy enough to move the account back West if that's the path you want
to take.

For the time being I will leave this ticket open while we dig around and we
can hope to catch the problem while it's happening.  If it is a server problem
it is likely affecting other people as well (although I have yet to see their
tickets) so I would like to get it resolved.

I hope you understand the difficulty in troubleshooting an intermittent issue
such as this, and I appreciate your patience in the matter.

--
Best Regards

Anyone notice slow page loads in the last week? Today it sometimes takes a minute or longer. At other times it's fast. Other websites are fast.

No, Brian - I for one have noticed no change in the responsiveness of this website.