Unusual GotApex Slowness

[TheCorp]

As some of you may know, GotApex has seemed a bit, shall we say, under the weather for the past few days, and right now we are working on trying to find out why exactly GotApex.com and its forums are loading slowly.

The first assumption would be that all the cool GotApex users are overloading the server and it can't handle the traffic, but since loads are amazingly low (since the server upgrade) we are almost 100% sure thats not the case. The next possibility some of you might be thinking is that the problem may be on your (the users) end. With the volume of complaints lately from so many different areas, we have pretty much ruled that out as well. SO, through the process of elimination we came to the conclusion that it must be the fibre optic connection to our server, but once again that isn't the case because during our testing, other sites on the RealGN server (same server, same HD, even same forum software) seem to be running smooth as ever. SO, what we need you to do is go ahead and check out the following list of sites and tell us whether you think they are running slow or not.

http://www.gamingin3d.com/forum/ <-- Vbulletin forums
http://www.gamingin3d.com/
http://www.darklights.com
http://forums.tfoutpost.com <-- Vbulletin forums

No this isn't a cheap ploy to get more hits to our sites (although that is an interesting point that someone pointed out to me ;) we are just trying to fix the slow GotApex situation as quick as we can, and only you can help! So just make a post here about your experiences with these sites, and if the GotApex site is getting any better, we would really appreciate it, thanx!

-TheCorp "Your Favorite RealGN Admin" =)

[Edited by TheCorp on 09-14-2000 at 02:19 PM]

[coleslaw]

All of those sites seem a little on the slow side for me. I noticed that some of the threads take a while to load just like the ones here recently. hmm...

[TheCorp]

I see, thats weird, I talked to all their admins and not one has complained yet about slowness, and darklights.com is about 5X the size of GotApex so I really dont know what to think.

Well I guess we will have to wait for more info from the rest of the forum squad. Thanx a lot coleslaw we really do appreciate the info.

-TheCorp

[AmRivlin]

Yep. I am sitting here on a Ultra 10 on a LAN with direct access to a T3. Thanks out to KSU, anyway here are the stats!

http://www.gamingin3d.com/forum/ 13.2 Seconds
http://www.gamingin3d.com/ 29.8 Seconds
http://www.darklights.com 13.7 Seconds
http://www.tfoutpost.com/ 82.3 Seconds
http://forums.tfoutpost.com 6.? Seconds

All times are provided by Timex, Indiglo mode no active. These results should not be used for any other purpose than to correct the problem involving APEX forums, if you have any questions contact my secretary LPM.

YUP 82 Friggin seconds, LOST PRODUCTIVITY PEOPLE!!! I'm calling my lawyer!

[bicycleKICK]

the first 2 were good, the 3rd one was really fast and the last 2 were good.

[NiceMann]

Not a problem on those boards with me. All loaded up with no problems at all. I am on cable. But Apex has been lagging with me the last day or so.

~NiceMann

[sbp]

The first assumption would be that all the cool GotApex users are overloading the server...
Immediately they thought of blaming us for this. :\ Now he tries to make it up to us by calling us cool. We see right through that transparent brownnosing attempt there buddy. http://sbp777.homestead.com/files/facehappy.gif


but since loads are amazingly low...Now he's saying we don't post enough and we're paltry. Bah! :mad:


The next possibility some of you might be thinking is that the problem may be on your (the users) end.They kept blaming us for everything and said we had crappy computers. :\


With the volume of complaints lately from so many different areas, we have pretty much ruled that out as well. Finally they listened to us. :/


No this isn't a cheap ploy to get more hits to our sites...Oh sure its not. :rolleyes: We're on to your Corp!

------------------------------------------

Both the gamingin3d site and forums are fast.
The tfoutpost site and forums are a bit slow. Uggh all that green. :sick:
darklights.com loads ok. Its not fast but not slow.

-sbp "Your favorite pain in the arse poster at the Got|Apex BBS. Why because he's been here so long and ran out of beer!"

[Edited by sbp on 09-14-2000 at 02:22 PM]

[TheCorp]

Oh come on Sbp, you know I didn't mean that stuff in that way, I was just going through our thought process, thats all.

Note: The TFOupost.com design loads relatively slowly because of the design not the connection, so we should rule that one out I think, ill take it out.

But the others are fair game, so I guess we will see with some time. Oh and btw, here is our current server load for those of you who know what this stuff means ;)

2:19pm up 11 days, 10:48, 2 users, load average: 0.72, 0.90, 0.86

And you can check out the load at any time at http://www.realgn.com on the bottom right.

-TheCorp "about to beat sbp for doubting me" =)

[Nanotech9]

I'm at work on a T3 (I think, I've hear rumors it might be an optical conection), but the part of the intranet I'm on is only running on a 10Mb (Mbit) hub. Anyways, the first two on the list loaded up OK, an then I got to darklights... it took a while. I cruised around a little on their site, and hit one of the topics on their forums that took about 12 secs to load up (compared to 3 or 4 sec for most other sites that don't overuse images/tables).

As I type, the bottom half of the page (the part w/ everyone else's replys that you can see while you type your reply) is still loading. Oh, hange on, it just finished... about 2 mins load time :(.

[Newbiee]

For me:
http://www.gamingin3d.com/ and its forum were much quicker than gotapex
http://www.darklights.com was a bit slow but still faster than gotapex
and the tfoutpost you said to disregard

[TheCorp]

Hmm well if anyone has any ideas as to why GotApex may be the only slowish site on the server, I am open to suggestions, in the mean time, I am apologize for the slowness, and we should have an answer for you soon...

-TheCorp

[coleslaw]

Yes, there is a distinct "stuttering" in the thread loading. It will load the page to a certain point, stop for a while, load some more, stop again, and then load the rest of the page. :hmm:

[coleslaw]

I forgot to mention that this "slowness" has only occurred in the past two days. When the new forum design was implemented, the pages loaded super fast... now it's just S...L.....O..........W

[dcsonic2k]

I've noticed the same thing...Sometimes its really fast, othertimes its really slow....Just to load the forum index took about 30 seconds. This reply page loaded really fast, other things took a while. I'm on a modem, so nothing loads blazingly fast, but I could definetly tell it was slowed WAY down.

I'm not an expert, but I think it might be a switch or something like that. Anybody else know anything more?

[coleslaw]

I just checked Slashdot and every page (even the articles with over 300 replies) loaded within a second. Also, I ran a bandwidth speed test and it shows that I am connected at over 1.5 Mbps. :hmm:

[Edited by coleslaw on 09-14-2000 at 03:05 PM]

[importedpimp]

No problems here with those sites.

[RaginRaj]

all those sites seem to running ok off my DSL at work... got | apex? on the other hand seems to be taking it's time. even this reply page took longer than normal to load

[Justin42]

I haven't noticed any out of the usual problems with ANY GotApex site either at home (cable modem) or work (T3)... everything seems much faster lately (the last couple weeks) if anything!!

[MonkeyBrains]

those sites loaded reguraly for me. gptpaex is still loadeding slow though

[50ftqueenie]

I loaded each site four times each. The first time, all of the sites loaded regularly. Second time, all of the sites loaded extremely slowly --except for the gaming site which actually loaded quicker than the first time. WTF? (Whoever described it as stuttering was right-- they sort of hang there for 30 seconds or longer after starting off fine.) Third time, none of the sites loaded-- got that DNS error message. Fourth time, same results as the first.

Gotapex is definitely loading the slowest of all, though. And I agree that this just started two days ago. Before that, everything was gravy.

[grooveman]

Evening everyone! =)

Ah - 'tis a firewall! =)Please disregard the following.

The network gurus probably are on such a thing, but I seem to be having trouble once my computer reaches 63.210.161.154. My local traceroute gets there and reports "Destination Net Unavailable" A similar traceroute from the cerf.net looking glass reports the following (Asterisks generally indicate timeouts) :

Type escape sequence to abort.
Tracing the route to gotapex.com (63.214.181.197)

1 fe8-0-0.san-bb1.cerf.net (134.24.38.225) [AS 1740] 0 msec 4 msec 0 msec
2 fe0-0-0.san-bb2.cerf.net (134.24.29.102) [AS 1740] 0 msec 4 msec 4 msec
3 so2-1-2-155M.san-bb4.cerf.net (134.24.29.181) [AS 1740] 0 msec 0 msec 4 msec
4 pos3-0-622M.lax-bb5.cerf.net (134.24.29.14) [AS 1740] 8 msec 4 msec 8 msec
5 pos4-0-622M.lax-bb5.cerf.net (134.24.33.174) [AS 1740] 4 msec 4 msec 4 msec
6 inter-connect.cerfnet.level3.net (209.245.82.49) [AS 3356] 4 msec 8 msec 4 m
sec
7 so-4-0-0.mp2.LosAngeles1.level3.net (209.247.10.201) [AS 3356] 8 msec 8 msec
4 msec
8 loopback1.hsipaccess1.SanDiego1.Level3.net (209.244.2.4) [AS 3356] 12 msec 8
msec 12 msec
9 63.210.161.154 [AS 3356] 12 msec 8 msec 12 msec
10 * * *
11 * * *
12 *
gotapex.com (63.214.181.197) [AS 3356] 44 msec *
route-server.cerf.net>

Just for another confirmation, bbc.co.uk reports the following :

1 rt1.kw.bbc.co.uk (132.185.132.2) 4 msec 0 msec 0 msec
2 rt2-HSSI5-0.thdo.bbc.co.uk (212.58.255.30) [AS 9156] 4 msec 4 msec 4 msec
3 rt0.thdo.bbc.co.uk (212.58.224.1) 4 msec 0 msec 0 msec
4 195.66.225.77 [AS 5459] 4 msec 4 msec 4 msec
5 loopback0.core2.London1.Level3.net (212.113.2.66) [AS 9057] 0 msec 4 msec 4 msec
6 atm11-0-103.core1.london1.level3.net (209.244.3.198) [AS 3356] 76 msec 76 msec 76 msec
7 so-6-0-0.mp2.NewYork1.level3.net (209.247.10.45) [AS 3356] 80 msec 80 msec 76 msec
8 loopback1.hsipaccess1.SanDiego1.Level3.net (209.244.2.4) [AS 3356] 152 msec 152 msec 152 msec
9 63.210.161.154 [AS 3356] 156 msec 152 msec 152 msec
10 * * *
11 *
gotapex.com (63.214.181.197) [AS 3356] 184 msec *



It seems as if there is a routing problem between 63.210.161.154 and gotapex.com (63.214.181.197). It should be noted, however, that this problem seems to be confined to certain localities. Within these localities, all packets make it to 63.210.161.154, but that machine does not seen to send a good deal of them from itself to gotapex.com. At times, it should be noted, that the machine seems to send all traffic over to gotapex.com OK, but at other points it dies for certain connections. (I.e., at points my connection to GotApex will not be too bad, at others it crawls pretty bad) Other places do seem to work fine, though. For instance, if I run traceroute from UWisconson, it ends up with "16 gotapex.com (63.214.181.197) 85.307 ms 85.563 ms 85.180 ms". There do not seem to be any time outs from there. UMaryland traceroute reports the same. Again, I'm sure the network gurus probably are on this already, but just thought I might toss in my two cents. **plonk, plonk**

Cheers everyone,
Andy

[Edited by grooveman on 09-14-2000 at 07:20 PM]

[renots]

...maybe it's Carnivore (http://www.stopcarnivore.org/)

;0)

[TheCorp]

Ahh as a note to those who were doing traceroutes to our server, those very last asterisks you were getting were because of the firewall we reected to make sure there was no DoS attacks on us.

Essentially you can't ping our box, and its for everyone's security, so thats not the problem, but that was definitly a good line of thinking!

Anyways, I really appreciate everyone's input its all going to be put together into some bigger picture we hope to see soon =)

-TheCorp

[jstreet]

Corp,

I'm not a network guru so I can't give you any suggestions, all I can say is that - oddly - I've had much better luck using the forum from home over a 44k connection then from my school's quarter-T1. I've tested GotApex forums against other sites at school, and they for the most part load very rapidly, but the forums here come in at between 1.2 and 1.5k per second. At home, however, I get around 2-3k per second. Now, this may be due to traffic, because I'm at school during the day and at home in the evenings, but that seems a little counterintuitive because one would think that traffic would be heavier at night than during the day.

I know we've already ruled out the traceroute, but I figured I'd paste mine nevertheless.

Tracing route to gotapex.com [63.214.181.197]
over a maximum of 30 hops:

1 105 ms 102 ms 102 ms connect-tnt-1.efortress.com [205.181.169.1]
2 100 ms 102 ms 101 ms CONNECT-RTR.efortress.com [38.150.212.1]
3 112 ms 109 ms 108 ms s6-0-11.bstnma1-cr1.bbnplanet.net [4.24.80.137]

4 103 ms 105 ms 101 ms p5-0.bstnma1-ba1.bbnplanet.net [4.24.4.209]
5 106 ms 98 ms 105 ms p7-0.bstnma1-br1.bbnplanet.net [4.24.7.117]
6 109 ms 100 ms 99 ms p2-0.bstnma1-br2.bbnplanet.net [4.24.7.114]
7 111 ms 111 ms 109 ms p4-0.washdc3-br1.bbnplanet.net [4.0.1.245]
8 110 ms 108 ms 111 ms p3-0.washdc3-br2.bbnplanet.net [4.24.4.146]
9 115 ms 112 ms 110 ms p2-0.washdc3-ba2.bbnplanet.net [4.24.4.141]
10 113 ms 134 ms 111 ms pos2-1.core2.Washington1.Level3.net [209.0.227.1
01]
11 115 ms 112 ms 108 ms so-6-0-0.mp2.Washington1.level3.net [209.247.10.
77]
12 180 ms 180 ms 192 ms loopback1.hsipaccess1.SanDiego1.Level3.net [209.
244.2.4]
13 181 ms 181 ms 181 ms 63.210.161.154
14 63.210.161.154 reports: Destination net unreachable.

Trace complete.

[Sandman]

Does the problem lie in the site or with the message board?

I just recently had problems with our Raq3 server:
-weird php errors (GIGS of them)
-runaway processes causing the server to have an insane load

I cycled the raq killing all the runaways also,did an upgrade and upgraded the versions of php and mysql as well as playing with the meta tags It is much faster now, and found this:

I am not sure if anyone here has encountered this before; however,I did find two links on phpbuilder's website where someone was asking about a similar problem:

http://www.phpbuilder.com/mail/php-general/2000072/1760.php
http://www.phpbuilder.com/mail/php-general/2000072/2051.php

as well as a slightly older link on the same list:

http://www.phpbuilder.com/mail/php3-list/199909/1313.php

I do not know if they will prove helpful to you, but perhaps they may be a start.

[John Short]

Forums almost always load quickly from home. At work forums hang for about 30 sec, waiting for connection to ads.tracker banner stuff. This happens as each forums screen loads. I do not know if this is related to the delays talked about here.

[TheCorp]

Well I would have to say that the Apex crowd has some of the smartest on the block, here are some smarter things we heard through the night...

grooveman : "It seems as if there is a routing problem between 63.210.161.154 and gotapex.com (63.214.181.197). It should be noted, however, that this problem seems to be confined to certain localities."

This definitly gave us a heads up on one thing that we had been checking out (which I will explain below)

sandman : "-runaway processes causing the server to have an insane load"

Definitly could be a problem, but after running some logging things we found that the load never went over 1.8, so thats not the problem, but definitly good try, we have heard of that prob before as well.

coleslaw : "Yes, there is a distinct "stuttering" in the thread loading. It will load the page to a certain point, stop for a while, load some more, stop again, and then load the rest of the page."

I think by far this was our biggest clue, the whole stuterring meant it couldn't actually be the connection, so thats when we went in to do some more exotic tests.

We ran speed tests (like FTP downloads) and were getting insane speeds and then bam, it would crawl for a few seconds, then we tried other things and to no avail, so finally I think at id say 12:30 AM we thought it might be a hardware problem. What could possibly be causing this? Well I can't remember who tipped us off, or when it struck me, but we think its the switch! SO, Roman (the real admin ;)) hauled his ass off down to the facility and plonked in a replacement one, and SO far, I think things are looking better.

SO just tell me how things are, our tests turned up better, actual visual loading time is looking better, overall much nicer, but still its up to you guys to tell us its better. AND since the new switch is only temporary, we are probably going to be going out to buy a nice Cisco 2900 XL 12 Port SNMP 100mbit switch, so if anyone has any deals on something like that please tell me!

-TheCorp "Tired yet satisfied" =)

[TheCorp]

I knew I saw it in this thread, but someone else on my ICQ list also notified me, but it was dcsonic2k

"I'm not an expert, but I think it might be a switch or something like that. Anybody else know anything more?"

Smart cookie =) Well he will be a smart cookie if we find out thats the problem

-TheCorp

[TheCorp]

Hehe who likes how many posts I have? ;)

Btw you can completly disregard it, I was just testing something =)

It would be nice to have a 5 digit post count though =)

-TheCorp

[sbp]

{Begins a revolution against TheCorp :smash:} Down with the man! We're taking over!

[hapoo]

maybe The corp is keeping the server busy, I mean damn 16436 posts in a couple days!!!! no wonder its slow

[hapoo]

The Corp: I still think its the database cause when i try to load a page it imediately loads the top of the page but freezes for a while when it gets to actually printing out the messages/forums/threads

[coleslaw]

Could there possibly be something goofy going on with the PHP calls to the database?

[coleslaw]

I want to smack it in the back of the head!!

T-t-t-t-t-t-today junior!!! :laugh:

[sbp]

Yes its TheCorp's fault for all this. More of a reason for treason. :D

That's exactly whats happening Hapoo. The top of the page loads then the rest of the page takes forever to comes up. :\

[coleslaw]

I have another idea... It appears that the text of the posts comes up very quickly, but the small icons and smilies and other graphics take a bit longer to display. Are the got|apex graphics (such as the badges, smilies, message icons) stored on the same RAID as the database? Or are they stored elsewhere? If the vBulletin software is having to pull data from the database and the graphics folder on a different drive, that could cause some of the problems because it may be waiting for the graphics in each of the table cells to load before moving downward.

Also, I am running Internet Explorer 5.5. What browser is anyone else having problems in? I haven't tried Netscape recently, so I will update on that in a bit...

[coleslaw]

Unfortunately, Netscape 4.72 is even slower than IE 5.5. So there goes that theory. It took Netscape almost 15 seconds to load the main Forum page and about 10 seconds to load a thread. And it is also suffering from the "Stuttering Red-headed Step-Child Who Stole Something Syndrome."

*Smack* *Smack* *Smack*

[lucky_sob]

It appears that you are hosting at Level 3 in San Diego, If this incorrect then disregaurd the rest of the message. I am too so that mean my connections never leave the building and it still takes a long time. So this leads me to believe the problem is server side, because the sites I host are still very fast as well as others in the same building.

Also they have a feature that lets you analize your network traffic on a webpage, very detailed, but I believe you have to ask someone for access to it and to have it set up.

Sorry I had to post this but you disabled the email feature on the web site.

[TheCorp]

Grr SBP is beginning to annoy me ;) As for the rest of you, all very good points!

So let me reply to a few I guess...

1. The server is definitly not the problem, and nor is the software because our loads at the moment are really low and MySQL (the database we use) is chugging along just fine.

2. As for the idea about the data being on a different drive than the images and other content, well no, we keep all the RealGN data on one 9.1 Gig Ultra Scsi 2 HD, which I think more than cuts it.

3. Finally, yes SOB, we are running at the Level3 facility in San Diego, which by now you probably know is an amazing place, and yes we have been using their stats programs in the past few days.

Essentially we are almost absolutely sure now that it has to do with the switch we are using. Right now the Intel switch is doing better than the old Dlink one, which we thought wasn't going to be a problem, but I guess you get what you paid for (which actually was a lot, so go figure ;). Well we are going to be buying the Cisco switch asap, so I think things should be golden pretty soon.

Thanx again for all your help guys, and Ill set my post count back since I can see the envy in your eyes now ;)

-TheCorp

[hapoo]

Dlink is the Hyundai of Network equipment

[Leon]

Thanx again for all your help guys, and Ill set my post count back since I can see the envy in your eyes now ;)

-TheCorp


I hate it when Admins beef up their post counts!!!!

[revil]

Damn Leon, I didn't think you posted this much:
http://www.engr.orst.edu/~hendripa/leon.jpg

[DPvball]

http://www.gamingin3d.com/forum/ <-- a little slow
http://www.gamingin3d.com/ <-- was very slow
http://www.darklights.com <-- very fast
http://forums.tfoutpost.com <-- was very slow

[LPMiller]

i was wondering if it was the switch...the slowness seems random, and affects newspro, the site and the forum...but only on a whim.

[coleslaw]

Has the Cisco switch been installed? Things seem a little faster! Keep up the good work gentlemen!

[AmRivlin]

It hasn't gotten any faster, in fact it reminds me of the olden days when I used an ISP!!! COME ON CORP, do your damn job! Or whatever you have to do to speed this site up. :wavey: Please? :wavey:

[Edited by AmRivlin on 09-18-2000 at 08:54 PM]

[lucky_sob]

Don't make me go down there and fix it myself. Grrrrrrrrrrr.

Just jokin don't sweat it, I sure it is the stupidest little thing causing the error.

[Bluto]

ummm.... whens the Cisco switch supposed to be up and running??? cause it ain't any faster yet...
just curious....

:puke:

[TheCorp]

There is going to be a big update on this topic later tonight or tommorow hopefully, we found that the problem technically isnt the switch.

I know it all sounds very confusing, but I promise we will be explain it all very soon. For those who thought you may be dreaming and thought it was fast for some part of today, well you were right, we were testing stuff and found out some things.

Well update coming soon..

-TheCorp

[dcsonic2k]

I did notice that it was faster today, but it also slowed down part of the time. I'm dying to know what was wrong with the server.

[AmRivlin]

Lot faster whats the dillyo?

[AmRivlin]

I hope you left the Cisco switch in!?! We need another hit of nitros!!!