The Furry Forums
TFF Central => Announcements => Topic started by: Tweak on October 01, 2009, 04:12:51 PM
-
Nothing to be worried about but I wanted to pass the message on from our technicians.
We will be performing essential network maintenance between midnight and 4am BST on Friday 9th October 2009.
During this time there will be three brief interruptions to network connectivity in our data centre. Each service interruption should last approximately one minute.
Most likely it'll not even be noticed but you've now been notified.
-
Thanks for the notice Tweak :P
-
yep, thankies :3
-
Apologies for the downtime lately, we're currently investigating the issue fully and trying to solve the problem.
-
Ok, we now have a reason for the system downtime but no reason for that reason. It's a serious technical issue which should not be ocurring, the server is stable again and we'll be investigating the issue all night.
An update to this problem should be available sometime tomorrow. Apologies for the inconvenience.
-
As always, we are forever grateful for your endless toil.
If I had enough money, i would express it in wealth.
-
Gaaah, you guys are always working too hard. :) Thanks for all the work you've done for the forums anyway :)
-
And the plot thickens >_>
Server was stable all weekend and then became unresponsive an hour or so ago. We've got a sysadmin booked to take a good look at the server tomorrow.
Apologies to everyone, once we know the cause, we can work on fixing it.
-
Curiouser and curiouser...
So a data centre sysadmin took a look at the server today and it's strange. The memory and swap dump is happening so quickly that the server has no time to log any issues, so at present we have no clue as to the problem. Some new logs have been set up to try catch the issue when it next occurs but we're currently suspecting there is a memory leak in one of the scripts hosted.
I will be taking some time to look through the newer hosted sites to see if anything is obvious, although I do have an idea which it may be (whether or not I'm right though). Rest assured though, this problem will be found and fixed and that all data is perfectly safe. Also rest assured it's not the forums or anything associated with them.
-
Thanks for the update Tweak, glad to hear that the forums isn't in any danger!
-
Thanks for the update Tweak, glad to hear that the forums isn't in any danger!
Now now, I didn't say that. I just said this issue wasn't caused by them! :P
-
Still glad! x3 Any idea yet as to how long till the issue's fixed?
-
Can't say for sure. Until the server burns up again we can't start to check what it is. I have done a few code changes on one of the sites on the server, we'll see if that makes a difference.
-
As some may have notice we lost it again yesterday evening, unfortunately the logs are not showing anything so it'll be some trial and error changes to try to prevent it from occurring.
I have installed a new proxy server over the current webserver to try reduce some load, this forwards all requests through to the normal server but due to its improve handling of requests the main webserver can do less work overall, this is still being adjusted so please report any slow loading or other problems through out bugs board.
Apologies for all these problems, the easiest way would be to put another server in and load balance them, unfortunately at this time I cannot afford to do that. But the server is more than powerful enough, I just need to get it perfect.
-
how very interesting. Let's just hope this can have some kind of amicable resolution. No need to do it fast, just that it's done at all is more than enough.
-
Thanks again Tweak and good luck :P
-
In case anyone noticed the short outage earlier. It only affected us for a few seconds while the network redistributed it's load. Report from my datacentre is below.
Between 14.58 and 15.02 this afternoon there was a brief network outage on some parts of the network. This was caused by a denial of service attack on one of our customers. Engineers enabled the DOS protection system within a few minutes.
Also in regards to the crashing issues, it seems the server is stable again, I have issued some limits on another site on the server which seemed to be causing them a few months ago and since then everything has appeared stable. A forced cache has also been introduced to cut the number of calls for images on the forums here, as we rarely change the images they are now cached on your system. This should improve load time to some extent.
Additionally a bug seems to have been squashed which was causing some issues with the server. Although they tended not to become too apparent as I managed to jump in and reboot the web server before it became an issue there were a couple of incidents where connections were refused. This was caused by a bug in a module within the server software causing processes to freeze and use slots, this slowly built up until the software thought it had hit its limit. This appears to now have been rectified.
-
Apologies for the outage we experienced earlier, report from the datacentre is below.
We have just recovered from a very serious Distributed Denial of Service attack which took out parts of our network. Many customers will have seen very serious connectivity problems starting at about 16.30 BST this afternoon. The attack is ongoing but all services are now back, and we will have more details shortly.
All services have been back to normal since 17.13 BST and although the attack continues it is being completely mitigated by the DDOS solution. We have identified the target of the attack and removed it from the network. (The target was a server purchased earlier today which seems to have been purchased with the express intent of damaging our network.)
Once again apologies but this issue was outside our control but was dealt with in cooperation with our technicians.
-
It's good to see the attackers are gone :D
-
They aren't, they just aren't causing a disruption now.
-
IRC connectivity should now be available again.
IRC has been online all afternoon but due to DoS attacks on the servers hosting the web-based client we use all network access to it was disabled. We worked with their technical team to ensure all was working before switching back on the link a few moments ago.
Apologies for anyone who was unable to get into IRC, all should be fine now.
-
Received from my datacentre:
On the night of Sunday 23rd May we are going to be completing a network upgrade. You will experience a short period of downtime sometime between midnight and 3am BST. The downtime will be no longer than 10 minutes.
Additionally to this improved network a solution should go live to completely nullify any future DDoS attacks.
Apologies for any issues this may cause.
-
yay! i didnt even know you could nullify dos attacks
-
If it's to give us further protection from DDoS attacks then any inconveniences it could cause should be more than worth it. It's a pain that the network is targeted so frequently.
-
The work above was completed at 4am BST.
-
Apologies for the extended downtime last night, unfortunately I was stuck in Chicago for 4 hours and have only just had a chance to sit down and solve the issue, everything should be fine now.
-
Good work *claps* :)
-
Pawsome, but what was the issue if i may ask?
-
No idea, was too tired to check.
-
No idea, was too tired to check.
Just rebooted the server? :3
-
Apologies again for the downtime on the 22nd - 23rd June, this was caused by a database corruption which I couldn't handle until I got back home (was at the airport again!) The problem was easily fixed and only affected the forums.
---
In the next few weeks the IRC server will be undergoing some changes:
Daemon and Services upgrade - Compiled and configured, just needs to go liveNew statistics softwareChannel registration will be made universally available- Chanserv will become invisible in channel
- The storage for users, channels, etc will be moved into a database
- Encryption will be changed
The new statistics package will provide real-time statistics on the server including the online users, channel information and more. Channel registration was always restricted but it has been decided to be lifted for all to use as wanted.
The storage and password encryption changes are occurring to support simpler backups and to provide a base for more features later on.
Any questions, ask as always.
-
The statistics software has been operating for a few days and will be made live once it is definitely working correctly.
We did suffer an attempted DoS attack on the IRC service today, DEFCON modes were activated and the attack was brought under control, anyone trying to connect during this time may have been banned for protection to the server, please let me know if you cannot access the IRC.
-
Apologies for the downtime experienced early this evening.
The network was subjected to another DDoS attack but due to the improvements mentioned previously it had no direct affect on the network and as a result no alarms were triggered. However an upstream link was almost saturated and connectivity to some areas was lost.
Work is ongoing with this provider to ensure future attacks are mitigated.
-
Just to reassure and quash any rumours that have been circulating.
The forums were in no way attacked or hacked. The ban message was an error caused by the server IP being added into a ban made on a troll. We are investigating how the issue occurred and any actions we can take to remove this risk.
-
That's a relief, I was stressed about that last night.
-
I don't see why, the forums are using up-to-date software with no known security issues. The easiest way in would be through a weak password at which point access is given to more areas than just the ban system. Personally banning everyone (something I wasn't even aware would be the affect of that) would be a pretty crap hack, deleting everything would be far more of an attack.
-
I think it was more or less a scare because not only was everyone banned, but being banned prevented us from seeing if anything was deleted or not and it's not something that's the norm. I'll admit I was a little worried last night as well.
-
I have installed a new proxy server over the current webserver to try reduce some load, this forwards all requests through to the normal server but due to its improve handling of requests the main webserver can do less work overall
Some of you may remember the above change being made. I completely forgot that I made a few changes to this recently.
Although the majority of changes will not affect you directly the proxy server is now set up to serve all requests for static content (HTML, CSS, Javascript, images, etc) all dynamic things such as PHP are sent through to the main server and processed. This reduces the overall web-server load by about 85%. We are investigating if a cache server may also aid in performance, this is ongoing.
-
We are aware of the current avatar display issue and are working to resolve it. All avatars are still stored, it's the act of getting them linked properly is the current issue.
We apologise for any inconvenience this may cause.
-
In the next few weeks the IRC server will be undergoing some changes:
Daemon and Services upgrade - Compiled and configured, just needs to go liveNew statistics softwareChannel registration will be made universally available- Chanserv will become invisible in channel
- The storage for users, channels, etc will be moved into a database
- Encryption will be changed
This is currently on hold pending investigation into the potential introduction of a different daemon.
The forums were in no way attacked or hacked. The ban message was an error caused by the server IP being added into a ban made on a troll. We are investigating how the issue occurred and any actions we can take to remove this risk.
This was found to be an error with the proxy server forwarding requests. It is fixed and such an event is no longer a possibility.
-
Some of you may have encountered an error when attempting to access the forum today. Although restricted to those trying to view the front page of the forums this was a server-wide issue caused by an issue with the /tmp directory failing to purge our proxy server sessions.
This has been fixed and measures put in place to prevent the same from happening again.
It was also discovered recent high loads may have also been caused by this and our load average has dropped significantly as shown in the graph attached.
The gap in the reading was due to this issue.
[attachment deleted by admin]
-
ANNOUNCEMENT :: SERVER MAINTENANCE
24th April 2010 :: 8am GMT+1
To ensure all kernel patches are applied correctly, the server will be rebooted at 8am GMT+1 on 24th April 2010.
There should only be a few minutes of interrupted service and we apologise for any inconvenience this may cause.
-
ANNOUNCEMENT :: NETWORK MAINTENANCE
10th May 2010 :: 00:00 - 06:30 GMT+1
I have just received the following notice from the data centre.
We will be performing essential maintenance on a cabinet which houses at least one of your dedicated servers.
The maintenance slot starts at 2011-05-10 00:10:00 and ends at 2011-05-10 06:30:00. The total downtime for your server(s) is estimated to be no more than three hours but services should be considered 'at risk' for the entire period.
We will be cutting power to this cabinet so if you wish to perform a graceful shutdown this should be performed between 00:00 and 00:10.
As a result the forums and IRC will be offline during this period. Unfortunately the maintenance is critical and there is no way to avoid it. I am investigating the possibility of fast-tracking our backup plans which may enable us to provide a temporary IRC.
My apologies for the inconvenience this may cause.
-
Oh well, I'll be at work anyways :P
-
As mentioned in the above post, I have expedited the plans for our backup system and started to get it online.
The backup system will always be available through the following link: http://backup.abanix.com.
At present this system is still being set up and only displays a holding page. More information will be released once it is completed.
During tomorrow's maintenance this backup will be online providing a chatroom for everybody to use, it will be online for the entire maintenance cycle.
Please make a note of the link above as this post will be inaccessible.
-
Thanks for a job well done Tweak :D
-
All services are now back online.
-
As well as patching the main security hole found in Apache today I also installed a new caching system.
This is now being used for the forums and should, in theory, provide a much smoother service to everyone. It will not work to it's full potential due to the security settings on the server in other places but should work better than the file-based caching we had before.
-
! Security rule signatures updated.
Please report any issues with 403 errors should they occur.
-
ANNOUNCEMENT :: SERVER MAINTENANCE
26th October 2011 :: 9am GMT+1
After a upgrade to the Operating System we need to ensure all kernel patches are applied correctly, the server will be rebooted at 9am GMT+1 on 26th October 2011.
There should only be a few minutes of interrupted service and we apologise for any inconvenience this may cause.
-
Is it the end of the world already?
-
Ask me that in 11 hours if the server fails to come back online!
-
This was completed without incident.
-
Hooray, the world remains safe!
-
Apologies for the spurts of downtime this evening, I had some problems reconfiguring some of the server software. It is now operating as expected.
-
love it when forums post seerver info and updates, but i would enjoy looking into the more detailed stuff
/technonerd
-
Due to some issues with our backup server there is currently an on-peak backup being run.
This may cause a detrimental effect to the forums and for this we apologise. We'll update when it has been completed.
-
Backups are now completed. Having problems setting up the share but will complete this tomorrow.
This has now been completed.
-
! Security rule signatures updated.
Please report any issues with 403/406 errors should they occur.
-
! MySQL server was upgraded. Expect better response times.
-
! Made some changes to some security systems to reduce the amount of false error-reports.
This will have a knock-on effect due to less IO usage.
! Also fixed an error with the main index page for the root domain, this will have the same effect as above - though to a lesser degree.
-
Today's downtime was caused by a data centre failure.
At approximately 11am Central time a HVAC failure occurred causing a major switch to overheat and shut down for safety reasons. On repair of the HVAC system it failed again and was subsequently replaced.
The server was not affected in anyway other than a loss of network connectivity, as soon as the ambient temperature was reduced and the networking infrastructure was restarted the systems came back online immediately.
Apologies for the downtime.
-
I wish to extend my apologies for the extended downtime experienced over the past couple of days.
Since our infrastructure changes we have had much less downtime and a much more stable service for all customers here including those with high-activity accounts.
Unfortunately we experienced a large scale Denial of Service attack, again our infrastructure is built to withstand most DoS attacks without a degraded service but due to the scale of this attack our upstream providers were forced to set the main server IP to a null-route (basically make it point nowhere) to reduce the huge stream of traffic from the attack.
As the attack is still ongoing and to protect the general network infrastructure the IP is to remain offline and so the need to migrate to another IP and refresh licenses caused some delay in getting everything online.
This is one of the rare force majeur events that occur in which we cannot predict or do anything but work as swiftly as possible to restore service.
I once again extend my apologies for the downtime and as always extend my gratitude for your continued business.
-
! Security rule signatures updated.
Please report any issues with 403/406 errors should they occur.
-
! Security rule signatures updated.
Please report any issues with 403/406 errors should they occur.
it says connection reset when I post, it was working yesterday but not today. is it related?
also it asked me to answer math questions when logging in earlier,it didn't do it before. :?
pretty sure I have no viruses, this is a mac and I check it for viruses.
I tried different browsers & they all say connection reset
I can post this with a proxy but without it I can't post anything not even a bug ticket. x_x
-
Is this still an issue? It shouldn't have been caused by my changes.
-
ANNOUNCEMENT :: NETWORK MAINTENANCE
13th March 2013 :: 02:00 - 04:00 GMT
I have just received the following notice from the data centre.
To improve network performance and resilience we will be conducting network upgrade maintenance on the morning of Wednesday 13th March 2013 betweek 02:00 and 04:00 GMT.
Whilst no outage is planned, connectivity to all services should be considered at risk during this time.
-
Unfortunately due to a database failure some data has been lost.
As a result, all posts, PMs, changes, etc made within the last 24 hours should be deemed lost. Unfortunately only daily backups are taken and as such no newer backup can be restored.
We apologise for this inconvenience.
-
We had another database failure yesterday which resulted in some data loss. Tweak is doing his best to make sure this sort of thing won't keep happening, but for now, I'm afraid that any posts and messages that were made in the past 24 hours have been lost.
We apologize if this has caused any inconvenience.
-
Apologies for the bad service as of late.
A few changes have been made and we'll be checking to ensure the infrastructure is once again sound.
-
Was there another Database failure? I can't seem to find some posts I made last night
-
There was, 99% of the database was recovered due to the changes made but it's possible some were unrecoverable.
-
It's 1am on the 4th day of only minimal sleep so I'll keep this short.
Unfortunately we were hit with a file system corruption leading to us having to rebuild to another machine - this was then met by the same error - resulting in another rebuild. At this time all services are online but may be inaccessible whilst everything updates.
Service may be a little slower than usual whilst we investigate the possibility of our software causing the file system issues. Please bare with us whilst we get back up to speed.
I want to apologise for the slowness of this fix, even with only 4-5 hours a night of sleep I lost a lot of time due to work.
-
I haven't been able to even reach the site since I was last on, good to finally get back.
Sounds like you are working your butt off trying to get this fixed, so I wouldn't worry, we appreciate your efforts.
-
Thanks Tweak, you're a legend
-
Yeah, Tweak, thank you so much man. Take care of yourself though, your health is more important to us than the forums.
-
Jeez man... You deserve a good long rest after all that. I'm sure everyone on this forums is thankful for your work, so owe it to yourself and rest.
-
Sorry for the blip there, managed to disable the NIC!
IRC is currently not working through the inbuilt client however can be accessing using the PJIRC client over at http://razi.alfreelancing.com/pjirc/ (http://razi.alfreelancing.com/pjirc/)
Once our Mibbit client is working I'll post here.
-
I'm not able to access IRC through my own client.
-
Any errors?
-
[12:25] * Looking up irc.furry-forum.com
[12:25] * Connecting to irc.furry-forum.com (78.40.39.252) port 6667...
[12:25] * Connection failed. Error: Connection timed out
-
Wow...guys...I am so glad to see this forum back online.
I was absent for a while and when I came back this place was gone. I wondered if it was some sort of attack. This forum is one of only two real alternatives to the Fur Affinity Forums...and since I just left that awful place...I really hated to see you guys down.
I've been noticing FDF refugees joining up at Furtopia and FAF before I left. I'll spread the word that they can come back home. :)
Thanks so much for your efforts. I'm sure those who left might make their way back.
Also....for what's its worth...have you considered perhaps changing to a different type of board? Perhaps an Invision or vBulletin board?
-
IRC is now connecting correctly.
-
Thanks man. I owe you.
-
All emails should now be going through correctly - GMail blocked our IP address due to some server issues. This has now been resolved.
-
Server is now back online. Took a little longer as a FSCK was required.
-
ANNOUNCEMENT :: NETWORK MAINTENANCE
24th February 2014 :: 2am - 7am GMT
The datacentre we're located in will be carrying out a major upgrade of the power systems.
The switch-over to the new systems will take place between 2 and 7am GMT on 24th February 2014. For this to happen, power will be cut. During this time the services will be unavailable for 30-60 minutes starting at 2am, no other problems should occur.
All services should be back online by 4am but the entire period is to be considered at risk.
Additionally, we are currently investigating the issues causing the downtime of late, we've diagnosed the problem but are still seeking out the cause. The services are being monitored thoroughly.
Also the previous file system corruption which led to rebuilds have been investigated and the cause solved, this cause will no longer result in such problems.
We're also in the process of making some changes both technically and within the forums. More information will be posted on this at a later time.
-
Does this look good to you guys?
[attachment deleted by admin]
-
Hard to say after such a short time frame xD
-
It is fairly high I'd say. My rule of thumb is usually 80% for upper limit, and if I go over that frequently I'd look into what's causing such high spikes and either see if I can fix it or if I'll need more memory.
-
The high point was what it was running steady on, the drop was after working on some optimizations. That drop has held steady. I basically cut memory usage by 20% in 3 minutes of work.
Simply put - I am awesome.
-
I am the werepyre and I approve this message.
-
More efficiency works for me.
-
Tweak, mostly the masses don't even know.
But some of us care and thank you :3
*hugs*
-
Tweak, mostly the masses don't even know.
But some of us care and thank you :3
*hugs*
Thanks, it's nice to see that people do appreciate the time I put into the forums.
-
Simply put - I am awesome.
gratz on awesome
-
Good work Tweak! :3
Have a virtual turnip; *Hands you a turnip*
-
Thanks dude ^_^
-
Nice.
(http://i.imgur.com/HzizKKk.gif)
-
And another 10% memory cut after some more tuning.
-
I'm noticing DRASTIC improvement!
Great job, man!
-
And another 10% memory cut after some more tuning.
BOOOM!
That's kinda like extending our lives because we're not wasting our time :3
*Hands top hat, monocle and moustache for being like a boss"
-
*pets Tweak and gives a cookie*
Good forum dragon.
-
Thanks all.
This could have been done a while ago but it can be hard to gain the motivation when nobody appears to actually care.
-
I care. :(
-
Oh we do care! I suppose though it's hard to show care for something we, as end users, don't notice directly other than in sudden increase (or decrease) in performance.
-
Thanks all.
This could have been done a while ago but it can be hard to gain the motivation when nobody appears to actually care.
We care! There's even a whole bunch of helpdesk tickets that people posted, so it shows people care about the forum and how well it's functioning :3
-
I care. :(
Daawww!
"Oh PLEASE do an epic Admin hug you two!"
*Clasps paws together* :D
-
*raises hand up in the air and jumps up & down* I care! I care!
-
Thanks all.
This could have been done a while ago but it can be hard to gain the motivation when nobody appears to actually care.
But people show that they care by frequenting and participating on the site. The knowledge that you have a site a lot of people like to visit and socialize on should be all you need. Yeah though, people can be very ungrateful sometimes.
-
Let there be hugs! X3
-
I care too. Great respect to you and your tech wizardry.
-
^Yes I agree Nikolai, an Admin hugfest WOULD be epic :D
-
ANNOUNCEMENT :: NETWORK MAINTENANCE
24th February 2014 :: 2am - 7am GMT
The datacentre we're located in will be carrying out a major upgrade of the power systems.
The switch-over to the new systems will take place between 2 and 7am GMT on 24th February 2014. For this to happen, power will be cut. During this time the services will be unavailable for 30-60 minutes starting at 2am, no other problems should occur.
All services should be back online by 4am but the entire period is to be considered at risk.
This maintenance has now been completed.