Page 1 of 2
Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 5:47 am
by advantagevoice
Last night, about 50 of the systems that I monitor (roughly half) all stopped responding over the course of a few hours. There is nothing they have in common. They are on different ISPs, different firmware, different models etc. They are not set up in multi-site. The only thing they had in common is they stopped responding to traffic on the WAN and LAN, all within an hour or two of midnight last night. I was able to powercycle a couple of dozen in a datacenter, but I can’t get to the ones located in customers’ offices. Post-reboot, I see no messages at all in the system event log before they froze. Post-reboot packet captures show nothing strange. They all have DNS server, FTP, HTTP, POP3, IMAP4 and SMTP disabled. The only two possible things I can think to explain this was either some sort of DOS attack against SIP PBXes or Allworx in general, or a bug in the firmware that caused systems to crash on April 7th/8th 2014. Neither seems like a good explanation.
Did anyone experience anything similar last night? Any possible explanations?
Re: Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 8:57 am
by deverett@ispe.org
Our froze this morning as well. It seemed to happen around 4am. We use Time Warner as our ISP, but our website and everything else seems fine. After a reboot, the server stayed up for 25-30 minutes then it would stop responding. I unplugged the WAN interface and it's been solid now for about 20 minutes. No errors in the System Events or on the SYSLOG server.
Re: Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 8:58 am
by advantagevoice
Quick update. Systems seem to be going back down after I powercycle. Not all, and no detectable pattern. I've mirrored the WAN port of a system that has gone down several times and am running a packet capture. Hopefully I'll catch it in the act.
Re: Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 9:00 am
by advantagevoice
deverett@ispe.org wrote:Our froze this morning as well. It seemed to happen around 4am. We use Time Warner as our ISP, but our website and everything else seems fine. After a reboot, the server stayed up for 25-30 minutes then it would stop responding. I unplugged the WAN interface and it's been solid now for about 20 minutes. No errors in the System Events or on the SYSLOG server.
Thanks for the report. The problem seems to be ISP agnostic. I have systems that froze that were connected to Time Warner, Comcast, Verizon, AT&T, Level 3, etc. In fact, I have several systems in a data center, all plugged into the same switch and some have froze while others did not.
Re: Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 11:13 am
by advantagevoice
One more update. We haven't had a system go down in a couple hours now. I'm 99% sure it was some sort of DOS attack, but it stopped for whatever reason.
Re: Dozens of Allworx systems froze last night
Posted: Tue Apr 08, 2014 1:37 pm
by dinda156
Our Allworx 6x server went down at 0130 hours. The server was rebooted. It went down again at 0345 and again at 0645.
Windstream is our ISP for our VoIP system.
Seriously considering dumping Allworx as these DOS attacks are increasing in frequency. Also the process in place to notify customers of new patches is just dysfunctional. As dysfunctional as the entire Windstream company.
We have had no issues since since 0700 hours this morning.
Re: Dozens of Allworx systems froze last night
Posted: Wed Apr 09, 2014 9:43 am
by Stephen
Yea, when I called Windstream yesterday to open a ticket they announced the DOS attack against systems. That's why we keeps ours setup internally only without internet access.
Re: Dozens of Allworx systems froze last night
Posted: Wed Apr 09, 2014 11:02 am
by Derek
Our Allworx 48x froze around 4-5am on the morning of 4/8 as well.
Re: Dozens of Allworx systems froze last night
Posted: Wed Apr 09, 2014 12:15 pm
by acticor
We also had many Allworx systems shutdown yesterday. They were a combination of 6x12,6x,48x. Some were in LAN host mode, others NAT/Firewall w/DMZ, none in stealth mode. I'm wondering if others had this issue with stealth mode enabled.
Re: Dozens of Allworx systems froze last night
Posted: Thu Apr 10, 2014 8:07 am
by lpie
Hi guys, we had 7-8 systems go down in the similar manner, int the syslog i found these and i have never seen them before.
Most of the systems came back by power down and up except two which suffered hardware failure, both of these systems were flashing lines red/green/orange and would not start up at all.
Code: Select all
7 04/07/2014 07:45:11pm tSip: Watching 37.8.28.109 messages.
2 04/07/2014 07:45:15pm tSip: Temporarily blocking 37.8.28.109 messages.
2 04/07/2014 07:47:33pm tSip: Done blocking 37.8.28.109 messages.
7 04/07/2014 07:50:59pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 07:51:12pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:12pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:20pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:23pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:24pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:31pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:51:48pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 07:57:24pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 07:57:32pm tMsmT0003: WEB Client: Could not connect to (192.168.3.5:8081) at 0xC0A80305
7 04/07/2014 07:57:32pm tMsmT0003: +++ IEC [7.4.18.2:msmWtp.c,3040]
7 04/07/2014 07:57:33pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:57:43pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:57:43pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:57:50pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:57:50pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:57:53pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 07:58:03pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:00:14pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:00:14pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:03:33pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:03:48pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:03:57pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:03:57pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:04:10pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:04:12pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:04:14pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:04:15pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:04:17pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:04:33pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:07:45pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:07:45pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:09:54pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:10:03pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:10:15pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:10:20pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:10:31pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:10:34pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:10:35pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:10:36pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:10:55pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:11:03pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:12:16pm tMsmT0001: WEB Client: Could not connect to (192.168.3.5:8081) at 0xC0A80305
7 04/07/2014 08:12:16pm tMsmT0001: +++ IEC [7.4.18.2:msmWtp.c,4299]
7 04/07/2014 08:16:21pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:16:45pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:16:47pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:16:58pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:17:05pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:17:10pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:17:13pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:17:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:22:48pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:23:03pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:23:05pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:05pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:23:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:23:20pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:23:21pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:26pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:30pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:33pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:23:34pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:34pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:23:40pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:23:48pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:29:17pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:29:33pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:29:35pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:29:40pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:29:48pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:30:05pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:30:05pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:30:11pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:30:12pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:30:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:30:21pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:30:21pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:30:25pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:30:33pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:35:45pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:36:03pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:36:08pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:36:12pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:36:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:36:29pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:36:29pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:36:32pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:36:38pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:36:48pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:36:49pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:36:49pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:37:00pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7 04/07/2014 08:37:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:42:09pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:42:18pm tSip: Done watching 37.8.28.109 messages.
7 04/07/2014 08:42:24pm tSip: Watching 37.8.28.109 messages.
7 04/07/2014 08:42:25pm tSip: +++ IEC [7.4.18.2:sipDB.c,3122]