Page 1 of 2

Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 5:47 am
by advantagevoice
Last night, about 50 of the systems that I monitor (roughly half) all stopped responding over the course of a few hours. There is nothing they have in common. They are on different ISPs, different firmware, different models etc. They are not set up in multi-site. The only thing they had in common is they stopped responding to traffic on the WAN and LAN, all within an hour or two of midnight last night. I was able to powercycle a couple of dozen in a datacenter, but I can’t get to the ones located in customers’ offices. Post-reboot, I see no messages at all in the system event log before they froze. Post-reboot packet captures show nothing strange. They all have DNS server, FTP, HTTP, POP3, IMAP4 and SMTP disabled. The only two possible things I can think to explain this was either some sort of DOS attack against SIP PBXes or Allworx in general, or a bug in the firmware that caused systems to crash on April 7th/8th 2014. Neither seems like a good explanation.
Did anyone experience anything similar last night? Any possible explanations?

Re: Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 8:57 am
by deverett@ispe.org
Our froze this morning as well. It seemed to happen around 4am. We use Time Warner as our ISP, but our website and everything else seems fine. After a reboot, the server stayed up for 25-30 minutes then it would stop responding. I unplugged the WAN interface and it's been solid now for about 20 minutes. No errors in the System Events or on the SYSLOG server.

Re: Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 8:58 am
by advantagevoice
Quick update. Systems seem to be going back down after I powercycle. Not all, and no detectable pattern. I've mirrored the WAN port of a system that has gone down several times and am running a packet capture. Hopefully I'll catch it in the act.

Re: Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 9:00 am
by advantagevoice
deverett@ispe.org wrote:Our froze this morning as well. It seemed to happen around 4am. We use Time Warner as our ISP, but our website and everything else seems fine. After a reboot, the server stayed up for 25-30 minutes then it would stop responding. I unplugged the WAN interface and it's been solid now for about 20 minutes. No errors in the System Events or on the SYSLOG server.
Thanks for the report. The problem seems to be ISP agnostic. I have systems that froze that were connected to Time Warner, Comcast, Verizon, AT&T, Level 3, etc. In fact, I have several systems in a data center, all plugged into the same switch and some have froze while others did not.

Re: Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 11:13 am
by advantagevoice
One more update. We haven't had a system go down in a couple hours now. I'm 99% sure it was some sort of DOS attack, but it stopped for whatever reason.

Re: Dozens of Allworx systems froze last night

Posted: Tue Apr 08, 2014 1:37 pm
by dinda156
Our Allworx 6x server went down at 0130 hours. The server was rebooted. It went down again at 0345 and again at 0645.

Windstream is our ISP for our VoIP system.

Seriously considering dumping Allworx as these DOS attacks are increasing in frequency. Also the process in place to notify customers of new patches is just dysfunctional. As dysfunctional as the entire Windstream company.

We have had no issues since since 0700 hours this morning.

Re: Dozens of Allworx systems froze last night

Posted: Wed Apr 09, 2014 9:43 am
by Stephen
Yea, when I called Windstream yesterday to open a ticket they announced the DOS attack against systems. That's why we keeps ours setup internally only without internet access.

Re: Dozens of Allworx systems froze last night

Posted: Wed Apr 09, 2014 11:02 am
by Derek
Our Allworx 48x froze around 4-5am on the morning of 4/8 as well.

Re: Dozens of Allworx systems froze last night

Posted: Wed Apr 09, 2014 12:15 pm
by acticor
We also had many Allworx systems shutdown yesterday. They were a combination of 6x12,6x,48x. Some were in LAN host mode, others NAT/Firewall w/DMZ, none in stealth mode. I'm wondering if others had this issue with stealth mode enabled.

Re: Dozens of Allworx systems froze last night

Posted: Thu Apr 10, 2014 8:07 am
by lpie
Hi guys, we had 7-8 systems go down in the similar manner, int the syslog i found these and i have never seen them before.

Most of the systems came back by power down and up except two which suffered hardware failure, both of these systems were flashing lines red/green/orange and would not start up at all.

Code: Select all

7	04/07/2014	07:45:11pm	tSip: Watching 37.8.28.109 messages.
2	04/07/2014	07:45:15pm	tSip: Temporarily blocking 37.8.28.109 messages.
2	04/07/2014	07:47:33pm	tSip: Done blocking 37.8.28.109 messages.
7	04/07/2014	07:50:59pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	07:51:12pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:12pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:20pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:23pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:24pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:31pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:51:48pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	07:57:24pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	07:57:32pm	tMsmT0003: WEB Client: Could not connect to (192.168.3.5:8081) at 0xC0A80305
7	04/07/2014	07:57:32pm	tMsmT0003: +++ IEC [7.4.18.2:msmWtp.c,3040]
7	04/07/2014	07:57:33pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:57:43pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:57:43pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:57:50pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:57:50pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:57:53pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	07:58:03pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:00:14pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:00:14pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:03:33pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:03:48pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:03:57pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:03:57pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:04:10pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:04:12pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:04:14pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:04:15pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:04:17pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:04:33pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:07:45pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:07:45pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:09:54pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:10:03pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:10:15pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:10:20pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:10:31pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:10:34pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:10:35pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:10:36pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:10:55pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:11:03pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:12:16pm	tMsmT0001: WEB Client: Could not connect to (192.168.3.5:8081) at 0xC0A80305
7	04/07/2014	08:12:16pm	tMsmT0001: +++ IEC [7.4.18.2:msmWtp.c,4299]
7	04/07/2014	08:16:21pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:16:45pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:16:47pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:16:58pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:17:05pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:17:10pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:17:13pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:17:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:22:48pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:23:03pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:23:05pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:05pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:23:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:23:20pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:23:21pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:26pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:30pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:33pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:23:34pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:34pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:23:40pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:23:48pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:29:17pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:29:33pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:29:35pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:29:40pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:29:48pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:30:05pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:30:05pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:30:11pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:30:12pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:30:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:30:21pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:30:21pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:30:25pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:30:33pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:35:45pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:36:03pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:36:08pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:36:12pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:36:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:36:29pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:36:29pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:36:32pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:36:38pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:36:48pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:36:49pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:36:49pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:37:00pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]
7	04/07/2014	08:37:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:42:09pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:42:18pm	tSip: Done watching 37.8.28.109 messages.
7	04/07/2014	08:42:24pm	tSip: Watching 37.8.28.109 messages.
7	04/07/2014	08:42:25pm	tSip: +++ IEC [7.4.18.2:sipDB.c,3122]