Posted on

NexGen Networks, the Worst Fiber ISP in Chicago

Our office recently moved from the Willis Tower to the Chicago Board of Trade Building. Part of this move was selecting a new Internet service provider. After looking at lots of options, we decided on a fiber carrier, NexGen Networks. Their cost was higher but we justified the higher price since we were told we could turn up service faster and the support would be better. This was not the case at all we came to find out.

My First Mistake

This all started on Thursday, our move in day. We had all of the other things to take care of with an office move and we thought our Internet would be the easy part as we had signed the contract and were assured the our service would be working. Actually, we were supposed to have the line turned up and tested Wednesday but that never happened. The hand-off to our office was single-mode 1000BaseLX fiber. We used a Cisco Catalyst 2960-CX switch and a 1000BaseLX SFP. We were going to use this switch basically as a glorified media converter with the fiber in on one port and copper out on another port on the same VLAN to pass to our copper Meraki MX Router. For testing, we hooked a laptop into the copper port with our static WAN IP just to test the connection with pings to make sure it was working ok. We were never able to get a single ping to go through.

Their “enterprise” tech support was a nice and well meaning guy who seemed to be pretty fresh out of college. He kept talking about how he will ask his professor about what he thought of the situation to see if we could get the line working. Ask your professor??? You are the guy that is supposed to know what is going on. While troubleshooting the line, we thought that maybe the 10Gtek branded SFP we were using was not working right so their on site people swapped out with a Cisco branded SFP since that would match what they had on their side. Still no Internet. The SFP port would not even turn Green on the port. We were all scratching our heads. I thought that maybe there was an issue with our Cisco Switch. I overnighted a dumb media converter and tried that when it arrived. Of course, that didn’t work either. We work with support  again and they told me to throw out the media converter because they stated “they never work.” Fine, our office has not been able to function all week. We sent our people home as we had no phones, Internet, or ability to perform any work there. The only thing their tech support would tell me is that I was using bad equipment.

Nobody’s Working on the Weekend (since we have no Internet)

On Friday night, the building fiber contractor came to our suite, after hours, and we hooked a microscope to our fiber patch cables, and the building fiber running from the NexGen Networks dmarc to our office. He was extremely thorough and double checked every connection to completely rule out the issue being the cabling. Big shout out to him and the Chicago Board of Trade Building for staying late on a Friday to try to help us out. The patch cable we were using was a little dirty so it was cleaned and we were still left with no connection at all.

We asked NexGen for a list of compatible hardware as their equipment seems to be extremely picky as to what devices it will negotiate with. We were never provided with a list or even a single part number. I am very familiar with the Aruba (HP Procurve) line of switches and I have configured dozens of them with fairly advanced routing and ACLs so I ordered a switch I was familiar with. I bought an Aruba 2530-24G switch and Aruba Branded SFP. It arrived Monday and we hooked it in. Guess what? There was still no connection. We updated the firmware on the switch and there was still no connection. To recap, here is what we have tried so far:

  • Cisco Catalyst Switch with 10Gtek SFP – NO CONNECTION
  • Cisco Catalyst Switch with Cisco Branded SFP (Borrowed from NexGen) – NO CONNECTION
  • Cisco Catalyst Switch with second Cisco Branded SFP (Borrowed from NexGen) – NO CONNECTION
  • 10Gtek Media Converter with new SFP – NO CONNECTION
  • Aruba (HP) 2530 switch with Aruba Branded SFP -NO CONNECTION

Insult to Injury

The email we received this morning from Jeffrey Barth at NexGen:

"We have worked with your team for many hours trying to get this fixed. We  have identified now that there is a compatibility issue between our  devices. (NexGen’s and Yours). NexGen tested our service at our  demarcation point prior to handing off the service so we do know that  this is working. As I stated it is a compatibility issue between the  NexGen Cisco POP device and your device."

I have asked for anything at all to get our office back to normal. I have ordered new equipment, spent countless wasted hours with NexGen’s useless tech support and we are still without a connection. I have asked that they loan us hardware or at least tell us what to buy. They have been completely silent so far on that. Apparently we have techs coming this afternoon but I really don’t have a bit of confidence in their ability to turn up a connection for us. My advice is if you are in Chicago and looking for a Fiber Internet provider, AVOID NEXGEN NETWORKS! They are based in New York and New Jersey it looks like and they really have no business being in Chicago and claiming to be an Enterprise Fiber ISP. If you are looking for service I recommend that you stick with a REAL ISP, not whatever NexGen is. I’ll update if they ever manage to deliver our service.

Update Friday, August 30th 2019

Nexgen techs had to schedule an emergency maintenance windows Thursday evening to replace their core switch at the board of trade. Today we are now actually able to use our equipment. We were down for over a week due to their faulty equipment and we now have lots of extra equipment that we didn’t need to buy. I have asked them to give me a service credit that they think would be appropriate for this whole debacle. We will see what they come back with.

Update Wednesday, October 30th 2019

We were working fine for a little while. Last week we started to see packet loss and high ping times even while we were not using much bandwidth. Their support has never gotten back to me with a status or any information other than a generic maintenance window while they I guess randomly replace equipment. We have had at least 3 maintenance windows since I reported the issues and we are still seeing the same issues. Our phones do not work as a result. Again, NEVER USE NEXGEN!What an awful company.

 

Posted on

Comcast Hijacking non-SSL sites to “inform” me

The alert embedded into my site.

When I opened my browser this evening, I was greeted with popup from my ISP about my data limit. I looked at the code and sure enough, Comcast intercepted the site I was viewing and sneakily inserted javascript to serve the popup to me. From a technical perspective I can see why they did this. It is a surefire way to alert customers because there is no authentication required. You just intercept the private web site the anonymous user it wanting to view, look through the code, and insert whatever you want. Like, tracking information, malware, subtle censorship, or a friendly reminder that Comcast has the ability and the will to edit the content of the the private web pages you view however they see fit. Unless you are going to a site that uses SSL, of course. It is just unsettling and shows that Comcast is willing to Hijack your private Internet traffic. Just because a door is unlocked, it’s not ethical to walk around in someone’s home and leave a creepy note on the kitchen table saying “You left your light on.”

If you see the locked padlock in your URL bar, then they can’t change the content at all because the content is encrypted from your device all the way back to the web server the page is hosted on. That is, if you trust the integrity of the certificate issuer. I’m looking at you, Symantec. I really need to enable SSL on this and lots of other sites I take care of. It is absolutely something that needs to happen in 2017 and with sites like Let’s Encrypt, it’s even free! Although, if your “secure” connection ends/ended up at a server that is/was a part of the NSA PRISM program then you’re out of luck anyway because they just look at your info after it is decrypted at the head end. Good lord, that was a bit of a tangent. I’m on a list now.

Posted on

How VR should be built and what it could mean

This is a really great talk about the possible future of VR. I just purchased an HTC Vive. I went with the Vive because it offered the most interaction and the most open platform. The Oculus Rift is owned by Facebook so that does not look like a great company to get behind for open software and the PlayStation VR is on a closed console. We have a long road ahead with better hardware and improved software to make the VR experience more meaningful in new ways.