Valid HTML 4.01 Transitional

Verizon FiOS Installation and Experience

James F. Carter <jimc@math.ucla.edu>, 2012-06-19

Table of Contents

Getting FiOS

We have been waiting since 2008 for Verizon FiOS service to be available at our house. The quadrant to the northwest has FiOS. The quadrant to the southeast, which we are in, doesn't. However, our house is too far from the southeast poles, and our wireline runs across the street into the northwest quadrant. The database that governs FiOS eligibility doesn't know this.

We have been bugging Verizon Customer Service and field installers since the FiOS backbone was installed. Finally my wife hit the jackpot: someone who knew someone who knew someone who understood the issue and knew how to fix it. We are now eligible for FiOS.

The plan we ordered provides:

POTS (Plain Old Telephone Service)

Unlimited calling nationwide, lots of features including anonymous call blocking. Server-side voicemail. The product is called Verizon Digital Voice, and the technology is VOIP.

Internet

15Mbit/s downlink, 5Mbit/s uplink. Considerably higher speeds are available. The ONT connects to the house router via MOCA or IEEE 802.3 ethernet (pick one; we're going to use MOCA). The provided router is an Actiontec MI424WR. It has these ports:

TV

We declined it, but they would love to sell you a bundle with TV.

Freebies

These items are included in the package:

Cost

$60/mo introductory price with 2 year contract. Includes $30/mo of discounts; we'll see how much of that evaporates after 2 years. Early termination penalty: $240 amortized over 2 years.

Ordering

Order date: 2012-06-09. Entirely online, no need to wait for or miscommunicate with a customer service rep. Installation promised: 2012-06-15 (could actually have been earlier). Installation was on time and not messed up.

Preparation for Installation

Here is what you need to have ready for them:

The router password: The Actiontec MI424RW router has a number of hardware and/or software revisions; the latest, which I got, is I (the letter). Verizon also sometimes used one of the Westell models. For the setup web interface, all the routers listen on 192.168.1.1 port 80, 443, and several others. They use a user name of admin (lower case). Older routers use a lame password of password or password1. After hacked access to the router was made useful, starting around 2008 they started using the router's serial number (case sensitive). My router (2012) has a truly random password of 8 bytes including punctuation, printed on the ID sticker on the bottom.

My ideal router configuration: It is not likely that the router firmware is going to be smart enough to do everything I want. The router is referred to below as Weasel. I have a real machine called Jacinth, a Koolu, which acts as our home server; among its various functions are routing, firewall, and NAT.

The actual router is:

Model: MI424WR-GEN3I
Hardware Version: I
Firmware Revision: 40.19.22

Older Actiontec MI424WR variants have a problem with their NAT table running out of space; some online games and file sharing protocols open large numbers of connections which exceed the router's capacity. However, the most recent revisions including version I have a much larger NAT table.

FiOS installation was relatively smooth. None of the planning items had to be revised on the fly. The job was done in two phases: first a tech ran the fiber from a suitable backbone location to our house, just leaving a roll of fiber on the roof. If utilities are underground, digging and burial would happen in this phase. Two days later (on the scheduled date) the technician did the inside wiring. That part of the job took about 4 hours. I gave the tech some help, pulling and untangling cable at ground level while he was on the roof stuffing it down the conduit.

Software and Hardware Reconfiguration Steps

Here is a forum discussion of various router configurations. This page has been around for a while but is actively updated, latest 2012-06-08 (4 days ago). They show 10 variant configurations. The goal on this page seems to be to use the user's router, often because the Actiontec's NAT table is too small or 802.11 on Actiontec is unsatisfactory.

  1. Plug another router's WAN port into a LAN port on Actiontec. If the other router is doing NAT, things get confusing from one LAN client to another, but you can communicate fine from clients to the outside world.

  2. Same as 1 but tell Actiontec that the other router is in the DMZ. Obviates port forwarding from Actiotec to the other router, but you still need router -> client port forwarding.

  3. Connect the other router's LAN port to Actiontec's LAN port. The other router will act as a switch. Much simpler than 1 or 2.

  4. From this point, Verizon does not support the configuration.

  5. Put Actiontec into bridge mode (see the howto). Verizon's CPE management interface will still work. But the switch ports on Actiontec are not connected to the LAN and are useless. The more usual outcome of bridge mode, at least on revision I, is to bridge the LAN ports to the WAN.

  6. Put Actiontec into bridge mode as in variant 4, assuming that you can bridge the MOCA WAN to the 802.3 (Ethernet) WAN port. Then from a LAN port of the other router connect to a LAN port of Actiontec. This will connect the LAN to the switch ports and to MOCA LAN. This looks like what I want, except I haven't yet figured out how to make it happen. See this diagram.

  7. Variants 6 to 9 assume a cat5 connection from the ONT to the router, which is not our case.

The following items may not be of general interest, since many items involve specific features of jimc's net.

Data Speed Tests

In their setup instructions, Verizon suggests that you check your speed using their tester, and the installation tech will want to do a speed test to see if you're getting the data rate you're paying for. Unfortunately it does the test and then doesn't display the results, using both Chromium-18.0 (libQtWebKit4-4.7.1) and Firefox-12.0 on Linux. Hence I did my own speed tests using iperf, available from the SuSE Build Service. Here are the command lines; the port varies depending on what the respective firewalls will let through that isn't being used.

Sender Command Server Command
Jacinth iperf -c harlech -p 443 Harlech iperf -s -p 443
Harlech iperf -c otter.mine.nu -p 2231 Jacinth iperf -s -p 2231

The machine connected to FiOS (via Weasel) is called Jacinth. The remote partner used for these tests is called Harlech. Most of the tests lasted 10 seconds. Speeds reported by the server are shown; the speed reported by the client is consistently about 200Kbit/sec higher, probably due to data buffered by the kernel that isn't properly accounted for. See below about Netpolice.

Media Sender Server Netpol. Speed kbit/sec Comments
Service level for DSL is 768 kbit/sec in, 128 kbit/sec out
DSL Jacinth Harlech Yes 122
DSL Harlech Jacinth Yes 776
DSL Jacinth Harlech No 382 Over service level
DSL Harlech Jacinth No 1010 Kills audio
DSL Jacinth Harlech No 381 50 sec test
DSL Harlech Jacinth No 1000 (no difference)
Service level for FiOS is 15000 kbit/sec in, 5000 kbit/sec out
FiOS Jacinth Harlech Yes 4420 Audio OK
FiOS Harlech Jacinth Yes 12200 Audio OK
FiOS Jacinth Harlech No 5210 Kills audio
FiOS Harlech Jacinth No 15100 Audio OK

Netpolice is significant: it reduces throughput by a few percent to avoid queue clogging on intermediate routers. Without it, competing traffic kills shell interaction and streaming audio/video. It looks to me like the ONT, unlike on DSL, regulates the client's data rate using an algorithm similar to that of netpolice, so high volume incoming data doesn't kill other connections. This would happen if a client were streaming video-on-demand and were downloading a movie on DVD at the same time, so likely Verizon received customer complaints and configured their ONT to enforce sharing the bandwidth.

Remaining setup steps:

Verizon's Mail Service and Web Hosting

The keyword for the web hosting is Verizon FiOS Personal Web Space. You get 10Mb with the free service (more can be purchased). They provide a web-based Site Builder tool. I didn't investigate it much beyond reading the introductory material.

I investigated their mail service slightly more deeply.

Putting the Actiontec MI424WR (Revision I) into Bridge Mode

All of these procedures refer to the STP checkboxes in the bridge group. STP means Spanning Tree Protocol, governed by IEEE 802.1d and extensions. The issue here is that if there are multiple routers and network segments, troublesome loops are possible, which can be detected and suppressed by STP. Each router sends out packets once per second to announce its role in the agreed-on spanning tree and to detect changes in the network. In the typical SOHO (Small Office / Home Office) situation there is only one router and no possibility of loops. In this case jimc says that STP is irrelevant and might as well be turned off.

DaDrgon's Procedure

Jimc's summary/rewrite of the HOWTO for activating bridge mode by DaDrgon (2007-01-20).

  1. Get logged in to the router's GUI; DaDrgon discusses this extensively, but as of 2012 they set up the router with a truly random password printed on the sticker on the bottom of the router.

  2. Once on the router, hit My Network -> Network Connections. Click on "advanced" at the bottom of the connections list.

  3. You should see:

  4. Click on Broadband Connection -> Settings. Key items should already set like this:

  5. Procedure for releasing the DHCP lease and killing the Internet connection. If you forget this step, you can wait 2 hours for the lease to expire, or call Verizon Customer Service and beg them to release it from their end. I'm not sure what promptly really means, but you don't want the router to decide to get a new lease before you finish.

  6. Back to Network Connections, click on Network(Home/Office) -> Settings. Look for Bridge. On the row for Broadband, click on STP.

  7. At this point, jimc deviates from DaDrgon's procedure.

  8. Plugged Jacinth eth1 -> router WAN port and LAN (switch) into router LAN port. Lost communiction with Weasel which is on the LAN ports. tcpdump on eth1 shows only our transmissions, no incoming packets. Reverted (eth1 to router LAN)

  9. Try again:

  10. Broadband is back to getting IP automatically. Repeat the procedure. Also turn off firewall. Let's reboot again to make sure it "took". It didn't. Several times. This was not a success.

Celestil's Procedure

See also: http://www.dslreports.com/faq/verizonfios/3.1_Actiontec#15992

Jimc's summary of the howto for getting into bridge mode by Celestil (2009-05-25). Celestil does the apply procedure after each individual change. Jimc thinks this is anally retentive: it seems to be OK to make all the changes on one page, then apply them all at once. But if you navigate to a different page before applying, you would lose the changes.

Also see the discussion above of STP (Spanning Tree Protocol).

  1. He hard resets the router and restores defaults.

  2. Advanced -> System Setting -> Disable auto WAN detection (apply) Actually, on the iteration that worked I left this turned on, with the default of continuous retries.

  3. Wireless -> Basic Security Settings -> Wireless Off (apply)

  4. Firewall -> General -> (pick Minimum) (apply)

  5. MyNetwork -> Connections -> Advanced (should stay set in following steps). The following are all inside MyNetwork -> Connections

  6. Broadband -> Settings -> Relese Lease (apply??)

  7. Broadband -> Settings -> IP Distribution, pick Disable (should already be set; else apply)

  8. Broadband -> Settings -> DNS Server, pick None (apply)

  9. Broadband -> Settings -> IP Address, pick None (apply) (when it's in bridge mode you won't see any of these settings.)

  10. Network(H/O) -> Settings -> IP Distribution, pick Disable (apply)

  11. Network(H/O) -> Settings -> DNS Server, pick None (apply)

  12. Network(H/O) -> Settings -> IP Address, pick "Use This Address" and fill in an unoccupied address on the LAN. (apply)

  13. Network(H/O) -> Settings -> In the Bridge group, left column, uncheck Coax and Wireless; check Broadband and also the STP box; apply after each step (jimc says: turn off STP). STP = Spanning Tree Protocol. Jimc says: Do check Ethernet/Coax! This is what you're bridging to!

  14. This actually puts it in bridge mode. There's a warning box, hit Apply. The router reboots.

  15. Network(H/O) -> Settings -> Network -> Broadband, he doesn't say any changes, but he hits Apply anyway.

  16. He clones the other router's MAC address so both the Actiontec and the other router are using the same MAC. Jimc believes this is only necessary if the ISP cues on the router's MAC address (like Time Warner does), and if this is done the other router should be altered to use a different MAC address. Jimc omitted this step entirely.

  17. Connect other router's WAN port to Actiontec's LAN port.

  18. Power cycle both routers. Other router should get an IP, Actiontec should not. This worked.

Continuing to activate local MOCA (not including local LAN ports which are bridged to WAN):

  1. This is all going to be in MyNetwork -> Connections.

  2. Make sure that Broadband(coax) still has no DNS, no IP.

  3. Broadband(Ether) -Settings - Network(H/O) - no changes but hit apply.

  4. (in Network Connections) -> Add -> Network Bridging -> Next

  5. "A bridge already exists" … Add a new bridge.

  6. Next screen lets you pick devices that aren't already bridged. In older router versions you would see the unused WAN ethernet port and LAN MOCA, but in revision I only the wireless is available. Useless. Cancel all. This phase of the procedure was a failure.

Further Investigation

Several times I have investigated the WAN ether port. The lamp indicating connectivity never lights, it never emits packets, and packets sent to it vanish without a trace. This interface is definitely not up.

On the other hand, the LAN ether ports are up and active. Using two machines I was able to show that they are truly a switch: unicast traffic sent to one port is not seen by another, though broadcast packets to go all of them. The role of LAN MOCA is unclear.

Wonders never cease! Open source download page for the MI424WR. Here you can find the hacked elements of the firmware that are under GPL or LGPL, and a readme telling how to install a ARM toolchain and compile the (hacked) sources. To get it onto the machine you need a serial debug board -- and where do you plug that in? (Presumably a header on the motherboard.)

Sparkfun website: This vendor seems to have reasonably priced breakout boards and cables, needed to hack devices like the MI424WR.

Unfortunately this is not quite the treasure trove I had hoped. The main part of the software is OpenRG by Jungo. Despite the name this is proprietary material. It seems to be all written in Java and to include the bridge management infrastructure: the sources do not include brctl or recognizable predecessor command-line tools.

I'm preparing a message to Actiontec. Here is Actiontec's support form. The message:

(Date installed: 2012-06-15; firmware 40.19.22; MOCA to ONT.)

I have special security issues requiring that my gateway machine be the primary "router". I have my MI424WR in bridge mode and it's working nicely, but this is bridging WAN MOCA/Ethernet to the LAN ether ports plus LAN MOCA, making LAN MOCA useless for data transport on my LAN.

What I would really like to do is: WAN MOCA bridged to WAN ether, LAN MOCA bridged to LAN ethers, wireless disabled (some people would want it on the LAN), DNS and DHCP (client and server) handled by the other machine. This URL tells how to do that: http://www.dslreports.com/forum/r17679150-Howto-make-ActionTec-MI424WR-a-network-bridge (by DaDrgon, 2007), but I believe this is for revision D, and with revision I you can only activate one of WAN MOCA or WAN ether, not use them as a bridge pair, as far as I can see.

Do you have any suggestions for bridging WAN MOCA to WAN ether on revision I?

Their answer: basically, you can't do that.

More Drastic Intervention in the Actiontec MI424WR Revision I

OpenWRT is an excellent open source alternative to the Jungo OpenRG management software, which runs on quite a variety of router-type machinery beyond the Linksys WRT-54G for which it was originally developed. OpenWRT has an official port to the Actiontec MI424WR, but only revisions A, C or D, which use a different CPU and devices. Here is a link to OpenWRT's wiki page for the MI424WR. I don't think I want to take the lead in bringing OpenWRT to revision I, but revision I is very impressive and someone else could get a lot of result from such an effort.

Something like this would be within my time limits and actually would probably be necessary as a first step in porting OpenWRT: to evict the stock firmware and put Debian for ARM on the machine, with a non-obsolete kernel and a complete set of modules. Here are the results of preliminary reconnaissance.

Open Ports

Here is what I found using nmap:

Port Service
23/tcp telnet
80/tcp http Same management GUI on all HTTP ports
443/tcp https
992/tcp telnets telnet over SSL
4567/tcp unknown See Botnet Service
8080/tcp http-proxy
8443/tcp https-alt
123/udp ntp
Plus 60 nonresponsive UDP ports that aren't closed.
Using Telnet

To connect to the telnet port, you need to enable it in the management GUI. Advanced -> Local Administration -> Allow telnet on port 23. Connect using telnet and give the loginID and password, same as on the web. You get a sort of serial version of the Jungo OpenRG management software. Do help all followed by help $category or help $command (substituting the item you are interested in), for a very brief description of the available commands. help -s $string will search for items containing that string.

If you give a category command you're in a kind of submenu. exit will get you out.

I'm not sure if there are items in the serial interface that you don't get in the GUI, but the GUI seems very complete, and a medium intensity review of what's available didn't turn up any treasures except for these:

The command bridge info told about the same story as the GUI does, but it listed a mysterious second bridge called br_s0, containing the Network(H/O) bridge itself (I thought recursive bridging was illegal), the LAN Ethernet switch, and the broadband device (MOCA).

The command system shell will spawn a shell. This is lash from Busybox. Hacker support is very limited; there is no grep, more, or find. It does, however, have both cp and dd. You have to use cat (and your scrollback buffer) to view system files.

Content of System Files

/proc/version

Linux version 2.6.16.14feroceon #1 Tue Oct 11 12:32:09 PDT 2011

/proc/cpuinfo

  • Processor: ARM926EJ-Sid(wt) rev 1 (v5l)
  • BogoMIPS: 1196.03 (AMD Geode 500MHz: 996.00)
  • Hardware: Feroceon-KW2
/proc/meminfo:

MemTotal: 123916 kB (interpreted as 128Mb really total)

Filesystems

From /proc/mounts and "df": The root filesystem is likely from cramfs (1.9Mb); can't tell how they manage read-write. A jffs2 filesystem of 32Mb (10Mb used, 22.7Mb free) is also mounted from somewhere. Looks like saved instances of the conf file(s), software images, and a log file.

/sys/class/net

  • ath0: Wifi (Atheros)
  • ath1: Wifi (Atheros)
  • br0: Network(H/O) bridge, 192.168.1.1
  • br_s0: Very mysterious
  • eth0: LAN ether switch
  • eth1: Broadband
  • lo: Normal loopback, 127.0.0.1
  • ppp0: PPPoE, not used for FiOS
  • wifi0: Wifi (Atheros)

I copied about 4.4Mb of stuff, but the byte counts on br_s0 increased by only a few kbytes. In kernel 3.2.9 you can look at /sys/class/net/br_s0/brif/ to find out its members, but not in kernel 2.6.16. There is no brctl program (or recognizable bridge admin tools).

ath0 and ath1 are definitely wireless devices, not MOCA; they have wireless statistics that are irrelevant on MOCA. The MOCA device driver apparently does not create network interfaces.

/proc/modules

These are a few key modules; not every module is shown.

  • usb_storage (seems unused, see /proc/filesystems)
  • tcp_mss (for MSS clamping, but I don't see an option in the GUI)
  • ath_pci and friends (Atheros wifi)
  • mv88e60xx (Marvell MV88E61{32,61,65} N-port ethernet switch)
  • EN2510_clnkdrv (Entropic c.LINK EN2510 MOCA 1.1)
  • kleds_mod (can't tell what this is, but the box has lots of LEDs)
  • one_module (looks like Busybox for kernel; may come with OpenRG)

For hacking the MOCA, this thread (O.P. rtoledo) (2007-11-25) on page 9 has a comment by azuretech (2008-09-08) pointing out that Entropic has a SDK available with Linux driver sources, so it should be possible to figure out how to get net packets out of the MOCA devices.

The MOCA chips do not give rise to network devices, and clearly the need to handle MOCA in userspace makes flexible bridging (like I want to do) very difficult. I wonder if a tun/tap interface would be helpful.

/proc/filesystems

Only has cramfs and jffs2. If you plugged something into the USB port you could read it as a block device but could only interpret it as cramfs because jffs2 only works on MTD, not USB mass storage.

Power Measurements

9W when booting, 12W in normal operation. The wall wart dissipates more power than I would expect.

More Attempts with MOCA

I tried bringing MOCA LAN back to life by putting in a separate MOCA channel station. I have pairs of Actiontec ECB2200 (not the V version, for Verizon) and Netgear MCA1001; results with either one were similar. Unfortunately I'm getting inter-band interference. Here are the bands:

Symptoms:

Routing variants:

Port 4567: CPE Maintenance, or Verizon Botnet Service

The router's webserver listens on several web-related ports, as well as port 4567 on the wild side. In older revisions it required a loginID and password, not the one given to users but same for all routers and widely known on the web. This authentication style was widely discussed in terms of a swarm of millions of bots. Verizon may have taken this threat seriously: revision I will not deliver any content to an ordinary user, returning 403 Unauthorized (HTTP, not HTTPS). This is connecting from a LAN port with the router in bridge mode.

In my configuration it isn't really feasible to chew on this port from the wild side and I can't say for sure what security measures they may or may not have, either at the router, in the ONT, or in the upstream routers. However, someone with cat-5 to the router could tap into this connection, i.e. connect the ONT, the Actiontec and his own laptop all to a network hub, and could dig up some more information.

If I were designing the port 4567 service, I would configure it to require a X.509 certificate from the client, signed by my own (self-signed) root certificate, and I would replace this root cert on the router frequently, in case a Verizon employee went over to the dark side and used the client certificate issued to him for nefarious purposes. The OpenRG software by Jungo in fact can do this configuration on any of the ports it listens to.

Since I can't get service from port 4567 I can't report what that service might be if I had proper credentials, but I saw a screenshot from an older router version which looked like the normal web interface. I would guess that 4567 is used by customer service people for troubleshooting: for example, checking if the user's computer or set top box has an IP address and will answer ping, all of which is available from the web interface. Or to jigger the user's configuration if it somehow got messed up.

The OpenRG software by Jungo also can be configured to check a server for updates, and to install them automatically. I saw one forum post where the user said he downgraded his firmware to restore a function that disappeared, and the router automatically updated again, annoying him considerably. I believe this feature is not tied in with port 4567.

Glossary and Acronyms

FiOS

When initially introduced the acronym was FOIS Fiber Optic Information Services, but it conflicts with the harmony grammar for English and so everyone pronounced it "FiOS", so Verizon went along with that spelling.

Wikipedia article about FiOS. Some details from that article: Verizon follows the ITU-T G.984 standard for the physical protocol. In particular, optical fiber has three frequency/wavelength bands that are particularly suitable for data transmission; they are assigned like this:

ONT

Optical Network Terminal. This device translates between the three signal bands on the fiber (POTS, TV, data or Internet) and their normal format on copper conductors in the customer's premises.

MOCA

Multimedia Over Cable Alliance, referring to an industry organization that maintanins and enforces a physical protocol for transmitting 802.3-type (Ethernet) data over coaxial cable. MOCA can coexist with conventional TV signals as it operates at a higher frequency. As utilized by Verizon there are two major bands: the 1000MHz band is used to connect the ONT to the house router, and the 1150MHz (or higher) band distributes data to MOCA-equipped client machines such as set-top boxes. The 1000MHz band has subchannels but they are used as a group to give about 250MHz bandwidth. The router uses one high band channel individually for about 50MHz datarate.

Many MOCA to Ethernet bridges only can use the 1150MHz band, designated D-band, and cannot do anything on the 1000MHz band. E-band refers to the 500-600MHz band and is used by satellite TV (DirecTV). Most MOCA bridges can't handle that either.

VOIP

Voice Over Internet Protocol. The voice signal is digitized, then compressed by a choice of codecs, and is sent to the other end using a normal TCP/IP connection. Incoming VOIP is decompressed and then performed by the client's sound card.

POTS

Plain Old Telephone Service. This involves copper wiring at baseband to the phones. From the ONT outward the signal is handled by VOIP on the normal data bands.

NAT or NAPT

Network Address (and Port) Translation. Internal clients send packets to outside servers via the NAT router. It alters the packets to appear to come from its own wild-side IP address on a randomly chosen port. Replies get the inverse treatment. Some protocols put addresses and ports inside the packets, and a helper module (Application Layer Gateway) is needed to do NAT on these internal items.

SOHO

Small Office / Home Office.

WAN

Wide Area Network, also known as the wild side or cloud or Internet. Hackers reside there.

LAN

Local Area Network. Generally the hosts on the LAN are under your administrative control and they have only one route to the wild side, through your router, referred to as the default route.

STP

Spanning Tree Protocol, defined in IEEE 802.1d. If multiple routers are connected to multiple network segments, loops are possible. The STP is a procedure by which each router, communicating only with the neighbors it is connected to directly, can break loops by disabling ports. In the typical SOHO environment there is only one router and two network segments (LAN and WAN), and STP is irrelevant.