If you have zero interest in networking and/or associated troubleshooting, you may stop reading now. Otherwise, I found the following presentation from NANOG very informative and discovered there's a lot more to tracert than I knew of ... Cheers, - Chris ---------- Forwarded message ---------- From: Brent Chapman <brent@netomata.com> Date: Fri, Nov 13, 2009 at 1:55 PM Subject: [lopsa-tech] So you think you know traceroute... To: tech@lopsa.org Most network engineers and sysadmins would probably say that they're intimately familiar with 'traceroute<http://en.wikipedia.org/wiki/Traceroute>', and consider it one of their fundamental network troubleshooting tools... I certainly do. But you might be amazed to learn, as I did, how much you * don't* know about traceroute. Richard Steenbergen of nLayer Communications, Inc., did an *excellent* presentation on traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf> at this month's NANOG <http://www.nanog.org/> (North American Network Operators Group) meeting: - A Practical Guide to (Correctly) Troubleshooting with Traceroute<http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N47_Sun.pdf>, by Richard A. Steenbergen<http://www.linkedin.com/pub/richard-steenbergen/0/8b/398> of nLayer Communications <http://www.nlayer.net/>. - http://www.nanog.org/meetings/nanog47/presentations/Sunday/RAS_Traceroute_N4... Among other things, this presentation shows you: - How traceroute works - What you can learn from the DNS hostnames returned by traceroute - Where the ISP/carrier boundaries are - Where the equipment is located, geographically (do you know what a CLLI code is?) - What type of equipment the ISP/carrier is using - What the round trip times reported by traceroute *really* mean - How you can be led astray by ICMP prioritization, rate limiting, asymmetric paths, and load balancing One of the coolest tricks I learned from this presentation is, to find out more about what's at the other end of some hop that appears to be a point-to-point link, assume that the IP address you see is one of the two addresses in a /30 subnet (as is commonly assigned to point-to-point links), and do a DNS reverse lookup of the other address in the /30. This is useful, for example, in figuring out which egress port a packet went out on, since traceroute normally only shows you the ingress ports for each device along the way. For example, let's say I was looking at the following traceroute output, and wanted to know the egress port on router #3, as the packet moved to router #4: *brent%* traceroute www.google.com traceroute: Warning: www.google.com has multiple addresses; using 208.67.219.230 traceroute to google.navigation.opendns.com (208.67.219.230), 64 hops max, 40 byte packets 1 192.168.0.1 (192.168.0.1) 3.145 ms 2.573 ms 2.382 ms 2 75-101-29-1.dsl.static.sonic.net (75.101.29.1) 9.555 ms 9.054 ms 9.089 ms 3 127.at-X-X-X.gw3.200p-sf.sonic.net (208.106.96.193) 9.510 ms 9.871 ms 9.194 ms 4 200.ge-0-1-0.gw.equinix-sj.sonic.net (64.142.0.210) 11.965 ms 11.870 ms 11.839 ms 5 0.as0.gw2.equinix-sj.sonic.net (64.142.0.150) 11.928 ms 12.519 ms 12.394 ms 6 GigabitEthernet3-1.GW2.SJC7.ALTER.NET (157.130.194.17) 11.360 ms 16.257 ms 11.268 ms 7 0.so-0-0-1.XL4.SJC7.ALTER.NET (152.63.51.50) 11.729 ms 11.679 ms 11.403 ms 8 0.so-7-0-0.XL2.PAO1.ALTER.NET (152.63.113.21) 14.775 ms 17.455 ms 0.so-5-0-0.XL2.PAO1.ALTER.NET (152.63.48.9) 15.548 ms 9 POS7-0.GW6.PAO1.ALTER.NET (152.63.55.14) 12.886 ms 13.143 ms 13.029 ms 10 65.203.37.46 (65.203.37.46) 13.517 ms 14.708 ms 16.566 ms 11 * * * 12 * * * ^C To find out more about router #3's egress port, I look at the IP address for router #4 (64.142.0.210), figure out what would be the other IP address in the same /30 (64.142.0.209; hint: the lower address in a /30 pair always ends in an odd number, and the higher address always ends in an even number, so if the address you know ends in an odd number, the other address in the same /30 is going to be the next *higher* number, and if the address you know is even, the other is going to be the next *lower* number), and do a DNS reverse lookup of that address: *brent%* dig -x 64.142.0.209 ; <<>> DiG 9.4.3-P3 <<>> -x 64.142.0.209 ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49382 ;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0 ;; QUESTION SECTION: ;209.0.142.64.in-addr.arpa. IN PTR ;; ANSWER SECTION: 209.0.142.64.in-addr.arpa. 259200 IN PTR 200.ge-6-3-0.gw3.200p-sf.sonic.net. ;; Query time: 31 msec ;; SERVER: 208.67.222.222#53(208.67.222.222) ;; WHEN: Fri Nov 13 09:42:05 2009 ;; MSG SIZE rcvd: 91 Another handy tip from the presentation is that, since light travels through fiber optic cable at about 200 km (or 125 miles, if you prefer) per millisecond, each 1 ms of delay shown by traceroute (which, remember, is*round trip* delay) should represent about 100 km (62.5 mi) of distance if the delay were due entirely to the distance travelled (i.e., no queuing or processing delays). Using that fact, you can see that 40ms for a packet to go from San Francisco to New York (about 2500 miles, or 4000km) would be "normal", but 40ms for a packet to go from San Francisco to San Jose (about 50 miles, or 80km) would indicate a problem; it should take the packet less than 1ms to cover that distance and back, so something else (congestion or processing delays, for example) must account for the other 39ms. There's a *lot* more in this presentation, about more complex issues such as - how the way in which routers handle traceroute packets can produce biased results (most routers handle traceroute packets much more slowly than they handle "real" data packets, which can make things look much worse than they are) - how asymmetric paths can lead you astray (traceroute only shows you the path *to* a system, but if you're pulling lots of bytes from the system, as would typically be the case with a remote server, you probably care more about the path back *from* the system, which might be totally different - how using MPLS, which is increasingly common in carrier networks, can lead to very confusing round-trip times in traceroute Anyway, if you ever use traceroute, I highly recommend that you review this excellent presentation. I think you'll be pleasantly surprised at how much you learn. Thanks to Strata Chalup of Virtual.net<http://www.linkedin.com/in/virtualnet> for bringing this very informative presentation to my attention. -Brent -- Brent Chapman <brent@netomata.com> Netomata, Inc. -- www.netomata.com Making networks more cost-effective, reliable, and flexible by automating network configuration