Map of your typical LLC connecting to internet, for beginners

Notice: Page may contain affiliate links for which we may earn a small commission through services like Amazon Affiliates or Skimlinks.

tinfoil3d

QSFP28
May 11, 2020
881
406
63
Japan
Okay, trying to figure out, in general, how to become multihomed at home. Or, say, in a countryside building.
Is this how you generally connect to internet?

Code:
 [ME]---(fiber to ISP1)---[ISP1] ---(direct fiber to hugeIX)
   |                                            |
   |                                            |
   |                                            |
(fiber to ISP2)                        [The big hugeIX]-------[Over the pond IX]----...etc
   |                                            |
   |                                            |
[ISP2]----------------------------(direct fiber to hugeIX)
If I'd want to do BGP from my site which isn't in a preexisting DC, I just apply for transit to ISP1 and ISP2 and then go pay APNIC or whoever for membership, AS and IPs(yes, they still have /24 unlike RIPE). Right?
But what if I'm in ISP1 or ISP2 position but I don't want to be a transit network, just and end customer with BGP. I pay several brazillion dollars to a huge company that routes fiber across the country, is that how this works? Does anyone know if it's brazillion or half a brazillion to get two fibers? Or if I'm far away, is it generally me who cares about the attenuation or the fiber operator, and I can just use more broadly available modules, without setting up a whole submarine-style DWDM machinery that's also several brazillion dollars?
How does it happen? Does anyone know anyone who's in ISP position on this chart? I just can't get my full picture of internet, from a perspective of an AS that will be operating from the edge of the world. Not from within a pre-existing datacenter.
To put that more into perspective, I have only one ISP which doesn't announce any ipv6. There's no other end-user-ready fiber cable available outside. And that whole one is probably entirely maintained by that single ISP, it was specifically pulled into this area several years ago, until that there was no nothing here. How can I work with this? I was playing around with different crazy ideas in my head like finding a place along the cable route where there's one more ISP available, then connect those fibers directly and then rent a section between that place and my location from this current ISP, that way I'd technically have two uplinks, while coming off one trunk cable though.
But in general just wondering what amount of zeroes in the quotes can I be looking at with all this stuff. And do I even get the stuff right.
 

sko

Active Member
Jun 11, 2021
249
131
43
Either get your own prefix (/24) and 2 uplinks with BGP peering, then announce your prefix to both of these ISPs with different metrics (or different paths). If one goes down, traffic will come over the second link.
This involves arrangements with both ISPs about metrics and route preferences and you might have to deal with asymmetric traffic. Also you won't be able to just get a /24 without a proper reason (and usually not as a regular person without a company in the background).
There's also another caveat to maintaining a /24 and/or a global ASN: usually RIRs require that any AS or prefix has to be reachable/announced over at least 2 BGP paths - so you'd need to either go with 2 ISPs or arrange some tunneled (MPLS) or multi-hop-peering (which is a PITA to arrange with ISPs/peers, and doesn't help if your physical line or primary peer goes down). Also you have to maintain full BGP tables, which requires decent hardware and requires you to properly safeguard your announcements and what routes you accept - otherwise you'll probably end up hijacking routes/prefixes or acting as a transit network and drown your uplinks (and upset your ISPs...)
With a single ISP you can simply receive a default route via each link and call it a day. Any responsible ISP will only accept announcements for the specific prefix they delegated you, so the risk of accidental route/AS hijacking is (theoretically) zero.

So for a home or even small to midsized company uplink you can simplify this A LOT by going with a single ISP:
Some ISPs offer backup links e.g. via directional radio (or a second fiber or even old copper infrastructure). That's exactly what we are using at my workplace. The primary link is a symmetric gbit fiber connection, the backup link is a 500/500mbit mmWave radio link. We have a peering over both links and announce our prefix with different metrics. Thanks to BFD with a 50ms*5 interval, rerouting is super fast and except for a short ~1-2sec interrupt on active VoIP calls, the users don't even recognize a transfer.
Of course you should check with the ISP if they route fiber and radio through the same node near your location (=very low redundancy) or if those links only interconnect at one (or different) of their edge datacenters - only the second variant gives you redundancy and protection from the typical backhoe incidents or e.g. a failing switch in the ISPs infrastructure. Of course, if that ISP has routing/peering problems at the edge, both of your connections still might be affected. You have to decide if this risk is acceptable and/or if the (huge!) premium on going with 2 separate fiber uplinks (at different physical locations of the premise) from different ISPs and all the additional effort mentioned above is justified. (also with 2 ISPs your transfer times will usually be much higher, often i

The big advantage of going with one ISP: you don't need to buy/rent a full /24 prefix. The ISP will give you a small slice of one of its /24 prefix, which is WAY cheaper (and absolutely sufficient even for most mid-sized non-IT companies), but cannot be announced globally (i.e. when announcing over different ISPs) because anything smaller than a /24 is not allowed and usually gets filtered/dropped.

As for your problem of the single ISP in your area: Look for local ISPs that push expansion of fiber infrastructure in your region. The big ISPs usually block this as much as they can (because they can still milk that copper cow if there's no competition...) and charge absurd fees for anything that actually belongs in this decade.
Small/local ISPs usually have a much more realistic price tag and are more flexible when it comes e.g. to multi-hop peerings (to place the BGP peer behind the edge router that does the actual heavy lifting) or transfer networks with more than 2 usable IPs for redundant routers. With big ISPs you usually get "not possible, go away" or "we can do that if you use our even more premium service for only N-hundred bucks more per month..."
 
  • Like
Reactions: BoredSysadmin

tinfoil3d

QSFP28
May 11, 2020
881
406
63
Japan
@sko Thanks a lot.
I don't want a paper-only multihoming(as a prequisite for APNIC and ASN registration and IP alloc), I'd rather have it in real life. My ISP actually does have a lot of downtime for maintenance, sometimes they even break something and no other AS is reachable anymore.
If I wanted I could colo anytime and play around there with my router. But I want to do it right here, on site.
BGP aspects are many, yep, I'm aware of bgp leaks and unintentional transit, that's all software stuff(and mostly taken care of by ISP filtering out anything but your AS and other means), I'm rather more interested in how is it all done at physical level and what are the relationships and overall organisational aspects of the process of "just setting up an own semi-redundant AS"(semi is because the bottom line is, just one cable, nothing as fancy as RF, we don't even have that type of equip licensed here for japan, like, mikrotik or ubiquiti kilometers-long RF extenders, and we may never have it). I can either try renting out a strand of fiber from the trunk that comes here from my current ISP and connect that to another fiber from another ISP along the run where both trunks cross, or be super creative with some technologies I don't have understanding of, MPLS or a currenly unavailble anymore offer of BGP tunnels from he.net or something via LTE carrier or maybe starlink or something. Maybe that's something that can theoretically be available for me now, if I was a member with ASN already which I'm not. Maybe I could have applied, starting off at a datacenter location, which will have multiple carriers, probably, but I'm really looking at something way bigger than me. I just don't understand how you connect to the internet from those DCs. Who do you pay for that? Say, one of DC just says in their rather short whitepaper, "we're using NTT backbone", which is likely exactly what I'm asking about, it's the operator of a nation-wide fiber link that connects you with tier 1(such as NTT itself) and all the others in japan IX over there in Tokyo...
I'm probably asking a lot of stupid questions that are easily answered on a DC tour. Is it a normal thing to request a physical DC tour nowadays? I've never been to one so I won't know. The one big thing I'm missing in my picture is how and where and who's routers are where exactly, usually, and how are they physically connected to each other. Not the software or bgp magic is magical to me. The real-world interconnect is.
So it's also kinda like, don't want to look like an idiot out of place, begging for someone to say "if you don't know how it all works, why the heck would you need bgp in the first place?"
I do know what bgp is and why would I need it, and it's simple lines connecting router vms in gns3. When it comes to real world, those aren't simple lines anymore, there's a lot of money and expensive eqiupment involved to operate that as an imaginary line like it looks in gns3. In a real world you don't simply connect some fiber in a countryside to the heart of JPNAP or whatever big IX.
I feel pretty dumb to contact the operators of my ISP(they're one of a few in JP with own AS and have a contact on peeringdb, a guy who's almost like CEO there) if I don't have understanding of these relatively simple things.