RIPE 82
.
IPv6 Working Group session
.
20 May 2021
.
10:30 (UTC + 2)
.
JEN LINKOVA: Welcome everyone to IPv6 Working Group, another virtual one. We have a packed agenda. Can I ask if anyone has any suggestions or agenda bashing, but I suspect we don't have time for anything, but you can suggest privately to the Chair if you have any amendments to the agenda and we can consider it. We will do our best to accommodate you.
.
Etiquette: When you go to the microphone, please state your name and affiliation if you have one. I hope everyone is now familiar with how to use Meetecho. If you are not, there is a button for you to send a video and send the audio, to please join video and audio if you would like to ask questions. You can also ask questions in the Q&A section and we will read it out loud.
.
Housekeeping: I probably should do the poll right. How many people read the meeting minutes from the last one? I actually have. I don't have any comments. I would say please take a look, and check if those minutes are correct. I believe RIPE sent them to the list, and before the end of this session, please let the Chairs know if you disagree with the current versions of the meeting minutes, otherwise we will considered them approved.
.
Rate the talks.
.
And now I am passing the microphone to Ray, who is going to talk about the Chair re‑election.
RAYMOND JETTEN: Good morning, everyone. Jen has been doing a great job for the last three years and she was trying to step down but we don't let her really, so, after a small conversation, we had acknowledgment that she is willing to continue if anyone else doesn't want her job.
.
We put that stuff on the mailing list, and guess what? No one dared to step up and say 'I want her job', so, with that, we confirm here that Jen is again, for the next couple of years, with us. Thank you.
JEN LINKOVA: Thank you. I guess I need to upgrade ‑‑
RAYMOND JETTEN: Geoff has already upgraded. Cannot hear you.
GEOFF HUSTON: I have exactly ten minutes so I'll be as quick as I can.
.
This is a presentation on an update on IP fragmentation. I don't seem to have control of the slides, Jen, so you are going to have it.
.
So this started with an RFC 7872 published in 2016, and it actually referred to work done in the previous two years. They sent fragmented v6 packets towards well‑known v6 servers using the Alexa top million and the world IPv6 launch set and found the following drop rate there underneath that blue arrow and for web servers they were seeing upwards of around a 30% drop rate at the most optimistic level, with some numbers getting worse than that.
.
That was certainly a shakeup. It's kind of, does the fragmentation work?
.
So at APNIC we decided to actually replicate this experiment but actually use the opposite data flow from the server towards the end client. So again, what we're doing is using our measurement platform that uses ad‑based measurement and across at the time, August 2017, across 1.6 million v6 clients, we sent them fragmented v6 packets and in this case we actually tried to understand if they got them, and the way they did that was look for the ACK. So we send the frags, we're listening at the server, did you get them? If the ACK comes through on the TCP session, yeah, you got it. Fragmentation loss rate: 21%. Getting better.
.
Now, at 21%, that's still pretty shocking, in in fact worse than pretty shocking. Thumbs down, can't fragment. We thought this year we might rework that experiment and look at this again using exactly the same kind of methodology with one subtle variation. So it's the server to client, we're fragmenting large packets as they head towards the client, it's an HTTPS web session, it's what everyone does and we actually constructed a middle box that did this fragmentation which is actually a subtle variation of IPv6 NAT and incredibly precise control on the way fragmentation is actually going to happen because we are doing the fragmentation almost by hand code. And we detect a drop by looking and listening for the ACKs that come back because the other end if they are receiving all the fragments will reassemble the packet, will send to TCP engine and we get an ACK of those session numbers. We get the ACK you reassemble ACK. If the session hands, thumbs down.
.
This is the twist, this time we were team mucking around with what's the fragmentation size that works best? And when you vary that size, does it alter the loss behaviour? And we have been doing this now every day for the last three months.
And it's kind of interesting, we started in early February with a 10% loss rate, which is much better than 20%, and then very quickly it swung down, and for the last few months it's been hovering around 6 to 7%, moving up to 8 at points. This is amazing. There are ten times the number of v6 users since 2017 ‑ 30 million to 300 million. Fragmentation rate appears, on average, to have dropped by two thirds. So are we getting better at v6 fragmentation? Yes. You know, it's actually working better.
.
That's still not the best number, but it's certainly not the worst.
.
The problem with any kind of one point measurement is that there is a world of detail behind it. Red is a 10% drop rate or worse. Anything less than that goes yellow and then goes green. And this is, by and large, an odd map.
.
Where we have a lot of v6 that's been around forever, US is certainly one of those cases, deployments in Europe, the loss rate is about 10%. And it's actually only in India, Iran, bits of Russia, Myanmar and over there in Panama and I think that's ‑‑ no, I went guess, you can figure it out, just below Guatemala, really, really good. So, what we are finding here is a conclusion that the more recent v6 deployments are a doing a whole whole lot better than the older ones. Maybe the older ones are being masked by happy eyeballs or some other thing, but loss rates vary a lot.
.
Drop rate by size. Now this is the bizarre thing. You can see a very, very slight bump at 1,280 where 7.4 average drop becomes 7.5. I don't know quite why but it is very marginal. What's this? That's completely unexpected. Why is 1,408 octets getting almost a one‑and‑a‑half percent higher drop rate closer to 2% than the smaller ones? I am like if are encapsulating 80 bytes of encapsulation, that's a lot of tunnelling and it's kind of, this doesn't seem to make a whole lot of sense, and so we dug a bit deeper.
.
And what we found was even more confusing. In North America, the drop rate is around 11% irrespective of size. European providers: Well, subtly different, because up until around 1,380 bytes or so the drop rate is around 12, 12.5%, and then it swings up a little bit higher to 13% for the larger packets. I don't know what you guys are doing over there in Europe, but that's just weird. That's bad. Bizarre.
.
Asia. Almost a bit like Europe, it's slightly more gentle but you notice the absolute failures it's 4.5 up to around 6% there. So Asia is not quite as bad, but again, larger packets are dropped more frequently. No explanation on my side. I don't know.
.
Of course, we have enough data now that we can look inside service providers. If you are curious, AS 852 is TELUS in Canada, major provider there. The drop rate has got better, it's swung down from 100% to 80%, well done.
.
On the other side. One of the largest v6 deployments we have ever seen, which is Bharti Airtel in India, which has a client base in v6 population numbering in the hundreds of millions, drop rate 1%. It can be done guys, in Canada. TELUS, it really can be done, you can make it work brilliantly.
.
Why?
.
Well, it could be local security policies. There is a whole bunch of networks that don't like fragmented v6. Microsoft servers, for example, just simply block it and a number of other service providers simply go, just don't fragment, please, we're not going to accept it.
.
More likely is that the extended header options in v6 trigger slow path processing. Once you go into the slow path, the drop rates get higher. Now, that's not 100% drop, but it is sort of a slightly higher than nominal drop rate because the slow path invariably actually has a higher drop rate and it's hard to believe, but there might be some IPv6 path MTUOs but don't forget our fragmentation policy is, we send a packet a lot smaller than 1,440 or 1,500, we are sending an initial fragmentation packet down between 1,200 and 1,400 at most. So, it's unlikely to be path MTU, but who knows? It's the Internet.
.
There is a daily report we produce this every single day. It's an ongoing measurement. There is an URL, if you screen snap in the next two seconds, that blotchy thing down at the bottom, where the QR code will lead you to that page. Too late.
.
Summary:
.
We're getting better. It's improving a lot. Five years ago, thinking about fragmentation in v6 was, you have got to be kidding, that's the worst idea you have ever had today, don't fragment. These days, the answer is, you may have had better ideas but it's not completely insane. It will work most of the time unless you are in Canada. But it will work more likely if you are inside a country or a network that has had a relatively recent v6 deployment and they are doing much better handling of v6 fragments. The more mature deployments the news is still bad.
.
For some reasons the smaller frags are more robust than larger ones, if you are going to fragment a packet to produce the larger one and put the trailing small bits afterwards, which is what most hosts do, don't do that.
.
Slice and dice evenly. Keep the packets smaller and you'll actually stand a better chance of getting the fragmented packet through for reasons that I don't know, maybe someone could tell me, why packets higher than 1,280 have an accelerated loss probability until you get up around 1,400, where it's sharply higher in some networks under some circumstances. And I think that's it.
.
Yeah! Questions. Or not?
JEN LINKOVA: I see a question in Q&A session. Okay.
.
"Hi Geoff. From how many vantage points do you measure, is really the end host that fail or might there just be a few misconfigured choke routers close to your vantage point through which is most tragic... for the available devices. If so, could it be that your numbers might just give the impression that device is route with a choke router?
GEOFF HUSTON: We use four server points and if you actually have a look at Canada and the US, you'll find that different networks have different loss rates, and so it's actually far more likely that this is close to the edge, not close to us. Also, we measure basically entire continents. So that we're actually not finding that the choke points are close to us, it's way more likely that the choke points are actually closer to the user, and they are either very close to the user or in the network that last top access network.
.
Next question, I'll go quickly because I know time. Did I my test include atomic fragments? I don't know what you mean by "atomic fragments". So the answer is meh!
JEN LINKOVA: I think actually, I think ‑‑ basically ‑‑ so you said two fragments, initial larger one and smaller one, it was just one packet with fragment header but no subsequent fragment.
GEOFF HUSTON: An old fragment header. That's coming up. Tune in in about a month's time or so and we'll have some data. We're trying to understand whether, if you had a null fragment header, just having an EH that doesn't mean anything is going to do anything or not. So, no, I haven't measured that yet.
.
Firewall set the MSS around 1,380, is the next question from. Yes, but some seem to go at 1,400. That setting of when to drop a packet, how big does it need to be? If a firewall sets an MSS down that low it's asking for trouble. It's just do you hate your traffic that you are passing through? It's a bizarre kind of thing to do. It's very low.
.
David: Have I done tests with fragment sizes less than 1,200?
.
No, it's the smallest we did.
.
Oliver: What's the baseline drop rate for v6?
.
Different talk, different time, I have run out of time. It's around 2%, from memory. I haven't looked in the last couple of weeks. But that's a different report in the site we use. At this point, I think the base rate is 2%.
.
Example configuration for ISPs and enterprise to show how to make it work.
.
Yeah, don't drop fragments. Sorry, it's as simple as that. Don't drop them.
It's more actually, get your router vendor not to drop packets with extension headers.
.
What can people inside the particular countries do to make this better?
.
I don't know, Jan. I think equipment has changed. I think what was on the market five years ago is now different to what's on the market today. I don't build networks and I don't buy this equipment, but I think the folk who are buying today actually are getting better equipment and I don't know exactly what aspects of that have improved but something sure as hell has got a whole lot better.
.
Will I publish the data?
.
Yes, that's all on that URL I gave you. Just follow the link.
.
Have I considered a drop rate of fragmented packets with other v6 options had a trigger slow path?
.
I was actually discussing this this morning, what other extension header options other than a null fragment actually makes sense? No one uses nimrod, so the nimrod routing header doesn't work, there is a whole bunch of extension headers in v6 but most of them are crap. So if you can find the extension headers that people want to use in anger on the network I'll test them but I haven't done so at the moment. And I think we have exhausted the Q&A. So I'm going to hand it back to you. So thank you all.
JEN LINKOVA: Thank you very much, Geoff, for such a positive presentation. I am so excited and thank you for doing measurements, which I would always like to do to see how we improve. I think we'll continue the discussion offline and I am looking forward for more measurements from you which proves that we can make that thing tock.
.
Thank you. Next is Tim.
TIM CHOWN: Hello, everyone. So I am here as part of a group that's working on an update to RIPE554. It's myself, Merike, Sander, Tim Winters and Jan.
.
So this is the RIPE document on IPv6 ICT tender guidance. The previous version, the second version is almost ten years old. So, we agreed at the last RIPE meeting when we presented this, that we would get on with updating it. We have had some discussion amongst the group of authors, and we feel what we should really focus on is to update the content as it is without adding new categories, new sections to the document. We feel if we can publish a new 554 with everything moved forward with ten years of new RFCs and new operation experience, that's useful, and if we feel at a later date that someone feels they should add extra categories we can look at it then.
.
That's the scope of what we're doing.
.
We are keeping the linkage to the IPv6 Ready Logo programme in because that's still an active programme. As part of that, we brought in Tim Winters as an extra author because he has experience with that.
.
If you want to look at the current draft of the update, there is a Google doc there that everyone has view access to, so you can look at that. I also e‑mailed a PDF of that to the list two or three days ago.
.
And, please, if you have got comments, send them to the v6 Working Group list so we can have discussion there.
.
Next slide.
.
So, the sort of status of the work. We have actually, having said we wouldn't add extra categories, we have actually removed one. So there was a category of mobile devices in there but we agreed that since that also led into 3G PP requirements and things certainly the author team didn't have expertise in we felt it would be better for the time being at least if we removed the mobile device category and just consider mobile devices as hosts that might be connecting to local infrastructure. So really you are thinking of the wi‑fi connectivity for mobile devices rather than the 4G, 5G, whatever.
.
We have added reference to transition mechanisms. One that weren't well mentioned ten years ago. But the focus there is on the on the v6 only access networks to legacy. So that's probably mainly included in the CPE category. I remind again the categories are hosts, switches, routers, security equipment, CPEs and load balancers, those are the six categories of equipment in this document at the moment.
.
IPsec remains optional. We see no reason to change that. But there was a large discussion about that last time. And most of the work today other than that has been about updating references to RFC. So, for example, 2460 has become 8210 for the IPv6 core spec. There's been a lot of RFCs published in the last ten years, it's been about updating the ones that have been updated and adding new ones that we feel are mandatory. In doing that, there are a lot of additions an updates so it's quite possible that people in the audience will think well maybe they shouldn't have added that, or maybe they have missed my favourite RFC, so, kind of the stage we're at at the moment now keeping the scope of the document as it was was to get feedback on the list about which RFCs are added, maybe shouldn't be or ones that we have missed, so that's the main thing we'd like to hear from you.
.
As an example of that, RFC 8601 is now listed as mandatory, that's the RFC about including DNS server info in RAs, for example. That's the sort of thing we need feedback on.
.
Next slide.
.
So, things we know are on the to do list still. Merike has still to sync the security section with the quite massive now IETF Working Group group, v6 security draft, that I ended up being a reviewer of. That's 50 to 70 page document with lots of guidance on applying v6 security practices. V6 is quite a good document for people to be looking at. That's hopefully going to the ISG soon and therefore being published soon, but we should sync that with what we're saying here. That's the key point.
.
And then once all the RFCs that we are going to include in the mandatory sections in this update are agreed, Tim Winters will go through and check how that correlates against the IPv6 Ready Logo programme which I think is important thing to do. So we can indicate in the document with asterisks this is the case now which of our mandatories and optionals are also listed in that programme.
.
We have also noticed we probably need to look the a things like netconf, YANG, restconf, where at the moment we are only mentioning SNMP. That's a few other minor things we need to consider whether things like Bonjeur/mDNS, DNS SD should be in there as optionals. What we need are comments on the document as it is now. Please send those to the list.
.
So, that's it. You can e‑mail me directly. That's my e‑mail address. But we would much prefer it if you sent questions to the list. I have had one e‑mail sent to me about a specific RFC and whether it should be included. The thing there was it was published in February or March this year, so I think since we first did a parse of all the new RFCs, there is probably been some published since then so we probably need to do that again. So that will be also on our to‑do list. But please send things to the list.
.
Are there any questions?
JEN LINKOVA: It looks like there are some questions in the questions section. Tim, can you read them?
.
TIM CHOWN: I see the one from Jordi. He has contributed a lot on this particular topic in the IETF. I think 464 XLAT will be in there. I'm not sure about Lightweight 406. Please post to the list your comment on that. I think 46 will be mentioned along with NAT64 and DNS 64.
.
Yes, thank you, Jordi. Your input would be very welcome.
.
There is questions in the chat ‑‑ or are those ‑‑ there ‑‑ there should be a living document.
RAYMOND JETTEN: That's not used for questions.
JEN LINKOVA: Yes.
TIM CHOWN: I will ignore chat.
JEN LINKOVA: If anyone has questions, please put them in the questions section so we clearly know it's a question and not just a discussion in the chat. Okay.
TIM CHOWN: If there are no questions, we can move on, but, yeah, comments to the list. What we'd like to do is to get all those comments in and the document in to a sort of a final draft by the end of May, might be optimistic, but ‑‑ and then we can ask the Chairs to consider how we would go forward from there once we think we have a final draft that no one is objecting to.
JEN LINKOVA: Okay, I think we have Jan who would like to say something.
JAN ZORZ: Hello. I would just like to emphasise what Tim said. This is just the update of the document, the current update. We would like to publish it as soon as possible because lots of people is using it, and then if you want to add sections, if you want to add things, we will be aiming for an update of this or something like this. But I would really like to see all the comments in as soon as possible so we can just publish the update and get on with work with additional comments ‑‑ additional sections.
TIM CHOWN: Great. Thanks. Thanks, Jen.
JEN LINKOVA: Next presentation is Yannis.
YANNIS NIKOLOPOULOUS: Hello. Hi. Hopefully you can hear me. Can you hear me? Thanks for joining in.
.
This was the original title of my presentation, but the new improved title is this one. So this is a presentation about the rocky road towards IPv4 as a service and this is the third chapter in this ‑‑ MAP‑E.
.
Just to get you up to speed a little bit. We have been doing, as an ISP we have been doing quite a lot of IPv6 in the previous 15 years. As far as these measurements goes, I think we're about 80% on the IPv6 deployment measurements. And as I said, we have been doing IPv6 for a few years now. We have a project running since 2009, and we have been providing services to our end users since 2013, all residential service users, business users, fixed line BGP, and in the past two or three years I think, we have been providing a dual stack service for our mobile users as well.
.
Almost all of our residential users have native IPv6, and we have been convinced for a long time now that IPv6‑only is, you know, the way forward.
.
We did skip some of the transition mechanisms like DS‑Lite /6rd, not because they were not good but because we were preparing for the next generation of mechanisms. If people want more information about the our IPv6 deployment as a whole, you can check out this link.
.
So, this is a brief timeline for our IPv4 AAS service journey, which started about six years ago. April and May 2015, when we did our first MAP‑T and Map‑E trials respectively. The first ones we did were a Cisco machine with, I think it was still the experimental MAP‑T code, and with the only CPE that was supporting MAP‑E and T at the time. Same thing with MAP‑E, we just used a different Cisco, and we also used a service module at the time.
.
Then came lightweight 4/6, there is a link there if you want to find out more information about that deployment. It lasted about two years. We had to put it to rest for several well documented problems. And around the beginning of 2020, we thought we'd give MAP‑E another go, and here we are.
.
So, for people that might be starting outright now and thinking about transitioning to a technology such as MAP‑E or MAP‑T or even lightweight 4/6. You have to know what are you competing against? In our case, first it was easy, we were competing against a single stack CGN and it's easy because it was a temporary solution, I mean honestly, we keep saying that to ourselves own, like, five, six years later, but anyway, it is expensive to scale that. I mean, those service modules cost a lot of money, and as I said before, we were convinced that you can't have a future without IPv6. We take that as a given ‑‑ we still do. But then we had an epiphany and we converted our CGN to dual stack and it wasn't so easy to compete against that, because okay, we have to get rid of the expensive service modules, but what else? Honestly, we couldn't think of any more arguments that, you know, had any essence.
.
So, also, keep in mind that, yeah, I mean, people like us might be doing dual stack for a long time, but still, there is still reluctance towards the change towards IPv6. I mean, the operations teams, even the engineering teams, are reluctant to treat IPv6 the same way as they do IPv4, and, you know, it's 2021 and we still hear turn off IPv6 when things go wrong. This shouldn't be a thing but it still is.
.
So how can you make the correct choice? So, there is two ways to go about it. There is the wrong way to go about it and there is the slightly less wrong way to go about it. The wrong way to go about it is just, you know, produce arguments like mapping makes more than than MAP‑T, or, you know, it feels right. Why?
.
And also, let's not spend a little bit more time on the CPE development because timing is of real essence in an ISP environment. Everything is critical. Everything needs to be done by today, and the same goes with money. Money, you know ‑‑ let's not try to spend a lot of money on the CPE development, let's put in the really very basic stuff just to get it working and then we're good to go. And also let's just keep it under the radar, management don't care about it. They don't need to know much about the project. I mean, if they don't, they let us do our job, you know, isn't that how it is? No, no it's definitely not.
.
So the slightly less wrong way to go about this to actually research all available transition methods in depth and that really requires a lot of time and effort put into it. We did quite a bit of research, but not in depth, and then you have to find the best fit in the network, in your network. Having worked for an ISP for many many years now, I can safely say, and you probably are already aware of it, that there is no two networks that are alike, so you really have to find the best fit. And the CPE must really be future proof. A lot of work must be put into the CPE development by either you or the vendor.
.
And management really needs to be engaged. They really need to be made part of the journey. They really need to know all the importance of the actual project, they need to know the struggles.
.
So what about MAP‑E?
.
This isn't a crash course on MAP‑E. People need more information about MAP‑E, they can look it up, but as per the RFC 7597, MAP‑E is a mechanism for transporting IPv4 packets across IPv6 networks using encapsulation, and it also is a generic mechanism for mapping between IPv6 addresses and IPv4 addresses as well as ports.
.
Hence the A + B, address + port.
.
What is the components, or what are the important stuff? Mapping.
.
So, we have a border relay which receives the IPv4 traffic from the Internet, encapsulates it into IPv6 and sends it back to the CPE. And, of course, the other way, where the CPE sends IPv4 over IPv6 traffic towards the BR, which then decapsulates it and sends it over to the Internet.
.
There is a CE function, which is basically a NAT 44+ encapsulation, so the packet, the IPv4 packet is NATed using a public IPv4 address which has been magically provisioned by a provisioning mechanism such as DHCP v6. RFC 7598 which describes all the options, not just for MAP‑E, but MAP‑T and lightweight.
.
If we just lake a look at the very simple diagram, you have your IPv4 Internet on the right. The map in the middle and the MAP‑CE on the left, the DHCP server and you have a configuration, which is not something essential but it's ‑‑ you know, it helps if you have one. And what happens is that you get ‑‑ you provision configuration to the MAP‑BR and the DHCP server. Excuse me for one second.
.
You provision the relevant configuration to the MAP‑BR and the DHCP server which will then, in turn provision this to the MAP‑C. And what sort of information does the BR and the C need to know? They need to know the BR address end point, which is IPv6, the IPv6 and IPv4 prefixes, plus prefix lengths and some information about the port range that will be used by each user, because each user is provisioned with a public IPv4 address they can use, plus a port range that they can use with it.
.
Since this is a deterministic way of producing this information, the BR will produce the same information as the CE so it knows how to forward the packets to the relevant user.
.
What the building blocks in our case. We use the ISC DHCP, the latest version, and we use our BNG just as a proxy. We have two locations which are kind an in active standby mode, and when I say kind of an active standby mode, the configuration is ‑‑ the configuration in the actual bindings are synchronised between them. They both announce the same end point via BGP and one of them is preferred in our network. So, you know, if the preferred one goes down, then the other one will take over.
.
It's not a hundred percent seamless, but it does work.
.
We have two MAP‑CEs. In our network, we provision for different CPEs to our customers, the officially supported ones.
.
And two of them which represent the account for at least 90% of the total number of CPEs deployed have the MAP‑E feature developed for them. We have the user selection and provisions is done also automatically. We have scripts that select users to be provisioned with a mapping service. The main criterion here is that the user is a CGN user because our main goal is to offload or CGN.
.
We also have a configuration repo in order to feed the DHCP and the BR, this is a GitLab repo, and we use an [inaudible] 2.
.
Finally, we have 2 Cisco ASR 99s acting as our MAP‑BRs, they are deployed in an Anycast fashion for now, and I am saying 'for now' because there are some issues, some cases when you deploy many BRs in an Anycast fashion. But, okay, I mean for now, that's how they are deployed.
.
The BRs are dedicated even though at first we were going for shared function along with our, you know, the usual DC router function that they did before. We had to use dedicated boxes because of some limitations that Cisco has.
.
And finally, the MAP domain would only use one MAP domain. We use one MAP domain per BNG, we have about 128 BNGs, so, we have the same amount of domains.
.
The initial deployment was ‑‑ I mean, the initial trials were successful. We took a small batch of 20 users, some volunteers, some internal users and then we started ‑‑ since we saw that everything was working out fine and we let those users try out the service for a few weeks and then we started rolling out small batches, like 500 users per batch at first, and if our customers complained, there is a roll‑back mechanism in place where ‑‑ which is implemented for our support operators to use, you know, whenever there were complaints.
.
So, the initial percentage of rolled‑back users was actually smaller than CGNs. So, we thought that this was some good news. But things started to change a bit later. As deployment progressed, this percentage started to increase and the complaints started to become a little bit louder, and we had to ‑‑ well, we started troubleshooting the cases that were presented to us, and, I mean, at first we thought that they were unrelated, they were just customers couldn't browse specific sites, but each one could browse a different website, it's not that there was a common website for all, and then other customers couldn't attach large files, but then we found out quite a few users that were unable to use their PoS machines and that was a bit of a problem, also some users couldn't access their VPN services, and during a pandemic this was also a big problem. So we started to think that maybe it wasn't unrelated after all.
.
So, after a while, it took us a while to realise, you know, to get the bottom of it, but here is what the problem was.
.
This ‑‑ again, this is a very simplified diagram of the network. You have from right to left. We have the IPv4 Internet route stream provider, our edge router, the MAP‑BR, the BNG and the MAP‑CE. The MTU size between our upstream and the edge router is 1,500, and the MTU side inside is 1,492, which means that any IPv4 packet larger than 1,452 when the BR would encapsulate that packet into IPv6 and add another 40 bytes, then it would be more than the MTU size on or MAP domain, which is 1492, as I said, and the BNG would would have to cut the packet and send back a packet to big message, which the BR would happily ignore. That was the first thing we noticed.
.
That was something that we solved quite easily. I think it was just a case of MTU path discovery not being enabled by default. So we did enable the MTU path discovery on the BR, but still we couldn't ‑‑ we didn't get the packet to our destination, and that's where we started engaging Cisco more and more, and we had a lot of troubleshooting sessions with them, and finally realised that the BR was not handling those packets. They were getting lost somewhere between the fast and the slow path. But, finally Cisco did manage to produce a path that would take those packets and fragment ‑‑ I mean, that would fragment those packets and send them back to the destination. But unfortunately, this could only be true for a limited packet rate because those packets, as mentioned, were being punted, and now we were violating the control plane policy of the box. So, we could only get a small packet rate of those packets and the solution would really not scale at all, we would need a lot of boxes, so that was a no‑go at the time.
.
So, this is just an intermission to just remind everyone that the RFCs need to be read and studied carefully. I mean it is mentioned that it's strongly recommended that the MTU inside the MAP domain be well‑managed so that no fragmentation occurs within the boundary of the MAP domain.
BENEDIKT STOCKEBRAND: About two minutes left.
YANNIS NIKOLOPOULOUS: So the solution that we're trying to implement right now is to increase the MTU to 1,540 inside the MAP domain and that's not as easy as it sounds because there is a lot of Layer2 equipment between the BNG and the MAP‑CE and we have to check them, and then a feature that needs to be developed for the CPE. And we have to investigate the possibility of having different MTU sizes on the CPE between dual stack and MAP‑E subscribers so that problematic subscribers can return to the MTU that works for them.
.
What did we learn? Not too many things. Just very quickly. When procuring a new CPE, try to include as many mechanisms as possible and try to use RFC 8026 for prioritising those mechanisms.
.
For the CPE, you always have to assume that the MAP algorithm will be implemented incorrectly, definitely, each and every time.
.
Extra attention should be paid to the MTU across the MAP domain and the provisioning should be automated as much as possible and take care of your MTU.
.
One last thing that we learn is that people are not very open with their implementations. I mean, I personally try to contact at least three ISPs that are known to have similar deployments. I didn't get an answer from any of those, unfortunately.
.
So basically, I am just going to end my presentation with a call, a call for collaboration, and a question to you is to ‑‑ basically, how can we fix this?
.
Thank you very much.
JEN LINKOVA: So I guess we have a few questions in the Q&A section. Would you mind answering them?
YANNIS NIKOLOPOULOUS: Sure, of course. Just reading the first one from Blake Willis: "What was the driver that pushed you to switch from MAP‑T to MAP‑E?"
.
Okay. We only ever tried MAP‑T in a very, very brief trial six years ago, as I mentioned, so we didn't really switch from MAP‑T to MAP‑E. When the time came and after we had tested both in the trials we, just went with MAP‑E because it felt right. Sorry about that.
.
Another question from Jan: "What were the top three operational issues with ‑‑"
The CPE ‑‑ actually, yeah, the CPE and the AFTR, the problem is that the AFTR couldn't scale well to a large number of subscribers and the vendor couldn't put up the extra development effort because there was no interest from other operators.
.
So that's a top 2. But I can't remember a third one right now.
.
A question from Richard Patterson: "Could you tell us more about the issues with Anycast and the BR?"
.
Yeah, yeah, well issues is that you might get sent ‑‑ for example, an ICMP v6 packet too big might be sent to the wrong BR if it's Anycasted, and about the policy‑based routing, yeah, since MAP‑E is implemented as a policy‑based routing mechanism in Cisco, Cisco only limits you to a number ‑‑ a specific number of PBR that you can implement on the box. I think it's either one or two per direction. That's the limitation.
.
Question from Jordi: "Slide 24. You talk about S468026, I think it's much better to use 8585, which includes also support for 46 X Etisalat. Very good point. I have got more success in several broadband deployments, cellular broadband and just cellular than all the other options. Thanks, Jordi, for that.
.
Working now on a deployment for 25 million subscribers, slowed down because of Covid. Thanks for the suggestions, Jordi.
.
Question from Sia Saatpoor:
.
"Thanks for the presentation. That was too technical for me."
Sorry about that.
"However, regarding activation of IPv6 and use of stimulation, I think that the major role and responsibility for the support for the use of IPv6 lies with the government. In the Netherlands, the highest policy body of the government has decided that all digital Dutch government must be accessible through IPv6.
That's great. By the end of 2021. That's even better.
.
"I am curious how the role of the government in Greece or other parts of... in the industry."
.
I can only talk about the Greek government. It's not too far behind, but it's a little bit behind. We, as operators, are operating part of the governmental network, so we are trying to push IPv6 as much as we can. I'm not aware of a government‑specific initiative at this point from Greece.
.
And finally, one more question.
JEN LINKOVA: If you could be very quick, because we are running out of time.
YANNIS NIKOLOPOULOUS: One more question, from Kostas Zorbadelos:
"After all this experience, would software mechanism be the way to go or could we just stick to CGN for the before part and dual stack deployment? Great work and best regards."
.
I really do hope, I really, really do hope that software mechanisms are still the way to go. Thank you very much.
JEN LINKOVA: Okay. Thank you very much. So next, Paolo.
PAOLO VOLPATO: Good morning everyone, I am with Huawei Europe. By the way, I will present on behalf of the different co‑authors you will see listed in my presentation.
.
So, before we begin, let me thank the Chairs of the IPv6 Working Group for having accepted our proposal to present the results of this analysis that we started around the year ago on the status of IPv6.
.
Since the beginning of our work, we have been involved in some activities and one of them is the publication of a drafter at IETF dedicated to the status of IPv6 deployment, and I will reference the draft in a minute.
.
So, moving to the agenda. Okay. So, I will just introduce the draft I have just mentioned. Then, I think we will enter the biggest part of the presentation, which is dedicated to the findings of our analysis. And based on our understanding, I would say that there are two main findings.
.
First one, the numbers of IPv6 seem relatively good. To an extent that we see, let's say, the importance of what we have called the IP service layer, the layer where we have both the end points of a connection, the user terminal and the content. It seems that we are ready to speak of the IPv6 service layer as an enabler for the transition to IPv6‑only.
.
Clearly, there are still areas which need an improvement. There are open issues and that will be the part ‑‑ the final part of my presentation before moving to the Q&A session.
.
Now, I have just mentioned the draft. You see the name on the title of this Charter. So, the initial proponents contributed to a white paper that you see referenced as well on this slide, and then decided to move to IETF to, as I say, bring it to a more technical layer. The intention was to update an old RFC, RFC 6036, which is more than ten years old, about the status of IPv6, identify what is missing for the transition to IPv6‑only and other motivations. We presented the draft at IETF 109 for the first time and, after IETF 110, it was decided to adopt the draft as if it were a group draft.
.
The reason was mainly because there was a lot of mail exchange on the mailing list, so we had a lot of activity, several ideas came up. We opened a space for many more research, let's say, activities, and that was also the trigger to come and present our work at RIPE.
.
By the way, you see the full list of the authors of the draft listed at the bottom of this chart.
.
Now, some early indications.
.
As I said, the numbers of IPv6 seem good.
.
We are going to see them in a minute.
.
We see, and there is a consensus, let's say, at least looking at the mail activity at IETF, that the connection end points, as I said, the user terminal on the one end and the content on the other end by the support IPv6. Such we can say ‑‑ we can speak of native end‑to‑end IPv6 connectivity, and that connectivity stays at the IP service layer.
.
As I said, one of the ideas that emerged during the discussion is that if this is true, probably we have to look at the IP service layer, sometimes it is referred to as overlay as the enabler, for let's say, moving further in the transition to IPv6‑only. Why that? Because there, we find a lot of transition mechanisms available, a couple of them are listed here: 464XLAT and DS‑Lite, and it's not by chance that we have put those two specific technologies there, they will be referenced in the next slides, and thanks to the availability, thanks to the fact that those mechanisms are already deployed worldwide, probably we are ready for the introduction of more services based on a NAT IPv6 transport, on new applications. Clearly, even if probably most of the work has been done, something still remains to be completed, we have to think about how to move forward in transition and here I say there is a kind of question for the community. So your feedback is welcome to identify what is missing, what could be, let's say, the solutions to the open questions that are listed at the end of the presentation.
.
Now, quickly, numbers.
.
So, you see that the ratio between IPv6‑capable users and the total inter‑population is growing over the years and is probably growing quite well. So ‑‑ the ratio is between IPv6 and the total population is more than one‑fourth. And what is specifically interesting is the compound annual growth rate; you see that it is ten times higher for the IPv6 population if compared to the total Internet population. By the way, those numbers have been processed taking into consideration the distribution reports by Dr. Geoff Huston and the APNIC team. So you see they are there.
.
That is for the user, so the first end of our connection.
.
Moving to the content, probably the ‑‑ here, the status is a bit more, let's say, difficult to be, let's say, fully understood. Why am I saying that? Because if we just look at the numbers, we have found these statistics from W3Techs. They generate 21, less than 20% of the worldwide support IPv6. If you look at the details of the analysis they took the top ten million websites worldwide and they have analysed the way there is support for IPv6 or not. But there are other analytics available.
.
For example, one from Cisco 6Labs shows that if you restrict the analysis to the top 500 websites worldwide, 40% of them support the IPv6. And second, that group represents the players that provide most of the content. So we are speaking of the hyper‑scalers: Amazon, Google, Microsoft, NetFlix, whatever.
.
If we consider that there are analyses showing that just considering Facebook and Google together, they provide more than half of the traffic ‑‑ of the mobile traffic worldwide, well probably the greatest part of the content is available on IPv6. There was a hypothetical question on the mailing list asking what happens if we shut down, overnight, IPv4? Does the IPv6‑based content survive? And the, let's say, the consensus, so the answer, it's a purely hypothetical, is that the content should survive on IPv6.
.
Then, moving on with numbers, we have just, as I say, processed again the same source, just looking at how we ‑‑ how we are in the RIPE region and specifically in the European Union. I think that the RIPE community, there are even more granular data, but just to have a, let's say, a quick look at the ‑‑ where we stand.
.
You see that, looking at the European Union, we are on par to the worldwide progression. Looking at the wider RIPE region, well, probably we are a bit behind, a bit lower, showing that maybe there is a space for their actions to push IPv6.
.
Now, there are a lot of sources where you can find even more information about the deployment of IPv6. They are listed here. In addition, we developed two independent policies that we submitted. The first one to the European operators, a good bunch of the European operators at the end of 2020, and there was also another survey developed by the Industry Network Technology Council and submitted to a group of large enterprises, mainly in North America.
.
If you look at the draft, you see ‑‑ you will find all the related information, but just to look at some findings.
.
You see, on the left, the status of IPv6 in large enterprises. You see that, let's say, the result is not so good, or at least not so exciting. Most of them have not deployed any IPv6 box. In many cases, the engineering department underwent only some basic trainings on IPv6, even if ‑‑ let's say the presence of IPv6 devices is good and, in some cases, we have, let's say, IPv6 in several places.
.
On the right‑hand side, you just find one of the answers from the European operators. Let's say that 80% of them do have plans for IPv6, and for those who are running IPv6, you see the situation, which is represented by the two pies on the right. The first one, the top right, is the deployment of IPv6 in cellular networks. The one bottom right is IPv6 in wire line, and you see that dual stack is the main technology considered, and for those who are already moving towards IPv6 only, 464XLAT is the preferred technology in mobile wire line.
.
Now, the reasons to move to IPv6, there are several motivations: The depletion of IPv4. There is also one point of attention, they are not wanting to use private IPv4 addresses. There are business motivations, so 5G, IoT are the most referenced technologies or applications requiring IPv6 because of the lack of available addresses, but there are also, say, business motivations for that; for sure, saving the cost about NAT, but here the debate is open because it depends on the management of investment on the CGN. We heard from Yannis before that this is a key point for an operator.
.
There is also some push from, let's say, the regulatory side. Entities who push for the deployment of IPv6 in specific countries, USA, China, are two examples, and there are also examples of, let me call it, moral situation, to move forward with the IPv6 specific as a requirement fora signing a 5G licence and this is an example coming from France, ARCEP.
.
Practical suggestions to move to IPv6‑only. An operator that starts this journey in general has to face two steps in this transition. The first one is the introduction of IPv6, and we have seen in that case the preferred technology is dual stack. It's quite simple. There are also a lot of documents helping an operator, or even an enterprise to deploy dual stack.
.
The second step is IPv6‑only. Sometimes it's also called IPv4 as a service. In that case, the requirement is that the network should move to IPv6 as well, but again, here the debate is also open because clearly that cannot happen in a very short time frame. So, there is more space, but, let's say, the second step is the one who, let's say, grant us that we are really moving to IPv6‑only.
.
And as I said, the IP service layer is playing a role here.
.
If you look at another analysis, which is different from ours, and that was conducted probably two or three years ago, you find the reference just to look at where you can find the original source of data. That is the situation considering the deployment in a real network.
.
So, on the left‑hand side, you have the technologies used by mobile carriers. On the right, those used by fixed carriers. Once again, you see that the results are confirmed.
.
So an operator either deploys dual stack for the first step in the transition, or considers other technologies and the preferred ones are 464XLAT for mobile and Dual Stack Lite for fixed.
.
Now, clearly there are open points, there are areas for improvement. Now, it's not my intention to go through this table that we can save time for the discussion, but just focusing on the operators' area.
.
We have seen that many comments in the mailing list where okay, for example, for fixed operators, there is a consensus that it's good to move to IPv6‑only. But they have to consider the installed base. They would love to move to some transition technologies, deploy that, but there is a, let's say, the point of granting the return of investment of ‑‑ on what has been deployed so far, and that is specifically associated to the role of CGN.
.
Another point, they have to wait for the update cycle of the CPEs which are considered another point of attention, at least for moving to IPv6‑only.
.
As far as mobile operators are concerned, there are other types of problems, probably they are a bit ahead if we compare them to fixed operators. In some cases, regulation is a positive effect to push to IPv6‑only. There is currently a debate: Where is the right point to move from dual stack to, for example, 464XLAT? Is there a matter of reaching a certain threshold of IPv6 traffic volume? And as I said, the question is open. And by the way, we would like to hear from your experience if that is the case.
.
There are other stakeholders, enterprises, Cloud and content providers. I just highlight that, for example, the community highlighted, even the CPEs have issues because many of them support IPv6 for sure, but there are exceptions, and Smart TVs, setup boxes, consoles, are examples of not complete, not widely supported for IPv6.
.
And, for sure, many of the respondents of the poll, and that was also in the mailing list, that showed that there is a role for regulation.
.
Now, moving towards the conclusion of the presentation.
.
There are perceived open issues. Just to name a few of them. Probably the most important:
.
Network operations. There are a lot of misconceptions IPv6 is complex to be deployed.
.
Performance: In general, people tend to compare IPv4 versus IPv6, and when the result of the comparison is not good, in general, the behaviour is that, okay, so IPv6 is not good for me, which is not true.
.
Security: Which is also perceived as a critical area, and many people tend to think that it's very complex.
.
So, we give the slide a title 'A Call for Action', because really we'd like to have your feedback on those points. If you have analysis about best current practices, even tools that could be referenced via work, that would be great and we would be happy to listen to you.
.
So, concluding the presentation:
.
Basically what we have found:
.
IPv6 is growing quite steadily, so the growth is good.
.
IPv6 has a value for the whole industry players.
.
In general, deployment is handled considering two stages.
.
As I said, there is a role for regulation to give a push, to give the support to the transition to IPv6.
.
Clearly, there are challenges, and we have seen the areas where challenges apply.
.
And as I said, there is an open question and we'd like to hear from you: Is there a threshold to move from IPv4 to IPv6, and specifically to IPv6‑only?
.
And I will say that I stop here, so I leave some time for questions. Thank you.
JEN LINKOVA: Thank you very much, Paolo. I think we have one question in the questions section and we have two minutes, so if you could answer that.
PAOLO VOLPATO: I see a question from Christian:
.
"Did you find any business reasons to adopt IPv6, did you find now any business reasons to adopt IPv6?"
.
I think this is one of the most interesting and critical questions. I am sorry to say we don't have a definite answer because that is also part of the discussion. In some cases, business reasons do exist, so if you look at the poll, spend time maybe reading our draft and providing comments. Some of the operators that moved to IPv6 clearly stated that it's a move for the future, we need to be open, let's say, to innovation. It's for enabling new applications even if those applications are not, let's say, available now. And there is a kind of consensus that probably 5G and IoT are two cases for something demanding IPv6. It's not completely true. Some other operators have a slightly different opinion, but that could be, let's say, many reasons.
.
There is another motivation, which is probably not clear yet, but it is the role of governments and, as I said, national authorities' actions in favour of IPv6. There are cases, two of them are USA and China, where federal agencies, government agencies, have ‑‑ began to demand not only the support of IPv6, but probably to have just IPv6‑based applications. That is the current status.
.
Jen, I see there is another question, do you want me to answer?
JEN LINKOVA: Very quickly. Yes.
PAOLO VOLPATO: Very quickly. Just a comment:
"The deduction of v6 does not immediately relieve the pressures of IPv4 address pool. Only IPv6‑only does help with IPv4 address shortage. It would be worth clarifying it on the relevant slide."
Let me say that the answer is, I fully agree. We will do that.
JEN LINKOVA: Okay. Thank you very much for your talk. Now we have the last presentation of today. You have ten minutes and you have 22 slides, which makes more than 2 slides per minute. I hope you can make it.
JENS LINK: You can hear me now, right. So, welcome to Berlin, at least Google told me on Monday that RIPE 82 is in Berlin this week.
.
So, Happy Eyeballs considered harmful. I recently ran across some problems with Happy Eyeballs and I want to share some thoughts about that.
.
So, first, who am I? I am not Jen; some people confuse us sometimes. I am a freelance consultant doing IPv6 since 2007, and I am organising a monthly meeting in Berlin ‑ well, right now on the Internet, and I come to that in a bit.
.
What is Happy Eyeballs, it's mainly implemented in web browsers. It's trying to work around broken IPv6 connections on the Internet. It's around since 2011. And some people claim that it was one of the reasons why the big players dared to enable IPv6 at all.
.
How does it work? Basically, it uses both IPv6 and IPv4. It prefers IPv6 to start with and tries to connect to the remote server and then choses a working on the better connection.
.
So, what's my problem with with Happy Eyeballs?
.
Nobody has noticed something ‑‑ how do you notice that something is broken. And what's the incentive to fix a problem if it apparently works? And what are the side effects we might see?
.
People might have heard about Happy Eyeballs, but don't really eyes what problems it may cause. So, yeah...
.
So, one example, and that's from our monthly meeting, the university here where we normally met in real life, has a big blue button for us. Users can only connect via IPv4. And unless you are looking closely you don't notice. So there is a Firefox plug‑in, IPvFoo you can install or somebody tested with RIPE Atlas code or the server has a AAAA record, routing works, but there is the firewall in place, and other people are setting ‑‑ running the server don't talk to the people running the firewalls, and yeah, that might lead to some problems.
.
Another example, a customer is using IPv6 moving to IPv6‑only internally where possible.
.
All new websites are dual stacked from the outside.
.
And they had a new site, and, while testing, it was only reachable via VPN.
.
And sometimes it would work, sometimes it didn't work.
.
And many people were guessing why.
.
One of the main reasons was, the VPN currently only supports IPv4. And so, IPv4 went through the VPN tunnel and the browser decided, yes, we use IPv4 now, and other times IPv6 was used and users who tested the site got error messages.
.
The short‑term solution was to disable IPv6 DNS lookups in Firefox, and the long term solution will be to set up a new VPN solution that supports dual stack until they can move to IPv6‑only in maybe 40 years.
.
So, you should be aware of Happy Eyeballs and the problems you might face. You should know your infrastructure. If you offer services to the Internet, monitor from the outside. That's always a good idea. Try to automate things, or at least have a checklist work flow for setting up new devices, new services, and don't use IPv4 on VPN tunnels.
.
So ‑‑ and maybe someone from the scientific community would like to do some measurements if we really need Happy Eyeballs today. So, are there that many broken IPv6 networks on it?
.
And now, I am ready for questions and discussions. And I prepared for the discussion, so I have a big bowl of popcorn.
JEN LINKOVA: Thank you. So I think we have some questions, or there is a question from Jordi, in the question section. Oh, more questions coming in. Can you see them?
JENS LINK: Yeah. So, Jordi is not a question, just a statement:
.
"Yes, other people see the same problem."
JEN LINKOVA: I will ask you to read the questions out when you are answering them so if someone is watching the video later they know what we're talking about.
JENS LINK: Okay. "I described the problem a long time ago and that's why I proposed in IETF reporting of Happy Eyeball Version 2 failures and so ISPs know that something is broken."
And he refers to the graph there, I won't read the URL.
.
And then Blake Willis says: "To add to Jordi's question, is the community aware of a tool that users can use to catalogue IPv6 problems or is the ecosystem hopelessly fragmented too?"
.
And there is something in the way ‑‑ okay. "Can you suggest a browser plugin to show Happy Eyeballs problems like IPvFoo?"
.
I am not aware of a plugin and I guess it's internal in the browser and you have to talk to the browser windows.
.
"Do you think adoption of error network error logs is causing this type of problem?"
.
It might be. "Is it enabled somewhere and analysed if it's enabled?"
.
Yeah, Gert Döring: "Another example we saw Apache ACLs were not right for IPv4, I think, and browsers were flip‑flopping IPv4/IPv6 and user reports were inconclusive."
.
Any more questions?
BENEDIKT STOCKEBRAND: There was someone on the audio.
JEN LINKOVA: I think Alexander asked his question in the Q&A section.
.
Okay, we have finished perfectly. Great job everyone. Thank you to everyone who presented, everyone who participated and commented. Please rate the talks so we know what to bring for the next session next year. So thank you very much, guys. See you soon.
.
(Lunch break)
