CarrierEthernet.Raj JainWashington University in Saint LouisSaint Louis, MO [email protected] slides and audio/video recordings of this class lecture are at: jain/cse570-13/Washington University in St. Louis jain/cse570-13/7-1 2013 Raj Jain

OverviewTechnologies for Data Center Interconnection:PDH, SDH, MPLS, PWE32. Metro Ethernet Services: E-Line, E-LAN, E-Tree, E-Access3. Administration and Management Ethernet Services4. Extensions to Ethernet for carriers:PB (Q-in-Q), PBB (MAC-in-MAC), PBB-TENote: Although these technologies were originally developed forcarriers, they are now used inside multi-tenant data centers(clouds)1.Washington University in St. Louis jain/cse570-13/7-2 2013 Raj Jain

Options to Connect Two Data Centers?Danforth Campus1. CampusDedicated Optical fiber (leased from the phone company)Ethernet over Optical Transport Network (all-Optical Switches)Ethernet over Wavelength Division Multiplexing (DWDM)Ethernet over Synchronous Digital Hierarchy (SDH)Ethernet over Plesiochronous Hierarchy (PDH)Ethernet over Pseudo-wire over MPLSEthernet over Micro-waveSingle Pair High-Speed Digital Subscriber Line (SHDSL)Ethernet with enhancementsWashington University in St. Louis jain/cse570-13/7-3 2013 Raj Jain

Plesiochronous Digital Hierarchy (PDH) Plesios Synchronous Near synchronousPhone Line 64 kbps 1 User channelNorth America T1 1.544 Mbps 24 User channels T2 6.312 Mbps 96 Channels T3 44.736 Mbps 480 ChannelsEurope: E1 2.048 Mbps 32 Channels E2 8.448 Mbps 128 Channels E3 139.264 Mbps 2048 ChannelsWashington University in St. Louis jain/cse570-13/7-4T1 2013 Raj Jain

SONET/SDHEEthernet ESSSS SDHSONET Synchronous optical networkStandard for digital optical transmissionStandardized by ANSI and then by ITU Synchronous Digital Hierarchy (SDH)Protection: Allows redundant Line or pathsFast Restoration: 50ms using ringsSophisticated managementIdeal for Voice: No queues. Guaranteed delayFixed Payload Rates: OC1 51.84 Mbps, OC3 155M,OC12 622M, OC48 2.4G, OC192 9.5GRates do not match data rates of 10M, 100M, 1G, 10GStatic rates not suitable for bursty trafficOne Payload per Stream jain/cse570-13/ High CostWashington University in St. Louis7-5 2013 Raj Jain

Multiprotocol Label Switching (MPLS)PBX1 PBX3523Allows virtual circuits in IP Networks (May 1996)Each packet has a virtual circuit number called ‘label’Label determines the packet’s queuing and forwardingCircuits are called Label Switched Paths (LSPs)LSP’s have to be set up before useAllows traffic engineeringWashington University in St. Louis jain/cse570-13/7-6 2013 Raj Jain

Label Switching ExampleEthernet Header IP Header PayloadEthernet Header64IP Header Payload3 64 ALabelR15 3 5 R3B 5 5Washington University in St. LouisR2 2 3 C23 jain/cse570-13/7-7 2013 Raj Jain

IP over MPLS over EthernetCEPECarrier NetworkPECEPE Provider EdgeCE Customer EdgeDest. Adr Src. Type MPLSCustomerCustomerL3 CRCNext Hop Adr /Len Tag Dest. MAC Adr Src. MAC Adr PDU48b48b 16b32b48b48b32bLabel ExperimentalStack Position TimeCoS/Drop-Preced. 1 Bottom to Live20b3b1b8b Allows 220 Label switched paths (LSP)Each path can have reserved capacity MPLS became a very popularWashington University in St. Louis jain/cse570-13/7-8 2013 Raj Jain

Martini Draft 1995-1999: IP over ATM, IP over EthernetL3L2 L2L3 EthernetIPATMPPP2000 : ATM over IP, Ethernet over IPSONET over IPEthernetATMIPPPPIdea proposed by Luca Martini ofLevel 3 Communications and thenCiscoWashington University in St. Louis jain/cse570-13/7-9 2013 Raj Jain

Pseudo Wire: L2 Circuits over IPCE1ATMNetworkPE1PE2Pseudo WireEmulated ServiceAFrame RelayNetworkEthernetCE2ATMNetworkBIPNetworkFrame RelayNetworkEthernetTunnel Hdr Demux Field [Control] ATM ATM ATM ATMPayload Type How to de-assemble payloadMPLS/GRE/L2TP- How to get to egress jain/cse570-13/Washington University in St. Louis 2013 Raj Jain7-10

Ethernet over PWE3 over MPLSMPLS LabelPW Label[Control]Ethernet Frame w/o FCSPID Flags FRG Length Sequence #4b 4b2b6b16b PW1MPLSLSPPW2Pseudo-Wire Emulation Edge-to-Edge (PWE3)Multiple pseudo-wires per MPLS LSPPW3Payload ID (PID): 5 Untagged Ethernet, 4 VLAN tagged, 4 VLAN tag put by carrier and customers may or may not berelevent for forwarding. Determined administratively by PE.Flags: Payload specific. FRG: Used for fragmentationPause frames are obeyed locally. Not transported.May put 802.1p priority in exp field of MPLS labelWashington University in St. Louis jain/cse570-13/7-11 2013 Raj Jain

Virtual Private LAN Service (VPLS) Allows multi-point Ethernet servicesover MPLS networks using pseudo-wiresNon-Hierarchical VPLS: Edge routers do complete routingtable, label distribution, multicast/broadcast replication Suitable for small deploymentsHierarchical VPLS (H-VPLS): Edge routers communicate withonly other edge devices and do not have large routing tables Suitable for large APECEBCE: Customer EdgePE: Provider EdgeRef: H. Saboowala, M. Abid, S. Modali, "Designing Networks and Services for the Cloud: Delivering business-grade cloudapplications and services," Cisco Press 2013, ISBN:1587142945 jain/cse570-13/Washington University in St. Louis 2013 Raj Jain7-12

Differentiated Services A way for IP routers to provide QoSExpedited Forwarding (EF): Also known as Premium Service Virtual leased line Guaranteed minimum service rate Policed: Arrival rate Minimum Service Rate Not affected by other forwarding classesAssured Forwarding (AF): Four Classes: No particular ordering Three drop preference per class:Low, Medium, HighBest Effort ServiceDifferentiated Service Code Point (6 bits) encode the service,E.g., 101110 EFWashington University in St. Louis jain/cse570-13/7-13 2013 Raj Jain

Carriers vs. EnterpriseWe need to exchangetopology foroptimal routing.Washington University in St. LouisSorry, We can’t tell youanything about ourinternal network. jain/cse570-13/7-14 2013 Raj Jain

Issue: UNI vs Peer-to-Peer Signaling Two Business Models: Carrier: Overlay or cloud Network is a black-box User-to-network interface (UNI)to create/destroy light paths (in OIF) Enterprise: Peer-to-Peer Complete exchange of informationUNIWashington University in St. LouisUNI jain/cse570-13/7-15 2013 Raj Jain

UNI vs. ENNI User to Network Interface (UNI): Separates responsibilities between the user and the provider.(Troubleshooting, failures etc). Like the wired phone box outside your home. Only one customer’s traffic.External Network to Network Interface (ENNI): Separates responsibilities between two providers. Many customer’s traffic passes through an ENNI Tier 2 operators sell services to Tier 3 service providers.CustomerUNIProvider 1Tier 3ENNI Provider 2Tier 2Ref: Fujitsu, “Carrier Ethernet Essentials,” jain/cse570-13/Washington University in St. Louis 2013 Raj Jain7-16

Operator Virtual Connection (OVC) Between UNI and ENNI or between two ENNIs.For wholesale service providersTwo types: Point-to-Point and Multipoint-to-MultipointUntagged or single tagged frames at NNI. Q-in-Q at ENNIUNIs may be 10 to 100 Mbps. ENNIs at 1 to 10 Gbps.UNIUNIENNIUNIENNIWashington University in St. LouisENNIENNIUNI jain/cse570-13/7-17 2013 Raj Jain

Metro Access Ethernet Private Line Access Ethernet Private Line (Access-EPL): Port-based service for Internet accessLike the service at your home. Ends at your access provider, where many other AccessEPLs may end Access provider has only one interfaceShared by many Access-EPLs Different from p2p EPL.Customer 1ENNICustomer nE-AccessWashington University in St. Louis jain/cse570-13/7-18 2013 Raj Jain

End-to-End Metro Ethernet Connection An EC may go through multiple service providers Multiple OVCs can be concatenated to create an ECENNIWholesaleAccess ProviderOVC1UNISubscriberWashington University in St. LouisENNIWholesaleTransport ProviderOVC2RetailService ProviderOVC3EC jain/cse570-13/7-19UNISubscriber 2013 Raj Jain

Ethernet Virtual Connections (EVCs) Port-based ECs: Forwarding not based on VLANs.Frames delivered to remote UNI/ENNI for P2P orBased on destination address for P2MPVLAN-based ECs: Forwarding based on VLAN tag. Multiple Virtual UNIs Ethernet Virtual Connection (EVC)More cost-effective for Enterprise customersEVCTypes of EVCs:UNI EVC1. Ethernet Virtual Private Line (EVPL)EVC2. Ethernet Virtual Private Tree (EVP-Tree)3. Ethernet Virtual Private LAN (EVPLAN)4. Access Ethernet Virtual Private Line (Access EVPL)Note: Service providers always share an ENNI for multipleconnections OVCs are always virtual No OCsWashington University in St. Louis jain/cse570-13/7-20 2013 Raj Jain

Metro Ethernet Service Attributes Bandwidth Profiles: Limits on data dates Ingress Profile: Incoming data rate Egress Profile: Outgoing data ratePer UNI, Per EVC or OVC, orDataPer EVC/OVC per Class of Service (CoS)CoS is indicated by the 3-bits in the priority field orCBS4-bit Differentiated Services Code Point (DSCP)Rate specified by 5 parameters1. Committed Information Rate (CIR)CIREBS2. Committed Burst Size (CBS)Dropped3. Excess Information rate (EIR)Mark EIR4. Excess Burst Size (EBS)DEI5. Color Mode (CM): Customer does/does notForwardedmark drop eligibility indicator (DEI)Washington University in St. Louis jain/cse570-13/7-21 2013 Raj Jain

Metro Ethernet derMEGUNI-CSubscriberWholesaleServiceProviderUNI-N ENNI ENNI UNI-N Operation, Administration and Maintenance (OAM)Defined in IEEE 802.1ag, IEEE 802.3ah, and ITU Y.1731Maintenance end points (MEPs)Maintenance Intermediate Points (MIPs)Maintenance Entity Group (MEG): Level of AdministrationUNI-C SubscriberMEPMIPService Provider MEGSubscriber MEGWashington University in St. Louis jain/cse570-13/7-22 2013 Raj Jain

Metro Ethernet OAM eProviderENNISubscriberUNI-N Performance Monitoring: Measure throughput and latencyConnectivity Fault Management: Monitor downtime Service Fault Management Link Fault ManagementUNI-C SubscriberIEEE 802.3ah Link OAMLinkOAMITU-T Y.1731 End-to-End Performance MonitoringIEEE 802.3ag End-to-End Connectivity Fault ManagementWashington University in St. Louis jain/cse570-13/7-23ServiceOAM 2013 Raj Jain

Metro Ethernet OAM iceProviderENNISubscriberUNI-N Continuity Check Message (CCM) in both directions(Similar to IP Ping)Link Trace Message (LTM): Locates fault. Link TraceResponse (LTR) is returned by each end point and intermediatepoint (similar to IP trace route)UNI-C SubscriberService Provider MEGLTMLTRsWashington University in St. Louis jain/cse570-13/7-24 2013 Raj Jain

Metro Ethernet Use Cases2.InternetSubscriber 1Building BUNIP2P EVCRetailServiceProviderUNIUNISubscriber 1Building AAccess EVCENNIHead office to Satellite offices and/or InternetUNI1.Customers to Cloud Service ProviderRetailServiceProviderWashington University in St. er criber 2UNIOVCAENNIUNISubscriber 1UNIEVC1 2013 Raj Jain

Ethernet Provider Bridge (PB)Customer AVLANs 1-100CEPECustomer BVLANs 1-100CEPE Service ProviderS-VLAN 1S-VLAN 2PECECustomer AVLANs 1-100PECECustomer BVLANs 1-100IEEE 802.1ad-2005 incorporated in IEEE 802.1Q-2011Problem: Multiple customers may have the same VLAN ID.How to keep them separate?Solution: Q-in-Q. Provider inserts a service VLAN tagAllows 4K customers to be serviced. Total 16M VLANsC-DA C-SA Type S-VID Type C-VID Type Payload88A88100Ref: D. Bonafede, “Metro Ethernet Network,” 0Bonafede.pdfRef: P. Thaler, et al., “IEEE 802.1Q,” IETF tutorial, March 10 EEE-8021-Thaler.pdfWashington University in St. Louis jain/cse570-13/7-26 2013 Raj Jain

Provider Bridge (Cont) 8 Traffic Classes using Differentiated Services Code Points(DSCP) for Assured n University in St. LouisCFI S-VLAN ID Type/Length1b12b16bClassStrict PriorityAF1AF2AF3AF4AF5AF6Best Effort jain/cse570-13/7-27 2013 Raj Jain

Provider Backbone Network (PBB)SubscriberProviderSubscriberSubscriber erProblem: Number of MAC addresses passing throughbackbone bridges is too large for all core bridge to rememberBroadcast and flooded (unknown address) frames giveunwanted traffic and security issuesSolution: IEEE 802.1ah-2008 now in 802.1Q-2011Add new source/destination MAC addresses pointing to ingressbackbone bridge and egress backbone bridge Core bridges only knowedge bridge addresses jain/cse570-13/Washington University in St. Louis 2013 Raj Jain7-28

MAC-in-MAC Frame Format Provider backbone edge bridges (PBEB) forwardto other PBEB’s and learn customer MACaddresses PB core bridges do not learn customer MACsB-DA Destination backbone bridge address EdgeDetermined by Customer Destination AddressEdgeBackbon