From the Cloud to SoNIC: Precise Realtime Software

From the Cloud to SoNIC: Precise Realtime Software

From the Cloud to SoNIC: Precise Realtime Software Access and Control of Wired Networks Prof. Hakim Weatherspoon Joint with Ki Suh Lee and Han Wang Cornell University Stanford University April 17, 2014 The Rise of Cloud Computing The promise of the Cloud A computer utility; a commodity Catalyst for technology economy Revolutionizing for health care, financial systems, scientific research, and society SEATTLE The Rise of Cloud Computing The promise of the Cloud ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. NIST Cloud Definition SEATTLE The Rise of Cloud Computing The promise of the Cloud ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. NIST Cloud Definition SEATTLE The Rise of Cloud Computing The promise of the Cloud

ubiquitous, convenient, on-demand network access to a shared pool of configurable computing resources (e.g., networks, servers, storage, applications, and services) that can be rapidly provisioned and released with minimal management effort or service provider interaction. NIST Cloud Definition Requires fundamentals in distributed systems Networking Computation Storage The Rise of Cloud Computing The promise of the Cloud Switch VM App VM 33KB Guest OS VM VM Xen-Blanket VMM App Guest OS Xen-Blanket Networking Computation Storage 02/03/2020

VMM SoNIC 6 My Contributions Cloud Networking SoNIC in NSDI 2013 and 2014 Wireless DC in ANCS 2012 (best paper) and NetSlice in ANCS 2012 Bifocals in IMC 2010 and DSN 2010 Maelstrom in ToN 2011 and NSDI 2008 Chaired Tudor Marians PhD 2010 (now at Google) Cloud Computation & Vendor Lock-in Plug into the Supercloud in IEEE Internet Computing-2013 Supercloud/Xen-Blanket in EuroSys-2012 and HotCloud-2011 Overdriver in VEE-2011 Chaired Dan Williams PhD 2012 (now at IBM) Cloud Storage Gecko in FAST 2013 / HotStorage 2012 RACS in SOCC-2010 SMFS in FAST 2009

Antiquity in EuroSys 2007 / NSDI 2006 The Rise of Cloud Computing The promise of the Cloud Xen-Blanket Switch VMM VM VM VM Networking Computation Storage 02/03/2020 VM Xen-Blanket 33KB VMM SoNIC 8 Cloud Networking: Challenges Challenges remain: Performance Packets are still lost Why is it so hard to move data between clouds over the wide-area? [NSDI 2014, 2013, 2008, FAST 2009, IMC 2010, DSN 2010] 02/03/2020 SoNIC 9

Cloud Networking: Challenges Uncover the ground truth Network changes inter-packet gap Traffic sent: Packet Interpacket gap Traffic received: Bursty traffic induced by packet chaining Inter-packet gaps higher layers (Software) Access to are theinvisible physicaltolayer is required 02/03/2020 SoNIC 10 Cloud Networking: Opportunities Why access the physical layer from software? Application Transport Issue: Programmers treat layers 1 and 2 as black box Network Data Link Physical 64/66b PCS

PMA Opportunities Network Measurements Network Monitoring/Profiling Network Steganography PMD Can improve security, availability, and performance of the Cloud Networks Cloud Networking: Opportunities Understanding Cloud Networks via Software-defined Network InterfaCe (SoNIC) Application Transport Network Data Link Physical Improve understanding of network Improves security, availability, and performance of the network SoNIC: Software-defined NIC Access the PHY In real-time In software 64/66b PCS PMA PMD Separates Netslice: Software Router [ANCS 2012] Enables Software-defined Networks Big Data-in-network solutions via Deep Packet Inspection (DPI) at 40Gbps

what is sent (software) how it is sent (hardware) 12 Outline Motivation SoNIC Network Research Applications Conclusion / Research Agenda 02/03/2020 SoNIC 13 10GbE Network Stack Application Data Transport Network Data Link Preamble Physical 64/66b PCS Encode Decode Scrambler

Descrambler Gearbox Blocksync PMA Eth Hdr 64 bit /S/ /D/ L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data Idle characters (/I/) 2 bit syncheader /D/ /D/

/D/ CRC Gap 10.3125 Gigabits /T/ /E/ /E/ 16 bit 011010010110100101101001011010010110100101101001011010010110100101101 PMD 02/03/2020 SoNIC 14 10GbE Network Stack Application Data Transport Network SW Data Link Physical 64/66b PCS Encode Decode

Scrambler Descrambler Gearbox Blocksync PMA PMD 02/03/2020 L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data Packet i Preamble Eth Hdr Packet i+1 CRC Gap HW /S/

/D/ /D/ /D/ Packet i /D/ /T/ /E/ Packet i+1 011010010110100101101001011010010110100101101001011010010110100101101 Commodity NIC SoNIC 15 10GbE Network Stack Application Data Application L3 Hdr Data Transport L2 Hdr L3 Hdr

Data Network L2 Hdr L3 Hdr Data DataCRC Link SW Transport Network Data Link Preamble Eth Hdr Physical 64/66b PCS Encode IPG /S/ Packet i /D/ Decode SW Scrambler Descrambler Gearbox

Blocksync HW /D/ Packet/D/ i+1 IPD Gap Physical 64/66b PCS Encode /T/ Decode /E/ /D/ Scrambler Descrambler Gearbox Blocksync HW PMA PMD 02/03/2020 011010010110100101101001011010010110100101101001011010010110100101101 PMA SoNIC NetFPGA SoNIC PMD 16 SoNIC Design

Application Data Transport Network Data Link Preamble Eth Hdr L3 Hdr Data L2 Hdr L3 Hdr Data L2 Hdr L3 Hdr Data CRC Gap Physical 64/66b PCS Encode /S/ Decode

/D/ /D/ /D/ /D/ /T/ /E/ SW Scrambler Descrambler Gearbox Blocksync HW PMA PMD 02/03/2020 011010010110100101101001011010010110100101101001011010010110100101101 SoNIC SoNIC 17 SoNIC Design and Architecture Application Data L3 Hdr APP Data

Userspace L3 Hdr APP Data Kernel L2 Hdr TX MAC L3 Hdr Data Transport Network L2 Hdr Data Link Preamble Eth Hdr RX MACCRC Gap Physical 64/66b PCS Encode /S/ Decode /D/ /D/

SW Scrambler Descrambler Gearbox Blocksync HW PMA PMD 02/03/2020 /D/ /D/ /T/ TX PCS RX PCS Gearbox Blocksync /E/ Hardware 011010010110100101101001011010010110100101101001011010010110100101101 Transceiver Transceiver SFP+ SoNIC

SoNIC 18 SoNIC Design: API Hardware control: ioctl syscall I/O : character device interface Sample C code for packet generation and capture 1: #include "sonic.h" 2: 3: struct sonic_pkt_gen_info info = { 4: .mode = 0, 5: .pkt_num = 1000000000UL, 6: .pkt_len = 1518, 7: .mac_src = "00:11:22:33:44:55", 8: .mac_dst = "aa:bb:cc:dd:ee:ff", 9: .ip_src = "192.168.0.1", 10: .ip_dst = "192.168.0.2", 11: .port_src = 5000, 12: .port_dst = 5000, 13: .idle = 12, 14: }; 15: 16: /* OPEN DEVICE*/ 17: fd1 = open(SONIC_CONTROL_PATH, O_RDWR); 18: fd2 = open(SONIC_PORT1_PATH, O_RDONLY); 02/03/2020 19: /* CONFIG SONIC CARD FOR PACKET GEN*/ 20: ioctl(fd1, SONIC_IOC_RESET) 21: ioctl(fd1, SONIC_IOC_SET_MODE, PKT_GEN_CAP) 22: ioctl(fd1, SONIC_IOC_PORT0_INFO_SET, &info) 23 24: /* START EXPERIMENT*/ 25: ioctl(fd1, SONIC_IOC_START) 26: // wait till experiment finishes 27: ioctl(fd1, SONIC_IOC_STOP) 28: 29: /* CAPTURE PACKET */

30: while ((ret = read(fd2, buf, 65536)) > 0) { 31: // process data 32: } 33: 34: close(fd1); 35: close(fd2); SoNIC 19 Outline Motivation SoNIC Network Research Applications Measurement / traffic analysis Profiling / fingerprinting Covert channels Conclusion / Research Agenda 02/03/2020 SoNIC 20 Measurement / Traffic Analysis using SoNIC Uncover the ground truth Network changes inter-packet gap Traffic sent: Packet Interpacket gap Traffic received: Bursty traffic induced by packet chaining Inter-packet gaps are invisible to higher layers, but not SoNIC

02/03/2020 SoNIC 21 Measurement / Traffic Analysis using SoNIC Precise end-to-end instrumentation platform Measurement at large scale Towards an open measurement platform 02/03/2020 SoNIC 22 Profiling / Fingerprinting using SoNIC Profiling One Hop Through a switch 02/03/2020 SoNIC 23 Profiling / Fingerprinting using SoNIC Cisco Catalyst 6500 switch 1Gbps data (1518B) Socket 1 APP0 0.1 Frequency (normalized) Socket 0

1 0.01 0.001 0.0001 APP1 1e-05 TX MAC0 RX MAC0 TX MAC1 RX MAC1 TX PCS0 RX PCS0 TX PCS1 RX PCS1 TX SFP0 RX SFP0 TX SFP1 RX SFP1 02/03/2020 1e-06 109340

SoNIC 111340 113340 Interpacket gap (bits) 115340 117340 24 Profiling / Fingerprinting using SoNIC Router/ Switch Signatures 1 1 0.1 0.1 0.1 0.01 0.01 0.01 0.001 0.0001 1e-05 1e-06

Frequency (normalized) 1 Frequency (normalized) Frequency (normalized) Different Routers and switches have different response function. Improve simulation model of switches and routers. Detect switch and router model in real network. 0.001 0.0001 1e-05 0 5000 10000 15000 20000 Interpacket gap (bits) Cisco 4948 1e-06 0.001 0.0001 1e-05

0 5000 10000 15000 Interpacket gap (bits) Cisco 6509 20000 1e-06 0 5000 10000 15000 Interpacket gap (bits) IBM BNT G8264R 1500 byte packets @ 6Gbps 02/03/2020 SoNIC 25 20000 Profiling / Fingerprinting using SoNIC Router/ Switch Signatures

Different Routers and switches have different response function. Improve simulation model of switches and routers. Detect switch and router model in real network. NetFPGA 10G 1500 byte packets @ 6Gbps 02/03/2020 SoNIC 26 Profiling / Fingerprinting using SoNIC End-to-End Profile of GENI Network Modeling Network Elements Testbed for Network System Theory and Queue Theory Towards a Predictable Network What is the aggregate effect? 1 0.1 1 0.01 1 0.1 Frequency (normalized) 0.001 0.0001 1e-05

1e-06 Frequency (normalized) 0.1 Frequency (normalized) 0.01 0.001 0.0001 0.01 1e-05 0.001 1e-06 0.0001 0 5000 10000 Interpacket gap (bits) 15000 20000 1e-05 0 5000

10000 Interpacket gap (bits) 15000 20000 1e-06 0 5000 10000 Interpacket gap (bits) Stanford U. Wash Berkeley 15000 20000 Cornell Princeton UPenn 1 0.1 Frequency (normalized) 0.01

0.001 0.0001 1e-05 1e-06 02/03/2020 0 5000 10000 Interpacket gap (bits) 15000 SoNIC 20000 27 Profiling / Fingerprinting using SoNIC Challenges: Rogue routers 02/03/2020 SoNIC 28 Covert Channels in SoNIC Create / Detect / Prevent Covert Channels in Layers 1 and 2 02/03/2020

SoNIC 29 Covert Channels in SoNIC Application Hide transmission of data Transport Network Data Link Physical Storage Channel Writing/reading of a storage location 64/66b PCS PMA PMD Timing Channel Modulation of system resources 02/03/2020 SoNIC 30 Covert Channels in SoNIC Application Transport Network Data Link Existing Covert Channels TCP/IP headers, HTTP requests Packet Rate / Timing

Physical 64/66b PCS PMA PMD Increasing number of detection techniques Covert Channels at the Physical layer 02/03/2020 SoNIC 31 Covert Channels in SoNIC Syn c Application Data Block /E/ Network /S/ Data Link 64/66b PCS PMA PMD /S/ Start of Frame block /T/

End of Frame block /E/ Idle block /D/ Data block 8 D0 16 24 D1 D2 C0 C0 D1 C1 C1 D2 C1 Block Payload 32 D3 40 D4 48 D5

56 D6 D7 D4 D4 D4 C6 D6 D6 C6 C6 C6 C6 C6 C6 D5 D5 C7 D7 D7 C7 C7 C7 C7 C7 C7 C7 D6 /T/ /E/ Block Type

Transport Physical 01 0 65 /T/ 10 10 10 10 10 10 10 10 10 10 10 0x1e 0x33 0x78 0x87 0x99 0xaa 0xb4 0xcc 0xd2 0xe1 0xff D0 D0 D0 D0 D0

D0 D0 D1 D1 D1 D1 D1 D1 C2 C2 C3 C3 D3 C2 C2 D2 D2 D2 D2 D2 /D/ SoNIC /D/ /D/ C5 D5 D5 D4 C3 C3

C3 Ethernet Frame /S/ C4 C4 C4 C4 C4 D3 D3 D3 D3 C5 C5 C5 C5 C5 Gap /D/ 32 Covert Timing Channel in SoNIC Embedding signals into interpacket gaps. Large gap: 1 Small gap: 0 Packet i Packet i+1 Packet i Packet i+1

Covert timing channel by modulating IPGs at 100ns Overt channel at 1 Gbps, Covert channel at 80 kbps Over 9-hop Internet path with cross traffic (NLR) less than 10% BER (can mask BER w/ FEC) Undetectable to software endhost 02/03/2020 SoNIC 33 Covert Timing Channel in SoNIC Modulating IPGs at 100ns scale (=128 /I/s), over 4 hops 1 SoNIC Kernel 0.8 3562 /I/s 3562 - 128 /I/s 3562 + 128 /I/s CDF BER = 0.37% 0.6 0.4 0.2 1 0 0 500

1500 2500 1: 3562 + 128 /I/s 0: 3562 128 /I/s 02/03/2020 3500 Interpacket delays (ns) 4500 1: 3562 + a /I/s 0: 3562 a /I/s SoNIC 34 Covert Timing Channel in SoNIC Prevent Covert Timing Channels? 3562 /I/s 1 CDF 0.8 0.6 0.4 0.2 0 500 02/03/2020 1500

2500 3500 Interpacket delays (ns) SoNIC 4500 35 Covert Channels in SoNIC Challenges: Rogue end-hosts 02/03/2020 SoNIC 36 Outline Motivation SoNIC Applications Measurement / traffic analysis Profiling / fingerprinting Covert channels Discussion and Conclusion 02/03/2020 SoNIC 37 Overview of Collaborations and Resources Mini-Cloud Testbed DURIP Funds for 16 SoNIC boards and

Funds a small cloud: 38 nodes and 608 cores Funded by AFOSR NSF Future Internet Architecture Collaboration with Cisco and other Universities such as Washington, Penn, Purdue, Berkeley, MIT, Stanford, CMU, Princeton, UIUC, and Texas DARPA CSSP Funds research in three phases, we are currently in Phase 2 Requires Collaboration with non-DARPA DoD agency Collaboration with AFRL Collaboration with NGA Exo-GENI Cornell PI into national research network Layer 2 access nationally Research in Software-Defined Networks (SDN) like OpenFlow NSF CAREER and Alfred P. Sloan Fellowship Funds related basic research 38 SoNIC Contributions Network Research Unprecedented access to the PHY with commodity hardware A platform for cross-network-layer research Can improve network research applications Engineering Precise control of interpacket gaps (delays) Design and implementation of the PHY in software Novel scalable hardware design Optimizations / Parallelism

Status Measurements in large scale: DCN, GENI, 40 GbE 02/03/2020 SoNIC 39 Concluding Remarks SoNIC responds to network at the center of the cloud High precision network measurement Profiles and characterizes switches and routers Covert channel detection and prevention Understand and create more available and secure networks Status: SoNIC platform is available DURIP grant has seeded and paid for a number of boards SDNM: Software Defined Network Measurement SoNIC enabled SDN/Openflow networks (e.g. GENI) Collaboration: Deployment in experimental networks 40 Questions Contact: [email protected] Website: http://fireless.cs.cornell.edu, http://sonic.cs.cornell.edu, and http://www.cs.cornell.edu/~hweather 02/03/2020

SoNIC 41 My Contributions and Paper Trail Cloud Networking SoNIC in NSDI 2013 and 2014 Wireless DC in ANCS 2012 (best paper) and NetSlice in ANCS 2012 Bifocals in IMC 2010 and DSN 2010 Maelstrom in ToN 2011 and NSDI 2008 Chaired Tudor Marians PhD 2010 (now at Google) Cloud Computation & Vendor Lock-in Plug into the Supercloud in IEEE Internet Computing-2013 Supercloud/Xen-Blanket in EuroSys-2012 and HotCloud-2011 Overdriver in VEE-2011 Chaired Dan Williams PhD 2012 (now at IBM) Cloud Storage Gecko in FAST 2013 / HotStorage 2012 RACS in SOCC-2010 SMFS in FAST 2009 Antiquity in EuroSys 2007 / NSDI 2006

Recently Viewed Presentations

  • The reach of imperialism - World History

    The reach of imperialism - World History

    Cecil Rhodes . In the 1880's British policy in South Africa was influenced greatly by Cecil Rhodes. Cecil Rhodes founded a mining company that made him a fortune off of diamonds and gold. Settlers invaded the Transvaal territory when diamonds...
  • Econ 171 Economic Development

    Econ 171 Economic Development

    Classwork 5%Homework 25%Midterm 30%Final 40%. Midterm July 14th in class. Final Aug 11th in class. There will be 31 meetings (29 lectures, 2 exams) Grading & Exams. Econ N171 Economic Development Atanu Dey. 20-06-2011
  • CCSS ELA Writing - Digital Chalkboard

    CCSS ELA Writing - Digital Chalkboard

    students are deeply involved in generating ideas and gathering credible information from print and digital texts, then . they must learn to use writing process strategies and technological tools to produce, refine, and present written products that address a range...
  • Convention 2017 SECRETARY/TREASURER WORKSHOP The website has so

    Convention 2017 SECRETARY/TREASURER WORKSHOP The website has so

    As official record keeper of the PTA, Secretary is one of the most important positions on the board ... Summary of Treasurer's report (beginning balance, receipts, disbursements, and ending balance) ... All units must file 990, 990-EZ, or 990-N (e-postcard)...
  • Coarse Grain Reconfigurable Architectures

    Coarse Grain Reconfigurable Architectures

    massively saving energy. Reconfigurable Computing (RC): the intensive Impact. SGI Altix 4700 with RC 100 RASC compared to Beowulf cluster. Tarek El-Ghazawi. ... Die Relativitätstheorie befasst sich mit der Struktur von Raum und Zeit.
  • ACARA's EAL/D Teacher resource

    ACARA's EAL/D Teacher resource

    AISWA. EAL/D K-12 Sophia Sabatier. AISWA. EAL/D K-12 This workshop aims to provide background on the creation of ACARA's English as an Additional Language or Dialect Teacher Resource and to explore both the EALD Language Learning Progression and the EALD...
  • DEDUCTIVE vs. INDUCTIVE REASONING

    DEDUCTIVE vs. INDUCTIVE REASONING

    What is the point of my essay? I've got a topic, but what now? Thinking about how you want to approach your topic, in other words what type of claim you want to make, is one way to focus your...
  • LISA WILLIAMS PRESIDENT, EASTERN CAMPUS (INTERIM) ANDREW PEGMAN

    LISA WILLIAMS PRESIDENT, EASTERN CAMPUS (INTERIM) ANDREW PEGMAN

    newly tenured faculty. diana kovacic. assistant professor, nursing. tomasz kowalczyk. associate professor, biology. maureen leanard. assistant professor, hospitality ...