KVM PV DEVICES [email protected] Agenda Kernel Virtual Machine overview Paravirtualized drivers intro & brief history VirtIO Enhanced VirtIO with KVM support 2 Copyright © 2007 Qumranet, Inc. All rights reserved. Kernel Virtual Machine overview VM is a regular Linux process. KVM handles cpu & memory virtualization Qemu virtualizes IO devices 3 Copyright © 2007 Qumranet, Inc. All rights reserved. KVM Execution Model 4 Copyright © 2007 Qumranet, Inc. All rights reserved. Evolution of x86 virtualization 5 Copyright © 2007 Qumranet, Inc. All rights reserved. General & history Fully virtualized devices performs bad 55 Mbps for RTL Lots of io-exits per packet Decided to implement a modern e1000 Advantage: Only Qemu coding no guest tools involved Irq coalescing Only 2-3 io-exits per packet Can be the base of user-space PV But then came Ingo… 6 Copyright © 2007 Qumranet, Inc. All rights reserved. PV driver architecture VM VM VM HVM VM guest HVM guest R PV I R frontend HVM guest PV N I driver frontend guest G HVM R N PV G PV backend PV driver backend driver PV backend PV driverbackend driver I R frontend PV N I driver frontend G N driver G Qemu Qemu Qemu Qemu driver Linux kernel & KVM KVM 7 Copyright © 2007 Qumranet, Inc. All rights reserved. PV backend driver VM General & history V0 - leveraging Ingo Molnar’s PV code Make loadable module Add HVM support Add NAPI Add memory barriers and improved ring Keep running after performance & stability Merge to the kernel? Alternatives Xen Polished drivers Xen specific VirtIO was just published. 8 Copyright © 2007 Qumranet, Inc. All rights reserved. VirtIO An API for virtual I/O Implements network & block driver logic Written by Rusty Russell Motivation Many hypervisors of all types Hard to tune and maintain each one Code reuse – The KVM way ;) Implementations Lguest KVM Possible (Xen, UML, Qemu, VMware?..) 9 Copyright © 2007 Qumranet, Inc. All rights reserved. VirtIO VM Guest kernel VIRTIO NET VIRTIO BlK Add_buf Get_buf Sync Detach Restart Hypervisor Specifics PV code Qemu Kick Linux kernel & KVM KVM PV backend driver scatterlist sg[] for skb/blk_req data 10 Copyright © 2007 Qumranet, Inc. All rights reserved. VM VirtIO – Hypervisor specifics The front end logic is implemented by VirtIO The backend needs Probing & Bus services Enumeration Irq Parameters (mac,..) Shared memory with remote side Hypercalls Host driver/userspace backend 11 Copyright © 2007 Qumranet, Inc. All rights reserved. VM Guest kernel VIRTIO BlK VIRTIO NET VM Hypervisor Specifics PV code Qemu Linux kernel & KVM KVM PV backend driver Enhanced VirtIO – Reuse the reuser Motivation Increase re-use Allow operation with various bus types Make new devices code smallest VM Guest kernel VIRTIO BlK VIRTIO NET VM Hypervisor Specifics PV code Components Shared memory code With per hypervisor I/O hypercalls Bus (pci, virtual bus) Host backend 12 Copyright © 2007 Qumranet, Inc. All rights reserved. Qemu Linux kernel & KVM KVM PV backend driver Enhanced VirtIO VM Blk Backend Guest kernel Net Backend VIRTIO Net VIRTIO Enhanced VIRTIO Virtual Bus Shared mem backend Hypercalls Qemu Linux kernel & KVM KVM 13 VIRTIO BlK PV backend driver Copyright © 2007 Qumranet, Inc. All rights reserved. VM Enhanced VirtIO Status: Interface was enhanced KVM support PCI like configuration space added Actually we got re-used… 162VMLOC Blk Backend Guest kernel 195 LOC Net Backend Virtual Bus 14 Copyright © 2007 Qumranet, Inc. All rights reserved. Hypercalls Shared mem backend Qemu Linux kernel & KVM KVM VIRTIO BlK VIRTIO Enhanced VIRTIO Result: Makes backend driver tiny 620 Mbps throughput for network HVM Linux guest Before optimization Userspace backend driver VIRTIO Net PV backend driver VM Enhanced VirtIO – shared memory backend VirtIO backend Implements VirtIO interface Callbacks to hypervisor and Code consists of add_buf, get_buf, restart, detach_buf Only shared memory logic needed sync Ring logic IO pending hypercall be_virtqueue_interrupt handler VM Blk Backend Guest kernel Copyright © 2007 Qumranet, Inc. All rights reserved. VIRTIO Net Virtual Bus VIRTIO BlK VIRTIO Enhanced VIRTIO Hypercalls Shared mem backend Qemu Linux kernel & KVM KVM 15 Net Backend PV backend driver VM Enhanced VirtIO – Day in a life of packet 3.start_xmit VM Blk Backend Guest kernel Net Backend VIRTIO Net 4.add_buf VIRTIO 6.sync Enhanced VIRTIO Virtual Bus VIRTIO BlK Hypercalls 11.irq 2.Kernel Tcp/ip 1.User write 12. skb_recv_done 13. get_buf… Shared mem 5.add_buf VM backend 7.sync 8.Notify hcall Qemu Linux kernel &10.Trigger KVM KVM 16 irq PV backend driver Copyright © 2007 Qumranet, Inc. All rights reserved. 9.The pkts travels to tap/backend host device Enhanced VirtIO – shared memory details Based on lguest 1-1 shared memory Data structure Page of descriptors for rx, tx. Available pointers page controlled by guest Used pointers page controlled by host SG list is currently internal to descriptors Descriptors are chained by next pointer 17 Copyright © 2007 Qumranet, Inc. All rights reserved. Enhanced VirtIO – network be driver Implements kvm_virtnet_probe for pci bus Creates tx,rx be_new_virtqueue 195 LOC VM Blk Backend Probes virtnet Guest kernel Net Backend Request_irq Register hypercall - shared memory pfn Device key for enumeration taken from bus 18 Copyright © 2007 Qumranet, Inc. All rights reserved. Hypercalls Shared mem backend Qemu Linux kernel & KVM KVM VIRTIO BlK VIRTIO Enhanced VIRTIO Virtual Bus Irq# taken from bus VIRTIO Net PV backend driver VM Further work Basic Add readv/writev handlers to Qemu Complete the user-space block device Complete migration support Advanced (also simple) Optimize and stabilize Add host back end drivers Add virtual bus HVM improvements Test with PV kernel VM Blk Backend Guest kernel Net Backend VIRTIO Net VIRTIO Enhanced VIRTIO Virtual Bus Hypercalls Shared mem backend Qemu Linux kernel & KVM KVM 19 Copyright © 2007 Qumranet, Inc. All rights reserved. VIRTIO BlK PV backend driver VM Q&A Thank you ;) 20 Copyright © 2007 Qumranet, Inc. All rights reserved.