RootCell <--> ivshmem-demo in Qemu

[d***@gmail.com]

Hi everybody,

Im trying to set up a communication between the RootCell and the ivshmem-demo, but running into some problems. Im using the unmodified qemu-x86.c as well as the unmodified ivshmem-demo.c as configuration files.
When I enable Jailhouse and create, load and start the ivshmem-demo cell I get the following output (which looks right to me...) :
<--- BEGIN Console output --->
Initializing Jailhouse hypervisor v0.8 on CPU 3
Code location: 0xfffffffff0000060
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
Initializing processors:
CPU 3... (APIC ID 3) OK
CPU 0... (APIC ID 0) OK
CPU 2... (APIC ID 2) OK
CPU 1... (APIC ID 1) OK
DMAR unit @0xfed90000/0x1000
Reserving 24 interrupt(s) for device ff00 at index 0
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 0010 at index 24
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00d8 at index 29
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00fa at index 30
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00ff at index 31
Adding virtual PCI device 00:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 254/1480, remap 65607/131072
Activating hypervisor
Adding virtual PCI device 00:0f.0 to cell "ivshmem-demo"
Shared memory connection established: "ivshmem-demo" <--> "QEMU-VM"
Created cell "ivshmem-demo"
Page pool usage after cell creation: mem 272/1480, remap 65607/131072
Cell "ivshmem-demo" can be loaded
Started cell "ivshmem-demo"
CPU 2 received SIPI, vector 100
IVSHMEM: Found 1af4:1110 at 00:0f.0
IVSHMEM: shmem is at 0x000000003f1ff000
IVSHMEM: bar0 is at 0x000000003f200000
IVSHMEM: bar2 is at 0x000000003f201000
IVSHMEM: mapped the bars got position 1
IVSHMEM: 00:0f.0 sending IRQ
IVSHMEM: 00:0f.0 sending IRQ
...
<--- END Console output --->


I followed the instructions from https://github.com/henning-schild-work/ivshmem-guest-code/blob/jailhouse/README.jailhouse to set up the uio driver in the RootCell. It seems to work up to this point because I see two Inter-VM shared memory device when I run lspci:
<--- BEGIN Console output --->
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller (rev ff)
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:02.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
00:0e.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
00:0f.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
00:1b.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (rev 01)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
<--- END Console output --->


Also the interrupt count goes up, here is what i see when i run grep ivshmem /proc/interrupts:
<--- BEGIN Console output --->
30: 0 0 0 IR-PCI-MSI 229376-edge uio_ivshmem
31: 41 0 0 IR-PCI-MSI 245760-edge uio_ivshmem
<--- END Console output --->


But the test programs from ivshmem-guest-code just wont work as they should. Running ./tests/shmem_test.py for example just prints a bunch of printable and unprintable characters.
Running ./uio/tests/Interrupts/VM/uio_read /dev/uio0 <number> ends in waiting for interrupts forever and the pings from the program ./uio/tests/Interrupts/VM/uio_send /dev/uio0 <number> 0 0 don't affect the debug output from ivshmem-demo cell.

But when I connect two non root cells and run the inmates/demos/x86/ivshmem-demo.bin, they are sending and recieving interrupts from and to each other...

Does anybody have an idea what im doing wrong?
Maybe I am missing a step for ivshmem-guest-code which is not mentioned in README.jailhouse, but I dont know:/

FYI, thats the script which I use to launch my qemu VM:
<--- BEGIN bash script --->
qemu-system-x86_64 -machine type=q35,kernel_irqchip=split -m 1G -enable-kvm \
-smp 4 -device intel-iommu,intremap=on \
-cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx \
-drive file=ubuntu.img,id=disk,format=raw,if=none \
-cdrom mini.iso \
-device ide-hd,drive=disk \
-device intel-hda,addr=1b.0 -device hda-duplex \
-serial stdio -serial file:com2.txt \
-netdev user,id=net -device e1000e,addr=2.0,netdev=net -redir tcp:2222::22
<--- END bash script --->


Guest System OS: Ubuntu 17.10 (GNU/Linux 4.13.0-36-generic x86_64)

Guest Kernel CMD_LINE:
GRUB_CMDLINE_LINUX="memmap=66M\\\$0x3b000000 intel_iommu=off"

Qemu Version: 2.11.1

If you need any further information, let me know :)

[Jan Kiszka]

Post by d***@gmail.com
Hi everybody,
Im trying to set up a communication between the RootCell and the ivshmem-demo, but running into some problems. Im using the unmodified qemu-x86.c as well as the unmodified ivshmem-demo.c as configuration files.
<--- BEGIN Console output --->
Initializing Jailhouse hypervisor v0.8 on CPU 3
Code location: 0xfffffffff0000060
Using x2APIC
Page pool usage after early setup: mem 44/1480, remap 64/131072
CPU 3... (APIC ID 3) OK
CPU 0... (APIC ID 0) OK
CPU 2... (APIC ID 2) OK
CPU 1... (APIC ID 1) OK
Reserving 24 interrupt(s) for device ff00 at index 0
Adding PCI device 00:01.0 to cell "QEMU-VM"
Adding PCI device 00:02.0 to cell "QEMU-VM"
Reserving 5 interrupt(s) for device 0010 at index 24
Adding PCI device 00:1b.0 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00d8 at index 29
Adding PCI device 00:1f.0 to cell "QEMU-VM"
Adding PCI device 00:1f.2 to cell "QEMU-VM"
Reserving 1 interrupt(s) for device 00fa at index 30
Adding PCI device 00:1f.3 to cell "QEMU-VM"
Adding PCI device 00:1f.7 to cell "QEMU-VM"
Reserving 2 interrupt(s) for device 00ff at index 31
Adding virtual PCI device 00:0e.0 to cell "QEMU-VM"
Adding virtual PCI device 00:0f.0 to cell "QEMU-VM"
Page pool usage after late setup: mem 254/1480, remap 65607/131072
Activating hypervisor
Adding virtual PCI device 00:0f.0 to cell "ivshmem-demo"
Shared memory connection established: "ivshmem-demo" <--> "QEMU-VM"
Created cell "ivshmem-demo"
Page pool usage after cell creation: mem 272/1480, remap 65607/131072
Cell "ivshmem-demo" can be loaded
Started cell "ivshmem-demo"
CPU 2 received SIPI, vector 100
IVSHMEM: Found 1af4:1110 at 00:0f.0
IVSHMEM: shmem is at 0x000000003f1ff000
IVSHMEM: bar0 is at 0x000000003f200000
IVSHMEM: bar2 is at 0x000000003f201000
IVSHMEM: mapped the bars got position 1
IVSHMEM: 00:0f.0 sending IRQ
IVSHMEM: 00:0f.0 sending IRQ
...
<--- END Console output --->
<--- BEGIN Console output --->
00:00.0 Host bridge: Intel Corporation 82G33/G31/P35/P31 Express DRAM Controller (rev ff)
00:01.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:02.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
00:0e.0 Unassigned class [ff01]: Red Hat, Inc Inter-VM shared memory
00:0f.0 Unassigned class [ff00]: Red Hat, Inc Inter-VM shared memory
00:1b.0 Audio device: Intel Corporation 82801FB/FBM/FR/FW/FRW (ICH6 Family) High Definition Audio Controller (rev 01)
00:1f.0 ISA bridge: Intel Corporation 82801IB (ICH9) LPC Interface Controller (rev 02)
00:1f.2 SATA controller: Intel Corporation 82801IR/IO/IH (ICH9R/DO/DH) 6 port SATA Controller [AHCI mode] (rev 02)
00:1f.3 SMBus: Intel Corporation 82801I (ICH9 Family) SMBus Controller (rev 02)
<--- END Console output --->
<--- BEGIN Console output --->
30: 0 0 0 IR-PCI-MSI 229376-edge uio_ivshmem
31: 41 0 0 IR-PCI-MSI 245760-edge uio_ivshmem
<--- END Console output --->
But the test programs from ivshmem-guest-code just wont work as they should. Running ./tests/shmem_test.py for example just prints a bunch of printable and unprintable characters.
Running ./uio/tests/Interrupts/VM/uio_read /dev/uio0 <number> ends in waiting for interrupts forever and the pings from the program ./uio/tests/Interrupts/VM/uio_send /dev/uio0 <number> 0 0 don't affect the debug output from ivshmem-demo cell.

There are two ivshmem devices in the root cell config. Possibly, you are
addressing the networking device while the non-root cell is linked to
the second device. Try if there is uio1 as well.

Jan

Post by d***@gmail.com
But when I connect two non root cells and run the inmates/demos/x86/ivshmem-demo.bin, they are sending and recieving interrupts from and to each other...
Does anybody have an idea what im doing wrong?
Maybe I am missing a step for ivshmem-guest-code which is not mentioned in README.jailhouse, but I dont know:/
<--- BEGIN bash script --->
qemu-system-x86_64 -machine type=q35,kernel_irqchip=split -m 1G -enable-kvm \
-smp 4 -device intel-iommu,intremap=on \
-cpu kvm64,-kvm_pv_eoi,-kvm_steal_time,-kvm_asyncpf,-kvmclock,+vmx \
-drive file=ubuntu.img,id=disk,format=raw,if=none \
-cdrom mini.iso \
-device ide-hd,drive=disk \
-device intel-hda,addr=1b.0 -device hda-duplex \
-serial stdio -serial file:com2.txt \
-netdev user,id=net -device e1000e,addr=2.0,netdev=net -redir tcp:2222::22
<--- END bash script --->
Guest System OS: Ubuntu 17.10 (GNU/Linux 4.13.0-36-generic x86_64)
GRUB_CMDLINE_LINUX="memmap=66M\\\$0x3b000000 intel_iommu=off"
Qemu Version: 2.11.1
If you need any further information, let me know :)

[d***@gmail.com]

Yes, that was the Problem. Thanks for your help! :)

[k***@gmail.com]

Post by d***@gmail.com
But when I connect two non root cells and run the inmates/demos/x86/ivshmem-demo.bin, they are sending and recieving interrupts from and to each other...

I am also trying to do the ivshmem connectivity between two non-root cells but i am not able to get the interrupts in non-root cell and root cell, can you share your non-root cell configs it will great help for us.




Thanks