Eagerly populate guest physmem PTEs

[iximeow]

I noticed today that VM_MEMMAP_F_WIRED is present and does this, but we weren't passing it in VM_MMAP_MEMSEG. By passing this, we proactively set up guest PTEs via vmspace_populate() after unconditionally setting up the upper levels of guest physical memory with vmspace_map(). As a result, on the guest's first access to a page of memory, it no longer takes an NPF exit for bhyve to handle via VM_EXITCODE_PAGING. This comes at the cost of a bit more time spent setting up the VM's mappings when Propolis is getting going.

When guest memory comes from the reservoir, I think the extra time setting up PTEs proactively is the only cost here. When it does not come from the reservoir, I don't fully understand the interactions. Since a transient allocation is fundamentally an addition and later subtraction from the reservoir, guests backed by transient allocations should have memory just as pinned as the reservoir-ful. So .. does that mean illumos should operate as if VM_MEMMAP_F_WIRED is passed since b57f5d3 added the reservoir in the first place?

At least in our case it seems that, assuming a guest will actually use its available memory, this is a clear win. A guest NPF involves at least a page table walk to get to the PTE which must be populated, plus additional work in the exit, return, etc. Conservatively, one could assume a guest NPF must be at least four pointer chases. Simply walking the page table in entry order as we do in vmspace_populate() is approximately pages writes, where every 512 entries we go back up the tree 1-3 levels. So lazily touching all memory is something like 4x more total time (and visibly slower to the guest!) versus just doing it up front when we start the VM.

Measurements

The new scripts/lifecycle-times.d bears this out: vm_timing_eager says vm_mmap_memseg for a 16,000MiB (mind the weird number) VM takes a total of about 620ms now, where it took about 24.2ms before. Lazily initializing PTEs though means that when that guest actually uses 8GiB of memory, we spend another 1115ms in vmc_fault setting up mappings. Since that's about half of guest memory, touching the other 8GiB would be at least another second, so the numbers are close to what I'd have predicted..

Said differently, the guest workload I alluded to (a memset() benchmark) now looks pretty much identical now on a Linux host versus a Linux guest in propolis-standalone - it was more than twice as slow for the first pass over memory before.

This maybe saves ~50 milliseconds in getting the same Ubuntu guest to running systemd. On the whole I expect this adds on the order of 50ms-per-GiB to time between an instance start being requested and being at a running userland in a VM. But probably saves that much time when a guest is asked to do useful work? For the largest VMs, that's almost an extra minute..

Conclusion

This last bit is the rub, and why I've made this a draft. I'm not sure that a minute setting up PTEs is actually a win versus slowly experiencing that minute once the guest has gotten to start doing useful work. I could imagine, though, that we allow concurrent ioctls to vmspace_populate() different regions of guest memory, and populate ptes at vCPU-level concurrency. then for a large VM the extra minute would be.. half a second? less? seems fine!