summaryrefslogtreecommitdiff
path: root/sys/arch/amd64/include/pmap.h
AgeCommit message (Collapse)Author
2022-08-20x86: Move definition of struct pmap to pmap_private.h.riastradh
This makes pmap_resident_count and pmap_wired_count out-of-line functions instead of inline. No functional change intended otherwise.
2022-08-20x86: Split most of pmap.h into pmap_private.h or vmparam.h.riastradh
This way pmap.h only contains the MD definition of the MI pmap(9) API, which loads of things in the kernel rely on, so changing x86 pmap internals no longer requires recompiling the entire kernel every time. Callers needing these internals must now use machine/pmap_private.h. Note: This is not x86/pmap_private.h because it contains three parts: 1. CPU-specific (different for i386/amd64) definitions used by... 2. common definitions, including Xenisms like xpmap_ptetomach, further used by... 3. more CPU-specific inlines for pmap_pte_* operations So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h for 2, and then defines 3. Maybe we should split that out into a new pmap_pte.h to reduce this trouble. No functional change intended, other than that some .c files must include machine/pmap_private.h when previously uvm/uvm_pmap.h polluted the namespace with pmap internals. Note: This migrates part of i386/pmap.h into i386/vmparam.h -- specifically the parts that are needed for several constants defined in vmparam.h: VM_MAXUSER_ADDRESS VM_MAX_ADDRESS VM_MAX_KERNEL_ADDRESS VM_MIN_KERNEL_ADDRESS Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64 too, just to keep things parallel.
2022-08-20x86: Move struct vm_page_md to common x86/pmap.h.riastradh
2020-05-15Revert previous after thinking about it. It was wrong, don't need to usead
an atomic to clear a PTE or set initial version unless the circumstances call for it.
2020-03-17Always set PTEs using atomics. There are too many assumptions to go wrong.ad
2019-11-14Add support for Kernel Memory Sanitizer (kMSan). It detects uninitializedmaxv
memory used by the kernel at run time, and just like kASan and kCSan, it is an excellent feature. It has already detected 38 uninitialized variables in the kernel during my testing, which I have since discreetly fixed. We use two shadows: - "shad", to track uninitialized memory with a bit granularity (1:1). Each bit set to 1 in the shad corresponds to one uninitialized bit of real kernel memory. - "orig", to track the origin of the memory with a 4-byte granularity (1:1). Each uint32_t cell in the orig indicates the origin of the associated uint32_t of real kernel memory. The memory consumption of these shadows is consequent, so at least 4GB of RAM is recommended to run kMSan. The compiler inserts calls to specific __msan_* functions on each memory access, to manage both the shad and the orig and detect uninitialized memory accesses that change the execution flow (like an "if" on an uninitialized variable). We mark as uninit several types of memory buffers (stack, pools, kmem, malloc, uvm_km), and check each buffer passed to copyout, copyoutstr, bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory that leaves the system. This allows us to detect kernel info leaks in a way that is more efficient and also more user-friendly than KLEAK. Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot tolerate having one non-instrumented function, because this could cause false positives. kMSan cannot instrument ASM functions, so I converted most of them to __asm__ inlines, which kMSan is able to instrument. Those that remain receive special treatment. Contrary to kASan again, kMSan uses a TLS, so we must context-switch this TLS during interrupts. We use different contexts depending on the interrupt level. The orig tracks precisely the origin of a buffer. We use a special encoding for the orig values, and pack together in each uint32_t cell of the orig: - a code designating the type of memory (Stack, Pool, etc), and - a compressed pointer, which points either (1) to a string containing the name of the variable associated with the cell, or (2) to an area in the kernel .text section which we resolve to a symbol name + offset. This encoding allows us not to consume extra memory for associating information with each cell, and produces a precise output, that can tell for example the name of an uninitialized variable on the stack, the function in which it was pushed on the stack, and the function where we accessed this uninitialized variable. kMSan is available with LLVM, but not with GCC. The code is organized in a way that is similar to kASan and kCSan, so it means that other architectures than amd64 can be supported.
2019-11-01Fix KUBSAN: the kernel size now exceeds the mapping limit, so bump themaxv
limit.
2019-08-07Add support for USER_LDT in SVS. This allows us to have both enabled atmaxv
the same time. We allocate an LDT for each CPU in the GDT and map an area for it, in addition to the default LDT already present. In context switches between different processes, we choose between the default or the per-cpu LDT selector: if the user set specific LDT entries, we memcpy them to the per-cpu LDT and load the per-cpu selector. Tested by Naveen Narayanan (with Wine on amd64).
2019-05-29Add PCID support in SVS. This avoids TLB flushes during kernel<->usermaxv
transitions, which greatly reduces the performance penalty introduced by SVS. We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages in both ASIDs. The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether SVS+PCID is in use.
2019-03-09New software PTE bits.maxv
2019-02-11We reorganise definitions for XEN source support as follows:cherry
XEN - common sources required for baseline XEN support. XENPV - sources required for support of XEN in PV mode. XENPVHVM - sources required for support for XEN in HVM mode. XENPVH - sources required for support for XEN in PVH mode.
2018-11-19Introduce pl_pi, will be used soon.maxv
2018-11-19Rename 'mask' -> 'frame', we will use the real 'mask' soon.maxv
2018-08-29Remove the constants of the DMAP, they are unused, and move NL4_SLOT_DIRECTmaxv
into amd64/.
2018-08-20Add support for kASan on amd64. Written by me, with some parts inspiredmaxv
from Siddharth Muralee's initial work. This feature can detect several kinds of memory bugs, and it's an excellent feature. It can be enabled by uncommenting these three lines in GENERIC: #makeoptions KASAN=1 # Kernel Address Sanitizer #options KASAN #no options SVS The kernel is compiled without SVS, without DMAP and without PCPU area. A shadow area is created at boot time, and it can cover the upper 128TB of the address space. This area is populated gradually as we allocate memory. With this design the memory consumption is kept at its lowest level. The compiler calls the __asan_* functions each time a memory access is done. We verify whether this access is legal by looking at the shadow area. We declare our own special memcpy/memset/etc functions, because the compiler's builtins don't add the __asan_* instrumentation. Initially all the mappings are marked as valid. During dynamic allocations, we add a redzone, which we mark as invalid. Any access on it will trigger a kASan error message. Additionally, the compiler adds a redzone on global variables, and we mark these redzones as invalid too. The illegal-access detection works with a 1-byte granularity. For now, we cover three areas: - global variables - kmem_alloc-ated areas - malloc-ated areas More will come, but that's a good start.
2018-08-17Remove big outdated comment, remove unused macros, remove XXX that hasmaxv
nothing to do here, style.
2018-08-12More ASLR: randomize the location of the PTE area. The PTE slot is notmaxv
created in locore anymore, but a little later; by using the already entered L4 page, rather than the recursive slot itself (which doesn't exist yet). In the prekern we still map the slot - the prekern behaves as an external locore -, because we need it as part of the randomization/relocation work. The kernel then removes this slot, and regenerates a randomized one. Tested on GENERIC and GENERIC_KASLR, Xen doesn't have it and dom0 still boots fine.
2018-08-12Move the PTE area from slot 255 to slot 509. I've never understood why wemaxv
put it on 255; the "kernel" half of the VM space begins on slot 256, so if anything, the PTE area should have been above it, not below. Virtually extend the user slots in slotspace, because we don't want (randomized) kernel mappings to land on slot 255. The prekern is updated accordingly. Tested on GENERIC, GENERIC_KASLR and XEN3_DOM0.
2018-08-12Introduce PDIR_SLOT_USERLIM, which indicates the limit of the user slots.maxv
Use it instead of PDIR_SLOT_PTE when we just want to iterate over the user slots. Also use it in SVS, I had hardcoded 255 because there was no proper define (which there now is).
2018-08-12Randomize the main memory on Xen, same as native. Tested on amd64-dom0.maxv
2018-08-12More ASLR: randomize the kernel main memory. VM_MIN_KERNEL_ADDRESS becomesmaxv
variable, and its location is chosen at boot time. There is room for improvement, since for now we ask for an alignment of NBPD_L4. This is enabled by default in GENERIC, but not in Xen. Tested extensively on GENERIC and GENERIC_KASLR, XEN3_DOM0 still boots fine.
2018-07-27Remove KERN_BASE, unused. It has always been wrong anyway, the valuemaxv
should have been passed into VA_SIGN_NEG().
2018-07-25Remove NPTECL, unused.maxv
2018-05-19add experimental new function uvm_direct_process(), to allow of read/writesjdolecek
of contents of uvm pages without mapping them into kernel, using direct map or moral equivalent; pmaps supporting the interface need to provide pmap_direct_process() and define PMAP_DIRECT implement the new interface for amd64; I hear alpha and mips might be relatively easy to add too, but I lack the knowledge part of resolution for PR kern/53124
2018-02-22Remove svs_pgg_update(). Instead of manually changing PG_G on each page,maxv
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that. In addition, install CR4_PGE when SVS is disabled manually (via the sysctl). Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance completely, exactly as if SVS hadn't been enabled in the first place.
2018-02-22Improve the SVS initialization.maxv
Declare x86_patch_window_open() and x86_patch_window_close(), and globalify x86_hotpatch(). Introduce svs_enable() in x86/svs.c, that does the SVS hotpatching. Change svs_init() to take a bool. This function gets called twice; early when the system just booted (and nothing is initialized), lately when at least pmap_kernel has been initialized.
2018-02-18Add svs_enabled, which defaults to 'true' when SVS is compiled (no dynamicmaxv
detection yet).
2018-01-21Increase the size of the initial mapping of the kernel. KASLR kernels aremaxv
bigger than their GENERIC counterparts, and the limit will soon be hit on them.
2018-01-07Add a new option, SVS (for Separate Virtual Space), that unmaps kernelmaxv
pages when running in userland. For now, only the PTE area is unmapped. Sent on tech-kern@.
2017-06-17Increase the kernel heap size from 512GB to 32TB, in such a way that itmaxv
is able to map the maximum amount of ram supported twice (16TB x 2).
2016-11-11Remove useless values, and explain where some others come frommaxv
2016-07-22Remove pmap_prealloc_lowmem_ptps on amd64. This function creates levels inmaxv
the page tree so that the first 2MB of virtual memory can be kentered in L1. Strictly speaking, the kernel should never kenter a virtual page below VM_MIN_KERNEL_ADDRESS, because then it wouldn't be available in userland. It used to need the first 2MB in order to map the CPU trampoline and the initial VAs used by the bootstrap code. Now, the CPU trampoline VA is allocated with uvm_km_alloc and the VAs used by the bootstrap code are allocated with pmap_bootstrap_valloc, and in either case the resulting VA is above VM_MIN_KERNEL_ADDRESS. The low levels in the page tree are therefore unused. By removing this function, we are making sure no one will be tempted to map an area below VM_MIN_KERNEL_ADDRESS in kernel mode, and particularly, we are making sure NULL cannot be kentered. In short, there is no way to map NULL in kernel mode anymore.
2016-05-21Explain where this value comes from.maxv
2016-05-14KNF so it appears aligned on NXR, and fix a comment.maxv
2015-01-09Bump amd64 module map size to 32 MB.riastradh
For lack of anything better to do, after no progress in discussion on the matter: https://mail-index.netbsd.org/port-amd64/2014/08/22/msg002108.html Needed in order to load the (solaris module needed by) dtrace module.
2012-06-30Extend the xpmap API, as described in [1]. This change is mechanical andjym
avoids exposing the MD phys_to_machine/machine_to_phys tables directly. Added: - xpmap_ptom handles PFN (pseudo physical) to MFN (machine frame number) translations, and is under control of the domain. - xpmap_mtop is its counterpart (MFN to PFN), and is under control of hypervisor. xpmap_ptom_map() map a pseudo-phys address to a machine address xpmap_ptom_unmap() unmap a pseudo-phys address (invalidation) xpmap_ptom_isvalid() check for pseudo-phys address validity The parameters are physical/machine addresses, like bus_dma/bus_space(9). As x86 MFNs are tracked by u_long (Xen's choice) while machine addresses can be 64 bits entities (PAE), use ptoa() to avoid truncation when bit shifting by PAGE_SHIFT. I kept the same namespace (xpmap_) to avoid code churn. [1] http://mail-index.netbsd.org/port-xen/2009/05/09/msg004951.html XXX will document ptoa/atop/trunc_page separately.
2012-06-11allow more space for modules.chs
2012-02-19Removing remaining references to the alternate PTE space. Modify ↵cherry
documentation appropriately
2012-01-19pmap_pte_set() is not supposed to be atomic, so only raise IPL, no need tobouyer
take pte_lock
2012-01-15for xen on amd64 PDP_BASE points to the per-cpu ci->ci_kpm_pdir copy of ↵cherry
*pmap_kernel()*'s L4 pdir, which is an alias for ci->ci_kpm_pdir. This is unlike PAE, where PDP_BASE points to the per-pmap pm_pdir consisting of 4 pages, the last of which is the "shadow". This "shadow" is not used directly in an active pmap, since it duplicates the kernel space and, for PAE, xen dissallows multiple cpus pointing to the same L3[3] page. Therefore, we use a per-cpu copy of the pmap_kernel() pdir's L3[3] page, ci->ci_pae_l3_pdir[3], while L3[0-2] point to the original pmap's pm_pdir[0 - 2]. Thus the "shadow" pdir only exists on i386 PAE. Note that on PAE, the recursive PDIR_SLOT_PTE is not per-cpu, and therefore cannot be made to point to per-cpu pdirs via (L4_BASE + PDIR_SLOT_PTE), unlike xen x86_64 where this is exactly the case.
2012-01-09Make cross-cpu pte access MP safe.cherry
XXX: review cases of use of pmap_set_pte() vs direct use of xpq_queue_pte_update()
2011-11-06[merging from cherry-xenmp] make pmap_kernel() shadow PMD per-cpu and MP aware.cherry
2011-11-06[merging from cherry-xenmp] Make the xen MMU op queue locking api private. ↵cherry
Implement per-cpu queues.
2011-08-27Implement sparse dumps for amd64 (copied from i386). Disabled for now viachristos
sysctl. XXX: most of the code can be merged.
2011-08-13Add locking around ops to the hypervisor MMU "queue".cherry
2011-02-01udpate license clauses on my code to match the new-style BSD licenses.chuck
based on diff that rmind@ sent me. no functional change with this commit.
2010-11-14Move struct vm_page_md definition from vmparam.h to pmap.h, becauseuebayasi
it's used only by pmap. vmparam.h has definitions for wider audience. All GENERIC kernels build tested, except ia64. powerpc/include/booke/vmparam.h has one too, but it has no pmap.h, so it's left as is.
2008-10-26put the contents of these header files around #ifdef __x86_64__, andmrg
#include the <i386/foo.h> in the #else clause, making these files largely bit-size independant.
2008-01-23Merge the bouyer-xeni386 branch. This brings in PAE support to NetBSD xeni386bouyer
(domU only). PAE support is enabled by 'options PAE', see the new XEN3PAE_DOMU and INSTALL_XEN3PAE_DOMU kernel config files. See the comments in arch/i386/include/{pte.h,pmap.h} to see how it works. In short, we still handle it as a 2-level MMU, with the second level page directory being 4 pages in size. pmap switching is done by switching the L2 pages in the L3 entries, instead of loading %cr3. This is almost required by Xen, which handle the last L2 page (the one mapping 0xc0000000 - 0xffffffff) in a very special way. But this approach should also work for native PAE support if ever supported (in fact, the pmap should almost suport native PAE, what's missing is bootstrap code in locore.S).
2008-01-20Make first argument of Xen's pmap_pte_cas() volatile, fix a warningbouyer
building pmap.c.