| Age | Commit message (Collapse) | Author |
|
This makes pmap_resident_count and pmap_wired_count out-of-line
functions instead of inline. No functional change intended
otherwise.
|
|
This way pmap.h only contains the MD definition of the MI pmap(9)
API, which loads of things in the kernel rely on, so changing x86
pmap internals no longer requires recompiling the entire kernel every
time.
Callers needing these internals must now use machine/pmap_private.h.
Note: This is not x86/pmap_private.h because it contains three parts:
1. CPU-specific (different for i386/amd64) definitions used by...
2. common definitions, including Xenisms like xpmap_ptetomach,
further used by...
3. more CPU-specific inlines for pmap_pte_* operations
So {amd64,i386}/pmap_private.h defines 1, includes x86/pmap_private.h
for 2, and then defines 3. Maybe we should split that out into a new
pmap_pte.h to reduce this trouble.
No functional change intended, other than that some .c files must
include machine/pmap_private.h when previously uvm/uvm_pmap.h
polluted the namespace with pmap internals.
Note: This migrates part of i386/pmap.h into i386/vmparam.h --
specifically the parts that are needed for several constants defined
in vmparam.h:
VM_MAXUSER_ADDRESS
VM_MAX_ADDRESS
VM_MAX_KERNEL_ADDRESS
VM_MIN_KERNEL_ADDRESS
Since i386 needs PDP_SIZE in vmparam.h, I added it there on amd64
too, just to keep things parallel.
|
|
|
|
an atomic to clear a PTE or set initial version unless the circumstances
call for it.
|
|
|
|
memory used by the kernel at run time, and just like kASan and kCSan, it
is an excellent feature. It has already detected 38 uninitialized variables
in the kernel during my testing, which I have since discreetly fixed.
We use two shadows:
- "shad", to track uninitialized memory with a bit granularity (1:1).
Each bit set to 1 in the shad corresponds to one uninitialized bit of
real kernel memory.
- "orig", to track the origin of the memory with a 4-byte granularity
(1:1). Each uint32_t cell in the orig indicates the origin of the
associated uint32_t of real kernel memory.
The memory consumption of these shadows is consequent, so at least 4GB of
RAM is recommended to run kMSan.
The compiler inserts calls to specific __msan_* functions on each memory
access, to manage both the shad and the orig and detect uninitialized
memory accesses that change the execution flow (like an "if" on an
uninitialized variable).
We mark as uninit several types of memory buffers (stack, pools, kmem,
malloc, uvm_km), and check each buffer passed to copyout, copyoutstr,
bwrite, if_transmit_lock and DMA operations, to detect uninitialized memory
that leaves the system. This allows us to detect kernel info leaks in a way
that is more efficient and also more user-friendly than KLEAK.
Contrary to kASan, kMSan requires comprehensive coverage, ie we cannot
tolerate having one non-instrumented function, because this could cause
false positives. kMSan cannot instrument ASM functions, so I converted
most of them to __asm__ inlines, which kMSan is able to instrument. Those
that remain receive special treatment.
Contrary to kASan again, kMSan uses a TLS, so we must context-switch this
TLS during interrupts. We use different contexts depending on the interrupt
level.
The orig tracks precisely the origin of a buffer. We use a special encoding
for the orig values, and pack together in each uint32_t cell of the orig:
- a code designating the type of memory (Stack, Pool, etc), and
- a compressed pointer, which points either (1) to a string containing
the name of the variable associated with the cell, or (2) to an area
in the kernel .text section which we resolve to a symbol name + offset.
This encoding allows us not to consume extra memory for associating
information with each cell, and produces a precise output, that can tell
for example the name of an uninitialized variable on the stack, the
function in which it was pushed on the stack, and the function where we
accessed this uninitialized variable.
kMSan is available with LLVM, but not with GCC.
The code is organized in a way that is similar to kASan and kCSan, so it
means that other architectures than amd64 can be supported.
|
|
limit.
|
|
the same time.
We allocate an LDT for each CPU in the GDT and map an area for it, in
addition to the default LDT already present. In context switches between
different processes, we choose between the default or the per-cpu LDT
selector: if the user set specific LDT entries, we memcpy them to the
per-cpu LDT and load the per-cpu selector.
Tested by Naveen Narayanan (with Wine on amd64).
|
|
transitions, which greatly reduces the performance penalty introduced by
SVS.
We use two ASIDs, 0 (kern) and 1 (user), and use invpcid to flush pages
in both ASIDs.
The read-only machdep.svs.pcid={0,1} sysctl is added, and indicates whether
SVS+PCID is in use.
|
|
|
|
XEN - common sources required for baseline XEN support.
XENPV - sources required for support of XEN in PV mode.
XENPVHVM - sources required for support for XEN in HVM mode.
XENPVH - sources required for support for XEN in PVH mode.
|
|
|
|
|
|
into amd64/.
|
|
from Siddharth Muralee's initial work. This feature can detect several
kinds of memory bugs, and it's an excellent feature.
It can be enabled by uncommenting these three lines in GENERIC:
#makeoptions KASAN=1 # Kernel Address Sanitizer
#options KASAN
#no options SVS
The kernel is compiled without SVS, without DMAP and without PCPU area.
A shadow area is created at boot time, and it can cover the upper 128TB
of the address space. This area is populated gradually as we allocate
memory. With this design the memory consumption is kept at its lowest
level.
The compiler calls the __asan_* functions each time a memory access is
done. We verify whether this access is legal by looking at the shadow
area.
We declare our own special memcpy/memset/etc functions, because the
compiler's builtins don't add the __asan_* instrumentation.
Initially all the mappings are marked as valid. During dynamic
allocations, we add a redzone, which we mark as invalid. Any access on
it will trigger a kASan error message. Additionally, the compiler adds
a redzone on global variables, and we mark these redzones as invalid too.
The illegal-access detection works with a 1-byte granularity.
For now, we cover three areas:
- global variables
- kmem_alloc-ated areas
- malloc-ated areas
More will come, but that's a good start.
|
|
nothing to do here, style.
|
|
created in locore anymore, but a little later; by using the already
entered L4 page, rather than the recursive slot itself (which doesn't
exist yet).
In the prekern we still map the slot - the prekern behaves as an external
locore -, because we need it as part of the randomization/relocation
work. The kernel then removes this slot, and regenerates a randomized
one.
Tested on GENERIC and GENERIC_KASLR, Xen doesn't have it and dom0 still
boots fine.
|
|
put it on 255; the "kernel" half of the VM space begins on slot 256, so
if anything, the PTE area should have been above it, not below.
Virtually extend the user slots in slotspace, because we don't want
(randomized) kernel mappings to land on slot 255.
The prekern is updated accordingly.
Tested on GENERIC, GENERIC_KASLR and XEN3_DOM0.
|
|
Use it instead of PDIR_SLOT_PTE when we just want to iterate over the
user slots. Also use it in SVS, I had hardcoded 255 because there was no
proper define (which there now is).
|
|
|
|
variable, and its location is chosen at boot time. There is room for
improvement, since for now we ask for an alignment of NBPD_L4.
This is enabled by default in GENERIC, but not in Xen. Tested extensively
on GENERIC and GENERIC_KASLR, XEN3_DOM0 still boots fine.
|
|
should have been passed into VA_SIGN_NEG().
|
|
|
|
of contents of uvm pages without mapping them into kernel, using
direct map or moral equivalent; pmaps supporting the interface need
to provide pmap_direct_process() and define PMAP_DIRECT
implement the new interface for amd64; I hear alpha and mips might be relatively
easy to add too, but I lack the knowledge
part of resolution for PR kern/53124
|
|
we can disable the global-paging mechanism in %cr4 with CR4_PGE. Do that.
In addition, install CR4_PGE when SVS is disabled manually (via the
sysctl).
Now, doing "sysctl -w machdep.svs_enabled=0" restores the performance
completely, exactly as if SVS hadn't been enabled in the first place.
|
|
Declare x86_patch_window_open() and x86_patch_window_close(), and globalify
x86_hotpatch().
Introduce svs_enable() in x86/svs.c, that does the SVS hotpatching.
Change svs_init() to take a bool. This function gets called twice; early
when the system just booted (and nothing is initialized), lately when at
least pmap_kernel has been initialized.
|
|
detection yet).
|
|
bigger than their GENERIC counterparts, and the limit will soon be hit on
them.
|
|
pages when running in userland. For now, only the PTE area is unmapped.
Sent on tech-kern@.
|
|
is able to map the maximum amount of ram supported twice (16TB x 2).
|
|
|
|
the page tree so that the first 2MB of virtual memory can be kentered in
L1.
Strictly speaking, the kernel should never kenter a virtual page below
VM_MIN_KERNEL_ADDRESS, because then it wouldn't be available in userland.
It used to need the first 2MB in order to map the CPU trampoline and the
initial VAs used by the bootstrap code. Now, the CPU trampoline VA is
allocated with uvm_km_alloc and the VAs used by the bootstrap code are
allocated with pmap_bootstrap_valloc, and in either case the resulting VA
is above VM_MIN_KERNEL_ADDRESS.
The low levels in the page tree are therefore unused. By removing this
function, we are making sure no one will be tempted to map an area below
VM_MIN_KERNEL_ADDRESS in kernel mode, and particularly, we are making sure
NULL cannot be kentered.
In short, there is no way to map NULL in kernel mode anymore.
|
|
|
|
|
|
For lack of anything better to do, after no progress in discussion on
the matter:
https://mail-index.netbsd.org/port-amd64/2014/08/22/msg002108.html
Needed in order to load the (solaris module needed by) dtrace module.
|
|
avoids exposing the MD phys_to_machine/machine_to_phys tables directly.
Added:
- xpmap_ptom handles PFN (pseudo physical) to MFN (machine frame number)
translations, and is under control of the domain.
- xpmap_mtop is its counterpart (MFN to PFN), and is under control of
hypervisor.
xpmap_ptom_map() map a pseudo-phys address to a machine address
xpmap_ptom_unmap() unmap a pseudo-phys address (invalidation)
xpmap_ptom_isvalid() check for pseudo-phys address validity
The parameters are physical/machine addresses, like bus_dma/bus_space(9).
As x86 MFNs are tracked by u_long (Xen's choice) while machine addresses
can be 64 bits entities (PAE), use ptoa() to avoid truncation when bit
shifting by PAGE_SHIFT.
I kept the same namespace (xpmap_) to avoid code churn.
[1] http://mail-index.netbsd.org/port-xen/2009/05/09/msg004951.html
XXX will document ptoa/atop/trunc_page separately.
|
|
|
|
documentation appropriately
|
|
take pte_lock
|
|
*pmap_kernel()*'s L4 pdir, which is an alias for ci->ci_kpm_pdir. This is unlike PAE, where PDP_BASE points to the per-pmap pm_pdir consisting of 4 pages, the last of which is the "shadow". This "shadow" is not used directly in an active pmap, since it duplicates the kernel space and, for PAE, xen dissallows multiple cpus pointing to the same L3[3] page. Therefore, we use a per-cpu copy of the pmap_kernel() pdir's L3[3] page, ci->ci_pae_l3_pdir[3], while L3[0-2] point to the original pmap's pm_pdir[0 - 2]. Thus the "shadow" pdir only exists on i386 PAE. Note that on PAE, the recursive PDIR_SLOT_PTE is not per-cpu, and therefore cannot be made to point to per-cpu pdirs via (L4_BASE + PDIR_SLOT_PTE), unlike xen x86_64 where this is exactly the case.
|
|
XXX: review cases of use of pmap_set_pte() vs direct use of xpq_queue_pte_update()
|
|
|
|
Implement per-cpu queues.
|
|
sysctl.
XXX: most of the code can be merged.
|
|
|
|
based on diff that rmind@ sent me.
no functional change with this commit.
|
|
it's used only by pmap. vmparam.h has definitions for wider
audience.
All GENERIC kernels build tested, except ia64.
powerpc/include/booke/vmparam.h has one too, but it has no pmap.h,
so it's left as is.
|
|
#include the <i386/foo.h> in the #else clause, making these files
largely bit-size independant.
|
|
(domU only). PAE support is enabled by 'options PAE', see the new XEN3PAE_DOMU
and INSTALL_XEN3PAE_DOMU kernel config files.
See the comments in arch/i386/include/{pte.h,pmap.h} to see how it works.
In short, we still handle it as a 2-level MMU, with the second level page
directory being 4 pages in size. pmap switching is done by switching the
L2 pages in the L3 entries, instead of loading %cr3. This is almost required
by Xen, which handle the last L2 page (the one mapping 0xc0000000 - 0xffffffff)
in a very special way. But this approach should also work for native PAE
support if ever supported (in fact, the pmap should almost suport native
PAE, what's missing is bootstrap code in locore.S).
|
|
building pmap.c.
|