summaryrefslogtreecommitdiff
path: root/sys/arch/aarch64/include
AgeCommit message (Collapse)Author
2023-05-06s/Regiser/Register/ and s/regester/register/ in comments.andvar
2023-04-20Provide a shared pmap_devmap implementation and convert all pmap_devmapskrll
arrays to use DEVMAP_ENTRY{,_END}
2023-04-16Rename VM_KERNEL_IO_ADDRESS to VM_KERNEL_IO_BASE to match RISC-Vskrll
It's less letters, matches other similar variables and will help with sharing code between the two architectures. NFCI.
2023-02-25aarch64: curcpu() audit.riastradh
Sprinkle KASSERT (or KDASSERT in hot paths) for kpreempt_disabled() when we use curcpu() and it's not immediately obvious that the caller has preemption disabled but closer scrutiny suggests the caller has. Note unsafe curcpu()s for syscall event counting. Not sure this is worth changing. Possible bugs fixed: - cpu_irq and cpu_fiq could be preempted while trying to run softints on this CPU. - data_abort_handler might incorrectly think it was invoked in interrupt context when it was only preempted and migrated to another CPU. - pmap_fault_fixup might report the wrong CPU logs. (However, we don't currently run with kpreemption on aarch64, so these are not yet real bugs fixed except if you patch it to build with __HAVE_PREEMPTION.)
2022-12-21Rename pmap_md_pdetab_destroy to pmap_md_pdetab_fini to matchskrll
pmap_md_pdetab_init. Call pmap_md_pdetab_fini from pmap_segtab_destroy.
2022-12-01Improve tprof(4)ryo
- Multiple events can now be handled simultaneously. - Counters should be configured with TPROF_IOC_CONFIGURE_EVENT in advance, instead of being configured at TPROF_IOC_START. - The configured counters can be started and stopped repeatedly by PROF_IOC_START/TPROF_IOC_STOP. - The value of the performance counter can be obtained at any timing as a 64bit value with TPROF_IOC_GETCOUNTS. - Backend common parts are handled in tprof.c as much as possible, and functions on the tprof_backend side have been reimplemented to be more primitive. - The reset value of counter overflows for profiling can now be adjusted. It is calculated by default from the CPU clock (speed of cycle counter) and TPROF_HZ, but for some events the value may be too large to be sufficient for profiling. The event counter can be specified as a ratio to the default or as an absolute value when configuring the event counter. - Due to overall changes, API and ABI have been changed. TPROF_VERSION and TPROF_BACKEND_VERSION were updated.
2022-12-01PMCR.E should not be disabled from tprof.ryo
PMCR.E controls not only performance event counters but also the cycle counter operation, and the cycle counter may be used for cpu_counter. Similarly, the 31st bit in PMINTENCLR and PMCNTENCLR controls the cycle counter, not performance event counters, and should not be modified.
2022-11-03Provide MI PMAP support on AARCH64skrll
2022-10-29KNF + remove unnecessary bracketsskrll
2022-10-23KNF.skrll
2022-10-23Line continuation alignment whitespace. NFC.skrll
2022-10-15Use "non-posted" instead of "strongly ordered" to describe nGnRnE mappingsjmcneill
Rename the following defines: - _ARM_BUS_SPACE_MAP_STRONGLY_ORDERED to BUS_SPACE_MAP_NONPOSTED - PMAP_DEV_SO to PMAP_DEV_NP - LX_BLKPAG_ATTR_DEVICE_MEM_SO to LX_BLKPAG_ATTR_DEVICE_MEM_NP Rename the following option: - AARCH64_DEVICE_MEM_STRONGLY_ORDERED to AARCH64_DEVICE_MEM_NONPOSTED
2022-09-10fix misspellings of 'available' and nearby typosrillig
2022-08-19Fixed a bug that pte's __BIT(63,48) could be set when accessing addresses ↵ryo
above 0x0001000000000000 in /dev/mem with mmap().
2022-07-24aarch64/lock.h: Need <sys/param.h> for _HARDKERNEL.riastradh
Add include guard while here. XXX Why does this aarch64-specific file have #ifdef __aarch64__?
2022-07-20aarch64: Make cpufunc.h includable without sys/cpu.h first.riastradh
2022-06-25Remove GIC_SPLFUNCS.jmcneill
2022-06-25pic: Update ci_cpl in pic_set_priority callback.jmcneill
Not all ICs need interrupts disabled to update the priority. DAIF accesses are not cheap, so push the update of ci_cpl from pic_set_priority to the IC's pic_set_priority callback, and let the IC driver determine whether or not it needs interrupts disabled.
2022-05-30Use the ABI value for 'R_AARCH64_TLSLD_LDST128_DTPREL_LO12_NC'.jkoshy
2022-05-02Only print the appropriate PAR fields for PAR.F={0,1}skrll
Group the fields in the header.
2022-04-02Update to support EFI runtime outside the kernel virtual address spaceskrll
by creating an EFI RT pmap that can be activated / deactivated when required. Adds support for EFI RT to ARM_MMU_EXTENDED (ASID) 32-bit Arm machines. On Arm64 the usage of pmapboot_enter is reduced and the mappings are created much later in the boot process -- now in cpu_startup_hook. Backward compatiblity for KVA mapped RT from old bootaa64.efi is maintained. Adding support to other platforms should be easier as a result.
2022-01-31add support Hardware updates to Access flag and Dirty state (FEAT_HAFDBS)ryo
- The DBM bit of the PTE is now used to determine if it is writable, and the AF bit is treated entirely as a reference bit. A valid PTE is always treated as readable. There can be no valid PTE that is not readable. - LX_BLKPAG_OS_{READ,WRITE} are used only for debugging purposes, and has been superseded by LX_BLKPAG_AF and LX_BLKPAG_DBM. - Improve comment The need for reference/modify emulation has been eliminated, and access/permission faults have been reduced, however, there has been little change in overall performance.
2022-01-15Remove unnecessary bracketsskrll
2022-01-14Restore the previous pmap_remove_all behaviour as the new method meantskrll
the n1sdp couldn't complete a build. No noticeable change in kernel build performance.
2022-01-05fix ID_AA64ISAR0_EL1.ATOMIC field definitionryo
2021-11-05Normally, to support COMPAT_NETBSD32 we need to definethorpej
__HAVE_STRUCT_SIGCONTEXT in order to support the old "sigcontext" style of handlers for 32-bit binaries. However, we only support 32-bit EABI binaries on AArch64, and by happy accident (due to a libc bug introduced in 2006), 32-bit NetBSD EABI binaries never used "sigcontext" style handlers. So, we don't need to carry any of this baggage forward. This addresses the AArch64 case of PR kern/56487.
2021-11-02In order to prevent _mcount() from being recursively called when built with ↵ryo
COPTS=-O0, sprinkle `__always_inline' to make _mcount() be generated as a single function.
2021-11-01Fix a last minute rebase/merge botch so that the cpu_hatch commit actuallyskrll
works.
2021-10-31Fix crash(8) buildskrll
2021-10-31Rework Arm (32bit and 64bit) AP startup so that cpu_hatch doesn't sleep.skrll
The AP initialisation code in cpu_init_secondary_processor will read and initialise the required system registers and state for the BP to attach and report. Rework the interrupt handler code for this new sequence. Thankfully, this removes a bunch of code for bcm2836mp. The VFP detection handler on <= armv7 relies on the global undefined handler being in place until the BP attaches vfp. That is, after the APs have been spun up. gicv3_its.c has a serialisation issue which is protected against in the gicv3_its_cpu_init, which is called from cpu_hatch, with a spin lock. The serialisation issue needs addressing more completely. Tested on RPI3, Apple M1, QEMU, and lx2k Fixes PR port-arm/56264: diagnostic assertion "l->l_stat == LSONPROC" failed on RPI3
2021-10-31Annotate some cpu_info membersskrll
2021-10-27- In sendsig() and sigaction1(), don't hard-code signal trampolinethorpej
versions. Instead, use the version constants from <sys/signal.h> and automatically (and correctly) handle cases where multiple versions of a particular trampoline flavor exist. Conditionalize support for sigcontext trampolines on __HAVE_STRUCT_SIGCONTEXT. - aarch64 and amd64 don't use sigcontext natively, but do need to support it for 32-bit compatibility; define __HAVE_STRUCT_SIGCONTEXT conditionally on _KERNEL.
2021-10-26fix build with COPTS=-O0ryo
2021-10-26Add a comment and adjust whitespace to match style in this fileskrll
2021-10-23Whitespaceskrll
2021-10-23Typo in commentskrll
2021-10-10Use sys/uvm/pmap/pmap_tlb.c on Aarch64 in the same way that some Arm, MIPS,skrll
and some PPC kernels do. This removes the limitation of 256 processes on CPUs with 8bit ASID field, e.g. Apple M1. Additionally the following changes have been made - removed a couple of unnecessary aarch64_tlbi_all calls - removed any invalidation after freeing page tables due to _pmap_sweep_pdp. This was never necessary afaict. - all kernel mappings are marked global and userland mapping not-global. Performance testing hasn't show a significant difference. The data here is from building a kernel on an lx2k system with nvme. before 1489.6u 400.4s 2:40.65 1176.5% 228+224k 0+32289io 57pf+0w 1482.6u 403.2s 2:38.49 1189.9% 228+222k 0+32274io 46pf+0w 1485.4u 402.2s 2:37.27 1200.2% 228+222k 0+32275io 12pf+0w after 1493.9u 404.6s 2:37.50 1205.4% 227+221k 0+32265io 48pf+0w 1485.0u 408.0s 2:38.54 1194.0% 227+222k 0+32272io 36pf+0w 1484.3u 407.0s 2:35.88 1213.3% 228+224k 0+32268io 14pf+0w >>> stats.ttest_ind([160.65,158.49,157.27], [157.5,158.54,155.88]) Ttest_indResult(statistic=1.1923622711296888, pvalue=0.2990182944606766) >>>
2021-09-30Make tlb_asid_t unsigned int as pmap_tlb.c expects tlb_asid_t to be able toskrll
hold ASID_MAX + 1.
2021-09-26Use the yield instruction as SPINLOCK_BACKOFF_HOOK for aarch64.jmcneill
2021-09-18gic_splx: performance optimizationsjmcneill
Avoid any kind of register access (DAIF, PMR, etc), barriers, and atomic operations in the common case where no interrupt fires between spl being raised and lowered. This introduces a per-CPU return address (ci_splx_restart) used by the vector handler to restart a sequence in splx that compares the new ipl with the per-CPU hardware priority state stored in ci_hwpl.
2021-08-30Add FIQ support.jmcneill
2021-08-30Add definition for HCR_E2H bitjmcneill
2021-08-14Improved the performance of kernel profiling on MULTIPROCESSOR, and possible ↵ryo
to get profiling data for each CPU. In the current implementation, locks are acquired at the entrance of the mcount internal function, so the higher the number of cores, the more lock conflict occurs, making profiling performance in a MULTIPROCESSOR environment unusable and slow. Profiling buffers has been changed to be reserved for each CPU, improving profiling performance in MP by several to several dozen times. - Eliminated cpu_simple_lock in mcount internal function, using per-CPU buffers. - Add ci_gmon member to struct cpu_info of each MP arch. - Add kern.profiling.percpu node in sysctl tree. - Add new -c <cpuid> option to kgmon(8) to specify the cpuid, like openbsd. For compatibility, if the -c option is not specified, the entire system can be operated as before, and the -p option will get the total profiling data for all CPUs.
2021-08-08Re-applyskrll
Move 'struct pic_pending' from percpu to struct cpu_info. Saves a few instructions in splx. There is(/was) no need to use atomic operations on the percpu / cpu_info members, so don't. Finally removng the use of percpu should help avoid problems with "late" attaching cpus.
2021-06-19Do not try to initialize PMU if ID_AA64DFR0_EL1 reports a non-standardjmcneill
PMU implementation.
2021-06-19CNTV_CTL_EL0 is a 64-bit registerjmcneill
2021-05-31Include "opt_param.h" (ifdef _KERNEL_OPT) everywhere that MSGBUFSIZE issimonb
referenced since some sources include <machine/param.h>.
2021-05-29Deal with the pmap limitation of maxproc in a more complete way andskrll
recognise CPUs with only 8bit ASIDs.
2021-05-29Sort includes. NFCI.skrll
2021-05-27fix build error with options ARMV85_BTIryo