* Boot compatibility fixes for UE titles, GUI toggles, and DeS render/boot work
Checkpoint of the Monster Truck Championship and Demon's Souls boot work.
Each piece is independently useful and verified against the titles.
- playgo: scePlayGoGetLocus now returns BAD_CHUNK_ID for chunk ids outside
the known set, matching real firmware. Titles enumerate chunk ids until
that error; answering OK for every id made the scan wrap the ushort range
and spin forever. Missing-sidecar and no-app0 fallbacks report a
fully-installed single chunk 0 so scePlayGoOpen keeps succeeding.
- kernel: restore the SHARPEMU_WRITABLE_APP0 opt-in. Unpackaged UE dumps
write their Saved tree under /app0 during PS5 component init and treat
the denial as a fatal boot error.
- pad: accept handle 0 as the primary pad across all pad calls. Real
firmware hands out small non-negative handles and some titles read state
with handle 0.
- bthid: env-gated experiment hooks for the Thrustmaster wheel middleware
investigation (fail-only-RegisterCallback modes and a synthetic
enumeration callback with a zeroed event struct). All default off.
- gui: add SHARPEMU_LOG_IO and SHARPEMU_WRITABLE_APP0 toggles to the
Environment tab.
- videoout: per-swapchain-image render-finished semaphores (the shared
semaphore raced the swapchain); whole-mip-chain layout init for offscreen
guest images (sampled binds read mips stuck in Undefined); GPU-resident
texture availability now canonicalizes through the texture format table
and accepts compatibility-class aliases, cutting per-frame CPU texture
re-reads (143 GB -> 55 GB per 300 s in Demon's Souls, 0.2 -> 0.5 fps).
- hle: add sceSystemServiceGetNoticeScreenSkipFlag,
sceSystemServiceGetMainAppTitleId (title id published from the runtime),
and sceNpWebApi2CreateUserContext (refuses so the online layer backs off).
- rtc: SHARPEMU_RTC_PROBE_RANGE diagnostic dumps the code around a busy-wait
caller of sceRtcGetCurrentTick once; costs nothing when unset.
* [ShaderCompiler] Fix VReadlaneB32 scalar destination field
The scalar destination lives in the low vdst byte (bits 0-7); it was read
from bits 8-14, the VOP3B carry-out field readlane does not have, sending
every readlane result to s0. Verified against raw gfx10 encodings and
LLVM's assembler tests (v_readlane_b32 s5, v1, s2 -> low byte 0x05).
* [VideoOut] Survive device loss and flip-order asserts without dying
Two ways a frame could take down the whole presenter:
- Device loss between any two Vulkan calls in a frame unwound the window
thread, and the Dispose-time fence check then threw again, masking the
original error. Catch the loss at the frame boundary, retire
presentations and guest submissions whose fences can never signal, and
keep the window loop pumping so the game (audio, logic) carries on.
- The ordered-flip capture invariant is violated ~100 times per run by
Demon's Souls (PPSA01342); on debug builds the Debug.Assert fail-fasts
the process with nothing in the log. Downgrade it to a once-per-version
warning until the capture/wait ordering is understood.
* Fix Dead Cells shader cache regression
---------
Co-authored-by: StealUrKill <35749471+StealUrKill@users.noreply.github.com>
* [Core] Add POSIX native execution and PS5 SELF support
Extend the native backend, guest TLS, fixed-address memory, and loader paths needed by PS5 titles on Windows, Linux, and macOS. Keep workstation GC so high-core-count hosts do not reserve over fixed guest image bases.
* [HLE] Expand PS5 service and media compatibility
Add the kernel, threading, save-data, networking, audio, video-codec, font, dialog, and service exports required by newer PS5 software. Preserve every SysAbi NID currently registered by main while adding the compatibility surface used by ASTRO BOT.
* [AGC/Vulkan] Extend Gen5 shader and presentation support
Expand PM4 handling, Gen5 shader translation, MRT and packed export support, guest image tracking, depth initialization, texture aliasing, and Vulkan presentation. Add the performance overlay and address-filtered diagnostics used to validate ASTRO BOT with original shaders.
* [Core] Align static TLS reservation across hosts
* [Pad] Align primary user ID with UserService
* [Gpu] Preserve runtime scalar buffers across renderer seam
* [AGC] Restore omitted command helper exports
* [Vulkan] Reuse primary views for promoted MRT targets
* [Vulkan] Preserve scratch storage bindings in compute dispatches
* [SourceGenerators] Add the SysAbi export generator and analyzers (phase 0)
New SharpEmu.SourceGenerators Roslyn component, complete and tested but
consumed by nothing yet — the emulator projects adopt it in the
following commits.
Ps5Nid ports the PS NID derivation (base64 of the byte-reversed first
eight SHA1 bytes of name + fixed suffix) from
scripts/generate_aerolib_binary.py to C#, so what has always been a
manual, out-of-band computation becomes a compile-time capability.
SysAbiExportGenerator emits a per-assembly SysAbiExportRegistry whose
CreateExports(Generation) reproduces ModuleManager's reflection scan
exactly — same generation inheritance and filtering, same method-name
fallback, same libKernel default — with attribute-omitted NIDs derived
algorithmically (equivalent to the runtime catalog lookup, which is
built from the same computation). Parameterless handlers are adapted to
the SysAbiFunction shape; invalid declarations are skipped here because
the analyzer rejects them as build errors, so nothing drops silently.
SysAbiExportAnalyzer turns the runtime failure modes into diagnostics:
SHEM001 duplicate NID (across declared and derived forms), SHEM002
malformed NID, SHEM003 uncallable handler signature, SHEM004 NID
contradicting its export name (the class of drift previously fixed by
hand), SHEM005 unresolvable export, SHEM006 export name unknown to
ps5_names.txt when the catalog is wired as an AdditionalFile, SHEM007
handler not reachable by generated code.
The self-contained test suite drives both in-process against the real
SharpEmu.HLE metadata: known catalog NID pairs pin the algorithm, the
generated registry must itself compile, and each diagnostic has a
triggering fixture. Fittingly, the NID pinning test caught a wrong
pair in its own first draft — the exact mistake SHEM004 exists to stop.
* [SourceGenerators] Adopt the generated export registry in the emulator (phase 1)
SharpEmu.Libs consumes the generator and analyzers, with
scripts/ps5_names.txt wired as the AdditionalFile catalog. The runtime
now registers exports from the compile-time SysAbiExportRegistry
instead of the boot-time reflection scan; RegisterFromAssembly is
retained solely as the arbiter for a parity test that pins the two
tables identical — same NIDs, names, libraries, targets, and handler
methods — across Gen4, Gen5, and combined registration.
First contact between the analyzer and all 715 existing exports
surfaced real drift the old offline checker structurally missed
(scripts/check_sysabi_aerolib.py skipped any NID absent from
aerolib.bin): three exports whose friendly names collide with real
catalog symbols of different NIDs, now suppressed at-site with reasons
pending AGC API confirmation, alongside the established synthetic
Unknown* labels for uncatalogued NIDs, which prompted a rule
refinement — SHEM004 only hard-errors when the export name is a real
catalog symbol, since synthetic labels cannot be validated by hashing
and the NID is authoritative for them. The two allowlisted mismatches
in the python checker no longer trigger anything, and the checker is
deleted: the analyzer subsumes it with the semantic model instead of
regex, and validates every declared pair rather than only
catalog-known NIDs.
* [SourceGenerators] Generate aerolib.bin at build time from ps5_names.txt
The runtime NID -> name catalog is derived data and no longer lives in
the repository: a Framework-only MSBuild task (GenerateAerolibBinaryTask,
sharing the same Ps5Nid implementation the analyzers use) builds it
into the intermediate directory from scripts/ps5_names.txt — now the
single source of truth — and SharpEmu.HLE embeds it from there. The
output is byte-identical to the previously committed binary, verified
with cmp against git history; a new test pins that the embedded catalog
loads and resolves a known symbol both directions.
scripts/generate_aerolib_binary.py is deleted (its algorithm lives in
Ps5Nid, its invocation in the build); the REUSE annotation for the
binary goes with it. MSBuild's Inputs/Outputs check means the ~154k NID
hashes only recompute when the names file actually changes. The task
implements ITask against Microsoft.Build.Framework directly, keeping
the vulnerable-flagged Utilities.Core package out and the analyzer
project's file-IO ban suppressed only inside the task itself.
* [SourceGenerators] Emit typed-signature register thunks (phase 2)
[SysAbiExport] handlers can now be written with real signatures — a
CpuContext followed by up to six int/uint/long/ulong parameters — and
the generator emits the SysV unmarshalling thunk, mapping parameters
positionally to RDI/RSI/RDX/RCX/R8/R9 with the same unchecked-cast
idiom hand-written handlers use. SHEM003 accepts the new shape and
rejects register overflow and non-register-representable types. Both
shapes coexist, so migration is per-handler; sceKernelPollSema,
sceKernelSignalSema, and sceKernelCancelSema migrate as the
demonstration (the last showing raw ulong guest-address passthrough).
The reflection scan cannot represent typed handlers, so it retires
here: RegisterFromAssembly, its signature validation, and
ResolveExportInfo are deleted, and the parity test that pinned the
generated registry to the scan is replaced by content-invariant tests
(duplicate-free, full 715-export surface, catalog identity). Deleting
the scan surfaced a phase-1 latent regression — the pre-JIT warm sweep
enumerated only reflection-scanned assemblies, so the generated
registration path warmed nothing and re-exposed the guest-thread
fail-fast risk; the warm set is now derived from the registered
handler delegates themselves.
* [SourceGenerators] Marshal guest strings declaratively with [GuestCString] (phase 3)
A string parameter on a typed [SysAbiExport] handler, annotated
[GuestCString(maxLength)], now makes the generated thunk read the
null-terminated UTF-8 string from the argument register's guest
address before the handler runs, returning
ORBIS_GEN2_ERROR_MEMORY_FAULT to the guest when the read fails —
the exact prologue nearly every string-taking handler writes by hand.
The attribute lives in SharpEmu.HLE next to SysAbiExportAttribute;
SHEM008 rejects misuse (non-string parameter, non-positive MaxLength)
while a bare string parameter stays a SHEM003 signature error.
_open, open, and sceKernelOpen migrate as the demonstration; they were
chosen because their hand-written prologue faulted on a null pointer
the same way the thunk does (handlers that return INVALID_ARGUMENT for
null pointers, like sceKernelCreateSema, keep the raw shape so guest-
visible semantics stay untouched).
* [SourceGenerators] Apply review findings across the branch
Behavior: the open/_open/sceKernelOpen [GuestCString] demo migration is
reverted — the local compat reader falls back to host memory for paths
in loader-mapped regions that ctx.Memory cannot see, so the generated
thunk would have turned recoverable reads into MEMORY_FAULT. The
marshalling infrastructure stays, proven by generator/analyzer tests;
production migration waits for a handler whose semantics the thunk
reproduces exactly. A comment on the handler records why.
Build robustness: the aerolib target is skipped for design-time builds
(the IDE resolves project references without compiling them, so on a
fresh clone the task assembly does not exist yet), and the task/names
paths are centralized in properties. The generator now emits no
registry for export-free assemblies, so referencing the analyzer can
never mint a colliding SharpEmu.Generated type.
Cleanup and perf: the pragma-suppression sites left mis-indented by the
phase-1 relocation are reformatted and the restores moved after the
method body; the dead ExportsForTesting hook and its InternalsVisibleTo
are deleted; the aerolib task reuses one SHA1 instance across ~150k
names; the analyzer caches the parsed catalog per file snapshot instead
of re-parsing 150k lines every compilation start, shares the attribute
name constant with the generator, and computes the catalog-membership
check once.
* [CI] Run the test suites in the build workflow
The workflow compiled the test projects (they are in SharpEmu.slnx) but
never executed them. A solution-level dotnet test now runs between
build and publish, so any test failure fails the build — including the
AerolibCatalogTests/SysAbiRegistryTests that guard the build-generated
aerolib.bin and the generated export registry. Generation failures of
aerolib.bin itself already fail the build step: the MSBuild task logs
an error event and returns false, and a missing task assembly or
missing embedded output are hard MSBuild errors. The NuGet cache key
now also tracks the test projects' lock files.
* [SourceGenerators] Address review feedback
Multi-diagnostic analyzer tests no longer assume a stable diagnostic
order (analyzer execution is concurrent), and the aerolib task logs
the full exception instead of only its message so build failures keep
the type and stack trace.
* [SourceGenerators] Address second review round
Symbol-name comparisons in the shape rules and analyzer now pin an
explicit SymbolDisplayFormat.FullyQualifiedFormat instead of relying on
the display-format default, and the aerolib task fails loudly on a
symbol name that would overflow the format's ushort length prefix
instead of silently truncating it, with null-safe output-directory
handling made explicit.
* [SourceGenerators] Embed aerolib.bin via a target so design-time builds never reference it
The static EmbeddedResource item referenced the generated file even in
design-time builds, where the generation target is skipped — on a fresh
clone the IDE would try to embed a file that never existed. The item is
now created inside an EmbedAerolibBinary target gated on
DesignTimeBuild, separate from the generation target so an up-to-date
skip of GenerateAerolibBinary cannot drop the item with the rest of its
body, and hooked before AssignTargetPaths since dynamic resource items
added later miss the resource pipeline.
Verified fresh build, incremental rebuild (embedded catalog test both
times), and a simulated design-time compile with no artifacts present.
* [SourceGenerators] Regenerate test lock file after rebase onto main
Rebase fallout: main's package graph shifted under #200, so the
SourceGenerators.Tests lock file is re-evaluated to keep --locked-mode
restore green at the branch tip.
* [Build] Drop NuGet lock files; rely on central package management
Central package management was already in effect (ManagePackageVersionsCentrally
with all versions in Directory.Packages.props and no inline PackageReference
versions), so the per-project packages.lock.json files and the lock-mode
workflow only added maintenance overhead. This removes all eleven lock files,
drops RestorePackagesWithLockFile so restore no longer regenerates them, and
takes --locked-mode off the CI restore steps (re-keying the NuGet cache on the
central props files). Package versions remain centrally pinned in
Directory.Packages.props.
* [Host] Introduce host platform abstraction with IHostMemory
Add SharpEmu.HLE/Host with IHostPlatform/IHostMemory interfaces, neutral
page-protection/region enums, and a HostPlatform.Current factory that
resolves the Windows backend (or throws PlatformNotSupportedException on
other OSes, matching today's de-facto behavior). WindowsHostMemory wraps
the exact VirtualAlloc/VirtualFree/VirtualProtect/VirtualQuery calls used
across the engine today, with identical MEM_*/PAGE_* constants.
Migrate StubManager as the first consumer: its private kernel32 P/Invokes
and enums are replaced by IHostMemory calls that issue the same two
native operations (RWX commit+reserve of the PLT arena, release on
Dispose). No behavior change.
This is the first step toward supporting non-Windows hosts; subsequent
commits move the remaining direct P/Invokes in Core and Libs behind the
same seam.
* [Host] Route PhysicalVirtualMemory through IHostMemory
Replace the class's private VirtualAlloc/VirtualFree/VirtualProtect/
VirtualQuery P/Invokes with IHostMemory calls. Every site maps 1:1 onto
the exact native call it issued before: MEM_COMMIT|MEM_RESERVE ->
Allocate, MEM_RESERVE -> Reserve, fault-path commits -> Commit, and
MEM_RELEASE -> Free, with identical protection values produced by the
Windows backend.
IHostMemory gains ProtectRaw so the save/restore protection sequences in
TryWriteExclusive and TryTemporarilyProtectForRead round-trip the raw OS
protection word (including modifier bits the neutral enum cannot
represent) exactly as before. Raw PAGE_* constants remain only for the
internal region-classification helpers, which only ever see values this
class itself assigned.
The exact-address free-on-mismatch, lazy reserve-only threshold, prime
loop, and all trace strings are unchanged.
* [Host] Add IGuestAddressSpace and retire the reflection-based allocator lookup
Introduce IGuestAddressSpace in SharpEmu.HLE (fixed-address AllocateAt /
TryAllocateAtOrAbove and guest mprotect via TryProtect) with signatures
copied from PhysicalVirtualMemory, which now implements it. TryProtect
reproduces the read/write/execute decomposition that
KernelMemoryCompatExports.ResolveHostProtection performs, yielding the
same PAGE_* values through the Windows backend.
KernelVirtualRangeAllocator previously located AllocateAt via cached
MethodInfo reflection (because SharpEmu.Libs cannot see Core types) and
walked wrapper memories through an untyped 'Inner' property. Both are
now typed: ICpuMemoryWrapper exposes the decorated memory (implemented
by TrackedCpuMemory, whose Inner property already existed) and the
allocator type-tests for IGuestAddressSpace with the same bounded
unwrap depth. Failure paths keep the exact [LOADER][TRACE] strings.
* [Host] Move Kernel HLE memory exports off direct kernel32 P/Invokes
KernelMemoryCompatExports loses its private VirtualQuery/VirtualProtect/
VirtualAlloc/VirtualFree declarations and MemoryBasicInformation struct:
- Guest mprotect (sceKernelMprotect/sceKernelMtypeprotect) now routes
through IGuestAddressSpace.TryProtect resolved from ctx.Memory. The
orbis read/write/execute decomposition moves into a GuestPageProtection
conversion whose mapping is value-identical to the removed
ResolveHostProtection.
- The guarded libc heap and host-page accessibility checks go through
IHostMemory (same commit+reserve/protect/free sequence; guard-page and
protection-mask checks compare HostRegionInfo.RawProtection against the
same PAGE_* literals as before).
- HostMemory is exposed as a property so merely loading the type never
resolves the platform backend on non-Windows hosts.
KernelRuntimeCompatExports' RDTSC stub allocates its 16-byte RWX page via
IHostMemory.Allocate; the OperatingSystem.IsWindows() gate returning null
is unchanged.
* [Host] Abstract thread, TLS, and symbol primitives in the execution backend
Add IHostThreading (native TLS slots, current-thread id, affinity, raw
thread create/join, diagnostic register capture) and IHostSymbolResolver
(enum-keyed host function addresses baked into emitted stubs), with
Windows implementations wrapping the exact kernel32 calls the backend
made directly before.
DirectExecutionBackend takes an optional IHostPlatform (defaulting to
HostPlatform.Current) and routes every TlsAlloc/TlsFree/TlsSet/GetValue,
GetCurrentThreadId, SetThreadAffinityMask, GetModuleHandle/GetProcAddress
and the suspend+GetThreadContext diagnostic snapshot through it. The
snapshot moves wholesale into WindowsHostThreading (including the Win64
CONTEXT size/flags/offsets, which are Windows-specific by nature) and
returns a neutral HostCapturedRegisters.
NativeGuestExecutor resolves WaitForSingleObject/SetEvent/ExitThread via
the symbol resolver — the same addresses end up in the emitted run loop,
so stub bytes are unchanged — and creates/joins its raw worker thread
through IHostThreading with the same stack-reservation semantics. The
run-loop emitter itself does not move.
Marshal.GetLastWin32Error() in the affinity-failure log still observes
SetThreadAffinityMask's error because the wrapper makes no intervening
SetLastError call.
* [Host] Move fault handling and remaining backend memory ops behind the seam
Add IHostFaultHandling (handler-thunk creation, first-chance handler
install/remove, unhandled-filter set) with WindowsFaultHandling in a new
Cpu/Native/Windows/ folder. The exception-handler trampoline emitter
moves there whole — same pre-filtered NTSTATUS codes, same TEB gs:[8]/
gs:[0x10] stack-limit reads, same host-RSP TLS switch — parameterized
only by (managed callback, TLS slot, TlsGetValue address), which is
exactly what SetupExceptionHandler passed it before. Handler
installation order, the AddVectoredExceptionHandler(first=1) flag, the
SHARPEMU_DISABLE_RAW_HANDLER gate, and all install/teardown log strings
are unchanged.
Every remaining VirtualAlloc/VirtualProtect/VirtualFree/VirtualQuery/
FlushInstructionCache in the backend partials routes through IHostMemory
with 1:1 call mapping (RWX emit -> RX downgrade -> flush for stub
emission, reserve/commit for the PRT aperture and lazy-commit fault
path, raw-protection round-trips via ProtectRaw). HostRegionInfo gains
RawState/RawAllocationProtection so the lazy-commit trace lines and
protection-mask checks keep printing and comparing the exact native
values.
Windows semantics leaked as bare literals become named constants with
identical values: NTSTATUS codes (WindowsFaultCodes) and Win64 CONTEXT
byte offsets (Win64ContextOffsets, with the existing CTX_* constants
aliased to it and handler-local numeric offsets replaced by the names).
* [Host] Resolve the host platform explicitly at the composition root
SharpEmuRuntime.CreateDefault() now resolves HostPlatform.Current once
and passes it explicitly to PhysicalVirtualMemory and (via a new
optional CpuDispatcher parameter) to DirectExecutionBackend, replacing
the implicit default-argument fallbacks. On unsupported OSes boot now
fails at the root with PlatformNotSupportedException and a clear
message instead of on the first native call. A future Linux/macOS
backend plugs in by returning a different IHostPlatform here.
* [Host] Convert the platform backends to source-generated P/Invokes
Replace [DllImport] with [LibraryImport] in the four Windows backend
files added by this branch (WindowsHostMemory, WindowsHostThreading,
WindowsHostSymbolResolver, WindowsFaultHandling). Marshalling stubs are
now generated at compile time instead of JIT-emitted at runtime, which
fits the pre-JIT-everything boot model and keeps the backends
NativeAOT/trimming ready.
Interop stays zero-copy: all signatures are blittable, GetModuleHandleW
now pins the managed string via Utf16 marshalling instead of copying,
and GetProcAddress names marshal through a stack-allocated Utf8 buffer.
Implicit contracts become explicit where LibraryImport requires it:
TlsFree/TlsSetValue gain [MarshalAs(UnmanagedType.Bool)] (the 4-byte
Win32 BOOL DllImport assumed silently), and GetModuleHandle targets the
W entry point directly since LibraryImport never probes suffixes.
The CONTEXT snapshot buffer stays a NativeMemory allocation rather than
stackalloc: CONTEXT requires 16-byte alignment, now documented at the
call site. Native call sequences are unchanged.
* [Host] Address Copilot review: harden failure paths, honor injected platform
- Free the handler thunk page when the RX protection downgrade fails
(the leak predates this branch, but the failure path is boot-fatal so
releasing the page is unobservable).
- TraceThreadMode and the static diagnostics helpers now resolve host
primitives through the backend bound to the current thread, falling
back to HostPlatform.Current only when no run is active (identical on
supported configs, honors injection everywhere a backend exists).
- HostPlatform.Create additionally requires an x64 process so native
Windows ARM64 fails with the promised PlatformNotSupportedException
instead of emitting x86-64 stubs into an ARM64 process.
Spooks