Act nowcan affect a running system
FPC ARM atomics regain ordering as HTTP.sys WebSockets are rebuilt
This edition repairs missing memory barriers in mORMot locks on FPC ARM and AArch64, where protected data could still race despite the lock word. On Windows, HTTP.sys request and WebSocket handling is extensively rebuilt, then stabilized across connection lifetime, overlapped I/O, sending, shutdown and ping paths. Direct users of low-level HTTP.sys wrappers and THttpApiWebSocketServerProtocol connection indexes have source migrations.
TL;DR
- FPC ARM and AArch64 lock operations now impose full memory barriers; prior code could lose updates or corrupt shared state.
- The HTTP.sys request engine now handles chunked unlimited bodies, disconnects unread-body failures and isolates per-request state.
- HTTP.sys WebSocket connection management now uses TLockedList IDs, distinct receive/wake overlapped records and guarded shutdown.
- THttpServerRequest.FullUrl is now RawUtf8, HttpApiRequest is removed, and several low-level HTTP.sys signatures changed.
- optCleanMagicJson can strip JSON_BASE64_MAGIC from SOA responses; SetErrorText, HtmlEscapeShortVar and TLockedList.Find are new.
- Land on build 2.4.17497; do not stop on the HTTP.sys transition builds 2.4.17464 through 2.4.17496.
Upgrade advice
Land on final build 2.4.17497. FPC ARM/ARM64 deployments should upgrade for the memory-ordering repair and stress-test locks, thread pools and timers. Windows HTTP.sys/WebSocket deployments should avoid builds 2.4.17464–2.4.17496, migrate changed public symbols, then test chunked uploads, upgrades, concurrent send/close, client disconnects, shutdown and ping timeouts.
32 of 32 entries · 35 commits upstream
1
Critical
ARM atomic wrappers centralize the memory-barrier repair
The follow-up defines barrier-aware InterlockedIncrement and InterlockedDecrement wrappers for FPC weak-atomic targets, adds AArch64 assembly entry points, and routes lock, async connection and ECC reference counters through the corrected primitives.
Fixes
mormot.core.base+4
c1a131a8+158-133
core: huge refactoring on arm32/64 about atomic operations
Impact
This is the stabilized implementation for FPC ARM32/ARM64 ordering. It protects concurrent backend state beyond the lock word and is especially important for thread pools, timers and reference-counted work.
Upgrade recommended
FPC ARM/ARM64 production systems should move to 2.4.17497 and rerun concurrency and lifetime stress tests.
2
Critical
FPC non-Intel locks become full memory barriers
FPC_WEAKATOMICS marks non-Intel targets, custom AArch64 LDXR/STLXR paths add DMB ordering, and other FPC CPUs wrap RTL atomics with barriers across TLightLock, TRWLightLock, TRWLock, TMultiLightLock and TOSLightLock.
Fixes
mormot.core.base+2
b2b195da+190-8
core: Locked*() are now full memory barriers on FPC non-Intel CPUs
Impact
On FPC ARM and AArch64, lock-word exclusion previously did not order protected memory. The commit documents observed lost increments, lost TSynThreadPool contexts and TSynBackgroundTimer heap corruption.
Upgrade recommended
Treat this as mandatory for concurrent FPC ARM/ARM64 services, but use the final build containing the follow-up refactor.
3
Critical
The unsafe WebSocket connection class rewrite is reverted
This commit fully reverts the preceding class-based connection-list rewrite because TOverlapped must remain the first bytes of the structure passed through IOCP.
Fixes
mormot.net.server
1ff6986c+120-69
revert "net: rewritten http.sys WS connections list"
Impact
It removes an invalid Windows runtime layout that could make completion callbacks interpret object metadata as TOverlapped. The later record and TLockedList work supersedes it.
Upgrade recommended
Do not use the immediately preceding revision; land on the final build 2.4.17497.
Units
4
Critical
A class-based WebSocket connection list introduces an invalid IOCP layout
The commit temporarily converts THttpApiWebSocketConnection from a TOverlapped-rooted record/object into a class and restructures protocol connection arrays. It is completely reverted by the next non-merge commit.
Under the hood
mormot.net.server
91f43c69+69-120
net: rewritten http.sys WS connections list
Impact
This revision must not be deployed: IOCP requires TOverlapped at offset zero, which a class instance does not provide. It has no lasting API effect after the revert.
Upgrade recommended
Skip this revision and use the final build 2.4.17497.
Units
5
High
HTTP.sys WebSocket connections move from indexes to stable IDs
THttpApiWebSocketServerProtocol stores connections in TLockedList, gives each THttpApiWebSocketConnection an ID, and makes Send and Close accept THttpApiWebSocketConnectionID. Separate read and wake overlapped records coordinate IOCP lifecycle and shutdown.
Breaking changes
mormot.net.server
8bcdf7e7+281-196
net: http.sys WebSockets server now uses TLockList
Impact
Code that retained an array index or called protocol Send/Close by integer must retain Conn.ID and pass that stable ID instead. The runtime gains ABA-resistant recycling and safer concurrent shutdown.
Migration required
Search for THttpApiWebSocketServerProtocol.Send/Close and stored connection indexes; capture THttpApiWebSocketConnection.ID in callbacks instead.
Public API
THttpApiWebSocketConnectionIDaddedTHttpApiWebSocketConnection.IDaddedTHttpApiWebSocketServerProtocol.Sendsignature-changedTHttpApiWebSocketServerProtocol.Closesignature-changed
Units
6
High
Per-request WebSocket state replaces a shared last connection
THttpApiWebSocketServer stores the pending connection on THttpApiServerRequest instead of fLastConnection, making concurrent upgrades request-local. THttpServerRequest.FullUrl becomes virtual RawUtf8 and the public HttpApiRequest property is removed.
Breaking changes
mormot.lib.winhttp+1
c852c8cf+92-103
net: several fixed to THttpApiWebSocketServer
Impact
The documented non-thread-safe shared state is removed. Code expecting FullUrl as SynUnicode must use RawUtf8, and code reading HttpApiRequest must move to supported request fields or an internal Windows-specific extension.
Migration required
Search for FullUrl declarations/assignments and HttpApiRequest references; migrate before compiling, then test concurrent upgrades.
Public API
THttpServerRequest.FullUrlsignature-changedTHttpServerRequest.GetFullUrlsignature-changedTHttpServerRequest.HttpApiRequestremoved
7
High
Low-level HTTP.sys helpers change their output contracts
HTTP_RESPONSE.SetStatus now returns PRawUtf8 instead of filling a var RawUtf8, RetrieveHeadersAndGetRemoteIPConnectionID becomes a procedure with var Headers, and public LookupTokenVar plus VERB_TEXT are added.
Breaking changes
mormot.core.os.security+1
9c149153+41-16
lib: some http.sys API wrapper refinements
Impact
Direct users of these low-level Windows wrappers will fail to compile until call sites adopt the new return and output parameters. Typical high-level THttpServer users are unaffected.
Migration required
Search for SetStatus and RetrieveHeadersAndGetRemoteIPConnectionID calls, remove temporary status strings and pass a Headers variable.
Public API
HTTP_RESPONSE.SetStatussignature-changedRetrieveHeadersAndGetRemoteIPConnectionIDsignature-changedLookupTokenVaraddedVERB_TEXTadded
8
High
Failed WebSocket upgrades and receives now drain safely
The server finalizes failed upgrades, rejects new upgrades while stopping, validates complete writes, permits only one pending receive per connection, completes failed receive actions with zero bytes and delays recycling until pending I/O is gone.
Fixes
mormot.net.server
97ab4cc4+64-24
net: some THttpApiWebSocketServer fixes
Impact
Windows HTTP.sys WebSocket error and shutdown paths no longer recycle records while completions may still arrive. No new source migration is introduced.
Upgrade recommended
Upgrade to the final build and test failed handshake, abrupt disconnect and server shutdown under load.
Units
9
High
websocket.dll completion is treated as a void operation
WebSocketApi.CompleteAction is corrected from an HRESULT-returning function to a procedure, result checks are removed, and receive processing starts only after the HTTP 101 switching-protocols response.
Fixes
mormot.lib.winhttp+1
6f0344bf+19-45
net: fixed WebSocketApi.CompleteAction usage
Impact
The Windows declaration now matches websocket.dll and avoids consuming a nonexistent return value. Upgrade sequencing also stops reads before the HTTP handshake is committed.
Upgrade recommended
Use the corrected declaration and validate WebSocket upgrade success and failure.
Public API
WebSocketApi.CompleteActionsignature-changed
10
High
HTTP.sys response state and Range failures are hardened
The server marks responses before submission, exits after static-file open failure, returns 416 when a Range starts beyond the file, catches OnBeforeBody exceptions and cleans response state consistently.
Fixes
mormot.net.server
3b4da68d+34-23
net: some THttpApiServer fixes and cleanup
Impact
Windows HTTP.sys applications avoid duplicate or continued response work after failures and produce the correct status for invalid byte ranges.
Upgrade recommended
Upgrade and retest static-file Range requests plus OnBeforeBody exception handling.
Units
11
High
Concurrent WebSocket send and close operations are serialized
THttpApiWebSocketServerProtocol now locks around connection Close and Send, and deletion from the pending-close array receives the count pointer rather than the array pointer.
Fixes
mormot.net.server
3383af99+28-18
net: some THttpApiWebSocketServerProtocol fixes
Impact
Windows WebSocket backends avoid races between application threads and protocol maintenance, plus memory corruption from updating the wrong pending count.
Upgrade recommended
Upgrade rather than keeping the preceding connection-list refactor, and stress concurrent Send, Close and disconnect.
Units
12
High
The WebSocket send declaration catches up with its implementation
The missing mormot.net.server declaration update makes WriteData return success and the written count, removes obsolete shared-overlapped send use, and sends response entity bodies synchronously.
Fixes
mormot.net.server
c9bcf0ea+16-24
net: missing file in previous commit
Impact
The previous commit was incomplete without this unit and could fail to compile or retain mismatched send behavior. No public application API changes.
Upgrade recommended
Do not stop on the preceding revision; use the completed final series.
Units
13
High
HTTP.sys closes connections that still contain unread request bodies
SendError can now add Connection: close and HTTP_SEND_RESPONSE_FLAG_DISCONNECT. Oversize bodies, bot rejection, OnBeforeBody aborts and entity-body read errors use that path.
Fixes
mormot.net.server
6a2970d5+21-9
net: properly disconnect http.sys client when needed
Impact
The server no longer reuses an HTTP.sys connection whose request body was not fully consumed, preventing the next request from inheriting a poisoned stream state.
Upgrade recommended
Retest rejected uploads and keep-alive behavior after upgrading.
Units
14
High
Ping failures no longer kill the WebSocket guard
THttpApiWebSocketConnection.CheckIsActive now ignores inactive states, closes expired peers before pinging again, and catches Ping exceptions to close only the affected connection.
Fixes
mormot.net.server
21b62bc8+18-9
net: fixed http.sys ping process
Impact
Windows HTTP.sys WebSocket services retain their guard thread when a ping fails and avoid pinging connections already closing. Delphi and FPC application APIs are unchanged.
Upgrade recommended
Use build 2.4.17497 and exercise ping timeout and shutdown paths.
Units
15
High
HTTP.sys entity-body receive now matches the Windows API contract
The ReceiveRequestEntityBody declaration changes pBytesReceived from var cardinal to PCardinal. Synchronous request reads pass @bytesread, overlapped WebSocket reads pass nil, and write errors explicitly clear their byte count.
Fixes
mormot.lib.winhttp+1
e795d965+14-13
net: fixed http.sys API incorrect usage
Impact
The wrapper now supports the null output pointer required for overlapped I/O. Direct callers of the function-pointer field must pass an address or nil; the same commit temporarily inverted a send test, fixed next.
Migration required
Update direct ReceiveRequestEntityBody calls to pass @count or nil, and land on 2.4.17497 rather than this intermediate revision.
Public API
THttpApiFunction.ReceiveRequestEntityBodysignature-changed
16
High
WebSocket writes and peer-close replies take the corrected paths
WriteData now reports success only for a complete HTTP.sys entity-body write, the send loop tests failure with the correct polarity, and a client close frame receives the matching server close reply.
Fixes
mormot.net.server
072bba0e+11-11
net: THttpApiWebSocketConnection fixes
Impact
This corrects failures introduced during the WebSocket rewrite: builds before this fix could abort successful sends or mishandle close handshakes on Windows.
Upgrade recommended
Do not deploy the intermediate send-path build; use 2.4.17497 and test send plus peer close.
Units
17
High
WebSocket network sends report the bytes actually written
WEB_SOCKET_SEND_TO_NETWORK_ACTION now writes each websocket.dll buffer, accumulates the actual byte count for CompleteAction and disconnects when an HTTP.sys entity-body write fails.
Fixes
mormot.net.server
4730c107+9-6
net: fixed WEB_SOCKET_SEND_TO_NETWORK_ACTION with http.sys
Impact
Windows HTTP.sys WebSocket sends no longer claim the requested length when fewer bytes reached the network, improving failure detection and protocol state.
Upgrade recommended
Use the final build and test large or fragmented sends plus forced disconnects.
Units
18
High
Unlimited chunked HTTP.sys bodies no longer receive 413
The content-length guard now enforces MaximumAllowedContentLength only when that limit is greater than zero, so zero retains its documented unlimited meaning for chunked bodies.
Fixes
mormot.net.server
7b05e490+4-3
net: fixed #642 THttpApiServer chunked body gets 413
Impact
Windows HTTP.sys endpoints configured with the default unlimited value accept chunked uploads instead of rejecting them with HTTP 413.
Upgrade recommended
Upgrade if chunked requests use MaximumAllowedContentLength=0, and retest both limited and unlimited uploads.
Units
19
High
HTTP.sys request processing becomes a per-request object
The monolithic THttpApiServer.DoExecute path is split into a private THttpApiServerRequest with request, response, buffers, logging and send/error methods. Per-thread reuse remains, while response and exception state is made explicit.
Under the hood
mormot.net.http+1
5adc88f8+507-518
net: huge THttpApiServer refactoring/rewrite
Impact
Windows HTTP.sys backends gain clearer request isolation and error handling, but this is the start of a multi-commit rewrite and should not be treated as a standalone landing point.
Worth a look
20
High
WebSocket connection records keep overlapped storage at offset zero
Connections return to record/object storage with TOverlapped as the first field, while allocation, lookup and pending-close arrays are reorganized around pointers.
Under the hood
mormot.net.server
22a41512+61-94
net: some THttpApiWebSocketConnection refactoring
Impact
This restores the layout required by IOCP callbacks after the reverted class attempt, but remains an intermediate architecture subsequently replaced by TLockedList.
Worth a look
Units
21
Medium
Malformed compressed HTTP content no longer escapes as an exception
THttpSocketCompressList.UncompressContent wraps decompressor execution in try/except, clears the result and returns nil when a codec raises.
Fixes
mormot.net.http
f1ec718b+11-7
net: hardened THttpSocketCompressList.UncompressContent
Impact
Malformed gzip/deflate input becomes a normal decompression failure instead of propagating into a Delphi/FPC HTTP request path. Callers should already handle nil.
Upgrade recommended
Verify that custom callers treat nil as invalid compressed content rather than relying on a codec exception.
Public API
THttpSocketCompressList.UncompressContentbehaviour-changed
Units
22
Medium
Locked lists can resolve ABA-resistant sequence identifiers
A public TLockedListID alias types sequence fields and TLockedList.Find scans under the list lock for a nonzero ID. The 32-bit FPC LockedInc64 carry path also uses LockedInc32 for the required barrier.
New features
mormot.core.base+1
4824b5db+32-8
core: new TLockedList.Find method
Impact
Infrastructure can look up recycled list records without relying on unstable pointers. Existing Delphi/FPC callers are unaffected unless adopting the new helper.
Worth a look
Public API
TLockedListIDaddedTLockedList.Findadded
23
Medium
SOA methods can return JSON without binary magic prefixes
TInterfaceMethodOption gains optCleanMagicJson. TInterfaceFactoryServer.ExecuteJson invokes RemoveMagicFromJson on the response when the option is present.
New features
mormot.core.interfaces+1
2ff79bbd+6-2
soa: new optCleanMagicJson option for #402
Impact
Selected interface methods can emit JSON strings without the internal JSON_BASE64_MAGIC prefix. Existing services retain their current output unless they opt in.
Worth a look
Public API
TInterfaceMethodOption.optCleanMagicJsonadded
24
Medium
JSON magic removal is rewritten as a faster copy-on-write scan
RemoveMagicFromJson now scans string boundaries first, detects JSON_BASE64_MAGIC prefixes and allocates a writable copy only when a marker must be removed.
Performance
mormot.core.json
367cdf5a+46-43
core: rewritten RemoveMagicFromJson()
Impact
SOA and direct callers keep the same public procedure but avoid copying ordinary JSON. Behavior remains focused on magic-prefixed JSON strings.
Worth a look
Units
25
Low
System error text can be assigned directly into RawUtf8
A public SetErrorText procedure fills an existing RawUtf8 from an OS error code, defaulting to the current error. GetErrorText delegates to it and HTTP code reuses the new helper.
New features
mormot.core.os+2
f47c802c+263-237
core: new SetErrorText() function
Impact
Callers can reuse a destination string and avoid a result temporary; existing GetErrorText code remains compatible across Delphi and FPC.
Public API
SetErrorTextadded
26
Low
HTML escaping can fill a caller-owned short string
The public HtmlEscapeShortVar procedure writes escaped RawUtf8 text into a ShortString with the selected format; HtmlEscapeShort now delegates to the wrapper.
New features
mormot.core.fmt
5ec9601b+14-4
core: new HtmlEscapeShortVar() wrapper function
Impact
Delphi and FPC callers that already own the destination can avoid a function result temporary. Existing calls remain source-compatible.
Public API
HtmlEscapeShortVaradded
Units
27
Low
Async networking receives a non-functional cleanup
mormot.net.async is reformatted and simplified without changing the executed branches or public declarations.
Under the hood
mormot.net.async
29b15f39+17-17
net: some code cleanup with no functional change
Impact
No impact on Delphi or FPC application code is expected from this cleanup.
Units
28
Low
The rewritten HTTP.sys request object receives a cleanup pass
The follow-up simplifies and formats the new THttpApiServerRequest implementation after the large request-processing rewrite without changing its intended behavior.
Under the hood
mormot.net.server
6dfbaf34+13-17
net: THttpApiServer cleanup after previous commit
Impact
No application-code impact is visible in the diff; it is maintenance within the Windows HTTP.sys implementation.
Units
29
Low
The example documents its interface and implementation folders
The SAD, README and project guides describe the app/<entity>/interface and app/<entity>/implementation split, dependency direction and local-mode exception.
Docs
recommended-patterns documentation
76e144c0+147-119
Document the app/<entity>/interface|implementation layout
Impact
No mORMot runtime behavior changes; teams following the example receive clearer architectural guidance.
30
Low
CQRS layout documentation incorporates review corrections
Architecture and development documents clarify interface/implementation dependencies and adjust the example service references after review.
Docs
recommended-patterns documentation
4ecaf74e+16-15
Address review notes on the interface/implementation docs
Impact
No runtime or public framework API impact; the commit only improves the recommended-patterns explanation.
31
Low
Example build instructions and dependency lists are corrected
The recommended-patterns documentation removes stale build steps and aligns dependency lists with the reorganized example source tree.
Docs
recommended-patterns documentation
5b856937+22-8
Fix stale build steps and dependency list in the example docs
Impact
No framework or application runtime changes; only developers building the example see corrected guidance.
32
Low
The CQRS example separates contracts from implementations
Recommended-patterns query, command and DTO units move into app/<entity>/interface, while mappers and handlers move into implementation; Delphi and FPC project paths are updated.
Housekeeping
recommended-patterns example
323eb2c9+70-62
Move CQRS interfaces from dom/ to app/<entity>/interface
Impact
Only the example project layout and its build paths change. Framework users and production backends are unaffected unless they copied this example structure.