Act nowcan affect a running systemReviewed all 30 non-merge commits and excluded two merges. The total is 1,955 additions and 1,162 deletions after excluding mormot.commit.inc and mormot.commit-num.inc. Final source at 1f7febd219f9cb54175a1f87620e6862e4e670b8 was inspected. No commit carries superseded_by metadata.
AES-GCM authentication is restored while Tunnel and queue lifetimes are rebuilt
The most urgent fix restores GMAC verification in the AVX AES-GCM decrypt path, which could accept forged ciphertext, tags or associated data on eligible x86-64 systems. Tunnel lifetime, handshake and shutdown code is rebuilt to keep external calls outside locks and give the owner deterministic thread control. Queue and thread-pool waits gain OS futexes and new APIs, followed by fixes required for Delphi, Win8+ and Linux.
TL;DR
- Land on build 2.4.17356 at 1f7febd219f9cb54175a1f87620e6862e4e670b8.
- TAesGcmEngine.Decrypt now verifies GMAC after the AVX single-pass path; tests reject forged ciphertext, tags and AAD.
- TTunnelLocal now owns and joins its worker, transfers sockets explicitly and performs relay callbacks outside its state lock.
- TSynQueue uses OS futexes where available and adds WaitPopReset and Waiters while compared waits retain polling semantics.
- TSynThreadTasks, PendingTasks and WaitFor add managed task execution and completion waits; the final fixes cover Delphi, Win8+ and Linux.
- In-memory ZIP data descriptors, REST WebSocket callback teardown and THttpApiServer startup each receive production fixes.
Upgrade advice
Take the complete build 2.4.17356. Upgrade promptly when AES-GCM may use the x86-64 AVX path, when Tunnel or REST WebSocket clients shut down concurrently, or when THttpApiServer is used. Retest forged AES-GCM inputs, repeated Tunnel open/close cycles, queue finalization/reset, thread-pool WaitFor timeouts and HTTP.sys startup. The raw payload has no superseded_by markers, so no formal avoid range is asserted; do not cherry-pick the intermediate Tunnel or WaitFor commits.
30 of 30 entries · 32 commits upstream
1
Critical
AVX AES-GCM decryption no longer accepts unauthenticated data
TAesGcmEngine.Decrypt moves the final tag comparison after all single-pass branches, including AvxProcess. The dual-pass path marks an already verified tag so it is not checked twice.
Security
mormot.crypt.core
3c395af5+13-11
crypt: fixed missing GMAC check in TAesGcmEngine.Decrypt() AVX path
Impact
On eligible x86-64 systems using the AVX AES-GCM path, forged ciphertext, authentication tags or AAD could previously return success. Scalar and dual-pass paths already performed authentication.
Upgrade recommended
Upgrade immediately if TAesGcmEngine can dispatch to AVX, and add negative tests that alter ciphertext, tag and AAD.
Public API
TAesGcmEngine.Decryptbehaviour-changed
Units
2
High
Tunnel socket transfer and delayed frames become race-safe
OpenInternal takes its socket by var, clears it after thread ownership transfer, checks closure under the state lock and drains queued handshake frames outside the lock through RelayFrame.
Fixes
mormot.core.threads+1
1f37f3a4+111-82
net: fixed several races in mormot.net.tunnel
Impact
Open and OpenSocket avoid double-closing transferred sockets, and callbacks caused by delayed frames cannot re-enter while the parent state lock is held. Later Tunnel fixes complete this sequence.
Public API
TTunnelLocal.OpenSocketbehaviour-changed
3
High
Tunnel handshake waiting and close paths reach a stable lock model
TTunnelLocal replaces its reentrant lock with TOSLightLock, initializes and finalizes it explicitly, centralizes WaitForHandshake state retrieval and closes failed post-thread handshakes through ClosePort.
Fixes
mormot.net.tunnel+1
0900c1f2+75-68
net: still some fixes to mormot Tunnel
Impact
Concurrent handshake arrival, timeout, closure and destruction no longer depend on recursive locking or nullable event checks. The final code applies to Delphi and FPC on all Tunnel-supported platforms.
Upgrade recommended
Take the complete Tunnel sequence through 0900c1f2 and stress early frames, remote close, timeout and destruction during handshake.
Public API
TTunnelLocal.WaitForHandshakebehaviour-changedTTunnelLocal.ClosePortbehaviour-changed
4
High
Tunnel transport calls move outside the parent state lock
The parent lock is renamed fStateSafe, SendFrame captures counters and closed state under it, and TTunnelLocalThread routes outgoing frames through the parent helper before calling ITunnelTransmit.TunnelSend outside the lock.
Fixes
mormot.net.tunnel
7d9ec6c6+62-45
net: fixed TunnelSend() call
Impact
A transport implementation may block or re-enter without deadlocking the Tunnel state transition. This is an intermediate form later replaced by the final TOSLightLock design.
Units
5
High
In-memory ZIP readers can extract a final data-descriptor entry
TZipRead.Create points the last in-memory local entry at the following central directory. SearchFromDataDescriptor can therefore search backward without falling into the disk branch and dereferencing a nil fSource.
Fixes
mormot.core.zip+1
444fc47e+98-0
core: TZipRead in-memory access to a last entry with data descriptor
Impact
Opening a ZIP fully from memory no longer raises EAccessViolation when its last entry uses FLAG_DATADESCRIPTOR, a layout used by LibreOffice .xlsx and .ods files.
Upgrade recommended
Upgrade if ZIP, XLSX or ODS content is loaded into memory, and test extraction of the final archive entry.
Public API
TZipRead.Createbehaviour-changed
6
High
Tunnel workers become owned and joined by their parent
TTunnelLocalThread stops using FreeOnTerminate and a strong transmit reference. TTunnelLocal publishes a suspended worker, starts it explicitly, terminates it in ClosePort and frees it from the parent destructor after the inherited join.
Fixes
mormot.net.tunnel
be0c7136+42-35
net: fixed TTunnelLocalThread lifetime
Impact
Worker lifetime no longer races parent destruction or clears the owner's thread reference from the worker. Later commits are required to finish external-call and handshake locking.
Units
7
High
Tunnel forwarding no longer holds its state lock during external work
TunnelSend now uses the lock only for handshake and flag transitions, then forwards regular frames outside it. ClosePort captures notification state under the lock before aborting sockets and calling the transport.
Fixes
mormot.net.tunnel
bd4626d4+34-36
net: fixed Tunnel process thread safety
Impact
Transport callbacks, socket shutdown and frame delivery cannot deadlock by re-entering TTunnelLocal while its state lock is held. The subsequent commits refine closure ownership further.
Units
8
High
WaitFor now wakes on Win8+ and Linux task completion
TSynThreadPool adds fWaiters around the futex wait, and TaskAllDone calls OsWakeAllOnValue when the final pending task completes.
Fixes
mormot.core.threads
b7a92cc1+29-18
core: fixed TSynThreadPool.WaitFor on Win8+ and Linux
Impact
The newly added WaitFor no longer depends on timeout expiry after the pending count reaches zero on futex platforms. The following commit aligns the counter lifetime more precisely.
Public API
TSynThreadPool.WaitForbehaviour-changed
Units
9
High
Thread-pool completion waiters now match the futex lifecycle
TSynThreadPool.WaitFor increments fWaiters before computing its deadline and keeps the counter around the exact wait loop. TaskAllDone uses that counter to decide whether OsWakeAllOnValue is required.
Fixes
mormot.core.threads
fc943be2+8-8
core: tied up TSynThreadPool.WaitFor futex
Impact
Win8+ and Linux callers waiting for all work avoid a missed wake-up window. Other platforms continue using the one-millisecond fallback loop.
Upgrade recommended
Use the final WaitFor implementation from 2.4.17356 and test finite and INFINITE waits under concurrent completion.
Public API
TSynThreadPool.WaitForbehaviour-changed
Units
10
High
HTTP.sys servers resume automatic startup after construction
THttpApiServer.Create stops adding hsoCreateSuspended before calling THttpServerGeneric.Create. The base class can therefore set fStartAfterConstruction and start the server thread after the constructor chain completes.
Fixes
mormot.net.server
1f7febd2+5-6
net: fixed THttpApiServer.Create startup
Impact
Windows HTTP.sys backends created with default options no longer remain suspended after initialization. Delphi and FPC socket-based servers are not on this path.
Upgrade recommended
Upgrade to 2.4.17356 and verify that each THttpApiServer begins accepting requests immediately after construction.
Public API
THttpApiServer.Createbehaviour-changed
Units
11
High
REST WebSocket callbacks are released before their owner disappears
TRestClientUri.Destroy moves FreeAndNilSafe(fFakeCallbacks) into the inner finally immediately after InternalClose and before inherited Destroy releases the REST model and services.
Fixes
mormot.rest.client
376e6934+5-5
rest: fixed TRestClientUri WebSockets callbacks finalization
Impact
Concurrent WebSocket shutdown no longer leaves callback storage referring to an owner already being destroyed. The fix applies to Delphi and FPC REST clients using WebSocket callbacks.
Upgrade recommended
Upgrade and stress repeated connect, callback registration, disconnect and TRestClientUri destruction from concurrent threads.
Public API
TRestClientUri.Destroybehaviour-changed
Units
12
Medium
Queue futex use is limited to true unfiltered consumers
TSynQueue avoids futex waits for PopEquals and WaitPeekLocked, preserves their polling semantics, prevents a zero sequence after overflow and resets the sequence after binary loading.
Fixes
mormot.core.threads
53ff22a2+32-21
core: some TSynQueue fixes and enhancements
Impact
Filtered waits do not sleep indefinitely when a nonmatching item is at the head, while ordinary WaitPop retains efficient OS blocking.
Public API
TSynQueue.WaitPopbehaviour-changedTSynQueue.WaitPeekLockedbehaviour-changed
Units
13
Medium
A simple object-owning task pool joins the thread primitives
TSynThreadTask defines DoExecute, and TSynThreadTasks executes those objects through TSynThreadPool. Add always takes ownership, freeing rejected tasks and using TaskAbort for queued work during shutdown.
New features
mormot.core.threads+1
27d2e21b+77-5
core: new TSynThreadTasks simple thread-pool class
Impact
Delphi and FPC applications can submit heap-owned task objects without writing worker-thread plumbing. Callers must not free a task after passing it to Add, even when Add returns false.
Worth a look
When adopting TSynThreadTasks, remove caller-side Free calls after Add and implement DoExecute without assuming task reuse.
Public API
TSynThreadTaskaddedTSynThreadTasksadded
14
Medium
Thread pools expose total work and a final-task hook
TSynThreadPool tracks queued plus executing work in fPendingTasks, publishes PendingTasks, and calls the protected virtual TaskAllDone when the counter reaches zero after execution or abort.
New features
mormot.core.threads
f54b3fa0+47-20
core: new TSynThreadPool.PendingTask property
Impact
Monitoring can distinguish queued contexts from all outstanding work, and descendants can react when the pool becomes idle. Counter maintenance applies to IOCP and POSIX paths.
Worth a look
Review custom TSynThreadPool descendants that need idle notification and use the final 2.4.17356 counter semantics.
Public API
TSynThreadPool.PendingTasksaddedTSynThreadPool.TaskAllDoneadded
Units
15
Medium
Crypto acceleration checks become reusable runtime APIs
TX64CpuFeatures gains cpuAESGCM, computed from CLMUL, SSE4.1 and AES-NI. HasAesGcmAvx and HasKeccakAvx2 also account for DisabledAsm before reporting an accelerated implementation.
New features
mormot.core.base+1
7ee3b216+34-12
core: introducing cpuAESGCM and other crypto HW test functions
Impact
Delphi and FPC x86-64 code can query the same final dispatch decision used by the crypto engine instead of duplicating CPU tests.
Worth a look
Use HasAesGcmAvx or HasKeccakAvx2 when diagnostics or tests must follow actual runtime dispatch.
Public API
cpuAESGCMaddedHasAesGcmAvxaddedHasKeccakAvx2added
16
Medium
Finalized queues can be reactivated and report waiting consumers
TSynQueue.WaitPopReset clears the destroying flag only after all prior WaitPop calls unregister. TSynQueue.Waiters returns the protected waiter count.
New features
mormot.core.threads
808c57b2+35-3
core: new TSynQueue.WaitPopReset/Waiters methods
Impact
A queue stopped through WaitPopFinalize can be reused without reallocating stored items, provided reset succeeds. Monitoring can observe active consumers safely.
Worth a look
Call WaitPopReset only after WaitPopFinalize returns and handle false while an earlier waiter is still terminating.
Public API
TSynQueue.WaitPopResetaddedTSynQueue.Waitersadded
Units
17
Medium
Thread pools can wait until every accepted task is finished
TSynThreadPool.WaitFor watches PendingTasks until zero, using OsWaitOnValue when available and a short sleep fallback elsewhere. It supports finite timeouts and INFINITE.
New features
mormot.core.threads
c7512a04+34-2
core: new TSynThreadPool.WaitFor method
Impact
Applications gain a completion barrier for accepted pool work. This first implementation needs the three following platform and wake-up fixes, so it should not be cherry-picked alone.
Public API
TSynThreadPool.WaitForadded
Units
18
Medium
Queue consumers block on native OS wait primitives
TSynQueue adds a sequence counter and uses OsWaitOnValue, OsWakeOnValue and OsWakeAllOnValue on Win8+ and Linux. Push wakes one registered consumer; finalization and binary load wake all.
Performance
mormot.core.threads+1
ac7c87e4+71-27
core: rewritten TSynQueue to use the OS futex if possible
Impact
Ordinary WaitPop no longer spins while idle on supported systems, reducing CPU use and wake latency. Unsupported platforms retain the existing SleepStep fallback.
Public API
TSynQueue.WaitPopbehaviour-changed
19
Medium
WaitFor becomes visible to Delphi IOCP builds
The TSynThreadPool.WaitFor declaration moves outside the non-IOCP conditional section, matching its implementation and allowing Delphi Windows builds to compile.
Compiler & platform
mormot.core.threads+1
5541a4f1+5-6
core: fixed Delphi compilation
Impact
Intermediate builds with the new method could fail to compile for the Windows IOCP configuration. FPC and non-IOCP declarations were already visible.
20
Medium
Tunnel closure state is consolidated into one protected flag set
The separate fClosed boolean is folded into TTunnelLocalFlags, and frame handling is split into handshake-state capture and RelayFrame processing. CallbackReleased maps remote release directly to ClosePort(FromNotify=true).
Under the hood
mormot.net.tunnel
d60cc6e8+46-48
net: internal Tunnel flags refactoring
Impact
Closure state becomes atomic with the other protected flags and remote notification is idempotent. This is an intermediate refactor followed by additional race fixes.
Units
21
Low
Test failures gain an exception-aware reporting helper
TSynTestCase.FailedRaised formats an Exception through ObjectToJsonDebug before calling TestFailed. Stack-symbol filtering also removes mormot.core.log and mormot.core.test redirection frames.
New features
mormot.core.log+2
311e29ed+45-36
core: introducing TSynTestCase.FailedRaised method
Impact
Custom mORMot test suites can report unexpected exceptions consistently, with cleaner caller locations. Production runtime paths are unchanged.
Public API
TSynTestCase.FailedRaisedadded
22
Low
Internal atomic increments use one pointer-based helper
Four InterlockedIncrement calls in sequence generation, thread-pool accounting, ECC reference counting and tests are replaced by LockedInc32 with explicit addresses.
Compiler & platform
mormot.core.base+3
883ee3ae+9-9
core: use LockedInc32() instead of InterlockedIncrement()
Impact
Behavior is unchanged; the code uses the framework atomic wrapper consistently across Delphi and FPC targets.
23
Low
Queue concurrency tests adopt the new owned-task pool
test.core.threads replaces dedicated push and wait thread classes with TSynThreadTask descendants executed by one TSynThreadTasks instance, preserving finalize, reset and multi-waiter coverage.
Tests
test.core.threads
c18eb6c5+340-333
test: rewrite TSynQueue using TSynThreadTasks
Impact
Application behavior is unchanged. The regression suite now exercises the new task-pool ownership model while validating TSynQueue concurrency.
Units
24
Low
Queue regression coverage moves into the thread test unit
The complete TSynQueue value, persistence and worker tests move from test.core.base into test.core.threads, next to the concurrency primitives they exercise.
Tests
test.core.base+2
d73f877c+200-233
test: moved TSynQueue tests to test.core.threads.pas
Impact
Application behavior is unchanged; test organization now matches the implementation area.
25
Low
Queue futex paths gain concurrent wake and shutdown tests
test.core.threads adds delayed producers, eight simultaneous WaitPop consumers, WakeOne distribution, WaitPopFinalize WakeAll behavior and post-finalization rejection checks.
Tests
test.core.threads
db598d3b+257-4
test: completed TSynQueue coverage
Impact
No application code changes. Windows and Linux futex behavior is now exercised with actual concurrent waiters.
Units
26
Low
Queue reset receives repeated multi-waiter regression coverage
test.core.threads adds finalize/reset cycles with eight waiting threads, verifies wake-one delivery after reset and confirms that a second finalization and reset leave the queue operational.
Tests
test.core.threads
3e909874+121-40
test: enhanced TSynQueue coverage
Impact
Application behavior is unchanged. The new WaitPopReset contract is exercised under concurrent sleepers instead of only sequential calls.
Units
27
Low
Regression tests report unexpected exceptions with structured details
ZIP, network and ORM tests replace ad-hoc Check(false, E.Message) handlers with TSynTestCase.FailedRaised. The ZIP data-descriptor fixture is rewritten into a fixed temporary buffer.
Tests
mormot.core.test+4
fc0d9757+80-75
test: some cleanup especially after #630
Impact
Application behavior is unchanged. Test failures retain exception class and structured fields, making regressions easier to diagnose.
28
Low
Crypto tests select accelerated paths through public capability helpers
TTestCoreCrypto replaces compile-time AVX assumptions with HasKeccakAvx2 and HasAesGcmAvx, extends forged AES-GCM cases and verifies plaintext is not mistaken for authenticated output.
Tests
test.core.crypt
f9ad68c8+43-25
test: refined TTestCoreCrypto content
Impact
Application behavior is unchanged. Tests now exercise only acceleration supported by both hardware, OS state and DisabledAsm settings.
Units
29
Low
AES-GCM tests reject altered ciphertext, tags and associated data
_AES_GCM encrypts and verifies lengths 1 through 64 in scalar and available AVX modes, then independently modifies ciphertext, the authentication tag and AAD and requires each verification to fail.
Tests
test.core.crypt
5f464423+24-1
test: cover forged AES-GCM ciphertext, tag and AAD in _AES_GCM
Impact
Application behavior is unchanged. The regression directly covers the authentication bypass fixed in the preceding commit.
Units
30
Low
Tunnel locking and ownership rules are documented beside the code
mormot.net.tunnel records the concurrency rules used by the implementation: short state locks, no external calls under locks, owner-managed workers and two-phase shutdown. TTunnelLocalThread.Abort also tolerates a nil self reference.
Docs
mormot.net.tunnel
2ad1823e+27-2
net: add some comments about Tunnel internal algorithms & patterns
Impact
The comments do not change application behavior. The nil guard makes internal shutdown calls defensive.