Skip to content

test: harden the allocation counter, benchmarks, and golden output - #29

Merged
Reddimus merged 2 commits into
mainfrom
test/support-fixes
Sep 25, 2026
Merged

Reddimus merged 2 commits into
mainfrom
test/support-fixes

Conversation

@Reddimus

Copy link
Copy Markdown
Owner

Summary

The benchmark suite's allocation counter could report wrong numbers without failing, and two test helpers could hide problems. This fixes both. The spc-cpp release review found most of these. No library code changes.

  • With LTO on macOS, ld64 internalized the replacement operator new, so allocations made inside libc++ skipped the counter: BM_Parse/storm_reports reported 5,482 allocations instead of 7,602. All 20 replacement operators are now marked used with default visibility on GCC and Clang.
  • The benchmarks now check at startup that the counter sees one direct operator new call and one allocation made inside the standard library. If it misses either, they print the likely cause and exit 1. Before, every counter read 0 under Valgrind and the run still exited 0. CONTRIBUTING now names the Valgrind flag that fixes this.
  • Configuring with SPC_BUILD_BENCHMARKS and either sanitizer option now stops with an error. A static sanitizer runtime also defines operator new, so the link used to fail with "multiple definition".
  • CountsEveryOperatorNewAndDeleteForm calls all 8 new forms and all 12 delete forms, up from 5 and 6.
  • The standard-library test grew a std::string, which GCC inlines from C++20 on, so on GCC it never called into libstdc++. It now constructs a std::runtime_error, a library call on both libc++ and libstdc++.
  • make bench runs CMake directly, as make tidy does, so CMAKE_ARGS='-DCMAKE_OSX_ARCHITECTURES="arm64;x86_64"' is no longer split at the semicolon.
  • A golden-test failure now escapes both excerpts, so a stray carriage return shows as \x0d instead of two lines that look the same.
  • Each golden renderer destructures its model with a structured binding, so a new field fails to compile until the renderer prints it. The golden files are unchanged.
  • Golden coverage skips dotfiles such as .DS_Store, and AllocationProbe zeroes its counters before it starts counting, so an allocation on another thread can't be counted and then wiped.

Testing

  • LTO on macOS (Apple Clang 21): before, SeesAllocationsMadeInsideTheStandardLibrary failed 0 vs 1 and nm -m no longer listed __Znwm as external. Now 130/130 pass, and all 28 benchmarks report the same heap counters as a build without LTO. With the attribute removed, the benchmarks exit 1 with the LTO message.

  • Valgrind 3.22 with GCC 13: before, memcheck and massif reported allocs=0 and exited 0. Now both exit 1 with the Valgrind message, and with --soname-synonyms=somalloc=nouserintercepts they report 9,439 allocations for BM_Parse/storm_reports, the same as a native run.

  • Linking the counter against GCC's static TSan runtime reproduces the "multiple definition of operator new" error; configuring with benchmarks and a sanitizer now fails at CMake time instead.

  • make bench with the universal archs builds an arm64 and x86_64 binary and runs every benchmark on both slices, x86_64 under Rosetta 2. Before, the shell ran x86_64 as a command.

  • Adding a field to StormReport now fails to compile in payload_text.hpp; SPC_UPDATE_GOLDEN=1 leaves tests/golden unchanged.

  • macOS: make test && make lint && make lint-md && make fixtures-check && make test-consumers pass (130/130), ASan and UBSan pass 123/123, and Apple Clang 17 builds with -Werror for macOS 13.4 and passes 130/130.

  • Ubuntu 24.04 in Docker: GCC 13 with -Werror and benchmarks (130/130 and the dry run), Clang 18 with libc++ 18 (130/130), and make tidy as CI runs it all pass. Clang 19 with libstdc++ wasn't available locally, so its CI job is the first check there.

  • make test and make lint pass

  • CHANGELOG.md: no entry, since only tests and build tooling change

  • No public API changes

Export the replacement operator new and delete, so LTO cannot internalize
them and leave allocations inside the standard library uncounted. The
benchmarks now check at startup that the counter sees an allocation made
directly and one made inside the standard library, and exit with the
likely cause when it does not, as under Valgrind. Configuring the
benchmarks with a sanitizer now fails, since both replace operator new.

The counter test covers all eight new forms and twelve delete forms, and
the standard-library test uses std::runtime_error, whose message copy is a
library call on libc++ and libstdc++. GCC inlines std::string growth.

make bench calls cmake directly, so CMAKE_ARGS keeps its quoting. Golden
diffs escape their excerpts, so a stray carriage return shows, and each
golden renderer destructures its model, so a new field breaks the build
until it is printed.
Copilot AI lite review requested due to automatic review settings September 25, 2026 22:07

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

@Reddimus
Reddimus merged commit d7d406d into main Sep 25, 2026
12 checks passed
@Reddimus
Reddimus deleted the test/support-fixes branch September 25, 2026 22:12
Reddimus added a commit that referenced this pull request Sep 25, 2026
Sets the version to 0.4.2 in `include/spc/version.hpp`, dates the
changelog section, and points the README's `GIT_TAG` at v0.4.2, which
`make test-consumers` checks. The `find_package(spc 0.4 REQUIRED)` line
stays, because 0.4.x is compatible.

v0.4.2 contains the performance and correctness work measured in each
PR: #16 and #28 (the storm report list is sized once, and only for real
reports), #19 (numeric strings parse the same way on every platform),
#21 (the typed ArcGIS queries parse each page once, about half the CPU
and allocations), #24 (the `parse_*` functions stay inside their
`string_view`), #25 (error docs), plus the CI and test-tooling changes
in #22, #26, and #29. No public API changes.

After merge, tagging `v0.4.2` runs the release workflow. It checks the
tag against `version.hpp`, runs the tests and consumer checks, and
publishes the changelog section as the release notes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants