Makefile and build overhaul - encore

[dlmiles]

Makefile and build system overhaul - encore

The clankers have been busy. The clankers have turned my local attempts in these areas into a
more comprehensive solution in line with the original intention/goals they were moving towards.

---

CHANGE LOG NOTES:

• Previously the documentation advised make database/database.h to seed the main compile. This
  has not been technically required since the last build-system overhaul, and the label has changed
  to make prepare so as to remove the hard dependency on the location of a specific file and use a
  more generic target unrelated to file existence. This is still optional for most/all builds, and
  for in-tree builds make database/database.h continues to work, but downstream builders should
  update their scripts.

• INSTALL_MacOS.md notes a new optional package 'coreutils' for 'gdate' (GNU date). Some effort is
  made to be compatible with the Mac BSD-based 'date', but it is recommended to have 'gdate' on
  $PATH if you are packaging for third parties.

• CAD_ROOT / TCL_DIR change, see above - not really expecting this to affect anyone.

---

configure - is now relocatable (out-of-tree builds).

cd magic; mkdir build-tmp; cd build-tmp; ../configure; make; ./run_magic.sh; make install DESTDIR=/tmp/stage

This now all works, this is probably the biggest blocker to magic being easily adopted by a modern
distribution as a regular package as the magic project predates autoconf and didn't get all the new
autoconf features as they arrived and out-of-tree building is the default where the sources are
treated read-only and the outputs end up in a $builddir.

in-tree building still continues to work

building multiple trees from the same source in different builddir using different configure
options and such should all now work with incremental build benefits.

We are still on autoconf 2.69 that is expected to be reviewed next time to look towards 2.71 / 2.72
even if we do not make use of new features from these, because they have become the new Linux
distro baseline.

---

Per-module Makefile.in migration - this is the mechanism that makes the out-of-tree build work.

Each module's Makefile is now a template (Makefile.in) that configure turns into a generated
Makefile in the $builddir via @srcdir@ substitution, rather than every subdir sharing one Makefile
that computed paths at make-time. config.status creates the build subdir with its Makefile, so
recursion is native (make -C

) and VPATH points back at the read-only source. In-tree the
build top and source top are the same directory, so nothing changes for that case.

---

A series of scripts run_*.sh are created in the $builddir and are designed to allow magic to be
run directly from the builddir without a make install:

run_magic.sh - GUI magic (Cairo/X11)
run_magicnull.sh - batch / no-GUI (-dnull)
run_magic_gdb.sh - magic under gdb
run_magic_valgrind.sh - magic under valgrind

On first run each one self-stages a $builddir/CAD_DIR directory (a tree of symlinks into the build
and source trees, no copies) and exports CAD_ROOT so magic finds its runtime files there. The gdb
and valgrind wrappers accept a "--" separator so you can pass options to the tool on the left and to
magic on the right.

---

CAD_ROOT / TCL_DIR - Due to making the "run from $builddir" feature work (without needing to
install), the TCL_DIR hardwired default is now constructed from $CAD_ROOT/magic/tcl at run time
when the CAD_ROOT environment variable is set and non-empty; otherwise the fallback to the hardwired
compile-time default still occurs. (CAD_ROOT is the runtime environment variable magic already
uses to relocate its sys files; CAD_DIR is the compile-time default and the name of the staged
directory.) If there is a downstream issue maybe we can add a $TCL_DIR override, but it would be
interesting to understand better the non-standard install layouts being used there if so; plus you
can always drop a symlink under $CAD_ROOT/magic/tcl. In the future the TCL_DIR compile-time
constant could perhaps be removed entirely.

---

The old legacy Depend mechanism has been removed. The SED processing that went with it may have
been to address a legacy compiler/toolchain matter that no longer exists today.

A more modern .deps/.d fragment is built per source file and transparently maintained by the
toolchain (compiler -MMD).

The depend system is only advisory anyway - it is not required to function to build everything, it
is only useful for those developing magic and wanting incremental build speeds. As a result there
is no longer a serialization barrier forcing dependencies to be built before compilation can start.

---

Parallelism - some work was done on this in the past to remove the worst offenders from the
compile/link phases. Now the build should be entirely parallelized so your make -j$(nproc) should
see full top-down rebuild benefits.

ccache is supported by default, if found at configure time; --disable-ccache exists to have
configure ignore it. configure --enable-ccache-prefix-map also exists to help scenarios where
independent build trees want to share compiler output. Since compile time is below 10s maybe this
is performance overkill at this time.

---

Reproducible build - the new configure option --disable-magic-builddate will remove the
MAGIC_BUILDDATE from the project, making the final executables/DSO more compatible with reproducible
build goals some package maintainers may be trying to achieve (due to recent supply-chain security
concerns). This means given the same environment/distro/toolchain and build options the output
executables/DSO can be compared against another system doing the same, to allow verification of
toolchain/supply-chain compromise. If there are other concerns found in this area please raise
issues.

If you are a developer maybe you also want to consider --disable-magic-builddate in your configure
since it removes the requirement to recompile buildinfo.o to update MAGIC_BUILDDATE every time you
re-link to test. (The build-info defines were isolated into a single small translation unit,
utils/buildinfo.c, which is what makes this - and ccache stability - possible.)

SOURCE_DATE_EPOCH is also honored (configure detects GNU vs BSD date); not fully tested but should
assist in the reproducible build space.

---

Bundled readline has been removed. The bundled copy was GNU readline 4.3 (over 23 years old;
doesn't really have any CVE security concern) but all platforms magic builds against can provide
their own more recent version, including SVR4/Solaris, the oldest tested against. This was done as
a deliberate three-patch series (require system readline, then remove the Makefile wiring, then
delete the readline/ subdirectory). When building non-Tcl/Tk the default now expects readline to be
present at configure time or it will fail with a clear message; --disable-readline still exists to
build without it, which is useful in slim/headless build modes.

---

WASM / Emscripten - the WebAssembly build was brought in line with the same out-of-tree principles
and unbroken against the current emsdk:

• each variant now builds out-of-source (npm/build.sh runs configure from a dedicated build dir,
  the source tree is read-only);
• --embed-file inputs are split by which tree they live in (generated tech from the build tree,
  verbatim glyphs/colour maps from the source tree) so out-of-source init works;
• INCOMING_MODULE_JS_API is pinned to emscripten's full default list plus wasmBinary, which fixes
  a hard "wasmBinary not included" abort seen on emsdk 6.x;
• magic is forced onto TCL's select()-based notifier because Emscripten's epoll is a non-working
  stub;
• the disposable out-of-source WASM build directories are gitignored.

---

LaTeX documentation - this was hidden away; it has now been made part of the default build output,
with configure auto-detecting the necessary programs and advising what you are missing. (The CI /
AppImage packaging was updated to install texlive so the documents actually regenerate and ship.)

[RTimothyEdwards]

I would point out that the LaTeX documentation was more or less hidden away on purpose, because I migrated all the documentation to HTML and the original .tex sources are considerably out of date (another job for "the clankers"?).

[RTimothyEdwards]

This is a very useful set of updates. My only concern is the mess of outdated information that is likely to end up in the compiled .tex documentation, which has not been kept up to date with the HTML documentation. However, making sure all forms of documentation (including the man pages) are all up to date and agreeing with each other can be done in a later commit.

I especially like having the "Depend" mechanism removed, as it was making the compile time slower than it used to be.

[RTimothyEdwards]

pulled and merged to opencircuitdesign.com

[dlmiles]

@RTimothyEdwards So what is being said, the docs/html/** information supersedes the docs/latexfiles/** in expected information quality ? and it is only the subdir latexfiles that could contain out-of-date information ? all the other subdirs (that are not regenerated) are also up-to-date ?

[RTimothyEdwards]

@dlmiles : Well, I did say that the files in man/ are also out of date. Stuff in psfiles/ really shouldn't be there at all because it's all generated from the latexfiles/. The psfigures/ are just figures for the documentation and tutcells/ are example layouts for the tutorials. Those are fine although the tutorials themselves need updating.

A long time ago I took the maint2.ps compiled from the LaTeX source and converted it to HTML, but it's on the website at opencircuitdesign.com and not in the repository (where it belongs). Currently, my nightly script updates a clone of the repository on my web server, and the web pages for magic link directly to the HTML documentation there, so I only need to update documentation in the repository source. I need to do the same thing with maint2.html. maint2.html on the website is up to date but not very well organized.