CKKS composite scaling: context generation fails with "EstimateLogP() failure" or never returns for valid-looking parameters #1358

Composite-scaling context generation has two failure modes on dev (1947bb4), both also present on main (v1.5), with clang 18 and gcc 14. Both were found running the examples before the v1.6 release and then characterized by generating contexts for 13,200 composite-scaling configurations: both techniques, register word size 24 to 48, composite degree 2 to 4, scaling modulus 40 to 120 bits, depth 1 to 16, numLargeDigits 0 to 5, HEStd_128_classic, HYBRID, one process per configuration with a 60-second limit.

1. "EstimateLogP() failure" for valid parameters

polynomial-evaluation-high-precision-composite-scaling aborts at its default parameters (first modulus 96 bits, scaling modulus 80 bits, register word size 32, depth 6, COMPOSITESCALINGAUTO):

terminate called after throwing an instance of 'lbcrypto::OpenFHEException'
  what():  src/pke/lib/schemerns/rns-cryptoparameters.cpp:l.161:PrecomputeCRTTables(): EstimateLogP() failure: expected sizeP [9], but got sizeP [7]

Cause. ParamsGenCKKSRNSInternal estimates the auxiliary modulus P with EstimateLogP to pick the ring dimension, before the moduli exist, and PrecomputeCRTTables later requires the real number of P primes to equal that estimate (the check added in #1158). With composite scaling the two partition Q into digits differently: the estimate splits numPrimes whole scaling primes (here 7 primes, 3 per digit, largest digit 256 bits), while the real setup splits the numPrimes * compositeDegree limbs (here 21 limbs, 7 per digit, largest digit about 203 bits). A digit boundary that falls inside a composite prime makes the estimate larger than the real value: 9 primes of 31 bits against 7.

Scope. 1,880 of the 13,200 configurations fail this way. In 1,872 of them the estimate is larger than the real value; in 8 it is smaller.

Proposed fix. Throw only when the real P is larger than the estimate:

         if (sizeP_estimate_global > 0) {
-            if (sizeP_estimate_global != sizeP) {
+            if (sizeP > sizeP_estimate_global) {

The estimate is used only for the ring-dimension choice, so an estimate larger than the real P selects a ring dimension for a larger modulus than is used, which is conservative; an estimate smaller than the real P could select a ring dimension that is too small, and still throws. Configurations that work today have estimate equal to real, so none of them changes.

With this change, 1,856 of the 1,880 configurations generate a context; the 8 underestimates are still refused, 12 reach a later "Overflow shrinking candidate" error and 4 hit the hang below. The example runs and gives the expected result, and the composite-scaling, FBT and CSvsFA unit tests pass (202 tests).

Making EstimateLogP model the limbs instead (splitting each composite prime into compositeDegree parts) is not a safe alternative on its own: it removes most mismatches but breaks 36 configurations that work today and raises the number of underestimates from 8 to 51, because the real limbs come out a fraction of a bit above their nominal size and the excess adds up across a digit. A more accurate estimate is worth doing together with a check of the real log(QP) against the security bound, but that is a larger change.

2. CompositePrimeModuliGen loops forever instead of throwing

For other parameters, context generation never returns: CompositePrimeModuliGen keeps searching for primes (the stack is in MillerRabinPrimalityTest called from CompositePrimeModuliGen). For example:

CCParams<CryptoContextCKKSRNS> p;
p.SetSecurityLevel(HEStd_128_classic);
p.SetScalingTechnique(COMPOSITESCALINGAUTO);
p.SetRegisterWordSize(24);
p.SetScalingModSize(40);
p.SetFirstModSize(41);
p.SetMultiplicativeDepth(6);
p.SetNumLargeDigits(2);
p.SetKeySwitchTechnique(HYBRID);
auto cc = GenCryptoContext(p);  // does not return

Scope. 2,505 of the 13,200 configurations did not finish within 60 seconds, and a sample of 8 was still running after 10 minutes. Most use COMPOSITESCALINGMANUAL with a composite degree too small for the register word size, which should be reported as a parameter error; 67 use COMPOSITESCALINGAUTO, where the library chose the degree itself.

Proposed fix. Bound the prime searches in CompositePrimeModuliGen and throw a parameter error when no suitable prime exists below the register word size, and check up front that scalingModSize / compositeDegree and firstModSize / compositeDegree fit the register word size (the existing check in ConfigureCompositeDegree only enforces a 19-bit minimum and carries a TODO about depth).

Validation plan

Rerun the 13,200-configuration sweep with both fixes: no configuration may hang, every failure must be a parameter error with a clear message, every configuration that generates a context today must still generate the same one, and the composite-scaling examples and unit tests must pass.