Strange failure when build Pulsar Docker image on M1

[tisonkun]

UPDATED: These issues should be all answered by #12944 (comment).

---

When debugging #12944, I encounter several strange failures. Here I'll share them and see if others encounter them also.

NOTE: These failures don't exist in CI. Thus I think they're strange.

1. cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent

[INFO] DOCKER> Step 14/25 : ENV JAVA_HOME /usr/lib/jvm/java-17-openjdk-amd64
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in a47a209de148
[INFO] DOCKER> Removing intermediate container a47a209de148
[INFO] DOCKER> ---> 6a9d5c8543b8
[INFO] DOCKER> Step 15/25 : RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security
[INFO] DOCKER> 
[INFO] DOCKER> ---> Running in d55c15e90ed1
[INFO] DOCKER> [91m/bin/sh: 1: cannot create /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security: Directory nonexistent

[INFO] DOCKER> Removing intermediate container d55c15e90ed1
[ERROR] DOCKER> Unable to build image [apachepulsar/java-test-image] : "The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2"  ["The command '/bin/sh -c echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security' returned a non-zero code: 2" ]

After add RUN mkdir -p /usr/lib/jvm/java-17-openjdk-amd64/conf/security before RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security in docker/pulsar/Dockerfile and test/docker-images/java-test-image/Dockerfile, this issue vanished.

2. FileNotFoundError: [Errno 2] No such file or directory: 'cc'

File "/tmp/pip-install-od2wltzl/grpcio/setup.py", line 149, in check_linker_need_libatomic
cc_test = subprocess.Popen(['cc', '-x', 'c++', '-std=c++11', '-'],
File "/usr/lib/python3.8/subprocess.py", line 858, in init
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.8/subprocess.py", line 1704, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'cc'

See details in #12944 (comment).

Does ubuntu:20.04 contain cc? If so, why does it fail on my local env. If not, why does it succeed in CI?

3. After I add gcc in the apt install list, it can find cc, but later it fails with Python.h not found.
4. After I add python-dev in the apt install list, it still fails with Python.h not found.

cc @lhotari @nodece @michaeljmarshall

[BewareMyPower]

You can debug it by executing the commands inside the ubuntu:20.04 container.

docker run -it -v $PWD:/pulsar ubuntu:20.04

---

Regarding the 1st question, I think it's caused by

pulsar/docker/pulsar/Dockerfile

Line 80 in e16a35d

RUN echo networkaddress.cache.ttl=1 >> /usr/lib/jvm/java-17-openjdk-amd64/conf/security/java.security

.

The directory might not be java-17-openjdk-amd64.

Regarding the 2nd question, yes, it doesn't contain cc by default. I think the problem happened during pip3 install grpcio. I suspect gRPC must be compiled from source in ARM arch (the arch of docker container is the same with the host) so it requires the GCC. Here are the outputs in my env, no C++ compilation is needed.

$ pip3 install grpcio
Collecting grpcio
  Downloading grpcio-1.47.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.5 MB)
     |████████████████████████████████| 4.5 MB 16 kB/s 
Requirement already satisfied: six>=1.5.2 in /usr/lib/python3/dist-packages (from grpcio) (1.14.0)
Installing collected packages: grpcio
Successfully installed grpcio-1.47.0

Regarding the 4th question, first you should install python3-dev because python-dev is for Python2. Then, the python3-dev is installed to /usr/local/python3.8 directory, which is not included in the default search path of GCC.

Maybe you can set the C_INCLUDE_PATH environment variable. See the following commands and outputs in my container:

$ ls /usr/include/python3.8/Python.h 
/usr/include/python3.8/Python.h
$ cat 1.c
#include <Python.h>

int main() {
	return 0;
}
$ gcc 1.c
1.c:1:10: fatal error: Python.h: No such file or directory
    1 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.
$ C_INCLUDE_PATH=/usr/include/python3.8 gcc 1.c
$ echo $?
0

[tisonkun]

Cool! Those sound reasonable. Multi-arch is hard >_<

[tisonkun]

For grpcio issues, we may try to upgrade the version if there's no regression. It seems we should upgrade the one inside bookkeeper first.

grpcio-1.47.0 still requires c++ compilation on ARM64:

$ pip3 install --no-cache-dir grpcio
Collecting grpcio
  Downloading grpcio-1.47.0.tar.gz (21.9 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 21.9/21.9 MB 3.3 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... done
Requirement already satisfied: six>=1.5.2 in /opt/homebrew/lib/python3.10/site-packages (from grpcio) (1.16.0)
Building wheels for collected packages: grpcio
  Building wheel for grpcio (setup.py) ... done
  Created wheel for grpcio: filename=grpcio-1.47.0-cp310-cp310-macosx_12_0_arm64.whl size=3782471 sha256=6f3f24c7454a7d9f2c3af8f1605b44c1ceffb6297220caeb2b849279b4ae1b7f
  Stored in directory: /private/var/folders/n2/rnpcrm6x3mb2c1jn1dkdb33c0000gn/T/pip-ephem-wheel-cache-uf4eynha/wheels/52/6c/59/4d81bb3e311148848df61c2eed3aac904414f3d18dcf365ace
Successfully built grpcio
Installing collected packages: grpcio
Successfully installed grpcio-1.47.0
$ arch
arm64

Reference:

• ISSUE 2377 : Relax grpcio requirement for python client to support python3.8 on mac os bookkeeper#2432
• [python-client 2.6.0] ContextualVersionConflict grpcio dependency conflict #7477
• Crosscompile python aarch64 wheels with dockcross grpc/grpc#25418

[BewareMyPower]

Oh it's right. The current grpcio version is 1.27, see

pulsar/pulsar-client-cpp/python/setup.py

Line 86 in e16a35d

"grpcio<1.28,>=1.8.2",

[tisonkun]

I think it's 1.27 :)