Evaluation of Python Returning Nothing to the log buffer

[thecontinium]

Hi

I use conjure extensively for clojure and it works great - thanks very much !
I would also like to use it for python.
I am unfortunatley finding the python experience rather eratic.

e.g. simply evaluating

import dumpy as np

The first time evaluate the "form" nothing is returned to the log buffer.
If I evaluate again I receive "None"

I have pasted the full log file of this interaction at the bottom of this message.

On the first attempt the debug is showing the following which seems to be formatting for an out of None but nothing is returned to the log.

# debug: remote.stdio.on-message; receive chunk >>"None↵"<<
nil
# debug: M.format-msg: >> None↵<<
nil

on the second attempt

# debug: M.format-msg: >> None↵<<
nil
None
# debug: M.format-msg: >> None↵<<
nil

I'm not sure why there is a duplicate but even with the same input the message None sometimes get output and other times it doesn't.

Unfortunately this is just one example of this happening.
Here I evaluate a file with no output to the log even though it seems to be formatting messages.

# debug: M.format-msg: >> ![image](plots/figure_1.png)↵1↵2↵3↵4↵5↵6↵7↵8↵9↵<<
nil
# debug: remote.stdio.on-message; receive source >>out<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>"None↵"<<
nil
# debug: M.format-msg: >> None↵<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<

All help appreciated.

I'm using python 3.12.6

Cheers

# Sponsored by @frenchy64 ❤
# Starting Python client...
# debug: remote.stdio.next-in-queue; stdin:write next-msg.code >>"import base64↵exec(base64.b64decode('aW1wb3J0IHN5cwpkZWYgY29uanVyZV9mb3JtYXRfb3V0cHV0KHZhbCk6CiAgICBwcmludChyZXByKHZhbCkpCnN5cy5kaXNwbGF5aG9vayA9IGNvbmp1cmVfZm9ybWF0X291dHB1dAoKX19uYW1lX18gPSAnX19yZXBsX18n'))↵"<<
nil
# --------------------------------------------------------------------------------
# python3 -iq (started)
# debug: remote.stdio.on-message; receive source >>out<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>"None↵"<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> >>> "<<
nil
# debug: M.format-msg: >> <<
nil
# debug: M.form-node?: node:type = "import_statement"
nil
# debug: M.form-node?: node:parent = #<<node module>>
nil
# --------------------------------------------------------------------------------
# eval (current-form): import numpy as np
# debug: M.eval-str opts >> {:action↵ "eval"↵ :code↵ "import numpy as np"↵ :file-path↵ "/Users/ggg/dev/thecontinium/hhh/tes.py"↵ :node↵ #<<node import_statement>>↵ :on-result↵ #<function: 0x010460ef48>↵ :origin↵ "current-form"↵ :preview↵ "# eval (current-form): import numpy as np"↵ :range↵ {:end [2 17] :start [2 0]}}<<
nil
# debug: remote.stdio.next-in-queue; stdin:write next-msg.code >>"import base64↵exec(base64.b64decode('aW1wb3J0IG51bXB5IGFzIG5w'))↵"<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<
nil
# debug: M.format-msg: >> <<
nil
# debug: M.format-msg: >> <<
nil
# debug: remote.stdio.on-message; receive source >>out<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>"None↵"<<
nil
# debug: M.format-msg: >> None↵<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<
nil
# debug: M.format-msg: >> <<
nil
# debug: M.form-node?: node:type = "import_statement"
nil
# debug: M.form-node?: node:parent = #<<node module>>
nil
# --------------------------------------------------------------------------------
# eval (current-form): import numpy as np
# debug: M.eval-str opts >> {:action↵ "eval"↵ :code↵ "import numpy as np"↵ :file-path↵ "/Users/ggg/dev/thecontinium/hhh/tes.py"↵ :node↵ #<<node import_statement>>↵ :on-result↵ #<function: 0x0104ea39b8>↵ :origin↵ "current-form"↵ :preview↵ "# eval (current-form): import numpy as np"↵ :range↵ {:end [2 17] :start [2 0]}}<<
nil
# debug: remote.stdio.next-in-queue; stdin:write next-msg.code >>"import base64↵exec(base64.b64decode('aW1wb3J0IG51bXB5IGFzIG5w'))↵"<<
nil
# debug: remote.stdio.on-message; receive source >>out<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>"None↵"<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<
nil
# debug: M.format-msg: >> None↵<<
nil
None
# debug: M.format-msg: >> None↵<<
nil
# debug: remote.stdio.on-message; receive source >>err<<
nil
# debug: remote.stdio.on-message; receive err >>nil<<
nil
# debug: remote.stdio.on-message; receive chunk >>">>> "<<
nil
# debug: M.format-msg: >> <<
nil

[russtoku]

It appears to be a quirk with importing numpy. I tried importing other modules before the first time importing numpy and the other modules evaluate to None. It's just the first time evaluating import numpy as np or import numpy that the evaluation doesn't return None in the Conjure log.

While missing from the Conjure help doc for the Python client, there is a setting, g:conjure#client#python#stdio#delay-stderr-ms, that forces the output on the stderr channel to be handled after the output on the stdout channel. It's default value is 10 milliseconds. On my M3 Macbook Air, 50 milliseconds seems to be able to order the messages so that the first evaluation of import numpy as np will evaluate to None as expected.

You can use one of these on the Nvim command line:

:let g:conjure#client#python#stdio#delay-stderr-ms = 80

or

:lua vim.g["conjure#client#python#stdio#delay-stderr-ms"] = 80

I used Python 3.12.7.

[russtoku]

For your second example:

Here I evaluate a file with no output to the log even though it seems to be formatting messages.

# debug: M.format-msg: >> ![image](plots/figure_1.png)↵1↵2↵3↵4↵5↵6↵7↵8↵9↵<<

Can you show the Python code? It would help to be able to replicate what you're experiencing.

[thecontinium]

Hi - thanks for the steer using the delay on the std-err.

I've tried a few different values but still get the problem on different forms I execute.

It seems to happen most often the first time I import.

with

      vim.g["conjure#client#python#stdio#delay-stderr-ms"] = 100

and executing these imports one after the other.

import numpy as np

import matplotlib.pyplot as plt

the first import is now fine the second is not. On a second attempt it works.

I've tried many, many other repl's for python but much prefer this interface and the way it focuses on the output.
Just need the reliability. Can the call not be synchronous because it's over stdio ?

I'm on a MB 2.4GHz 8 Core Intel i9.

Regarding running the file I think this problem was more to do with matplotlib so let's park that one for now.

[russtoku]

Can the call not be synchronous because it's over stdio ?

No, it can't be because Neovim uses libuv (luvit) to create and talk to a subprocess. libuv is an asynchronous library. If Neovim used a synchronous library, then it would hang until the response came back.

[russtoku]

@thecontinium, with #743 being merge, would you update your Conjure plugin and try things out with your Python code?