possible fix to get rid of _get_real_dir_from_pth_file empty folder error

[chrisdane]

Hi

While being on a rather old esm_tools branch, I suddenly had problems using venv with this error:

esm_runscripts <runscript_with_venv_true> -e expname

Traceback (most recent call last):
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/bin/esm_runscripts", line 7, in <module>
    sys.exit(main())
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages/esm_runscripts/cli.py", line 288, in main
    check_all_esm_packages()
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages/esm_motd/esm_motd.py", line 168, in check_all_esm_packages
    motd.motd_handler("esm_tools", esm_tools.__version__)
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages/esm_motd/esm_motd.py", line 130, in motd_handler
    esm_tools_path = esm_tools._get_real_dir_from_pth_file("")
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages/esm_tools/__init__.py", line 73, in wrapped_func
    return func(*args, **kwargs)
  File "/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages/esm_tools/__init__.py", line 154, in _get_real_dir_from_pth_file
    raise FileNotFoundError(
FileNotFoundError: Could not determine where 's path is inside the esm-tools installation! These were searched for info: ['/work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/lib/python3.10/site-packages']
Traceback (most recent call last):
  File "/home/a/a270073/.local/bin/esm_runscripts", line 33, in <module>
    sys.exit(load_entry_point('esm-tools', 'console_scripts', 'esm_runscripts')())
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/cli.py", line 289, in main
    setup()
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/sim_objects.py", line 119, in __call__
    self.config = prepexp.run_job(self.config)
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/prepexp.py", line 28, in run_job
    evaluate(config, "prepexp", "prepexp_recipe")
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/helpers.py", line 71, in evaluate
    config = esm_plugin_manager.work_through_recipe(
  File "/home/a/a270073/esm/esm_tools/src/esm_plugin_manager/esm_plugin_manager.py", line 159, in work_through_recipe
    config = getattr(submodule, workitem)(config)
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/virtual_env_builder.py", line 276, in venv_bootstrap
    _source_and_run_bin_in_venv(
  File "/home/a/a270073/esm/esm_tools/src/esm_runscripts/virtual_env_builder.py", line 69, in _source_and_run_bin_in_venv
    return subprocess.check_call(command, shell=shell)
  File "/sw/spack-levante/mambaforge-22.9.0-2-Linux-x86_64-kptncg/lib/python3.10/subprocess.py", line 369, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'source /work/ab1095/a270073/out/awicm-1.0-recom/awi-esm-1-1-lr_kh800/esm-hist_vdeta_p25_vdet_p25_i2/.venv_esmtools/bin/activate &&  esm_runscripts awicm_recom_levante_hist.run.yaml -e esm-hist_vdeta_p25_vdet_p25_i2 --contained-run' returned non-zero exit status 1.

Together with @pgierz we made the proposed changes.

Hope this helps,
Chris

[mandresm]

See my comment below

[mandresm]

Doesn't this mean that esm_tools_path will always be None and consequently the print below will never print the esm_tools_path?

This doesn't look like a fix, but just a patch to get things going.

@chrisdane, if MOTD is giving you troubles in your current branch I recommend you deactivate it by using --no-motd. If that doesn't fix it, then please open an issue.

@pgierz, is this problem existing on release? If it does can you please try to fix it? The function esm_tools._get_real_dir_from_pth_file("") should be able to give you the path of esm_tools independently from where it's installed.

[pgierz]

Well, ._get_real_dir_from_pth_file("") looks like at the very least like I badly named the function.

This happens on release too, by the way. I'll try to fix it

[pgierz]

OK, I have a better way to fix this now :-) @chrisdane, can you please try again and let me know if that works?

@mandresm, see if the code is to your liking

[mandresm]

OK, I have a better way to fix this now :-) @chrisdane, can you please try again and let me know if that works?

@mandresm, see if the code is to your liking

Yeah, this is now good code, thanks :)

I'll be trying it with the venv