Malformed numpy

[akzaidi]

From @andrie:

> use_condaenv("cntk-py35", conda = "/home/adevries/anaconda3/envs/cntk-py35/bin/conda", required = TRUE)
> py_config()

Error in py_run_string_impl(paste0("import sys; sys.path.append('", system.file("python",  :
  ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.
 
 
Detailed traceback:
  File "/home/adevries/anaconda3/lib/python3.6/site-packages/numpy/__init__.py", line 146, in <module>
    from . import add_newdocs
  File "/home/adevries/anaconda3/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/adevries/anaconda3/lib/python3.6/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/adevries/anaconda3/lib/python3.6/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/adevries/anaconda3/lib/python3.6/site-packages/numpy/core/__init__.py", line 24, in <module>
    raise ImportError(msg)

Update numpy from 1.12 -> 1.13 seems to fix, but this issue doesn't effect the python API, just the R API.

[andrie]

Additional information:

adevries@adv-ug-dsvm-1708:~$ source activate cntk-py35
(cntk-py35) adevries@adv-ug-dsvm-1708:~$ python
Python 3.5.4 |Continuum Analytics, Inc.| (default, Aug 14 2017, 13:26:58)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import cntk
>>> print(cntk.__version__)
2.1
>>> exit()

(cntk-py35) adevries@adv-ug-dsvm-1708:~$ pip install -U numpy
Requirement already up-to-date: numpy in ./anaconda3/envs/cntk-py35/lib/python3.5/site-packages

(cntk-py35) adevries@adv-ug-dsvm-1708:~$ pip freeze
cntk==2.1
h5py==2.7.0
numpy==1.13.1
scipy==0.19.1
six==1.10.0

> sessionInfo()
R version 3.3.3 (2017-03-06)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
 [4] LC_COLLATE=en_US.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
[10] LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] reticulate_1.1.0.9002 RevoUtilsMath_10.0.0  RevoUtils_10.0.3      RevoMods_11.0.0      
[5] MicrosoftML_1.3.0     RevoScaleR_9.1.0      lattice_0.20-34       rpart_4.1-10         

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.12           codetools_0.2-15       CompatibilityAPI_1.1.0 foreach_1.4.3         
 [5] grid_3.3.3             R6_2.2.0               jsonlite_1.3           iterators_1.0.8       
 [9] tools_3.3.3            mrupdate_1.0.1  

[akzaidi]

On a fresh DSVM, I pip-installed CNTK and then did the following:

> py_config()
Error in py_run_string_impl(paste0("import sys; sys.path.append('", system.file("python",  :
  ImportError:
Importing the multiarray numpy extension module failed.  Most
likely you are trying to import a failed build of numpy.
If you're working with a numpy git repo, try `git clean -xdf` (removes all
files not under version control).  Otherwise reinstall numpy.


Detailed traceback:
  File "/home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy/__init__.py", line 146, in <module>
    from . import add_newdocs
  File "/home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy/core/__init__.p
> q()

(cntk-py35) alizaidi@azcntkr:~$ pip install -U numpy
Collecting numpy
  Downloading numpy-1.13.1-cp35-cp35m-manylinux1_x86_64.whl (16.9MB)
    100% |████████████████████████████████| 16.9MB 87kB/s
Installing collected packages: numpy
  Found existing installation: numpy 1.12.1
    Uninstalling numpy-1.12.1:
      Successfully uninstalled numpy-1.12.1
Successfully installed numpy-1.13.1

(cntk-py35) alizaidi@azcntkr:~$ R

R version 3.3.3 (2017-03-06) -- "Another Canoe"
> library(reticulate)
> use_condaenv("cntk-py35")
> py_config()
python:         /home/alizaidi/.conda/envs/cntk-py35/bin/python
libpython:      /home/alizaidi/.conda/envs/cntk-py35/lib/libpython3.5m.so
pythonhome:     /home/alizaidi/.conda/envs/cntk-py35:/home/alizaidi/.conda/envs/cntk-py35
version:        3.5.3 |Anaconda 4.4.0 (64-bit)| (default, Mar  6 2017, 11:58:13)  [GCC 4.4.7 20120313 (Red Hat 4.4.7-1)]
numpy:          /home/alizaidi/.conda/envs/cntk-py35/lib/python3.5/site-packages/numpy
numpy_version:  1.13.1

python versions found:
 /home/alizaidi/.conda/envs/cntk-py35/bin/python
 /usr/bin/python
 /usr/bin/python3
> reticulate::py_module_available("cntk")
[1] TRUE

One difference now appears to be anaconda3 vs 2.7, but I don't think that's the issue. I'll keep looking.

[andrie]

I don't understand this. I created a new DSVM, followed these instructions, and I get:

> use_condaenv("cntk-py35")
Error: Unable to find conda binary. Is Anaconda installed?

If I also specify the conda environment, then I don't get an error, but R points to python 2.7

> use_condaenv("cntk-py35", conda = "/home/adevries/.conda/envs/cntk-py35/bin/conda", required = TRUE)
> py_config()
python:         /usr/bin/python
libpython:      /usr/lib/python2.7/config-x86_64-linux-gnu/libpython2.7.so
pythonhome:     /usr:/usr
version:        2.7.12 (default, Nov 19 2016, 06:48:10)  [GCC 5.4.0 20160609]
numpy:           [NOT FOUND]

python versions found: 
 /usr/bin/python
 /usr/bin/python3

I also don't understand how conda decides where to put the envs:

(cntk-py35) adevries@adv-ug-dsvm-1709:~$ conda env list
# conda environments:
#
cntk-py35             *  /home/adevries/.conda/envs/cntk-py35
py35                     /anaconda/envs/py35
root                     /anaconda

[andrie]

I tried again, this time forcing a reinstall of numpy, and now it seems to work:

 pip install --upgrade --force-reinstall numpy

[drdarshan]

Hi @andrie, is it OK to close this issue?