python -m spacy.en.download no longer valid?

[scevola44]

Running the line of code no longer download the spaCy's models, because there are now three different ones: en_core_web_sm, en_core_web_md, en_core_web_lg, en_vectors_web_lg.
Which one (if any) is the correct one to download during setup, and should the setup file be changed at line 125 in order to follow the change?

[kengz]

Hi, yep spaCy moved to v2.0.
Since wordvec is needed, download the with this new command python -m spacy download en_core_web_md.

If it works, could you please submit a pull request with the changes in the setup file?

[scevola44]

I see. I'll try to get it working, although in the file ./aiva/node_modules/spacy-nlp/src/py/nlp.py in line 2 where it says from spacy.en import English I'm unsure what changes I should make (I'm not an expert of python)

Does this mean though that if a language doesn't have the "_md" module it won't be possible to use the bot with that specific language?

[scevola44]

Also, even changing all files where the outdated command was used (6, IIRC), I'm still getting an error when launching npm start --debug :

aiva@4.2.0 start /home/scevola/aiva
bin/start

[ ------- Starting on local machine, not Docker ------- ]

aiva@4.2.0 development /home/scevola/aiva
mkdir -p ./logs; NODE_ENV=development ./node_modules/forever/bin/forever --minUptime=1000 --spinSleepTime=1000 --killSignal=SIGTERM --uid aiva-development -m 10 -a index.js | tee ./logs/aiva-development.log

[Wed Nov 15 2017 09:14:09 GMT+0000 (utc)] ERROR
[Wed Nov 15 2017 09:14:09 GMT+0000 (utc)] ERROR
error: Forever detected script exited with code: 0
error: Script restart attempt #1

And then the script restarts 10 times until it fails. In the log file (production.log, since development.log is empty) I have this:

[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Authenticated database successfully
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Starting aiva process
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Starting poly-socketio server on port: 6466, expecting 4 IO clients
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for js at 6466
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for python at 6466
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Deploying bot aivadev with adapter SLACK
[Tue Nov 14 2017 16:42:06 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for python at 6466
[Tue Nov 14 2017 16:42:07 GMT+0000 (utc)] �[32mINFO�[39m js 1WGMS9oIUx-nmeGsAAAA joined, 3 remains
[Tue Nov 14 2017 16:42:07 GMT+0000 (utc)] ERROR Invalid service token provided, please follow the upgrade instructions

[kengz]

oh that's no longer a spacy issue, it's now a token issue with the config file. What services are you using? (slack bot, ngrok, fb messenger)?

[scevola44]

I'm using slack, following your blog's advice. You think the issue might be with the bot's token?

[kengz]

ahh, unlikely it's Slack. On the config file default.json under ADAPTERS, could you delete the keys for services not used (so leave just Slack).

[scevola44]

I get the same output

[kengz]

did some digging, seems like an error towards your slack token. See the source of error method
It throws when ur token does not begin with the 'xoxb-', 'xoxp-' strings. Check if your slack token is correct and the latest?

[kengz]

gotta run will look again evening if it's still erroring, thanks for ur patience

[scevola44]

My token starts with 'xoxb-272[...]', so it would seem that it shouldn't raise that error.
I even tried "regenerating" the token on Slack and updating it in default.json, but the result is the same

[scevola44]

Okay, I managed to "move on", and get new error messages. I did a clean reset (recloned the aiva folder and -hopefully- fixed most of the python code related to the outdated linguistic models), and now the stdout is:

npm start --debug

> aiva@4.2.0 start /home/scevola/aiva
> bin/start

[ ------- Starting on local machine, not Docker ------- ]

> aiva@4.2.0 development /home/scevola/aiva
> mkdir -p ./logs; NODE_ENV=development ./node_modules/forever/bin/forever --minUptime=1000 --spinSleepTime=1000 --killSignal=SIGTERM --uid aiva-development -m 10 -a index.js | tee ./logs/aiva-development.log

[Thu Nov 16 2017 08:05:12 GMT+0000 (utc)] INFO Authenticated database successfully
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG 
Sequelize [Node: 8.9.1, CLI: 2.8.0, ORM: 3.30.4]

Loaded configuration file "config/db.json".
Using environment "development".
No migrations were executed, database schema was already up to date.

[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Starting aiva process
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG globalConfig {
  "BOTNAME": "labibot",
  "PORTS": {
    "NEO4J": 7476,
    "SOCKETIO": 6466,
    "SLACK": 8345,
    "TELEGRAM": 8443,
    "FB": 8545
  },
  "NGROK_AUTH": null,
  "ADMINS": [
    "alberto.bandini.93@gmail.com"
  ],
  "TEST": {
    "HUBOT_SHELL_USER_ID": "ID0000001",
    "HUBOT_SHELL_USER_NAME": "alice"
  }
}
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Starting poly-socketio server on port: 6466, expecting 4 IO clients
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG adapterConfig {
  "ACTIVATE": true,
  "HUBOT_SLACK_TOKEN": "xoxb-272773609590-91TJwfHhbMnDHAcUvTmirULH"
}
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG import js lib from client.js
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Starting socketIO client for js at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Starting socketIO client for python at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG Set SLACK PORT: 8345
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] DEBUG No WEBHOOK set for adapter SLACK
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Deploying bot labibot with adapter SLACK
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO Starting socketIO client for python at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] INFO js 0zyAW_VV2mx93iq4AAAA joined, 3 remains

[Thu Nov 16 2017 08:05:14 GMT+0000 (utc)] ERROR Traceback (most recent call last):
File "/home/scevola/aiva/node_modules/spacy-nlp/src/client.py", line 14, in

[Thu Nov 16 2017 08:05:14 GMT+0000 (utc)] ERROR from py import *
  File "/home/scevola/aiva/node_modules/spacy-nlp/src/py/nlp.py", line 2, in <module>
    from spacy import en_core_web_md  # NLP with spaCy https://spacy.io
ImportError: cannot import name en_core_web_md

[Thu Nov 16 2017 08:05:14 GMT+0000 (utc)] ERROR Traceback (most recent call last):
  File "/home/scevola/aiva/lib/client.py", line 14, in <module>
    from py import *

File "/home/scevola/aiva/lib/py/convo_classifier.py", line 5, in
from autocorrect import spell
ImportError: No module named autocorrect

[Thu Nov 16 2017 08:05:15 GMT+0000 (utc)] INFO Logged in as labibot of LabiTech
[Thu Nov 16 2017 08:05:18 GMT+0000 (utc)] INFO Slack client now connected
[Thu Nov 16 2017 08:05:18 GMT+0000 (utc)] DEBUG Started global js socketIO client for SLACK at 6466
[Thu Nov 16 2017 08:05:18 GMT+0000 (utc)] INFO global-client-js OncOSpTy8AyIcbU1AAAB joined, 2 remains
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] ERROR {}
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] ERROR Expected number of IO clients failed to join in time
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] INFO Shutting down
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] INFO Exit: killed ioClient.js children
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] INFO Exit: killed ioClient.js children
error: Forever detected script exited with code: 1
error: Script restart attempt #1

I highlighted the ones I think are the 2 main errors (hope it doesn't look too ugly). The first still has to do with spaCy's module. I didn't know how to fix line 2 of client.py and improvised something, but it doesn't seem to work.
As for the second error, I do have autocorrect installed, and the spell module too. It's all under the python3 folder, so I'm uncertain as to why it doesn't find it.

This is the content of aiva-development.log(only log file):

[Thu Nov 16 2017 08:05:12 GMT+0000 (utc)] �[32mINFO�[39m Authenticated database successfully
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Starting aiva process
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Starting poly-socketio server on port: 6466, expecting 4 IO clients
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for js at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for python at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Deploying bot labibot with adapter SLACK
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m Starting socketIO client for python at 6466
[Thu Nov 16 2017 08:05:13 GMT+0000 (utc)] �[32mINFO�[39m js 0zyAW_VV2mx93iq4AAAA joined, 3 remains
[Thu Nov 16 2017 08:05:15 GMT+0000 (utc)] INFO Logged in as labibot of LabiTech
[Thu Nov 16 2017 08:05:18 GMT+0000 (utc)] INFO Slack client now connected
[Thu Nov 16 2017 08:05:18 GMT+0000 (utc)] �[32mINFO�[39m global-client-js OncOSpTy8AyIcbU1AAAB joined, 2 remains
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] �[32mINFO�[39m Shutting down
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] �[32mINFO�[39m Exit: killed ioClient.js children
[Thu Nov 16 2017 08:06:53 GMT+0000 (utc)] �[32mINFO�[39m Exit: killed ioClient.js children

[scevola44]

So, I "solved" the first problem. in /home/scevola/aiva/node_modules/spacy-nlp/src/py/nlp.py line 2 and 3 should be

import spacy
nlp = spacy.load('en_core_web_md')

However, in this way it looks for the spacy package in python 2.7.x (default package for Ubuntu 16.04, at least), and not in 3.x. Not knowing much of Python coding, I simply downloaded spacy's language models into Python 2.7.x too, and now that ERROR is gone.

EDIT:
I tried the same procedure for the ImportError: No module named autocorrectand it works!
The bot's brain launched, but now I have a different issue, that I believe might be cause by the "mixed" imports from 2.7 and 3.x.

I tried the same commands you gave in your test example, but I don't get a "saved to brain" message, and in the terminal I get

[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG IO on pass, msg: {
  "hash": "global-client-js_8ccb1e14f9eb28e852fc9d929d8bd8c0",
  "from": "global-client-js",
  "input": "Bob brought the pizza to Alice.",
  "to": "convo_classifier.py",
  "intent": "classify"
} fn: undefined
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG IO on pass, msg: {
  "output": null,
  "from": "cgkb-py",
  "hash": "global-client-js_7def1ed0c267abf4bed2b368aa8bd522",
  "to": "global-client-js"
} fn: undefined
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG IO on pass, msg: {
  "output": null,
  "from": "py",
  "hash": "global-client-js_8ccb1e14f9eb28e852fc9d929d8bd8c0",
  "to": "global-client-js"
} fn: undefined
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hasher.handle invoking cb for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hasher.handle invoking cb for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] ERROR 
[Thu Nov 16 2017 09:56:33 GMT+0000 (utc)] INFO Knowledge saved to brain
[Thu Nov 16 2017 09:56:35 GMT+0000 (utc)] DEBUG [In log]: labibot Bob brought the pizza to Alice.

but no ERROR is present in development.log

This apparently happens also when giving it the sample command for booking a flight. Instead of the json output shown in the blog, I get this as an output

[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG Started global js socketIO client for SLACK at 6466
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG Added a callback to hasher for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] INFO global-client-js CRLj1zBnJoZWRRgpAAAB joined, -8 remains
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] INFO All 4 IO clients have joined
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG [Out log]: Your wish is my command
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG IO on pass, msg: {
"hash": "global-client-js_26e4da2159e48091e432980ad167244a",
"from": "global-client-js",
"input": "Book a flight from New York to London for Sunday",
"to": "nlp.cgkb-py",
"intent": "parse"
} fn: undefined
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG IO on pass, msg: {
"output": null,
"from": "cgkb-py",
"hash": "global-client-js_26e4da2159e48091e432980ad167244a",
"to": "global-client-js"
} fn: undefined
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG Added a callback to hasher for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG IO on pass, msg: {
"hash": "global-client-js_fce40c4ddcee0dad5825e419d4d71f8d",
"from": "global-client-js",
"input": "Book a flight from New York to London for Sunday",
"to": "convo_classifier.py",
"intent": "classify"
} fn: undefined
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hasher.handle invoking cb for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG IO on pass, msg: {
"output": null,
"from": "py",
"hash": "global-client-js_fce40c4ddcee0dad5825e419d4d71f8d",
"to": "global-client-js"
} fn: undefined
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hasher.handle invoking cb for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] DEBUG hash string exists for global-client
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] ERROR
[Thu Nov 16 2017 10:08:30 GMT+0000 (utc)] INFO Knowledge saved to brain

Also, the neo4j interface doesn't show me the graph

[scevola44]

PS: I also got a funny response after restarting the bot

And in the terminal it labeled my request as topic: weather

[kengz]

great to see u made progress :)
The error would be from the socket failing to post to python, the process needs start in python3 (cuz of some encoding error in the socket library in python2).
So the problem is 2 fold:

1. the socket connection needs to happen from Slack -> Nodejs -> sent over SocketIO to python3 -> processed in spacy -> sent back the parsed results to Nodejs -> Node records on Neo4j and replies to user. The last 2 steps is where it fails and hence an empty neo4j db. This is indicated by the output: null error message.
   Recommended solution: start and embed ur runtime in python3 with the installs, use something like Conda:

if which conda >/dev/null; then
  echo "Conda is already installed"
else
  curl -O https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
  bash Miniconda3-latest-Linux-x86_64.sh -b -u
fi
conda create -n aiva python=3.6 -c conda-forge -y
# every time when u use it, active the python3 environment in your terminal session:
source activate aiva
# now u can run the installations and AIVA commands from.
pip install -r requirements.txt
# ... install spacy and things u did earlier
# to deactivate, just do this from within the environment
source deactivate

Shd fix problem 1.

2. Problem 2 is with the weird response. So, Your wish is my command is the immediate reply triggered in the script when the input to spacy is received. Once done processing from the above chain (thru spacy and neo4j), it passes a output: {some response} instead of output: null, which is picked up and sent to Slack. Since in your terminal it was output: null, nothing is sent after.
   The No idea, I live in the cloud and it's always sunny ;) is from some other script that got triggered, likely default Hubot script.

Let me know how this goes!

[scevola44]

I'm unsure at what point I should run the code you sent. I tried running it in the terminal, but line 7 gives an error
conda: command not found
but it seemed to have installed conda when I gave the previous commands