Fixing Whoogle after Debian upgrade + updating Whoogle

Whoogle is the other service that is now unable to start after upgrading from Debian 10 to Debian 11. Just like with Searx, I suspect the problem will have something to do with Python, since both apps are built using Flask, which is a micro web framework written in Python. However, I have been postponing upgrading to the newest version of Whoogle for a while now, so I will use this opportunity to first perform an upgrade.

Upgrading Whoogle

Check and stop the service

$ systemctl status whoogle

● whoogle.service - Whoogle
     Loaded: loaded (/etc/systemd/system/whoogle.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-09-15 14:57:08 CEST; 48ms ago
   Main PID: 157075 (python3)
        CPU: 16ms
     CGroup: /system.slice/whoogle.service
             └─157075 /var/www/whoogle/venv/bin/python3 -um app --host 127.0.0.1 --port 5000

Sep 15 14:57:08 hostname whoogle[157075]:     __import__(pkg_name)
Sep 15 14:57:08 hostname whoogle[157075]:   File "/var/www/whoogle/app/__init__.py", line 1, in <module>
Sep 15 14:57:08 hostname whoogle[157075]:     from app.request import send_tor_signal
Sep 15 14:57:08 hostname whoogle[157075]:   File "/var/www/whoogle/app/request.py", line 1, in <module>
Sep 15 14:57:08 hostname whoogle[157075]:     from app.models.config import Config
Sep 15 14:57:08 hostname whoogle[157075]:   File "/var/www/whoogle/app/models/config.py", line 1, in <module>
Sep 15 14:57:08 hostname whoogle[157075]:     from flask import current_app
Sep 15 14:57:08 hostname whoogle[157075]: ModuleNotFoundError: No module named 'flask'
Sep 15 14:57:08 hostname systemd[1]: whoogle.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 14:57:08 hostname systemd[1]: whoogle.service: Failed with result 'exit-code'.

We can see that the service is trying to start but is unable to. Let's leave that for a while, stop the service and perform an upgrade.

$ sudo systemctl stop whoogle

Confirm that the service has stopped.

$ sudo systemctl status whoogle

● whoogle.service - Whoogle
     Loaded: loaded (/etc/systemd/system/whoogle.service; enabled; vendor preset: enabled)
     Active: inactive (dead) (Result: exit-code) since Wed 2021-09-15 14:57:26 CEST; 10min ago
    Process: 157082 ExecStart=/var/www/whoogle/venv/bin/python3 -um app --host 127.0.0.1 --port 5000 (code=exited, status=1/>
   Main PID: 157082 (code=exited, status=1/FAILURE)
        CPU: 105ms

Sep 15 14:57:24 hostname systemd[1]: whoogle.service: Main process exited, code=exited, status=1/FAILURE
Sep 15 14:57:24 hostname systemd[1]: whoogle.service: Failed with result 'exit-code'.
Sep 15 14:57:26 hostname systemd[1]: Stopped Whoogle.

Temporarily add shell to Whoogle user

Whoogle is also running under a separate user who owns its directory and since we will be performing management tasks as this user, we need it to have shell access. Replace /usr/sbin/nologin with /bin/bash on the whoogle line in /etc/passwd.

$ sudo vi /etc/passwd

whoogle:x:1001:1001:,,,:/home/whoogle:/usr/sbin/nologin

to

whoogle:x:1001:1001:,,,:/home/whoogle:/bin/bash

Backup existing config file

All configuration of Whoogle is done using the whoogle.env file located in the root of the Whoogle folder, in my case /var/www/whoogle. Backup this file before upgrading. This will copy the file to the home directory of Whoogle user.

(whoogle)$ cp /var/www/whoogle/whoogle.env ~/

Clone the newest version

Move to the home directory of Whoogle user.

(whoogle)$ cd

Once in there, clone the Git repository.

(whoogle)$ git clone https://github.com/benbusby/whoogle-search.git

Merge the config file

Across versions, config files may change in structure or add/remove options. Always check manually.

Move to the cloned repository and merge your old whoogle.env with the downloaded one. In this case, the newer config was structured a bit differently, so I kept the new one and replaced some values in it.

(whoogle)$ cd whoogle-search && vi whoogle.env

Delete and replace with new version

You could just copy and rewrite everything in /var/www/whoogle with new files, but that poses the risk of leaving some old files behind or running into permission issues. That's why I decided to completely remove files from the old directory and then copy in the new version. This is essentially a reinstall of the service, but everything we need is backed up in the config file, so we are not losing any data.

Heads up – This isn't exactly the fastest way to perform an upgrade and even if it can be done in 5 minutes, in availability dependent environments, consider options like rerouting traffic to another web server and switching back once everything is ready and tested.

There is a way to do it in one command, but I didn't want to spend more time on this.

Make sure you are deleting the right directory! This will delete everything without asking.

(whoogle)$ rm -rvf /var/www/whoogle/*
(whoogle)$ rm -rvf /var/www/whoogle/.*

Copy the download files to /var/www/whoogle, you have to be in the correct directory for it to work.

(whoogle)$ cp -av whoogle-search/. /var/www/whoogle/

Create Python virtual environment

Navigate to the root Whoogle folder.

(whoogle)$ cd /var/www/whoogle

Create venv folder and activate the virtualenv.

(whoogle)$ python3 -m venv venv
(whoogle)$ source venv/bin/activate

Install Python requirements

Check if you are now in the venv, which you can tell by having (venv) in the front of the terminal and install requirements using pip.

(whoogle)$ pip install -r requirements.txt

The output should look something like this:

Collecting attrs==19.3.0
  Using cached attrs-19.3.0-py2.py3-none-any.whl (39 kB)
Collecting beautifulsoup4==4.8.2
  Using cached beautifulsoup4-4.8.2-py3-none-any.whl (106 kB)
Collecting bs4==0.0.1
  Using cached bs4-0.0.1.tar.gz (1.1 kB)
Collecting cachelib==0.1
  Using cached cachelib-0.1-py3-none-any.whl (12 kB)
Collecting certifi==2020.4.5.1
  Using cached certifi-2020.4.5.1-py2.py3-none-any.whl (157 kB)
Collecting cffi==1.13.2
  Downloading cffi-1.13.2.tar.gz (460 kB)
     |████████████████████████████████| 460 kB 3.2 MB/s
Collecting chardet==3.0.4
  Using cached chardet-3.0.4-py2.py3-none-any.whl (133 kB)
Collecting Click==7.0
  Using cached Click-7.0-py2.py3-none-any.whl (81 kB)
Collecting cryptography==3.3.2
  Downloading cryptography-3.3.2-cp36-abi3-manylinux2010_x86_64.whl (2.6 MB)
     |████████████████████████████████| 2.6 MB 20.4 MB/s
Collecting Flask==1.1.1
  Using cached Flask-1.1.1-py2.py3-none-any.whl (94 kB)
Collecting Flask-Session==0.3.2
  Using cached Flask_Session-0.3.2-py2.py3-none-any.whl (7.4 kB)
Collecting idna==2.9
  Using cached idna-2.9-py2.py3-none-any.whl (58 kB)
Collecting itsdangerous==1.1.0
  Using cached itsdangerous-1.1.0-py2.py3-none-any.whl (16 kB)
Collecting Jinja2==2.11.3
  Using cached Jinja2-2.11.3-py2.py3-none-any.whl (125 kB)
Collecting MarkupSafe==1.1.1
  Downloading MarkupSafe-1.1.1-cp39-cp39-manylinux2010_x86_64.whl (32 kB)
Collecting more-itertools==8.3.0
  Using cached more_itertools-8.3.0-py3-none-any.whl (44 kB)
Collecting packaging==20.4
  Using cached packaging-20.4-py2.py3-none-any.whl (37 kB)
Collecting pluggy==0.13.1
  Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB)
Collecting py==1.10.0
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
     |████████████████████████████████| 97 kB 1.7 MB/s
Collecting pycodestyle==2.6.0
  Using cached pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB)
Collecting pycparser==2.19
  Using cached pycparser-2.19-py2.py3-none-any.whl
Collecting pyOpenSSL==19.1.0
  Using cached pyOpenSSL-19.1.0-py2.py3-none-any.whl (53 kB)
Collecting pyparsing==2.4.7
  Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB)
Collecting PySocks==1.7.1
  Using cached PySocks-1.7.1-py3-none-any.whl (16 kB)
Collecting pytest==5.4.1
  Using cached pytest-5.4.1-py3-none-any.whl (246 kB)
Collecting python-dateutil==2.8.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting requests==2.25.1
  Downloading requests-2.25.1-py2.py3-none-any.whl (61 kB)
     |████████████████████████████████| 61 kB 2.6 MB/s
Collecting soupsieve==1.9.5
  Using cached soupsieve-1.9.5-py2.py3-none-any.whl (33 kB)
Collecting stem==1.8.0
  Using cached stem-1.8.0.tar.gz (2.9 MB)
Collecting urllib3==1.26.5
  Downloading urllib3-1.26.5-py2.py3-none-any.whl (138 kB)
     |████████████████████████████████| 138 kB 19.0 MB/s
Collecting waitress==1.4.3
  Using cached waitress-1.4.3-py2.py3-none-any.whl (148 kB)
Collecting wcwidth==0.1.9
  Using cached wcwidth-0.1.9-py2.py3-none-any.whl (19 kB)
Collecting Werkzeug==0.16.0
  Using cached Werkzeug-0.16.0-py2.py3-none-any.whl (327 kB)
Collecting python-dotenv==0.16.0
  Using cached python_dotenv-0.16.0-py2.py3-none-any.whl (18 kB)
Collecting six>=1.4.1
  Downloading six-1.16.0-py2.py3-none-any.whl (11 kB)
Using legacy 'setup.py install' for bs4, since package 'wheel' is not installed.
Using legacy 'setup.py install' for cffi, since package 'wheel' is not installed.
Using legacy 'setup.py install' for stem, since package 'wheel' is not installed.
Installing collected packages: pycparser, MarkupSafe, Werkzeug, soupsieve, six, pyparsing, Jinja2, itsdangerous, Click, cffi, wcwidth, urllib3, py, pluggy, packaging, more-itertools, idna, Flask, cryptography, chardet, certifi, cachelib, beautifulsoup4, attrs, waitress, stem, requests, python-dotenv, python-dateutil, pytest, PySocks, pyOpenSSL, pycodestyle, Flask-Session, bs4
    Running setup.py install for cffi ... done
    Running setup.py install for stem ... done
    Running setup.py install for bs4 ... done
Successfully installed Click-7.0 Flask-1.1.1 Flask-Session-0.3.2 Jinja2-2.11.3 MarkupSafe-1.1.1 PySocks-1.7.1 Werkzeug-0.16.0 attrs-19.3.0 beautifulsoup4-4.8.2 bs4-0.0.1 cachelib-0.1 certifi-2020.4.5.1 cffi-1.13.2 chardet-3.0.4 cryptography-3.3.2 idna-2.9 itsdangerous-1.1.0 more-itertools-8.3.0 packaging-20.4 pluggy-0.13.1 py-1.10.0 pyOpenSSL-19.1.0 pycodestyle-2.6.0 pycparser-2.19 pyparsing-2.4.7 pytest-5.4.1 python-dateutil-2.8.1 python-dotenv-0.16.0 requests-2.25.1 six-1.16.0 soupsieve-1.9.5 stem-1.8.0 urllib3-1.26.5 waitress-1.4.3 wcwidth-0.1.9

Try running the server with ./run. Navigate to you URL and you should see it working.

(whoogle)$ ./run

Serving on http://0.0.0.0:5000

Revert changes to the Whoogle user

Now kill the server with Ctrl+C if you haven't already and logout of whoogle user.

This is important, before returning Whoogle into production, change the shell of whoogle account back to /usr/sbin/nologin in /etc/passwd. Use your regular sudo user for this purpose.

whoogle:x:1001:1001:,,,:/home/whoogle:/usr/sbin/nologin

Restart the service

$ sudo systemctl restart whoogle

Check if the service is running fine.

$ sudo systemctl status whoogle

● whoogle.service - Whoogle
     Loaded: loaded (/etc/systemd/system/whoogle.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2021-09-15 17:13:31 CEST; 4s ago
   Main PID: 157621 (python3)
        CPU: 868ms
     CGroup: /system.slice/whoogle.service
             └─157621 /var/www/whoogle/venv/bin/python3 -um app --host 127.0.0.1 --port 5000

Sep 15 17:13:31 hostname systemd[1]: whoogle.service: Succeeded.
Sep 15 17:13:31 hostname systemd[1]: Stopped Whoogle.
Sep 15 17:13:31 hostname systemd[1]: Started Whoogle.
Sep 15 17:13:31 hostname whoogle[157621]: Serving on http://localhost:5000

Navigate to your URL and Whoogle should be up and running again.

Fixing Whoogle after Debian upgrade + updating Whoogle

Upgrading Whoogle

Check and stop the service

Temporarily add shell to Whoogle user

Backup existing config file

Clone the newest version

Merge the config file

Delete and replace with new version

Create Python virtual environment

Install Python requirements

Revert changes to the Whoogle user

Restart the service

No Comments