Skip to main content

Fixing Searx after Debian 10 to Debian 11 upgrade

I have recently upgraded my Searx server instance from Debian 10 to Debian 11. Apart from not being able to boot on the first try due to a backported kernel, all of my services also stopped working. Today we will look at Searx which currently responds with teh following message:

image-1631648296900.png

Troubleshooting

Just like always when we have a problem, we need to start troubleshooting somewhere. Each person approcahes troubleshooting differently, here's what I will do in this case:

Restart Searx

Who doesn't know the phrase "have you tried turning it off and on again?". Most of you probably do. Sometimes a quick restart is all that's needed. However we first need to figure out what to restart by listing all services on the system. There are a couple of ways to do this, since Debian uses SystemD as its init system, we can issue the following command:

$ systemctl --type=service

or

$ systemctl list-units --type=service

Both should give you the same output, similar to this:

  UNIT                               LOAD   ACTIVE     SUB          DESCRIPTION
● apache2.service                    loaded failed     failed       The Apache HTTP Server
● certbot.service                    loaded failed     failed       Certbot
  console-getty.service              loaded active     running      Console Getty
  dbus.service                       loaded active     running      D-Bus System Message Bus
  filtron.service                    loaded active     running      filtron
  ifupdown-pre.service               loaded active     exited       Helper to synchronize boot up for ifupdown
  networking.service                 loaded active     exited       Raise network interfaces
  nginx.service                      loaded active     running      nginx - high performance web server
  ssh.service                        loaded active     running      OpenBSD Secure Shell server
  systemd-journal-flush.service      loaded active     exited       Flush Journal to Persistent Storage
  systemd-journald.service           loaded active     running      Journal Service
  systemd-logind.service             loaded active     running      User Login Management
  systemd-modules-load.service       loaded active     exited       Load Kernel Modules
  systemd-networkd.service           loaded active     running      Network Service
  systemd-remount-fs.service         loaded active     exited       Remount Root and Kernel File Systems
  systemd-resolved.service           loaded active     running      Network Name Resolution
  systemd-sysctl.service             loaded active     exited       Apply Kernel Variables
  systemd-sysusers.service           loaded active     exited       Create System Users
  systemd-tmpfiles-setup-dev.service loaded active     exited       Create Static Device Nodes in /dev
  systemd-tmpfiles-setup.service     loaded active     exited       Create Volatile Files and Directories
  systemd-update-utmp.service        loaded active     exited       Update UTMP about System Boot/Shutdown
  systemd-user-sessions.service      loaded active     exited       Permit User Sessions
  user-runtime-dir@1000.service      loaded active     exited       User Runtime Directory /run/user/1000
  user@1000.service                  loaded active     running      User Manager for UID 1000
  uwsgi.service                      loaded active     running      LSB: Start/stop uWSGI server instance(s)
  whoogle.service                    loaded activating auto-restart Whoogle

Searx runs on uWSGI, which can be described, as its creators call it "Swiss Army Knife for your network applications". I only have one service running on uWSGI, which means I can restart the entire server without worrying. However, if there are other working services depending on uWSGI, they will be restarted as well. Firstly, check status of the uWSGI service.

$ systemctl status uwsgi

As we can see, it is active and running and has loaded /etc/uwsgi/apps-enabled/searx.ini

● uwsgi.service - LSB: Start/stop uWSGI server instance(s)
     Loaded: loaded (/etc/init.d/uwsgi; generated)
     Active: active (running) since Fri 2021-09-10 10:42:33 CEST; 4 days ago
       Docs: man:systemd-sysv-generator(8)
    Process: 2837 ExecStart=/etc/init.d/uwsgi start (code=exited, status=0/SUCCESS)
        CPU: 48.725s
     CGroup: /system.slice/uwsgi.service
             ├─2938 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemoni>
             ├─2941 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemoni>
             ├─2942 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemoni>
             └─2944 /usr/bin/uwsgi --ini /usr/share/uwsgi/conf/default.ini --ini /etc/uwsgi/apps-enabled/searx.ini --daemoni>

Warning: some journal files were not opened due to insufficient permissions.

To restart, we need to elevate using sudo.

$ sudo systemctl restart uwsgi

Check if the service is running like before with systemctl status. Unfortunately in my case, the restart didn't help. Let's finally dig deeper.

Consult Logs

Logs for uWSGI are located in /var/log/uwsgi/app. We are looking for searx.log which should be present within this folder. Open it using your favorite method, I will use less and jump to end. Heads up – this file requires elevated privileges to be opened.

sudo less searx.log

The log is fairly verbose and understandable. This part describes the killing and initialization process of uWSGI and also the attempted start of Searx.

Tue Sep 14 23:12:10 2021 - SIGINT/SIGQUIT received...killing workers...
Tue Sep 14 23:12:10 2021 - gateway "uWSGI http 1" has been buried (pid: 2944)
Tue Sep 14 23:12:11 2021 - worker 1 buried after 1 seconds
Tue Sep 14 23:12:11 2021 - worker 2 buried after 1 seconds
Tue Sep 14 23:12:11 2021 - goodbye to uWSGI.
Tue Sep 14 23:12:12 2021 - *** Starting uWSGI 2.0.19.1-debian (64bit) on [Tue Sep 14 23:12:12 2021] ***
Tue Sep 14 23:12:12 2021 - compiled with version: 10.2.1 20210110 on 11 June 2021 09:08:33
Tue Sep 14 23:12:12 2021 - os: Linux-5.4.0-74-generic #83~18.04.1-Ubuntu SMP Tue May 11 16:01:00 UTC 2021
Tue Sep 14 23:12:12 2021 - nodename: hledej.nl01
Tue Sep 14 23:12:12 2021 - machine: x86_64
Tue Sep 14 23:12:12 2021 - clock source: unix
Tue Sep 14 23:12:12 2021 - pcre jit disabled
Tue Sep 14 23:12:12 2021 - detected number of CPU cores: 8
Tue Sep 14 23:12:12 2021 - current working directory: /
Tue Sep 14 23:12:12 2021 - writing pidfile to /run/uwsgi/app/searx/pid
Tue Sep 14 23:12:12 2021 - detected binary path: /usr/bin/uwsgi-core
Tue Sep 14 23:12:12 2021 - chdir() to /var/www/searx/searx
Tue Sep 14 23:12:12 2021 - your memory page size is 4096 bytes
Tue Sep 14 23:12:12 2021 - detected max file descriptor number: 1024
Tue Sep 14 23:12:12 2021 - lock engine: pthread robust mutexes
Tue Sep 14 23:12:12 2021 - thunder lock: disabled (you can enable it with --thunder-lock)
Tue Sep 14 23:12:12 2021 - *** Cache "searxcache" initialized: 11MB (key: 2136 bytes, keys: 4272000 bytes, data: 8192000 bytes, bitmap: 250 bytes) preallocated ***
Tue Sep 14 23:12:12 2021 - uWSGI http bound on 127.0.0.1:8888 fd 5
Tue Sep 14 23:12:12 2021 - uwsgi socket 0 bound to UNIX address /run/uwsgi/app/searx/socket fd 8
Tue Sep 14 23:12:12 2021 - setgid() to 1002
Tue Sep 14 23:12:12 2021 - setuid() to 1002
Tue Sep 14 23:12:12 2021 - Python version: 3.9.2 (default, Feb 28 2021, 17:03:44)  [GCC 10.2.1 20210110]
Tue Sep 14 23:12:12 2021 - PEP 405 virtualenv detected: /var/www/searx/searx-pyenv
Tue Sep 14 23:12:12 2021 - Set PythonHome to /var/www/searx/searx-pyenv
Tue Sep 14 23:12:12 2021 - Python main interpreter initialized at 0x564252505c10
Tue Sep 14 23:12:12 2021 - python threads support enabled
Tue Sep 14 23:12:12 2021 - your server socket listen backlog is limited to 100 connections
Tue Sep 14 23:12:12 2021 - your mercy for graceful operations on workers is 60 seconds
Tue Sep 14 23:12:12 2021 - mapped 218808 bytes (213 KB) for 2 cores
Tue Sep 14 23:12:12 2021 - *** Operational MODE: preforking ***
Tue Sep 14 23:12:12 2021 - added /var/www/searx/searx/ to pythonpath.
Tue Sep 14 23:12:12 2021 - spawned uWSGI master process (pid: 136045)
Tue Sep 14 23:12:12 2021 - spawned uWSGI worker 1 (pid: 136048, cores: 1)
Tue Sep 14 23:12:12 2021 - spawned uWSGI worker 2 (pid: 136049, cores: 1)
Tue Sep 14 23:12:12 2021 - cache sweeper thread enabled
Tue Sep 14 23:12:12 2021 - spawned uWSGI http 1 (pid: 136051)
ModuleNotFoundError: No module named 'searx'
Tue Sep 14 23:12:12 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Tue Sep 14 23:12:12 2021 - *** no app loaded. going in full dynamic mode ***
ModuleNotFoundError: No module named 'searx'
Tue Sep 14 23:12:12 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Tue Sep 14 23:12:12 2021 - *** no app loaded. going in full dynamic mode ***
Tue Sep 14 23:12:24 2021 - --- no python application found, check your startup logs for errors ---

It is all looking fine until these lines:

ModuleNotFoundError: No module named 'searx'
Tue Sep 14 23:12:12 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Tue Sep 14 23:12:12 2021 - *** no app loaded. going in full dynamic mode ***

Searx also includes helpful utilities and debug scripts. Try running:

sudo /var/www/searx/utils/searx.sh inspect service
service status & log
====================

sourced .config.sh :

  PUBLIC_URL          : http://hledej.nl01/searx
  SEARX_URL_PATH      : /searx
  SEARX_INSTANCE_NAME : searx@hledej.nl01
  SEARX_INTERNAL_HTTP  : 127.0.0.1:8888

ERROR: Service account searx does not exists!
ERROR: ~searx: python environment is not available!
ERROR: ~searx: Missing searx software!
INFO:  uWSGI app searx.ini is enabled.
INFO:  public URL   --> http://hledej.nl01/searx
INFO:  internal URL --> http://127.0.0.1:8888
INFO:  got 500 from http://127.0.0.1:8888
INFO:  got 301 from http://hledej.nl01/searx

Enable searx debug mode? [NO/yes]
No

Unit searx.service could not be found.
// use CTRL-C to stop monitoring the log
Wed Sep 15 00:01:07 2021 - spawned uWSGI http 1 (pid: 137252)
ModuleNotFoundError: No module named 'searx'
Wed Sep 15 00:01:07 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Wed Sep 15 00:01:07 2021 - *** no app loaded. going in full dynamic mode ***
ModuleNotFoundError: No module named 'searx'
Wed Sep 15 00:01:07 2021 - unable to load app 0 (mountpoint='') (callable not found or import error)
Wed Sep 15 00:01:07 2021 - *** no app loaded. going in full dynamic mode ***
Wed Sep 15 00:01:21 2021 - --- no python application found, check your startup logs for errors ---
Wed Sep 15 00:01:21 2021 - --- no python application found, check your startup logs for errors ---
Wed Sep 15 00:04:03 2021 - --- no python application found, check your startup logs for errors ---

The errors at the bottom are identical to what we got from the uWSGI log, however there is more information at the top of the command output. Apparently, there is not searx service account, which is false,

 

Restart SSH

This really has nothing to do with Searx, but when I connected to my server, I noticed the responsivnes has significantly decreased, letters appearing after a few seconds, generally slugish performance. First I

Back to top