2021-06-04 18:09:30 +03:00
|
|
|
from app.filter import clean_query
|
Add tor and http/socks proxy support (#137)
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
2020-10-29 03:47:42 +03:00
|
|
|
from app.request import send_tor_signal
|
2021-04-01 07:23:30 +03:00
|
|
|
from app.utils.session import generate_user_key
|
2021-03-08 20:22:04 +03:00
|
|
|
from app.utils.bangs import gen_bangs_json
|
2021-07-01 02:00:01 +03:00
|
|
|
from app.utils.misc import gen_file_hash
|
2020-01-21 23:26:49 +03:00
|
|
|
from flask import Flask
|
2020-06-02 21:54:47 +03:00
|
|
|
from flask_session import Session
|
2020-12-18 00:39:35 +03:00
|
|
|
import json
|
2021-04-09 16:26:16 +03:00
|
|
|
import logging.config
|
2020-01-21 23:26:49 +03:00
|
|
|
import os
|
Add tor and http/socks proxy support (#137)
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
2020-10-29 03:47:42 +03:00
|
|
|
from stem import Signal
|
2021-03-28 20:27:08 +03:00
|
|
|
from dotenv import load_dotenv
|
2020-01-21 23:26:49 +03:00
|
|
|
|
2020-12-18 00:06:47 +03:00
|
|
|
app = Flask(__name__, static_folder=os.path.dirname(
|
|
|
|
os.path.abspath(__file__)) + '/static')
|
2021-03-28 20:27:08 +03:00
|
|
|
|
|
|
|
# Load .env file if enabled
|
2021-11-21 02:34:37 +03:00
|
|
|
if os.getenv('WHOOGLE_DOTENV', ''):
|
2021-03-28 20:27:08 +03:00
|
|
|
dotenv_path = '../whoogle.env'
|
|
|
|
load_dotenv(os.path.join(os.path.dirname(os.path.abspath(__file__)),
|
|
|
|
dotenv_path))
|
|
|
|
|
2021-04-01 07:23:30 +03:00
|
|
|
app.default_key = generate_user_key()
|
2020-06-06 00:24:44 +03:00
|
|
|
app.config['SECRET_KEY'] = os.urandom(32)
|
2020-06-02 21:54:47 +03:00
|
|
|
app.config['SESSION_TYPE'] = 'filesystem'
|
Improve public instance session management (#480)
This introduces a new approach to handling user sessions, which should
allow for users to set more reliable config settings on public instances.
Previously, when a user with cookies disabled would update their config,
this would modify the app's default config file, which would in turn
cause new users to inherit these settings when visiting the app for the
first time and cause users to inherit these settings when their current
session cookie expired (which was after 30 days by default I believe).
There was also some half-baked logic for determining on the backend
whether or not a user had cookies disabled, which lead to some issues
with out of control session file creation by Flask.
Now, when a user visits the site, their initial request is forwarded to
a session/<session id> endpoint, and during that subsequent request
their current session id is matched against the one found in the url. If
the ids match, the user has cookies enabled. If not, their original
request is modified with a 'cookies_disabled' query param that tells
Flask not to bother trying to set up a new session for that user, and
instead just use the app's fallback Fernet key for encryption and the
default config.
Since attempting to create a session for a user with cookies disabled
creates a new session file, there is now also a clean-up routine included
in the new session decorator, which will remove all sessions that don't
include a valid key in the dict. NOTE!!! This means that current user
sessions on public instances will be cleared once this update is merged
in. In the long run that's a good thing though, since this will allow session
mgmt to be a lot more reliable overall for users regardless of their cookie
preference.
Individual user sessions still use a unique Fernet key for encrypting queries,
but users with cookies disabled will use the default app key for encryption
and decryption.
Sessions are also now (semi)permanent and have a lifetime of 1 year.
2021-11-18 05:35:30 +03:00
|
|
|
app.config['SESSION_COOKIE_SAMESITE'] = 'strict'
|
2021-11-21 02:34:37 +03:00
|
|
|
|
|
|
|
if os.getenv('HTTPS_ONLY'):
|
|
|
|
app.config['SESSION_COOKIE_NAME'] = '__Secure-session'
|
|
|
|
app.config['SESSION_COOKIE_SECURE'] = True
|
|
|
|
|
2021-12-09 03:57:22 +03:00
|
|
|
app.config['VERSION_NUMBER'] = '0.7.0'
|
2020-12-18 00:06:47 +03:00
|
|
|
app.config['APP_ROOT'] = os.getenv(
|
|
|
|
'APP_ROOT',
|
|
|
|
os.path.dirname(os.path.abspath(__file__)))
|
|
|
|
app.config['STATIC_FOLDER'] = os.getenv(
|
|
|
|
'STATIC_FOLDER',
|
|
|
|
os.path.join(app.config['APP_ROOT'], 'static'))
|
2021-07-01 02:00:01 +03:00
|
|
|
app.config['BUILD_FOLDER'] = os.path.join(
|
|
|
|
app.config['STATIC_FOLDER'], 'build')
|
|
|
|
app.config['CACHE_BUSTING_MAP'] = {}
|
2021-03-21 04:21:41 +03:00
|
|
|
app.config['LANGUAGES'] = json.load(open(
|
2021-08-31 02:23:19 +03:00
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'settings/languages.json'),
|
|
|
|
encoding='utf-8'))
|
2021-03-21 04:21:41 +03:00
|
|
|
app.config['COUNTRIES'] = json.load(open(
|
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'settings/countries.json')))
|
2021-05-25 00:03:02 +03:00
|
|
|
app.config['TRANSLATIONS'] = json.load(open(
|
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'settings/translations.json')))
|
2021-06-28 17:26:51 +03:00
|
|
|
app.config['THEMES'] = json.load(open(
|
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'settings/themes.json')))
|
2020-12-18 00:06:47 +03:00
|
|
|
app.config['CONFIG_PATH'] = os.getenv(
|
|
|
|
'CONFIG_VOLUME',
|
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'config'))
|
|
|
|
app.config['DEFAULT_CONFIG'] = os.path.join(
|
|
|
|
app.config['CONFIG_PATH'],
|
|
|
|
'config.json')
|
2021-04-27 17:36:03 +03:00
|
|
|
app.config['CONFIG_DISABLE'] = os.getenv('WHOOGLE_CONFIG_DISABLE', '')
|
2020-12-18 00:06:47 +03:00
|
|
|
app.config['SESSION_FILE_DIR'] = os.path.join(
|
|
|
|
app.config['CONFIG_PATH'],
|
|
|
|
'session')
|
|
|
|
app.config['BANG_PATH'] = os.getenv(
|
|
|
|
'CONFIG_VOLUME',
|
|
|
|
os.path.join(app.config['STATIC_FOLDER'], 'bangs'))
|
|
|
|
app.config['BANG_FILE'] = os.path.join(
|
|
|
|
app.config['BANG_PATH'],
|
|
|
|
'bangs.json')
|
2021-11-02 19:35:40 +03:00
|
|
|
app.config['RELEASES_URL'] = 'https://github.com/' \
|
|
|
|
'benbusby/whoogle-search/releases'
|
2021-06-15 17:14:42 +03:00
|
|
|
|
|
|
|
# The alternative to Google Translate is treated a bit differently than other
|
|
|
|
# social media site alternatives, in that it is used for any translation
|
|
|
|
# related searches.
|
|
|
|
translate_url = os.getenv('WHOOGLE_ALT_TL', 'https://lingva.ml')
|
|
|
|
if not translate_url.startswith('http'):
|
|
|
|
translate_url = 'https://' + translate_url
|
|
|
|
app.config['TRANSLATE_URL'] = translate_url
|
|
|
|
|
2021-03-07 22:04:05 +03:00
|
|
|
app.config['CSP'] = 'default-src \'none\';' \
|
2021-06-15 17:14:42 +03:00
|
|
|
'frame-src ' + translate_url + ';' \
|
2021-03-21 02:52:06 +03:00
|
|
|
'manifest-src \'self\';' \
|
2021-05-05 19:51:11 +03:00
|
|
|
'img-src \'self\' data:;' \
|
2021-03-07 22:04:05 +03:00
|
|
|
'style-src \'self\' \'unsafe-inline\';' \
|
|
|
|
'script-src \'self\';' \
|
|
|
|
'media-src \'self\';' \
|
2021-08-31 16:57:50 +03:00
|
|
|
'connect-src \'self\';'
|
2020-06-02 21:54:47 +03:00
|
|
|
|
|
|
|
if not os.path.exists(app.config['CONFIG_PATH']):
|
|
|
|
os.makedirs(app.config['CONFIG_PATH'])
|
|
|
|
|
2020-06-06 00:24:44 +03:00
|
|
|
if not os.path.exists(app.config['SESSION_FILE_DIR']):
|
|
|
|
os.makedirs(app.config['SESSION_FILE_DIR'])
|
2020-06-02 23:38:29 +03:00
|
|
|
|
Add tor and http/socks proxy support (#137)
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
2020-10-29 03:47:42 +03:00
|
|
|
# Generate DDG bang filter, and create path if it doesn't exist yet
|
2020-10-10 22:55:14 +03:00
|
|
|
if not os.path.exists(app.config['BANG_PATH']):
|
|
|
|
os.makedirs(app.config['BANG_PATH'])
|
Add tor and http/socks proxy support (#137)
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
2020-10-29 03:47:42 +03:00
|
|
|
if not os.path.exists(app.config['BANG_FILE']):
|
|
|
|
gen_bangs_json(app.config['BANG_FILE'])
|
2020-10-10 22:55:14 +03:00
|
|
|
|
2021-07-01 02:00:01 +03:00
|
|
|
# Build new mapping of static files for cache busting
|
|
|
|
if not os.path.exists(app.config['BUILD_FOLDER']):
|
|
|
|
os.makedirs(app.config['BUILD_FOLDER'])
|
|
|
|
|
|
|
|
cache_busting_dirs = ['css', 'js']
|
|
|
|
for cb_dir in cache_busting_dirs:
|
|
|
|
full_cb_dir = os.path.join(app.config['STATIC_FOLDER'], cb_dir)
|
|
|
|
for cb_file in os.listdir(full_cb_dir):
|
|
|
|
# Create hash from current file state
|
|
|
|
full_cb_path = os.path.join(full_cb_dir, cb_file)
|
|
|
|
cb_file_link = gen_file_hash(full_cb_dir, cb_file)
|
|
|
|
build_path = os.path.join(app.config['BUILD_FOLDER'], cb_file_link)
|
2021-07-02 22:28:27 +03:00
|
|
|
|
|
|
|
try:
|
|
|
|
os.symlink(full_cb_path, build_path)
|
|
|
|
except FileExistsError:
|
|
|
|
# Symlink hasn't changed, ignore
|
|
|
|
pass
|
2021-07-01 02:00:01 +03:00
|
|
|
|
|
|
|
# Create mapping for relative path urls
|
|
|
|
map_path = build_path.replace(app.config['APP_ROOT'], '')
|
|
|
|
if map_path.startswith('/'):
|
|
|
|
map_path = map_path[1:]
|
|
|
|
app.config['CACHE_BUSTING_MAP'][cb_file] = map_path
|
|
|
|
|
|
|
|
# Templating functions
|
|
|
|
app.jinja_env.globals.update(clean_query=clean_query)
|
|
|
|
app.jinja_env.globals.update(
|
|
|
|
cb_url=lambda f: app.config['CACHE_BUSTING_MAP'][f])
|
|
|
|
|
2020-06-06 00:24:44 +03:00
|
|
|
Session(app)
|
2020-01-21 23:26:49 +03:00
|
|
|
|
Add tor and http/socks proxy support (#137)
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
2020-10-29 03:47:42 +03:00
|
|
|
# Attempt to acquire tor identity, to determine if Tor config is available
|
|
|
|
send_tor_signal(Signal.HEARTBEAT)
|
|
|
|
|
2020-12-18 00:06:47 +03:00
|
|
|
from app import routes # noqa
|
2021-04-09 16:26:16 +03:00
|
|
|
|
|
|
|
# Disable logging from imported modules
|
|
|
|
logging.config.dictConfig({
|
|
|
|
'version': 1,
|
|
|
|
'disable_existing_loggers': True,
|
|
|
|
})
|