The python cryptography library isn't able to be built without the rust
compiler for some stages the cross-platform buildx workflow. This hasn't
been the case in the past, but for some reason since upgrading to the
latest cryptography version there are now errors with installing that
package. Since dependencies in the builder stage are not included in the
final build, we can safely include rust as a fallback to ensure this
doesn't happen without bloating the size of the final build.
The whoogle.env file previously needed to be created and enabled using
the WHOOGLE_DOTENV var. This removes the second step and loads the env
file if it's found during app init.
The Dockerfile has also been updated to copy in whoogle.env if it
exists.
Fixes#909
Adds support for encoding (and optionally encrypting) user config values as
a single string that can be passed to any endpoint with the "preferences" url
param.
Co-authored-by: Ben Busby <contact@benbusby.com>
Farside can now redirect quora links to querte instances and imdb links
to libremdb instances. This updates Whoogle to perform link replacements
for both services when site alts are configured.
* Relativization of search results
* Fix JavaScript error when opening images
* Replace single-letter logo and remove sign-in link
* Add `WHOOGLE_URL_PREFIX` env var to support relative path redirection
The `WHOOGLE_URL_PREFIX` var can now be set to fix internal app
redirects, such as the `/session` redirect performed on the first visit
to the Whoogle home page.
Co-authored-by: Ben Busby <contact@benbusby.com>
Bibliogram uses a slightly different URL format than Instagram, and
requires a "u/" before the username when replacing Instagram links. This
was already implemented everywhere else except the Dockerfile.
Wikipedia, imgur, and translate alternatives were all still using
hardcoded URLs when replaced with their respective alternative frontend.
This updates them to use farside instead.
If the config dir already exists, setting the mode (`-m 777`) doesn't
actually work as it should. This change splits the command into two
separate commands for directory creation and enabling the directory to
be writable by all.
Fixes#658
Depending on bash wasn't strictly necessary, as the two minimal scripts
in the repo were both nearly POSIX anyways.
Aside from simplifying the repo's dependencies a little bit, this also
helps reduce the overall Docker image size as an added bonus.
Outdated pip versions require a rust compiler to install the
cryptography package. Ensuring that pip is up to date should eliminate
the recent buildx errors where a prebuilt cryptography wheel is not
available.
* Integrate Farside into Whoogle
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
* Expand conversion of config<->url params
Config settings can now be translated to and from URL params using a
predetermined set of "safe" keys (i.e. config settings that easily
translate to URL params).
* Allow jumping instances via Farside when ratelimited
When instances are ratelimited (when a captcha is returned instead of
the user's search results) the user can now hop to a new instance via
Farside, a new backend service that redirects users to working instances
of a particular frontend. In this case, it presents a user with a
Farside link to a new Whoogle (or Searx) instance instead, so that the
user can resume their search.
For the generated Farside->Whoogle link, the generated link includes the
user's current Whoogle configuration settings as URL params, to ensure a
more seamless transition between instances. This doesn't translate to
the Farside->Searx link, but potentially could with some changes.
Closes#554Closes#559
With 843632a, whoogle.env is now gitignored and should only be created
by users from the whoogle.template.env file. Since the file no longer
exists, the docker build cannot copy it in by default. This just
conditionally copies the file in if it exists.
scribe.rip is a privacy respecting front end for medium.com. This
feature allows medium.com results to be replaced with scribe.rip links,
and works for both regular medium.com domains as well as user specific
subdomains (i.e. user.medium.com).
[scribe.rip website](https://scribe.rip)
[scribe.rip source code](https://git.sr.ht/~edwardloveall/scribe)
Co-authored-by: Ben Busby <noreply+git@benbusby.com>
* Add support for Lingva translations in results
Searches that contain the word "translate" and are normal search queries
(i.e. not news/images/video/etc) now create an iframe to a Lingva url to
translate the user's search using their configured search language.
The Lingva url can be configured using the WHOOGLE_ALT_TL env var, or
will fall back to the official Lingva instance url (lingva.ml).
For more info, visit https://github.com/TheDavidDelta/lingva-translate
* Add basic test for lingva results
* Allow user specified lingva instances through csp frame-src
* Fix pep8 issue
The wget method seemed to have a possible issue with creating endless
index.html copies (despite being specified to output to console only),
so this has been updated to use curl instead.
Also uses new non-authenticated "healthz" route to perform the
healthcheck.
Fix#316Fix#313
wget's default behavior to download contents to a file is not always a
valid method to use as a Docker healthcheck, due to permission issues.
This circumvents any issues there by redirecting the healthcheck output
to stdout, which shouldn't cause any issues regardless of context.
Fixes#272
This allows the user to enable their preferred settings in a variety of
ways, depending on their deployment preference. Values added to
whoogle.env can be enabled using WHOOGLE_DOTENV=1, in which case all
values in the env var file will overwrite defaults or user provided
settings.
Co-authored-by: Ben Busby <benbusby@protonmail.com>
Introduces a new script for quickly regenerating a Heroku instance
(typically with a new IP) to handle the rare circumstances where an
instance is flagged by Google and prompted for a captcha.
The config/ dir was renamed to misc/ to avoid confusion with the Docker
mounted config volume, and to more closely match its intended purpose
(which is to contain all miscellaneous features/scripts/etc that add
functionality to Whoogle, but are not critical for general use).
* Adds the ability to redirect reddit.com to libredd.it using the existing
"site alts" config setting.
This adds the WHOOGLE_ALT_RD environment variable for optionally
redirecting reddit links to libreddit
(https://github.com/spikecodes/libreddit).
* Include libreddit in home page site alt note
Pip installs of whoogle search were missing access to the misc/ folder,
which previously contained the language and country json files. These
have been moved to app/misc, and the previous root level misc/ was
renamed to config/ (since it now only contains the tor config files).
Bump to 0.3.1.
The Dockerfile was incorrectly assigning an empty string to the site alt
values, which caused the filter to update relevant sites with an empty
host. This replaces the empty string with the correct domain for each
site alternative.
* Add ability to configure site alts w/ env vars
Site alternatives (i.e. twitter.com -> nitter.net) can now be configured
using environment variables:
WHOOGLE_ALT_TW='nitter.net' # twitter alt
WHOOGLE_ALT_YT='invidio.us' # youtube alt
WHOOGLE_ALT_IG='bibliogram.art/u' # instagram alt
Updated testing to confirm results have been modified.
* Add site alt vars to docker settings and readme
The tor service is now started by calling a script which runs tor
according to the current container user. If the user is root, the script
will begin the tor service as normal. Otherwise, it runs tor as the
current user. This primarily is meant to address the issue with Heroku
builds (which don't have a root user) not being able to start tor as a
service.
Also refactored the rc/ dir to misc/ (with a tor/ subdir) since that
makes more sense.
* Add tor and http/socks proxy support
Allows users to enable/disable tor from the config menu, which will
forward all requests through Tor.
Also adds support for setting environment variables for alternative
proxy support. Setting the following variables will forward requests
through the proxy:
- WHOOGLE_PROXY_USER (optional)
- WHOOGLE_PROXY_PASS (optional)
- WHOOGLE_PROXY_TYPE (required)
- Can be "http", "socks4", or "socks5"
- WHOOGLE_PROXY_LOC (required)
- Format: "<ip address>:<port>"
See #30
* Refactor acquire_tor_conn -> acquire_tor_identity
Also updated travis CI to set up tor
* Add check for Tor socket on init, improve Tor error handling
Initializing the app sends a heartbeat request to Tor to check for
availability, and updates the home page config options accordingly. This
heartbeat is sent on every request, to ensure Tor support can be
reconfigured without restarting the entire app.
If Tor support is enabled, and a subsequent request fails, then a new
TorError exception is raised, and the Tor feature is disabled until a
valid connection is restored.
The max attempts has been updated to 10, since 5 seemed a bit too low
for how quickly the attempts go by.
* Change send_tor_signal arg type, update function doc
send_tor_signal now accepts a stem.Signal arg (a bit cleaner tbh). Also
added the doc string for the "disable" attribute in TorError.
* Fix tor identity logic in Request.send
* Update proxy init, change proxyloc var name
Proxy is now only initialized if both type and location are specified,
as neither have a default fallback and both are required. I suppose the
type could fall back to http, but seems safer this way.
Also refactored proxyurl -> proxyloc for the runtime args in order to
match the Dockerfile args.
* Add tor/proxy support for Docker builds, fix opensearch/init
The Dockerfile is now updated to include support for Tor configuration,
with a working torrc file included in the repo.
An issue with opensearch was fixed as well, which was uncovered during
testing and was simple enough to fix here. Likewise, DDG bang gen was
updated to only ever happen if the file didn't exist previously, as
testing with the file being regenerated every time was tedious.
* Add missing "@" for socks proxy requests
Executable renamed to "run" to avoid confusion with pip installed script
Updated heroku deploy button to use the heroku-app branch, which by
default enforces HTTPS
Added instructions for enforcing HTTPS on various deployment options,
with note about how this isn't a required task.
Updated setup.py description to use improved app description
* Adding HTTPS enforcement
Command line runs of Whoogle Search through pip/pipx/etc will need the
`--https-only` flag appended to the run command.
Docker runs require the `use_https` build arg applied.
* Update README.md
Moved https-only note to top of docker run command, updated pip runner help output
* Dockerfile: removed HTTPS enforcement, updated PORT setting
Dockerfile no longer enforces an HTTPS connection, but still allows for
setting via a build arg. The Flask server port is now configurable as a
build arg as well, by setting a port number to "whoogle_port"
* Fixed incorrect port assignment
* Use slim version of docker container
This massively reduces the size of the final container (330mb -> 60mb)
* Install libcurl and libssl explicitly in container
They appear to be missing from `-slim` variant containers
Co-authored-by: Ben Busby <benbusby@pm.me>
Co-authored-by: Ben Busby <benbusby@pm.me>
Ben Busby
João
gdm85
Albony Cal
ras07
DUO Labs
Roy Zuo
Ben Busby
Vansh Comar
Ben Busby
Suika
Shimul
Basti
Ben Busby
Jake Howard
Daniel Sendzik