celery: move send_email task to a better home in notification model

Avoid bundling everything from many different layers in one big task library.

This is more feasible now when we don't need kallithea.CELERY_APP set at import
time.

model/user: 'body' parameter to NotificationModel().create() is unused so remove it

The email template for new user registration mails does not use the 'body'
parameter, so don't pass anything.

model: remove unused 'subject' parameter of NotificationModel.create()

The subject of emails is determined with EmailNotificationModel._subj_map,
based on the notification type. The 'subject' parameter passed to
NotificationModel.create is completely unused.

Remove this parameter and update its callers, removing code that is now no
longer used.

hooks: import whole hooks module

Minimize impact if there should be cycles.

imports: try to use global imports unless it is a layering violation

To minimize the impact, do imports that violate the layering at runtime instead
of at import time.

model: changes toward import whole modules

*If* there should be circular dependencies, importing 'from' another module
could fail because the module at that time only was partially imported. That
had to be worked around by importing at runtime instead of globally.

Instead, try to always import whole modules. (But we should still try to avoid
cycles.)

lib: move webhelpers2 and friends to webutils

Gives less of the unfortunate use of helpers - especially in low level libs.

lib: move canonical_url & co to webutils

This gives less of the unfortunate use of helpers - especially in model.

lib: consistently use webutils.url instead of h.url

Templates are still the valid use case for accessing through h.

Simplify mocking of url() function.

pytest: use hmac.new instead of hmac.HMAC

According to documentation, hmac.new is the way to create a HMAC object ... and
the first argument is mandaatory and we don't want to name it.

This has no functional change but will address a pytype warning:

File "kallithea/model/user.py", line 304, in get_reset_password_token: Invalid keyword arguments (digestmod, key, msg) to function HMAC.__init__ [wrong-keyword-args]

isort: upgrade to 5.1.2

The changes to non-top-level imports seem nice.

db: drop SA caching_query and FromCache, and thus sql_cache_short beaker cache

It is not a good idea to have dead ORM objects. If we want caching, we should
do it explicit.

It is unknown how much this cache helps, but we can profile and introduce
better caching of simple data where relevant.

py3: remove safe_unicode in places where it no longer is needed because all strings (except bytes) already *are* unicode strings

(The remaining safe_unicode calls are still needed and can't just be removed,
generally because we in these cases still have to convert from bytes to unicode
strings.)

user: allow LDAP users with non-ASCII characters in the DN

The database and sqlalchemy use unicode, so any attempt at storing encoded data
is wrong and might fail.

Thus, use safe_unicode instead of encoding it using safe_str.

model: move notification types from Notification to NotificationModel

This commit is part of the removal of the UI notification feature from
Kallithea, which is not deemed useful in its current form. Only email
notifications are preserved.

As there is no database storage of notifications anymore, the Notification
class will be removed. However, the notification type definitions are still
used for email notifications, and need to live somewhere. As creating
notifications is always passing via NotificationModel, it makes sense to
move the types there.

tg: minimize future diff by some mocking and replacing some pylons imports with tg

No actual tg dependency yet, just a temporary hack faking tg as an alias for
pylons.

Based on work by Alessandro Molina.

model: remove BaseModel class

The only remaining purpose of this class was to provide the "sa" field,
allowing a custom SQLAlchemy session to be used for model operations.
However, this never actually worked, nor was it used anywhere.

There's always a global SQLAlchemy session associated with the current
thread; using another session for a single function call does not make
any sense (as sessions cannot be mixed), unless the code works carefully
to ensure the two sessions (and all objects loaded from them) are kept
completely separate. Suffice to say that Kallithea does no such thing,
thus there's no need to pretend to support multiple concurrent sessions.

cleanup: drop superfluous Session.add calls

Session.add should only be called on newly created database objects.

Per the Kallithea contribution guidelines:

When getting an object from the session (via Session().query or
any of the utility functions that look up objects in the database),
it's already part of the session, and should not be added again.

cleanup: refer less to User.DEFAULT_USER

Down the road we might want to identify the default user in another
way than by username.

db: clean up SQLAlchemy session flushes

Many calls to Session().flush() were completely superfluous and have
been removed. (See also the note on "flush" in the contributor docs.)
For the remaining calls, a comment has been added to explain why it's
necessary.

model: move code from __init__.py to base.py

Having too much code, in particular too much imports, inside a package's
__init__.py is a recipe for circular imports, and considered bad practice in
Python [1]

Move out everything from kallithea/model/__init__.py to a new file
kallithea/model/base.py and adapt the existing imports.

[1] http://docs.python-guide.org/en/latest/writing/structure/#packages

db: remove superfluous Session.add calls

Don't re-add objects to the SQLAlchemy Session just because they were
modified. Session.add is only for freshly constructed objects that
SQLAlchemy doesn't know about yet.

The rules are quite simple:

When creating a database object by calling the constructor directly, it
must explicitly be added to the session.

When creating an object using a factory function (like "create_repo"),
the returned object has already (by convention) been added to the
session, and should not be added again.

When getting an object from the session (via Session.query or any of the
utility functions that look up objects in the database), it's already
added, and should not be added again. SQLAlchemy notices attribute
modifications automatically for all objects it knows about.

auth: cleanup of EXTERN_TYPE_INTERNAL

Don't set it in top level namespace - it is a weak link between the database
and the actual implementation. Don't make it more than that.

Don't hardcode in that many places that 'internal' is the default - just call
it DEFAULT_AUTH_TYPE.

Don't use it for extern_name - it is only intended for use as extern_type.

Remove unused uses.

user: when updating a user, never set the password to None - the field is not nullable

We also don't allow setting empty password. Setting the database value to empty
string is secure but will never match any password.

celeryd: annotate tasks so they can be run directly without run_task

This also makes the system less forgiving about celery configuration problems
and thus easier to debug. I like that.

mails: restructure notification mail content

* Mention "The Thing" in the header, link to online version
* Show The Thing
* A dense summary of the essentials of the context below it

The html indentation is odd in order to make the next diff smaller.

The text version is based on:

for a in kallithea/templates/email_templates/*.html; do sed -e 's,<\([^%/>]\|/[^%>]\)*>,,g' -e 's,\.html",\.txt",g' -e 's,^ *,,g' -e 's/}/|n,unicode}/g' $a > ${a%%.html}.txt ; done

model: drop BaseModel.cls

It no longer serves a purpose.

users: fix crash when creating users with non ASCII characters

This was already changed by the cleaned up on the default branch in
330c671dd451 but happened to also fix this issue on the stable branch.

auth: only local passwords can be reset

Do for password reset what de9a3152c206 did for password change.

auth: use HMAC-SHA1 to calculate password reset token

The use of standard cryptographic primitives is always preferable, and
in this case allows us not to worry about length extension attacks
and possibly any number of issues that I'm not presently aware of.

Also fix a potential Unicode encoding problem.

auth: secure password reset implementation

This is a better implementation of password reset function, which
doesn't involve sending a new password to the user's email address
in clear text, and at the same time is stateless.

The old implementation generated a new password and sent it
in clear text to whatever email assigned to the user currently,
so that any user, possibly unauthenticated, could request a reset
for any username or email. Apart from potential insecurity, this
made it possible for anyone to disrupt users' workflow by repeatedly
resetting their passwords.

The idea behind this implementation is to generate
an authentication token which is dependent on the user state
at the time before the password change takes place, so the token
is one-time and can't be reused, and also to bind the token to
the browser session.

The token is calculated as SHA1 hash of the following:

* user's identifier (number, not a name)
* timestamp
* hashed user's password
* session identifier
* per-application secret

We use numeric user's identifier, as it's fixed and doesn't change,
so renaming users doesn't affect the mechanism. Timestamp is added
to make it possible to limit the token's validness (currently hard
coded to 24h), and we don't want users to be able to fake that field
easily. Hashed user's password is needed to prevent using the token
again once the password has been changed. Session identifier is
an additional security measure to ensure someone else stealing the
token can't use it. Finally, per-application secret is just another
way to make it harder for an attacker to guess all values in an
attempt to generate a valid token.

When the token is generated, an anonymous user is directed to a
confirmation page where the timestamp and the usernames are already
preloaded, so the user needs to specify the token. User can either
click the link in the email if it's really them reading it, or to type
the token manually.

Using the right token in the same session as it was requested directs
the user to a password change form, where the user is supposed to
specify a new password (twice, of course). Upon completing the form
(which is POSTed) the password change happens and a notification
mail is sent.

The test is updated to test the basic functionality with a bad and
a good token, but it doesn't (yet) cover all code paths.

The original work from Andrew has been thorougly reviewed and heavily
modified by Søren Løvborg.

spelling: fix inappropriate use of title case

These are not titles or labels, and should not be title cased.

auth: move UserModel.fill_data to AuthUser

Because the method is only used by AuthUser, and also because it's an
awful thing to do (copying a large but ill-defined set of attributes
from one object to another), and we don't want usage to spread.

auth: have fill_data take User object, not lookup key

Simplify the UserModel.fill_data interface by passing the actual db.User
object instead of a key by which it can be looked up. This also saves a
lookup in case the db.User is already loaded (which for now is only the
case in one place, but that'll change in the following changesets).

Also enhance fill_data docstring.

UserModel: remove methods that are redundant with db.User

UserModel().get_by_api_key is exactly equivalent to User.get_by_api_key.

UserModel's get_by_username and get_by_email are not exactly identical
to their db.User counterparts, due to a difference in the order of
optional arguments. Fortunately, these optional arguments are never
used.

cleanup: check for None object identity in cases where that is what the 'contract' says

Avoid applying bool() on complex objects - it might return something unexpected
such as the key (which might be 0 and thus be false). Checking for None is more
safe and faster.

validators: cleanup of message wording

NotReviewedRevisions was unused and could thus got the big cleanup ...

i18n: use plain strings on the english lookup side of translations

Unicode strings are unnecessarily complex and do not in any of our cases add
any value. It is thus better to consistently use plain strings.

utils: make API key generator more random

The API key generator abused temporary filenames in what seems to be an attempt
of creating keys that unambiguously specified the user and thus were unique
across users. A final hashing did however remove that property.

More importantly, tempfile is not documented to use secure random numbers ...
and it only uses 6 characters, giving approximately 36 bits of entropy.

Instead, use the cryptographically secure os.urandom directly to generate keys
with the same length but with the full 160 bits of entropy.

Reported and fixed by Andrew Bartlett.

UserModel: remove code duplication of User.get_by_username()

UserModel.get_by_username() and User.get_by_username() implement the same
logic and thus one should use the other instead of duplicating.

helpers: introduce user_or_none helper helper

Slight cleanup refactoring - will be useful later.

user: prevent deletion of users that are owners of a user group

TODO: make it possible to reassign group ownership

user: prevent deletion of users that are owners of a repo group

TODO: make it possible to reassign group ownership

cleanup: avoid some 'except Exception' catching - catch specific exceptions or log it and show what happened

This has a risk of introducing regressions ... but we want to get rid of all
exception muting and make the whole system less fragile and easier to debug.

email templates: send text/plain part as well

This change adds text parts to the email templates, as
HTML and text templates may be way to different to be
handled automatically. Also, use proper dash-dash-space
signature separator, so the email clients recognise it
for sure.

urls: introduce canonical_url config setting

All URLs that are shown or persisted or emailed will use this instead of the
current url.

This is convenient when the server has multiple names - for instance when
transitioning from one protocol or domain or hostname to another.

Correct licensing information in individual files.

The top-level license file is now LICENSE.md.

Also, in various places where there should have been joint copyright holders
listed, a single copyright holder was listed. It does not appear easy to add
a link to a large list of copyright holders in these places, so it simply
refers to the fact that various authors hold copyright.

In future, if an easy method is discovered to link to a list from those
places, we should do so.

Finally, text is added to LICENSE.md to point to where the full list of
copyright holders is, and that Kallithea as a whole is GPLv3'd.

Second step in two-part process to rename directories.
This is the actual directory rename.