Clicking "Add new" on /_admin/user_groups/1/edit/perms would flicker the input fields, and then show "User group permissions updated".
That turns out to be because HTML button tags default to 'submit' inside forms, and in this place (but not for user or repo group permissions) we didn't specify the button type as 'button'.
Deletion of a repository group that has a parent group (i.e. is not at the root of the repository group tree) failed as follows:
Traceback (most recent call last): [...] File ".../lib/python3.9/site-packages/tg/configurator/components/dispatch.py", line 114, in _call_controller return controller(*remainder, **params) File "<decorator-gen-5>", line 2, in delete
File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/lib/auth.py", line 572, in __wrapper return func(*fargs, **fkwargs) File "/home/tdescham/repo/contrib/kallithea/kallithea-release/kallithea/controllers/admin/repo_groups.py", line 271, in delete if gr.parent_group: File ".../lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 294, in __get__ return self.impl.get(instance_state(instance), dict_) File ".../lib/python3.9/site-packages/sqlalchemy/orm/attributes.py", line 730, in get value = self.callable_(state, passive) File ".../lib/python3.9/site-packages/sqlalchemy/orm/strategies.py", line 717, in _load_for_state raise orm_exc.DetachedInstanceError( sqlalchemy.orm.exc.DetachedInstanceError: Parent instance <RepoGroup at 0x7f1f2664f4c0> is not bound to a Session; lazy load operation of attribute 'parent_group' cannot proceed (Background on this error at: http://sqlalche.me/e/13/bhk3)
In the reference 'gr.parent_group', 'gr' is an SQLAlchemy object referring to the group being deleted, and 'gr.parent_group' is a lazy reference to its parent group. The 'lazy' means that the parent group object is not loaded automatically when 'gr' is assigned, but instead will be loaded on-the-fly when the parent group is actually accessed. See [1] and [2] for more information.
The problem was that the lazy 'parent_group' attribute was accessed _after_ deleting the database object it was part of.
Fix this by obtaining a handle to the parent group _before_ deleting the subgroup.
controllers: don’t pass start=0 to BaseRepository.get_changesets()
MercurialRepository.get_changesets() can fail if passing start=0 if the revision 0 is not in self.revisions. That can happen if revision 0 is not in the visible subset of the revisions in the repository. Before Kallithea changeset 7c43e15fb8bc7a73f17f577e59a4698589b6809d, it was working by chance because start=0 was treated like start=None in the relevant places (GitRepository.get_changesets still does that).
The intention of passing start=0 was seemingly to not limit the start. Therefore passing start=None (or nothing, as it’s the default value) should be correct.
I got the following traceback before this change:
Traceback (most recent call last): File "~/vcs/kallithea/kallithea/controllers/changelog.py", line 117, in index collection = c.db_repo_scm_instance.get_changesets(start=0, end=revision, File "~/vcs/kallithea/kallithea/lib/vcs/backends/hg/repository.py", line 529, in get_changesets start_pos = None if start is None else self.revisions.index(start_raw_id) ValueError: '4257f758b3eaacaebb6956d1aefc019afab956b8' is not in list
Running "kallithea-cli front-end-build" with npm 7.21.1 gave:
npm WARN old lockfile The package-lock.json file was created with an old version of npm, npm WARN old lockfile so supplemental metadata must be fetched from the registry. npm WARN old lockfile npm WARN old lockfile This is a one-time fix-up, please be patient...
files: fix raw download of repo files with names with unicode points above 256 in name
Raw download had apparently only been tested with non-ascii characters that were latin1. That was apparently a (too) simple case that worked without crashing.
Files with unicode code points above 256 in their name would fail to download, when Waitress failed like this, trying to get a real byte string by encoding WSGI headers to latin1: UnicodeEncodeError: 'latin-1' codec can't encode characters in position 84-85: ordinal not in range(256)
HTTP headers are of course byte strings on the network, but Python3 WSGI does unfortunately neither expose it as bytes nor as unicode strings to be encoded as utf-8. Instead, it uses unicode strings with byte values encoded as code points 0-255. That is achieved by decoding the utf-8 encoded bytes as latin1.
For raw downloads, the recommended download filename is provided in the Content-Disposition header. The problem is that it was provided as a real unicode string.
Fixed by applying the "proper" latin1-decoding of a utf8-encoding.
i18n: fix duplicate entries after fixing odd space before ? in "Don't have an account ?"
Dropping the odd spaces in 462064bd9489 gave some duplicate translation strings ... but there are no actual conflicting translations, so it can be resolved "trivially".
ini: set translate_forwarded_server = False when using paste prefix middleware
Paste Deploy PrefixMiddleware will be default trust HTTP headers that a proxy server might have set. That can be a problem if there is no proxy server or if the proxy just pass these headers through.
Change the configuration example to disable this translation.
To catch this case anyway, add a new check to verify that the parsed URL can roundtrip back to the original representation with urllib.parse.urlunparse .
The actual exception might vary, but one of them should always fire.
There is a risk that the new check will reject some URLs that somehow isn't normalized. No such cases have been found yet.
auth: only use X- headers instead of wsgi.url_scheme if explicitly told so in url_scheme_header - drop https_fixup setting
Before, several X- headers would be trusted to overrule the actual connection protocol (http or https) seen by the Kallithea WSGI server. That was mainly when https_fixup were set, but it incorrectly also kicked in if https_fixup or use_htsts were configured. The ambiguity of which headers were used also made it less reliable. The proxy server not only had to be configured to set one of the headers correctly, it also had to make sure other headers were not passed on from the client. It would thus in some cases be possible for clients to fake the connection scheme, and thus potentially be possible to bypass restrictions configured in Kallithea.
Fixed by making it configurable which WSGI environment variable to use for the protocol. Users can configure url_scheme_header to for example HTTP_X_FORWARDED_PROTO instead of using the default wsgi.url_scheme .
This change is a bit similar to what is going on in the https_fixup middleware, but is doing a bit more of what for example is happening in similar code in werkzeug/middleware/proxy_fix.py .
The semantics of the old https_fixup were unsafe, so it has been dropped. Admins that are upgrading must change their configuration to use the new url_scheme_header option.
auth: only use X- headers instead of REMOTE_ADDR if explicitly told so in remote_addr_header
Before, X-Forwarded-For (and others) headers would *always* be trusted blindly, also in setups without a proxy server. It would thus in some cases be possible for users to fake their IP, and thus potentially be possible to bypass IP restrictions configured in Kallithea.
Fixed by making it configurable which WSGI environment variable to use for the remote address. Users can configure remote_addr_header to for example HTTP_X_FORWARDED_FOR instead of using the default REMOTE_ADDR.
This change is a bit similar to what is going on in the https_fixup middleware, but is doing a bit more of what for example is happening in similar code in werkzeug/middleware/proxy_fix.py .
Trusting the *first* IP in HTTP_X_FORWARDED_FOR would allow clients to claim any IP, which could be used to bypass IP restrictions configured in Kallithea.
Instead, only trust the last proxy in the chain, and thus only use the *last* IP in HTTP_X_FORWARDED_FOR. (In setups where more than last IP should be trusted, the last proxy server in the chain must be configured rewrite the header accordingly.)
It was made mandatory in 9685f50a69d0. At that time it seemed like it always were present due to indirect dependencies, but apparently that is no longer the case.
pytype: add assertion to guide pytype through Whoosh usage
Mute pytype warnings:
File "kallithea/controllers/admin/admin.py", line 105, in _journal_filter: No attribute 'fieldname' on None [attribute-error] In Optional[Any] File "kallithea/controllers/admin/admin.py", line 106, in _journal_filter: No attribute 'text' on None [attribute-error] In Optional[Any] File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'startdate' on None [attribute-error] In Optional[Any] File "kallithea/controllers/admin/admin.py", line 107, in _journal_filter: No attribute 'enddate' on None [attribute-error] In Optional[Any]
pytype: add some assertions to guide pytype through const dict with tricky typing structure
These assertions also make the code more explicit and slightly more readable.
Mute pytype warnings:
File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'path' on str [attribute-error] In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str] File "kallithea/tests/vcs/base.py", line 76, in setup_repo: No attribute 'content' on str [attribute-error] In Union[kallithea.lib.vcs.nodes.FileNode, nothing, str]
pytype: add Python type annotations where necessary to guide pytype
Mute pytype warnings:
File "kallithea/lib/auth.py", line 142, in _cached_perms_data: No attribute 'DEFAULT_USER_ID' on module 'kallithea' [module-attr] File "kallithea/lib/vcs/backends/base.py", line 73, in ...: No attribute '...' on BaseRepository [attribute-error] File "kallithea/lib/vcs/backends/base.py", line 405, in ...: No attribute '...' on BaseChangeset [attribute-error] File "kallithea/tests/api/api_base.py", line 2397, in test_api_get_changeset: No attribute 'TEST_REVISION' on _BaseTestApi [attribute-error] File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_DST' on _BaseTestApi [attribute-error] File "kallithea/tests/api/api_base.py", line 2445, in test_api_get_pullrequest: No attribute 'TEST_PR_SRC' on _BaseTestApi [attribute-error] File "kallithea/tests/api/api_base.py", line 2467, in test_api_get_pullrequest: No attribute 'TEST_PR_REVISIONS' on _BaseTestApi [attribute-error] File "kallithea/tests/api/api_base.py", line 67, in api_call: No attribute 'app' on _BaseTestApi [attribute-error] File "kallithea/tests/base.py", line 154, in log_user: No attribute 'app' on TestController [attribute-error] File "kallithea/tests/base.py", line 169, in _get_logged_user: No attribute '_logged_username' on TestController [attribute-error]
pytype: mute errors from import of optional or platform specific modules
Mute pytype warnings:
File "kallithea/bin/kallithea_cli_iis.py", line 69, in iis_install: Can't find module 'isapi_wsgi'. [import-error] File "kallithea/config/post_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr] File "kallithea/config/pre_receive_tmpl.py", line 24, in <module>: No attribute 'setmode' on module 'msvcrt' [module-attr] File "kallithea/lib/compat.py", line 59, in kill: No attribute 'windll' on module 'ctypes' [module-attr] File "kallithea/lib/utils.py", line 242, in is_valid_repo_uri: Can't find module 'hgsubversion.svnrepo'. [import-error] File "kallithea/tests/scripts/manual_test_concurrency.py", line 203, in <module>: No attribute '_RandomNameSequence' on module 'tempfile' [module-attr]
Spotted by pytype --strict-import: File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'utils' on module 'kallithea.lib' [module-attr] File "kallithea/bin/kallithea_cli_db.py", line 73, in db_create: No attribute 'scm' on module 'kallithea.model' [module-attr]
It happened to work anyway due to the import chain ... but it is better to be explicit.
hooks: when overwriting non-Kallithea hooks, move the old hook to .bak
Note: This will only happen when actually overwriting an unknown hook. Repeated hook installation will thus not clobber the existing .bak file until a non-Kallithea hook is found again.
hooks: be more consistent in only using active Ui entries
There is no UI to control or display the ui_active value for custom hooks, but *if* they are inactive, they will be ignored in make_ui, and it will be misleading and confusing to show them in the list of active custom hooks.
There *should* never be any inactive hooks entries, but let's be consistent in handling the case *if* it should happen. (It happened for me while hacking around.)
celery: upgrade to Celery 5.0 ... and adjust for Click API
Celery 5 has apparently no relevant API or config changes.
Celery is however switching to click. run_from_argv goes away, and there is no simple way to do as before and start the worker with our Celery app but still use Celery's own command line parser.
Apply hacks to make sure it still is possible to run like: