Changeset - b66725ba01ed
[Not reviewed]
default
0 7 1
Thomas De Schampheleire - 7 years ago 2018-09-24 20:44:36
thomas.de_schampheleire@nokia.com
cli: add command 'kallithea-cli front-end-build'

Kallithea is under the GPL license, and we can thus only distribute any
generated code if we also ship the corresponding source.

We are moving towards a web front-end that use npm to download and compile
various open source components. The components might not be GPL, but if we
distribute any parts of their code (compiled or converted to other
representation), then we also must distribute the corresponding source under
the GPL.

It doesn't seem feasible for us to distribute the source of everything that
npm downloads and includes when we are building. It thus also doesn't seem
feasible for us to build and ship the compiled (possibly minified) front-end
code. Instead, we have to make it as smooth as possible for our users to
get up and running.

It doesn't seem feasible for us to ship or install npm. We must assume it is
available. That requirement must be documented clearly, and we must recommend
how to install npm for the most common platforms.

We could perhaps just document what manual steps to run. Kallithea doesn't
work out of the box anyway - it has to be configured and initialized. Extra
steps might not be a big problem.

Another approach is to call out to npm while pip is installing Kallithea and
download the requirements and build the files. It can be done by customizing
setuptools commands in setup.py. But: Python packaging is fragile. Even
though we only support pip, it really isn't built for things like this.
Custom output is muted and buffered and only shown if running with -v or the
command fails. And pip and setup.py can be used to build and install in so
many ways that we probably can't make it work reliably with all ways of
installing Kallithea.

The approach implemented by this commit is to add a custom cli command
'front-end-build' to run the required commands. This single user-facing
command can internally run various steps as needed. The only current
requirement is the presence of npm and an internet connection.

For now, this will just create/update style.css ... but currently probably
without any actual changes. The files created by npm (and the node_modules
directory) must *not* be a part of the release package made with 'setup.py
sdist'.

(Commit message is mostly written by Mads Kiilerich)
8 files changed with 64 insertions and 44 deletions:
0 comments (0 inline, 0 general)
docs/contributing.rst
Show inline comments
 
@@ -16,52 +16,51 @@ https://kallithea-scm.org/repos/kallithe
 
of Kallithea.
 

	
 
For now, we use Bitbucket_ for `pull requests`_ and `issue tracking`_. The
 
issue tracker is for tracking bugs, not for support, discussion, or ideas --
 
please use the `mailing list`_ or :ref:`IRC <readme>` to reach the community.
 

	
 
We use Weblate_ to translate the user interface messages into languages other
 
than English. Join our project on `Hosted Weblate`_ to help us.
 
To register, you can use your Bitbucket or GitHub account. See :ref:`translations`
 
for more details.
 

	
 

	
 
Getting started
 
---------------
 

	
 
To get started with Kallithea development::
 

	
 
        hg clone https://kallithea-scm.org/repos/kallithea
 
        cd kallithea
 
        virtualenv ../kallithea-venv
 
        source ../kallithea-venv/bin/activate
 
        pip install --upgrade pip setuptools
 
        pip install --upgrade -e .
 
        pip install --upgrade -r dev_requirements.txt
 
        npm install     # install dependencies - both tools and data
 
        npm run less    # for generating css from less
 
        kallithea-cli config-create my.ini
 
        kallithea-cli db-create -c my.ini --user=user --email=user@example.com --password=password --repos=/tmp
 
        kallithea-cli front-end-build
 
        gearbox serve -c my.ini --reload &
 
        firefox http://127.0.0.1:5000/
 

	
 
If you plan to use Bitbucket_ for sending contributions, you can also fork
 
Kallithea on Bitbucket_ first (https://bitbucket.org/conservancy/kallithea) and
 
then replace the clone step above by a clone of your fork. In this case, please
 
see :ref:`contributing-guidelines` below for configuring your fork correctly.
 

	
 

	
 
Contribution flow
 
-----------------
 

	
 
Starting from an existing Kallithea clone, make sure it is up to date with the
 
latest upstream changes::
 

	
 
        hg pull
 
        hg update
 

	
 
Review the :ref:`contributing-guidelines` and :ref:`coding-guidelines`.
 

	
 
If you are new to Mercurial, refer to Mercurial `Quick Start`_ and `Beginners
 
Guide`_ on the Mercurial wiki.
 

	
 
Now, make some changes and test them (see :ref:`contributing-tests`). Don't
docs/overview.rst
Show inline comments
 
@@ -48,53 +48,51 @@ Two different cases will pretty much cov
 
installed.
 

	
 
- The Kallithea source repository can be cloned and used -- it is kept stable and
 
  can be used in production. The Kallithea maintainers use the development
 
  branch in production. The advantage of installation from source and regularly
 
  updating it is that you take advantage of the most recent improvements. Using
 
  it directly from a DVCS also means that it is easy to track local customizations.
 

	
 
  Running ``pip install -e .`` in the source will use pip to install the
 
  necessary dependencies in the Python environment and create a
 
  ``.../site-packages/Kallithea.egg-link`` file there that points at the Kallithea
 
  source.
 

	
 
- Kallithea can also be installed from ready-made packages using a package manager.
 
  The official released versions are available on PyPI_ and can be downloaded and
 
  installed with all dependencies using ``pip install kallithea``.
 

	
 
  With this method, Kallithea is installed in the Python environment as any
 
  other package, usually as a ``.../site-packages/Kallithea-X-py2.7.egg/``
 
  directory with Python files and everything else that is needed.
 

	
 
  (``pip install kallithea`` from a source tree will do pretty much the same
 
  but build the Kallithea package itself locally instead of downloading it.)
 

	
 
.. note:: The front-end code is built with Node. Currently, it must be built
 
          locally after installing Kallithea. Assuming Node and the Node
 
          Package Manager is available, other tools and source code will be
 
          downloaded and installed. The front-end code can then be built from
 
          source locally.
 
.. note:: Kallithea includes front-end code that needs to be processed first.
 
The tool npm_ is used to download external dependencies and orchestrate the
 
processing. The ``npm`` binary must thus be available.
 

	
 

	
 
Web server
 
----------
 

	
 
Kallithea is (primarily) a WSGI_ application that must be run from a web
 
server that serves WSGI applications over HTTP.
 

	
 
Kallithea itself is not serving HTTP (or HTTPS); that is the web server's
 
responsibility. Kallithea does however need to know its own user facing URL
 
(protocol, address, port and path) for each HTTP request. Kallithea will
 
usually use its own HTML/cookie based authentication but can also be configured
 
to use web server authentication.
 

	
 
There are several web server options:
 

	
 
- Kallithea uses the Gearbox_ tool as command line interface. Gearbox provides
 
  ``gearbox serve`` as a convenient way to launch a Python WSGI / web server
 
  from the command line. That is perfect for development and evaluation.
 
  Actual use in production might have different requirements and need extra
 
  work to make it manageable as a scalable system service.
 

	
 
  Gearbox comes with its own built-in web server but Kallithea defaults to use
 
  Waitress_. Gunicorn_ is also an option. These web servers have different
 
@@ -123,24 +121,25 @@ There are several web server options:
 

	
 
The best option depends on what you are familiar with and the requirements for
 
performance and stability. Also, keep in mind that Kallithea mainly is serving
 
dynamically generated pages from a relatively slow Python process. Kallithea is
 
also often used inside organizations with a limited amount of users and thus no
 
continuous hammering from the internet.
 

	
 

	
 
.. _Python: http://www.python.org/
 
.. _Gunicorn: http://gunicorn.org/
 
.. _Waitress: http://waitress.readthedocs.org/en/latest/
 
.. _virtualenv: http://pypi.python.org/pypi/virtualenv
 
.. _Gearbox: http://turbogears.readthedocs.io/en/latest/turbogears/gearbox.html
 
.. _PyPI: https://pypi.python.org/pypi
 
.. _Apache httpd: http://httpd.apache.org/
 
.. _mod_wsgi: https://code.google.com/p/modwsgi/
 
.. _isapi-wsgi: https://github.com/hexdump42/isapi-wsgi
 
.. _uWSGI: https://uwsgi-docs.readthedocs.org/en/latest/
 
.. _nginx: http://nginx.org/en/
 
.. _iis: http://en.wikipedia.org/wiki/Internet_Information_Services
 
.. _pip: http://en.wikipedia.org/wiki/Pip_%28package_manager%29
 
.. _WSGI: http://en.wikipedia.org/wiki/Web_Server_Gateway_Interface
 
.. _HAProxy: http://www.haproxy.org/
 
.. _Varnish: https://www.varnish-cache.org/
 
.. _npm: https://www.npmjs.com/
docs/setup.rst
Show inline comments
 
.. _setup:
 

	
 
=====
 
Setup
 
=====
 

	
 

	
 
Preparing front-end
 
-------------------
 

	
 
Temporarily, in the current Kallithea version, some extra steps are required to
 
build front-end files:
 

	
 
Find the right ``kallithea/public/less`` path with::
 

	
 
    python -c "import os, kallithea; print os.path.join(os.path.dirname(os.path.abspath(kallithea.__file__)), 'public', 'less')"
 

	
 
Then run::
 

	
 
    npm install
 
    npm run less
 

	
 

	
 
Setting up Kallithea
 
--------------------
 

	
 
First, you will need to create a Kallithea configuration file. Run the
 
following command to do so::
 

	
 
    kallithea-cli config-create my.ini
 

	
 
This will create the file ``my.ini`` in the current directory. This
 
configuration file contains the various settings for Kallithea, e.g.
 
proxy port, email settings, usage of static files, cache, Celery
 
settings, and logging. Extra settings can be specified like::
 

	
 
    kallithea-cli config-create my.ini host=8.8.8.8 "[handler_console]" formatter=color_formatter
 

	
 
Next, you need to create the databases used by Kallithea. It is recommended to
 
use PostgreSQL or SQLite (default). If you choose a database other than the
 
default, ensure you properly adjust the database URL in your ``my.ini``
 
configuration file to use this other database. Kallithea currently supports
 
PostgreSQL, SQLite and MySQL databases. Create the database by running
 
the following command::
 

	
 
    kallithea-cli db-create -c my.ini
 

	
 
@@ -49,48 +33,52 @@ This will prompt you for a "root" path. 
 
Kallithea will store all of its repositories on the current machine. After
 
entering this "root" path ``db-create`` will also prompt you for a username
 
and password for the initial admin account which ``db-create`` sets
 
up for you.
 

	
 
The ``db-create`` values can also be given on the command line.
 
Example::
 

	
 
    kallithea-cli db-create -c my.ini --user=nn --password=secret --email=nn@example.com --repos=/srv/repos
 

	
 
The ``db-create`` command will create all needed tables and an
 
admin account. When choosing a root path you can either use a new
 
empty location, or a location which already contains existing
 
repositories. If you choose a location which contains existing
 
repositories Kallithea will add all of the repositories at the chosen
 
location to its database.  (Note: make sure you specify the correct
 
path to the root).
 

	
 
.. note:: the given path for Mercurial_ repositories **must** be write
 
          accessible for the application. It's very important since
 
          the Kallithea web interface will work without write access,
 
          but when trying to do a push it will fail with permission
 
          denied errors unless it has write access.
 

	
 
Finally, prepare the front-end by running::
 

	
 
    kallithea-cli front-end-build
 

	
 
You are now ready to use Kallithea. To run it simply execute::
 

	
 
    gearbox serve -c my.ini
 

	
 
- This command runs the Kallithea server. The web app should be available at
 
  http://127.0.0.1:5000. The IP address and port is configurable via the
 
  configuration file created in the previous step.
 
- Log in to Kallithea using the admin account created when running ``db-create``.
 
- The default permissions on each repository is read, and the owner is admin.
 
  Remember to update these if needed.
 
- In the admin panel you can toggle LDAP, anonymous, and permissions
 
  settings, as well as edit more advanced options on users and
 
  repositories.
 

	
 

	
 
Internationalization (i18n support)
 
-----------------------------------
 

	
 
The Kallithea web interface is automatically displayed in the user's preferred
 
language, as indicated by the browser. Thus, different users may see the
 
application in different languages. If the requested language is not available
 
(because the translation file for that language does not yet exist or is
 
incomplete), the language specified in setting ``i18n.lang`` in the Kallithea
 
configuration file is used as fallback. If no fallback language is explicitly
docs/upgrade.rst
Show inline comments
 
@@ -65,60 +65,49 @@ Verify that you are using the Python env
 
installed Kallithea in by running::
 

	
 
    pip freeze
 

	
 
This will list all packages installed in the current environment. If
 
Kallithea isn't listed, activate the correct virtual environment.
 
See the appropriate installation page for details.
 

	
 

	
 
4. Install new version of Kallithea
 
-----------------------------------
 

	
 
Please refer to the instructions for the installation method you
 
originally used to install Kallithea.
 

	
 
If you originally installed using pip, it is as simple as::
 

	
 
    pip install --upgrade kallithea
 

	
 
If you originally installed from version control, it is as simple as::
 

	
 
    cd my-kallithea-clone
 
    hg pull -u
 
    pip install --upgrade -e .
 

	
 
Temporarily, in the current version, an extra step is required to build
 
front-end files:
 

	
 
Find the right ``kallithea/public/less`` path with::
 

	
 
    python -c "import os, kallithea; print os.path.join(os.path.dirname(os.path.abspath(kallithea.__file__)), 'public', 'less')"
 

	
 
Then run::
 

	
 
    npm install
 
    npm run less
 
    kallithea-cli front-end-build
 

	
 

	
 
5. Upgrade your configuration
 
-----------------------------
 

	
 
Run the following command to create a new configuration (``.ini``) file::
 

	
 
    kallithea-cli config-create new.ini
 

	
 
Then compare it with your old config file and see what changed.
 

	
 
.. note::
 
    Please always make sure your ``.ini`` files are up to date. Errors
 
    can often be caused by missing parameters added in new versions.
 

	
 
.. _upgrade_db:
 

	
 

	
 
6. Upgrade your database
 
------------------------
 

	
 
.. note::
 
    If you are *downgrading* Kallithea, you should perform the database
 
    migration step *before* installing the older version. (That is,
docs/usage/customization.rst
Show inline comments
 
@@ -7,55 +7,52 @@ Customization
 
There are several ways to customize Kallithea to your needs depending on what
 
you want to achieve.
 

	
 

	
 
HTML/JavaScript/CSS customization
 
---------------------------------
 

	
 
To customize the look-and-feel of the web interface (for example to add a
 
company banner or some JavaScript widget or to tweak the CSS style definitions)
 
you can enter HTML code (possibly with JavaScript and/or CSS) directly via the
 
*Admin > Settings > Global > HTML/JavaScript customization
 
block*.
 

	
 

	
 
Style sheet customization with Less
 
-----------------------------------
 

	
 
Kallithea uses `Bootstrap 3`_ and Less_ for its style definitions. If you want
 
to make some customizations, we recommend to do so by creating a ``theme.less``
 
file. When you create a file named ``theme.less`` in the Kallithea root
 
directory, you can use this file to override the default style. For example,
 
you can use this to override ``@kallithea-theme-main-color``,
 
``@kallithea-logo-url`` or other `Bootstrap variables`_.
 

	
 
After creating the ``theme.less`` file, you need to regenerate the CSS files.
 
Install npm for your platform and run::
 
After creating the ``theme.less`` file, you need to regenerate the CSS files, by
 
running::
 

	
 
    npm install
 
    npm run less
 

	
 
in the Kallithea root directory.
 
    kallithea-cli front-end-build --no-install-deps
 

	
 
.. _bootstrap 3: https://getbootstrap.com/docs/3.3/
 
.. _bootstrap variables: https://getbootstrap.com/docs/3.3/customize/#less-variables
 
.. _less: http://lesscss.org/
 

	
 

	
 
Behavioral customization: rcextensions
 
--------------------------------------
 

	
 
Some behavioral customization can be done in Python using ``rcextensions``, a
 
custom Python package that can extend Kallithea functionality.
 

	
 
With ``rcextensions`` it's possible to add additional mappings for Whoosh
 
indexing and statistics, to add additional code into the push/pull/create/delete
 
repository hooks (for example to send signals to build bots such as Jenkins) and
 
even to monkey-patch certain parts of the Kallithea source code (for example
 
overwrite an entire function, change a global variable, ...).
 

	
 
To generate a skeleton extensions package, run::
 

	
 
    kallithea-cli extensions-create -c my.ini
 

	
 
This will create an ``rcextensions`` package next to the specified ``ini`` file.
 
See the ``__init__.py`` file inside the generated ``rcextensions`` package
kallithea/bin/kallithea_cli.py
Show inline comments
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 

	
 
# 'cli' is the main entry point for 'kallithea-cli', specified in setup.py as entry_points console_scripts
 
from kallithea.bin.kallithea_cli_base import cli
 

	
 
# import commands (they will add themselves to the 'cli' object)
 
import kallithea.bin.kallithea_cli_celery
 
import kallithea.bin.kallithea_cli_config
 
import kallithea.bin.kallithea_cli_db
 
import kallithea.bin.kallithea_cli_extensions
 
import kallithea.bin.kallithea_cli_front_end
 
import kallithea.bin.kallithea_cli_iis
 
import kallithea.bin.kallithea_cli_index
 
import kallithea.bin.kallithea_cli_ishell
 
import kallithea.bin.kallithea_cli_repo
kallithea/bin/kallithea_cli_front_end.py
Show inline comments
 
new file 100644
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 

	
 
import click
 
import kallithea.bin.kallithea_cli_base as cli_base
 

	
 
import os
 
import subprocess
 

	
 
import kallithea
 

	
 
@cli_base.register_command()
 
@click.option('--install-deps/--no-install-deps', default=True,
 
        help='Skip installation of dependencies, via "npm".')
 
@click.option('--generate/--no-generate', default=True,
 
        help='Skip generation of front-end files.')
 
def front_end_build(install_deps, generate):
 
    """Build the front-end.
 

	
 
    Install required dependencies for the front-end and generate the necessary
 
    files.  This step is complementary to any 'pip install' step which only
 
    covers Python dependencies.
 

	
 
    The installation of front-end dependencies happens via the tool 'npm' which
 
    is expected to be installed already.
 
    """
 
    rootdir = os.path.dirname(os.path.dirname(os.path.abspath(kallithea.__file__)))
 

	
 
    if install_deps:
 
        click.echo("Running 'npm install' to install front-end dependencies from package.json")
 
        subprocess.check_call(['npm', 'install'], cwd=rootdir)
 

	
 
    if generate:
 
        click.echo("Generating CSS")
 
        lesscpath = os.path.join(rootdir, 'node_modules', '.bin', 'lessc')
 
        lesspath = os.path.join(rootdir, 'kallithea', 'public', 'less', 'main.less')
 
        csspath = os.path.join(rootdir, 'kallithea', 'public', 'css', 'style.css')
 
        subprocess.check_call([lesscpath, '--relative-urls', '--source-map',
 
                '--source-map-less-inline', lesspath, csspath])
package.json
Show inline comments
 
{
 
  "name": "kallithea",
 
  "private": true,
 
  "dependencies": {
 
    "bootstrap": "3.3.7"
 
  },
 
  "devDependencies": {
 
    "less": "~2.7",
 
    "less-plugin-clean-css": "~1.5"
 
  },
 
  "scripts": {
 
    "less": "lessc --relative-urls --source-map --source-map-less-inline kallithea/public/less/main.less kallithea/public/css/style.css"
 
  }
 
}
0 comments (0 inline, 0 general)