Changeset - 213085032127
[Not reviewed]
default
3 10 2
Mads Kiilerich - 9 years ago 2016-09-12 17:50:11
madski@unity3d.com
gearbox: make a make-config sub-command available again

Drop the old kallithea-config command line tool and rework it to a replacement
for the make-config paster command. Make-config was experimental in paste, it
doesn't exist with gearbox, it was tied to pylons, and we already have the
luxury problem of having two ways of doing almost the same thing. This is a
good opportunity to get rid of one of them.

This replacement tool is Kallithea specific and doesn't need to be told that it
is Kallithea it has to create a config file for. The command should thus
perhaps be given a Kallithea specific name instead of the very generic name ...
13 files changed with 66 insertions and 693 deletions:
0 comments (0 inline, 0 general)
MANIFEST.in
Show inline comments
 
@@ -2,20 +2,19 @@ include           Apache-License-2.0.txt
 
include           CONTRIBUTORS
 
include           COPYING
 
include           LICENSE-MERGELY.html
 
include           LICENSE.md
 
include           MIT-Permissive-License.txt
 
include           README.rst
 
include           development.ini
 
recursive-include docs *
 
recursive-include init.d *
 
recursive-include kallithea/alembic *
 
include           kallithea/bin/ldap_sync.conf
 
include           kallithea/bin/template.ini.mako
 
include           kallithea/config/deployment.ini_tmpl
 
recursive-include kallithea/i18n *
 
recursive-include kallithea/public *
 
recursive-include kallithea/templates *
 
recursive-include kallithea/tests/fixtures *
 
recursive-include kallithea/tests/scripts *
 
include           kallithea/tests/test.ini
 
include           kallithea/tests/vcs/aconfig
docs/contributing.rst
Show inline comments
 
@@ -27,25 +27,25 @@ for more details.
 

	
 
Getting started
 
---------------
 

	
 
To get started with development::
 

	
 
        hg clone https://kallithea-scm.org/repos/kallithea
 
        cd kallithea
 
        virtualenv ../kallithea-venv
 
        source ../kallithea-venv/bin/activate
 
        pip install --upgrade pip setuptools
 
        pip install -e .
 
        TODO make-config Kallithea my.ini
 
        gearbox make-config my.ini
 
        gearbox setup-db -c my.ini --user=user --email=user@example.com --password=password --repos=/tmp
 
        gearbox serve -c my.ini --reload &
 
        firefox http://127.0.0.1:5000/
 

	
 
You can also start out by forking https://bitbucket.org/conservancy/kallithea
 
on Bitbucket_ and create a local clone of your own fork.
 

	
 

	
 
Running tests
 
-------------
 

	
 
After finishing your changes make sure all tests pass cleanly. Install the test
docs/dev/dbmigrations.rst
Show inline comments
 
@@ -9,25 +9,25 @@ If you are developing a Kallithea featur
 
changes, you should make a matching Alembic database migration script:
 

	
 
1. :ref:`Create a Kallithea configuration and database <setup>` for testing
 
   the migration script, or use existing ``development.ini`` setup.
 

	
 
   Ensure that this database is up to date with the latest database
 
   schema *before* the changes you're currently developing. (Do not
 
   create the database while your new schema changes are applied.)
 

	
 
2. Create a separate throwaway configuration for iterating on the actual
 
   database changes::
 

	
 
    TODO make-config Kallithea temp.ini
 
    gearbox make-config temp.ini
 

	
 
   Edit the file to change database settings. SQLite is typically fine,
 
   but make sure to change the path to e.g. ``temp.db``, to avoid
 
   clobbering any existing database file.
 

	
 
3. Make your code changes (including database schema changes in ``db.py``).
 

	
 
4. After every database schema change, recreate the throwaway database
 
   to test the changes::
 

	
 
    rm temp.db
 
    gearbox setup-db -c temp.ini --repos=/var/repos --user=doe --email doe@example.com --password=123456 --no-public-access --force-yes
docs/installation_win.rst
Show inline comments
 
@@ -157,25 +157,25 @@ added to the ``PATH`` environment variab
 
``gzip.exe`` are available.
 

	
 
Step 9 -- Configuring Kallithea
 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

	
 
Steps taken from `<setup.html>`_
 

	
 
You have to use the same command prompt as in Step 7, so if you closed
 
it, reopen it following the same commands (including the "activate"
 
one). When ready, type::
 

	
 
  cd C:\Kallithea\Bin
 
  TODO make-config Kallithea production.ini
 
  gearbox make-config production.ini
 

	
 
Then you must edit production.ini to fit your needs (IP address, IP
 
port, mail settings, database, etc.). `NotePad++`__ or a similar text
 
editor is recommended to properly handle the newline character
 
differences between Unix and Windows.
 

	
 
__ http://notepad-plus-plus.org/
 

	
 
For the sake of simplicity, run it with the default settings. After your edits (if any) in the previous command prompt, type::
 

	
 
  gearbox setup-db -c production.ini
 

	
docs/installation_win_old.rst
Show inline comments
 
@@ -195,25 +195,25 @@ The prompt will change into "(Env) C:\\K
 
Some warnings will appear, don't worry as they are normal.
 

	
 
Step 8 -- Configuring Kallithea
 
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 

	
 
steps taken from http://packages.python.org/Kallithea/setup.html
 

	
 
You have to use the same Visual Studio 2008 command prompt as Step7, so
 
if you closed it reopen it following the same commands (including the
 
"activate" one). When ready, just type::
 

	
 
  cd C:\Kallithea\Bin
 
  TODO make-config Kallithea production.ini
 
  gearbox make-config production.ini
 

	
 
Then, you must edit production.ini to fit your needs (network address and
 
port, mail settings, database, whatever). I recommend using NotePad++
 
(free) or similar text editor, as it handles well the EndOfLine
 
character differences between Unix and Windows
 
(http://notepad-plus-plus.org/)
 

	
 
For the sake of simplicity lets run it with the default settings. After
 
your edits (if any), in the previous Command Prompt, type::
 

	
 
 gearbox setup-db -c production.ini
 

	
docs/setup.rst
Show inline comments
 
@@ -2,25 +2,25 @@
 

	
 
=====
 
Setup
 
=====
 

	
 

	
 
Setting up Kallithea
 
--------------------
 

	
 
First, you will need to create a Kallithea configuration file. Run the
 
following command to do so::
 

	
 
    TODO make-config Kallithea my.ini
 
    gearbox make-config my.ini
 

	
 
This will create the file ``my.ini`` in the current directory. This
 
configuration file contains the various settings for Kallithea, e.g.
 
proxy port, email settings, usage of static files, cache, Celery
 
settings, and logging.
 

	
 
Next, you need to create the databases used by Kallithea. It is recommended to
 
use PostgreSQL or SQLite (default). If you choose a database other than the
 
default, ensure you properly adjust the database URL in your ``my.ini``
 
configuration file to use this other database. Kallithea currently supports
 
PostgreSQL, SQLite and MySQL databases. Create the database by running
 
the following command::
docs/upgrade.rst
Show inline comments
 
@@ -82,31 +82,29 @@ If you originally installed using pip, i
 
    pip install --upgrade kallithea
 

	
 
If you originally installed from version control, it is as simple as::
 

	
 
    cd my-kallithea-clone
 
    hg pull -u
 
    pip install -e .
 

	
 

	
 
5. Upgrade your configuration
 
-----------------------------
 

	
 
Run the following command to upgrade your configuration (``.ini``) file::
 

	
 
    TODO make-config Kallithea my.ini
 
Run the following command to create a new configuration (``.ini``) file::
 

	
 
This will display any changes made by the new version of Kallithea to your
 
current configuration, and attempt an automatic merge. It is recommended
 
that you check the contents after the merge.
 
    gearbox make-config new.ini
 

	
 
Then compare it with your old config file and see what changed.
 

	
 
.. note::
 
    Please always make sure your ``.ini`` files are up to date. Errors
 
    can often be caused by missing parameters added in new versions.
 

	
 
.. _upgrade_db:
 

	
 

	
 
6. Upgrade your database
 
------------------------
 

	
 
.. note::
kallithea/config/deployment.ini_tmpl
Show inline comments
 
deleted file
kallithea/lib/paster_commands/make_config.py
Show inline comments
 
modified file chmod 100755 => 100644
 
file renamed from kallithea/bin/kallithea_config.py to kallithea/lib/paster_commands/make_config.py
 
#!/usr/bin/env python2
 

	
 
# -*- coding: utf-8 -*-
 
# This program is free software: you can redistribute it and/or modify
 
# it under the terms of the GNU General Public License as published by
 
# the Free Software Foundation, either version 3 of the License, or
 
# (at your option) any later version.
 
#
 
# This program is distributed in the hope that it will be useful,
 
# but WITHOUT ANY WARRANTY; without even the implied warranty of
 
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 
# GNU General Public License for more details.
 
#
 
# You should have received a copy of the GNU General Public License
 
# along with this program.  If not, see <http://www.gnu.org/licenses/>.
 
"""
 
kallithea.bin.kallithea_config
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
configuration generator for Kallithea
 
kallithea.lib.paster_commands.make_config
 
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 

	
 
This file was forked by the Kallithea project in July 2014.
 
Original author and date, and relevant copyright and licensing information is below:
 
:created_on: Jun 18, 2013
 
:author: marcink
 
:copyright: (c) 2013 RhodeCode GmbH, and others.
 
make-config gearbox command for Kallithea
 

	
 
:license: GPLv3, see LICENSE.md for more details.
 
"""
 

	
 

	
 
import os
 
import sys
 
import uuid
 
import argparse
 
from mako.template import Template
 
TMPL = 'template.ini.mako'
 
here = os.path.dirname(os.path.abspath(__file__))
 

	
 
def argparser(argv):
 
    usage = (
 
      "kallithea-config [-h] [--filename=FILENAME] [--template=TEMPLATE] \n"
 
      "VARS optional specify extra template variable that will be available in "
 
      "template. Use comma separated key=val format eg.\n"
 
      "key1=val1,port=5000,host=127.0.0.1,elements='a\,b\,c'\n"
 
    )
 
from kallithea.lib.paster_commands.common import ask_ok, BasePasterCommand
 

	
 

	
 
class Command(BasePasterCommand):
 
    """Kallithea: Create a new config file
 

	
 
    parser = argparse.ArgumentParser(
 
        description='Kallithea CONFIG generator with variable replacement',
 
        usage=usage
 
    )
 
    make-config is part of a two-phase installation process (the
 
    second phase is setup-app). make-config creates a bare configuration
 
    file (possibly filling in defaults from the extra
 
    variables you give).
 
    """
 

	
 
    takes_config_file = False # at least not an existing one ...
 

	
 
    ## config
 
    group = parser.add_argument_group('CONFIG')
 
    group.add_argument('--filename', help='Output ini filename.')
 
    group.add_argument('--template', help='Mako template file to use instead of '
 
                                          'the default builtin template')
 
    group.add_argument('--raw', help='Store given mako template as raw without '
 
                                     'parsing. Use this to create custom template '
 
                                     'initially', action='store_true')
 
    group.add_argument('--show-defaults', help='Show all default variables for '
 
                                               'builtin template', action='store_true')
 
    args, other = parser.parse_known_args()
 
    return parser, args, other
 
    def take_action(self, args):
 
        _run(args)
 

	
 
    def get_parser(self, prog_name):
 
        parser = super(Command, self).get_parser(prog_name)
 

	
 
        parser.add_argument('config_file', nargs='?',
 
            help='application config file to write')
 

	
 
        parser.add_argument('custom', nargs=argparse.REMAINDER,
 
            help='custom values to write to config file')
 

	
 
        parser.add_argument('--show-defaults', action='store_true',
 
            help="Show the default values that can be overridden")
 

	
 
        return parser
 

	
 

	
 
def _escape_split(text, sep):
 
    """
 
    Allows for escaping of the separator: e.g. arg='foo\, bar'
 

	
 
    It should be noted that the way bash et. al. do command line parsing, those
 
    single quotes are required. a shameless ripoff from fabric project.
 

	
 
    """
 
    escaped_sep = r'\%s' % sep
 

	
 
@@ -81,80 +79,65 @@ def _escape_split(text, sep):
 
    unfinished = startlist[-1]
 
    startlist = startlist[:-1]
 

	
 
    # recurse because there may be more escaped separators
 
    endlist = _escape_split(after, sep)
 

	
 
    # finish building the escaped value. we use endlist[0] because the first
 
    # part of the string sent in recursion is the rest of the escaped value.
 
    unfinished += sep + endlist[0]
 

	
 
    return startlist + [unfinished] + endlist[1:]  # put together all the parts
 

	
 
def _run(argv):
 
    parser, args, other = argparser(argv)
 
    if not len(sys.argv) > 1:
 
        print parser.print_help()
 
        sys.exit(0)
 

	
 
def _run(args):
 
    if args.config_file is None:
 
        if not args.show_defaults:
 
            raise ValueError("Missing argument: config_file")
 
    else:
 
        if args.show_defaults:
 
            raise ValueError("Can't specify both config_file and --show_defaults")
 

	
 
    # defaults that can be overwritten by arguments
 
    tmpl_stored_args = {
 
        'http_server': 'waitress',
 
        'lang': 'en',
 
        'database_engine': 'sqlite',
 
        'host': '127.0.0.1',
 
        'port': 5000,
 
        'error_aggregation_service': None,
 
    }
 
    if other:
 
        # parse arguments, we assume only first is correct
 
    for custom in args.custom:
 
        # parse arguments
 
        kwargs = {}
 
        for el in _escape_split(other[0], ','):
 
        for el in _escape_split(custom, ','):
 
            kv = _escape_split(el, '=')
 
            if len(kv) == 2:
 
                k, v = kv
 
                kwargs[k] = v
 
        # update our template stored args
 
        tmpl_stored_args.update(kwargs)
 

	
 
    # use default that cannot be replaced
 
    tmpl_stored_args.update({
 
        'uuid': lambda: uuid.uuid4().hex,
 
        'here': os.path.abspath(os.curdir),
 
    })
 
    if args.show_defaults:
 
        for k,v in tmpl_stored_args.iteritems():
 
            print '%s=%s' % (k, v)
 
        sys.exit(0)
 

	
 
    # use default that cannot be replaced
 
    tmpl_stored_args.update({
 
        'uuid': lambda: uuid.uuid4().hex,
 
        'here': os.path.dirname(os.path.abspath(args.config_file)),
 
    })
 
    try:
 
        # built in template
 
        tmpl_file = os.path.join(here, TMPL)
 
        if args.template:
 
            tmpl_file = args.template
 

	
 
        with open(tmpl_file, 'rb') as f:
 
            tmpl_data = f.read().decode('utf-8')
 
            if args.raw:
 
                tmpl = tmpl_data
 
            else:
 
                tmpl = Template(tmpl_data).render(**tmpl_stored_args)
 
        with open(args.filename, 'wb') as f:
 
            tmpl = Template(tmpl_data).render(**tmpl_stored_args)
 
        with open(args.config_file, 'wb') as f:
 
            f.write(tmpl.encode('utf-8'))
 
        print 'Wrote new config file in %s' % (os.path.abspath(args.filename))
 
        print 'Wrote new config file in %s' % (os.path.abspath(args.config_file))
 

	
 
    except Exception:
 
        from mako import exceptions
 
        print exceptions.text_error_template().render()
 

	
 
def main(argv=None):
 
    """
 
    Main execution function for cli
 

	
 
    :param argv:
 
    """
 
    if argv is None:
 
        argv = sys.argv
 

	
 
    return _run(argv)
 

	
 

	
 
if __name__ == '__main__':
 
    sys.exit(main(sys.argv))
kallithea/lib/paster_commands/template.ini.mako
Show inline comments
 
file renamed from kallithea/bin/template.ini.mako to kallithea/lib/paster_commands/template.ini.mako
scripts/generate-ini.py
Show inline comments
 
#!/usr/bin/env python2
 
"""
 
Based on kallithea/bin/template.ini.mako, generate
 
  kallithea/config/deployment.ini_tmpl
 
Based on kallithea/lib/paster_commands/template.ini.mako, generate
 
  development.ini
 
  kallithea/tests/test.ini
 
"""
 

	
 
import re
 

	
 
makofile = 'kallithea/bin/template.ini.mako'
 
makofile = 'kallithea/lib/paster_commands/template.ini.mako'
 

	
 
# the mako conditionals used in all other ini files and templates
 
selected_mako_conditionals = set([
 
    "database_engine == 'sqlite'",
 
    "http_server == 'waitress'",
 
    "error_aggregation_service == 'appenlight'",
 
    "error_aggregation_service == 'sentry'",
 
])
 

	
 
# the mako variables used in all other ini files and templates
 
mako_variable_values = {
 
    'host': '127.0.0.1',
 
    'port': '5000',
 
    'here': '%(here)s',
 
    'uuid()': '${app_instance_uuid}',
 
}
 

	
 
# files to be generated from the mako template
 
ini_files = [
 
    ('kallithea/config/deployment.ini_tmpl',
 
        '''
 
        Kallithea - Example config
 

	
 
        The %(here)s variable will be replaced with the parent directory of this file
 
        ''',
 
        {}, # exactly the same settings as template.ini.mako
 
    ),
 
    ('kallithea/tests/test.ini',
 
        '''
 
        Kallithea - config for tests:
 
        initial_repo_scan = true
 
        sqlalchemy and kallithea_test.sqlite
 
        custom logging
 

	
 
        The %(here)s variable will be replaced with the parent directory of this file
 
        ''',
 
        {
 
            '[server:main]': {
 
                'port': '4999',
scripts/manifest
Show inline comments
 
@@ -66,29 +66,27 @@ init.d/supervisord.conf
 
kallithea/
 
kallithea/__init__.py
 
kallithea/bin/
 
kallithea/bin/__init__.py
 
kallithea/bin/base.py
 
kallithea/bin/kallithea_api.py
 
kallithea/bin/kallithea_backup.py
 
kallithea/bin/kallithea_config.py
 
kallithea/bin/kallithea_gist.py
 
kallithea/bin/ldap_sync.conf
 
kallithea/bin/ldap_sync.py
 
kallithea/bin/rebranddb.py
 
kallithea/bin/template.ini.mako
 
kallithea/config/
 
kallithea/config/__init__.py
 
kallithea/config/conf.py
 
kallithea/config/deployment.ini_tmpl
 
kallithea/config/environment.py
 
kallithea/config/middleware.py
 
kallithea/config/post_receive_tmpl.py
 
kallithea/config/pre_receive_tmpl.py
 
kallithea/config/rcextensions/
 
kallithea/config/rcextensions/__init__.py
 
kallithea/config/routing.py
 
kallithea/controllers/
 
kallithea/controllers/__init__.py
 
kallithea/controllers/admin/
 
kallithea/controllers/admin/__init__.py
 
kallithea/controllers/admin/admin.py
 
@@ -346,24 +344,25 @@ kallithea/lib/middleware/simplegit.py
 
kallithea/lib/middleware/simplehg.py
 
kallithea/lib/middleware/wrapper.py
 
kallithea/lib/paster_commands/
 
kallithea/lib/paster_commands/__init__.py
 
kallithea/lib/paster_commands/cache_keys.py
 
kallithea/lib/paster_commands/cleanup.py
 
kallithea/lib/paster_commands/install_iis.py
 
kallithea/lib/paster_commands/ishell.py
 
kallithea/lib/paster_commands/make_index.py
 
kallithea/lib/paster_commands/make_rcextensions.py
 
kallithea/lib/paster_commands/repo_scan.py
 
kallithea/lib/paster_commands/setup_db.py
 
kallithea/lib/paster_commands/template.ini.mako
 
kallithea/lib/paster_commands/update_repoinfo.py
 
kallithea/lib/pidlock.py
 
kallithea/lib/profiler.py
 
kallithea/lib/rcmail/
 
kallithea/lib/rcmail/__init__.py
 
kallithea/lib/rcmail/exceptions.py
 
kallithea/lib/rcmail/message.py
 
kallithea/lib/rcmail/response.py
 
kallithea/lib/rcmail/smtp_mailer.py
 
kallithea/lib/rcmail/utils.py
 
kallithea/lib/recaptcha.py
 
kallithea/lib/timerproxy.py
setup.py
Show inline comments
 
@@ -146,24 +146,25 @@ setuptools.setup(
 
    [console_scripts]
 
    kallithea-api =    kallithea.bin.kallithea_api:main
 
    kallithea-gist =   kallithea.bin.kallithea_gist:main
 
    kallithea-config = kallithea.bin.kallithea_config:main
 

	
 
    [paste.app_factory]
 
    main = kallithea.config.middleware:make_app
 

	
 
    [paste.app_install]
 
    main = pylons.util:PylonsInstaller
 

	
 
    [gearbox.commands]
 
    make-config=kallithea.lib.paster_commands.make_config:Command
 
    setup-db=kallithea.lib.paster_commands.setup_db:Command
 
    cleanup-repos=kallithea.lib.paster_commands.cleanup:Command
 
    update-repoinfo=kallithea.lib.paster_commands.update_repoinfo:Command
 
    make-rcext=kallithea.lib.paster_commands.make_rcextensions:Command
 
    repo-scan=kallithea.lib.paster_commands.repo_scan:Command
 
    cache-keys=kallithea.lib.paster_commands.cache_keys:Command
 
    ishell=kallithea.lib.paster_commands.ishell:Command
 
    make-index=kallithea.lib.paster_commands.make_index:Command
 
    upgrade-db=kallithea.lib.dbmigrate:UpgradeDb
 
    celeryd=kallithea.lib.paster_commands.celeryd:Command
 
    install-iis=kallithea.lib.paster_commands.install_iis:Command
 
    """,
0 comments (0 inline, 0 general)