.. _setup:

Setup

=====

Setting up the application

--------------------------

::

 paster make-config RhodeCode production.ini

- This will create `production.ini` config inside the directory

  this config contains various settings for RhodeCode, e.g proxy port, 

  email settings,static files, cache and logging.

::

 paster setup-app production.ini

- This command will create all needed tables and an admin account. 

  When asked for a path You can either use a new location of one with already 

  existing ones. RhodeCode will simply add all new found repositories to 

  it's database. Also make sure You specify correct path to repositories.

- Remember that the given path for mercurial_ repositories must be write 

  accessible for the application. It's very important since RhodeCode web interface

  will work even without such an access but, when trying to do a push it'll 

  eventually fail with permission denied errors. 

- Run 

::

 paster serve production.ini

- This command runs the RhodeCode server the app should be available at the 

  127.0.0.1:5000. This ip and port is configurable via the production.ini 

  file  created in previous step

- Use admin account you created to login.

- Default permissions on each repository is read, and owner is admin. So 

  remember to update these if needed.

Setting up Whoosh full text search

----------------------------------

Index for whoosh can be build starting from version 1.1 using paster command

passing repo locations to index, as well as Your config file that stores

whoosh index files locations. There is possible to pass `-f` to the options

to enable full index rebuild. Without that indexing will run always in in

incremental mode.

::

 paster make-index --repo-location=<location for repos> production.ini  

for full index rebuild You can use

::

 paster make-index -f --repo-location=<location for repos> production.ini

- For full text search You can either put crontab entry for

This command can be run even from crontab in order to do periodical 

index builds and keep Your index always up to date. An example entry might 

look like this

::

 /path/to/python/bin/paster --repo-location=<location for repos> /path/to/rhodecode/production.ini

When using incremental(default) mode whoosh will check last modification date 

of each file and add it to reindex if newer file is available. Also indexing 

daemon checks for removed files and removes them from index. 

Sometime You might want to rebuild index from scratch. You can do that using 

the `-f` flag passed to paster command or, in admin panel You can check 

`build from scratch` flag.

Setting up LDAP support

-----------------------

RhodeCode starting from version 1.1 supports ldap authentication. In order

to use ldap, You have to install python-ldap package. This package is available

via pypi, so You can install it by running

::

 easy_install python-ldap

::

 pip install python-ldap

.. note::

   python-ldap requires some certain libs on Your system, so before installing 

   it check that You have at least `openldap`, and `sasl` libraries.

ldap settings are located in admin->ldap section,

Here's a typical ldap setup::

 Enable ldap  = checked                 #controls if ldap access is enabled

 Host         = host.domain.org         #actual ldap server to connect

 Port         = 389 or 689 for ldaps    #ldap server ports

 Enable LDAPS = unchecked               #enable disable ldaps

 Account      = <account>               #access for ldap server(if required)

 Password     = <password>              #password for ldap server(if required)

 Base DN      = uid=%(user)s,CN=users,DC=host,DC=domain,DC=org

`Account` and `Password` are optional, and used for two-phase ldap 

authentication so those are credentials to access Your ldap, if it doesn't 

If all data are entered correctly, and `python-ldap` is properly installed

Users should be granted to access RhodeCode wit ldap accounts. When 

logging at the first time an special ldap account is created inside RhodeCode, 

so You can control over permissions even on ldap users. If such user exists 

already in RhodeCode database ldap user with the same username would be not 

able to access RhodeCode.

If You have problems with ldap access and believe You entered correct 

information check out the RhodeCode logs,any error messages sent from 

ldap will be saved there.

Nginx virtual host example

--------------------------

Sample config for nginx using proxy::

 server {

    listen          80;

    server_name     hg.myserver.com;

    access_log      /var/log/nginx/rhodecode.access.log;

    error_log       /var/log/nginx/rhodecode.error.log;

    location / {

            root /var/www/rhodecode/rhodecode/public/;

            if (!-f $request_filename){

                proxy_pass      http://127.0.0.1:5000;

            }

            #this is important for https !!!

            proxy_set_header X-Url-Scheme $scheme;

            include         /etc/nginx/proxy.conf;  

    }

 }  

Here's the proxy.conf. It's tuned so it'll not timeout on long

pushes and also on large pushes::

    proxy_redirect              off;

    proxy_set_header            Host $host;

    proxy_set_header            X-Host $http_host;

    proxy_set_header            X-Real-IP $remote_addr;

    proxy_set_header            X-Forwarded-For $proxy_add_x_forwarded_for;

    proxy_set_header            Proxy-host $proxy_host;

    client_max_body_size        400m;

    client_body_buffer_size     128k;

    proxy_buffering             off;

    proxy_connect_timeout       3600;

    proxy_send_timeout          3600;

    proxy_read_timeout          3600;

    proxy_buffer_size           8k;

    proxy_buffers               8 32k;

    proxy_busy_buffers_size     64k;

    proxy_temp_file_write_size  64k;

Also when using root path with nginx You might set the static files to false

in production.ini file::

  [app:main]

    use = egg:rhodecode

    full_stack = true

    static_files = false

    lang=en

    cache_dir = %(here)s/data

To not have the statics served by the application. And improve speed.

Apache reverse proxy

--------------------

Tutorial can be found here

http://wiki.pylonshq.com/display/pylonscookbook/Apache+as+a+reverse+proxy+for+Pylons

Apache's example FCGI config

----------------------------

TODO !

Other configuration files

-------------------------

Some extra configuration files and examples can be found here:

http://hg.python-works.com/rhodecode/files/tip/init.d

and also an celeryconfig file can be use from here:

http://hg.python-works.com/rhodecode/files/tip/celeryconfig.py

Troubleshooting

---------------

- missing static files ?

 - make sure either to set the `static_files = true` in the .ini file or

   double check the root path for Your http setup. It should point to 

   for example:

   /home/my-virtual-python/lib/python2.6/site-packages/rhodecode/public

- can't install celery/rabbitmq

 - don't worry RhodeCode works without them too. No extra setup required

- long lasting push timeouts ?

 - make sure You set a longer timeouts in Your proxy/fcgi settings, timeouts

   are caused by https server and not RhodeCode

- large pushes timeouts ?

 - make sure You set a proper max_body_size for the http server

.. _virtualenv: http://pypi.python.org/pypi/virtualenv

.. _python: http://www.python.org/

.. _mercurial: http://mercurial.selenic.com/

.. _celery: http://celeryproject.org/

.. _rabbitmq: http://www.rabbitmq.com/

#!/usr/bin/env python

# encoding: utf-8

# ldap authentication lib

# Copyright (C) 2009-2010 Marcin Kuzminski <marcin@python-works.com>

#

# This program is free software; you can redistribute it and/or

# modify it under the terms of the GNU General Public License

# as published by the Free Software Foundation; version 2

# of the License or (at your opinion) any later version of the license.

# 

# This program is distributed in the hope that it will be useful,

# but WITHOUT ANY WARRANTY; without even the implied warranty of

# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the

# GNU General Public License for more details.

# 

# You should have received a copy of the GNU General Public License

# along with this program; if not, write to the Free Software

# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston,

# MA  02110-1301, USA.

"""

Created on Nov 17, 2010

@author: marcink

"""

from rhodecode.lib.exceptions import *

import logging

log = logging.getLogger(__name__)

try:

    import ldap

except ImportError:

    pass

class AuthLdap(object):

    def __init__(self, server, base_dn, port=389, bind_dn='', bind_pass='',

                 use_ldaps=False, ldap_version=3):

        self.ldap_version = ldap_version

        if use_ldaps:

            port = port or 689

        self.LDAP_USE_LDAPS = use_ldaps

        self.LDAP_SERVER_ADDRESS = server

        self.LDAP_SERVER_PORT = port

        #USE FOR READ ONLY BIND TO LDAP SERVER

        self.LDAP_BIND_DN = bind_dn

        self.LDAP_BIND_PASS = bind_pass

        ldap_server_type = 'ldap'

        if self.LDAP_USE_LDAPS:ldap_server_type = ldap_server_type + 's'

        self.LDAP_SERVER = "%s://%s:%s" % (ldap_server_type,

                                               self.LDAP_SERVER_ADDRESS,

                                               self.LDAP_SERVER_PORT)

        self.BASE_DN = base_dn

    def authenticate_ldap(self, username, password):

        """Authenticate a user via LDAP and return his/her LDAP properties.

        Raises AuthenticationError if the credentials are rejected, or

        EnvironmentError if the LDAP server can't be reached.

        :param username: username

        :param password: password

        """

        from rhodecode.lib.helpers import chop_at

        uid = chop_at(username, "@%s" % self.LDAP_SERVER_ADDRESS)

        if "," in username:

            raise LdapUsernameError("invalid character in username: ,")

        try:

            ldap.set_option(ldap.OPT_X_TLS_CACERTDIR, '/etc/openldap/cacerts')

            ldap.set_option(ldap.OPT_NETWORK_TIMEOUT, 10)

            server = ldap.initialize(self.LDAP_SERVER)

            if self.ldap_version == 2:

                server.protocol = ldap.VERSION2

            else:

                server.protocol = ldap.VERSION3

            if self.LDAP_BIND_DN and self.LDAP_BIND_PASS:

            server.simple_bind_s(dn, password)

            properties = server.search_s(dn, ldap.SCOPE_SUBTREE)

            if not properties:

                raise ldap.NO_SUCH_OBJECT()

        except ldap.NO_SUCH_OBJECT, e:

            log.debug("LDAP says no such user '%s' (%s)", uid, username)

            raise LdapUsernameError()

        except ldap.INVALID_CREDENTIALS, e:

            log.debug("LDAP rejected password for user '%s' (%s)", uid, username)

            raise LdapPasswordError()

        except ldap.SERVER_DOWN, e:

            raise LdapConnectionError("LDAP can't access authentication server")

        return properties[0]

            if value.get('new_password'):

                try:

                    value['new_password'] = \

                        get_crypt_password(value['new_password'])

                except UnicodeEncodeError:

                    e_dict = {'new_password':_('Invalid characters in password')}

                    raise formencode.Invalid('', value, state, error_dict=e_dict)

            return value

class ValidPasswordsMatch(formencode.validators.FancyValidator):

    def validate_python(self, value, state):

        if value['password'] != value['password_confirmation']:

            e_dict = {'password_confirmation':

                   _('Password do not match')}

            raise formencode.Invalid('', value, state, error_dict=e_dict)

class ValidAuth(formencode.validators.FancyValidator):

    messages = {

            'invalid_password':_('invalid password'),

            'invalid_login':_('invalid user name'),

            'disabled_account':_('Your account is disabled')

            }

    #error mapping

    e_dict = {'username':messages['invalid_login'],

              'password':messages['invalid_password']}

    e_dict_disable = {'username':messages['disabled_account']}

    def validate_python(self, value, state):

        password = value['password']

        username = value['username']

        user = UserModel().get_by_username(username)

        if authenticate(username, password):

            return value

        else:

            if user and user.active is False:

                log.warning('user %s is disabled', username)

                raise formencode.Invalid(self.message('disabled_account',

                                         state=State_obj),

                                         value, state,

                                         error_dict=self.e_dict_disable)

            else:

                log.warning('user %s not authenticated', username)

                raise formencode.Invalid(self.message('invalid_password',

                                         state=State_obj), value, state,

                                         error_dict=self.e_dict)

class ValidRepoUser(formencode.validators.FancyValidator):

    def to_python(self, value, state):

        sa = meta.Session()

        try:

            self.user_db = sa.query(User)\

                .filter(User.active == True)\

                .filter(User.username == value).one()

        except Exception:

            raise formencode.Invalid(_('This username is not valid'),

                                     value, state)

        finally:

            meta.Session.remove()

        return self.user_db.user_id

def ValidRepoName(edit, old_data):

    class _ValidRepoName(formencode.validators.FancyValidator):

        def to_python(self, value, state):

            slug = h.repo_name_slug(value)

            if slug in ['_admin']:

                raise formencode.Invalid(_('This repository name is disallowed'),

                                         value, state)

            if old_data.get('repo_name') != value or not edit:

                if RepoModel().get_by_repo_name(slug, cache=False):

                    raise formencode.Invalid(_('This repository already exists') ,

                                             value, state)

            return slug

    return _ValidRepoName

def ValidForkType(old_data):

    class _ValidForkType(formencode.validators.FancyValidator):

        def to_python(self, value, state):

            if old_data['repo_type'] != value:

                raise formencode.Invalid(_('Fork have to be the same type as original'),

                                         value, state)

            return value

    return _ValidForkType

class ValidPerms(formencode.validators.FancyValidator):

    messages = {'perm_new_user_name':_('This username is not valid')}

    def to_python(self, value, state):

        perms_update = []

        perms_new = []

        #build a list of permission to update and new permission to create

        for k, v in value.items():

            if k.startswith('perm_'):

                if  k.startswith('perm_new_user'):

                    new_perm = value.get('perm_new_user', False)

                    new_user = value.get('perm_new_user_name', False)

                    if new_user and new_perm:

                        if (new_user, new_perm) not in perms_new:

                            perms_new.append((new_user, new_perm))

                else:

                    usr = k[5:]

                    if usr == 'default':

                        if value['private']:

                            #set none for default when updating to private repo

                            v = 'repository.none'

                    perms_update.append((usr, v))

        value['perms_updates'] = perms_update

        value['perms_new'] = perms_new

        sa = meta.Session

        for k, v in perms_new:

            try:

                self.user_db = sa.query(User)\

                    .filter(User.active == True)\

                    .filter(User.username == k).one()

            except Exception:

                msg = self.message('perm_new_user_name',

                                     state=State_obj)

                raise formencode.Invalid(msg, value, state,

                                         error_dict={'perm_new_user_name':msg})

        return value

class ValidSettings(formencode.validators.FancyValidator):

    def to_python(self, value, state):

        #settings  form can't edit user

        if value.has_key('user'):

            del['value']['user']

        return value

class ValidPath(formencode.validators.FancyValidator):

    def to_python(self, value, state):

        if not os.path.isdir(value):

            msg = _('This is not a valid path')

            raise formencode.Invalid(msg, value, state,

                                     error_dict={'paths_root_path':msg})

        return value

def UniqSystemEmail(old_data):

    class _UniqSystemEmail(formencode.validators.FancyValidator):

        def to_python(self, value, state):

            value = value.lower()

            if old_data.get('email') != value:

                sa = meta.Session()

                try:

                    user = sa.query(User).filter(User.email == value).scalar()

                    if user:

                        raise formencode.Invalid(_("This e-mail address is already taken") ,

                                                 value, state)

                finally:

                    meta.Session.remove()

            return value

    return _UniqSystemEmail

class ValidSystemEmail(formencode.validators.FancyValidator):

    def to_python(self, value, state):

        value = value.lower()

        sa = meta.Session

        try:

            user = sa.query(User).filter(User.email == value).scalar()

            if  user is None:

                raise formencode.Invalid(_("This e-mail address doesn't exist.") ,

                                         value, state)

        finally:

            meta.Session.remove()

        return value

class LdapLibValidator(formencode.validators.FancyValidator):

    def to_python(self, value, state):

        try:

            import ldap

        except ImportError:

            raise LdapImportError

        return value

#===============================================================================

# FORMS        

#===============================================================================

class LoginForm(formencode.Schema):

    allow_extra_fields = True

    filter_extra_fields = True

    username = UnicodeString(

                             strip=True,

                             min=1,

                             not_empty=True,

                             messages={

                                       'empty':_('Please enter a login'),

                                       'tooShort':_('Enter a value %(min)i characters long or more')}

                            )

    password = UnicodeString(

                            strip=True,

                            min=6,

                            not_empty=True,

                            messages={

                                      'empty':_('Please enter a password'),

                                      'tooShort':_('Enter %(min)i characters or more')}

                                )

    #chained validators have access to all data

    chained_validators = [ValidAuth]

def UserForm(edit=False, old_data={}):

    class _UserForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = True

        username = All(UnicodeString(strip=True, min=1, not_empty=True),

                       ValidUsername(edit, old_data))

        if edit:

            new_password = All(UnicodeString(strip=True, min=6, not_empty=False))

            admin = StringBoolean(if_missing=False)

        else:

            password = All(UnicodeString(strip=True, min=6, not_empty=True))

        active = StringBoolean(if_missing=False)

        name = UnicodeString(strip=True, min=1, not_empty=True)

        lastname = UnicodeString(strip=True, min=1, not_empty=True)

        email = All(Email(not_empty=True), UniqSystemEmail(old_data))

        chained_validators = [ValidPassword]

    return _UserForm

def RegisterForm(edit=False, old_data={}):

    class _RegisterForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = True

        username = All(ValidUsername(edit, old_data),

                       UnicodeString(strip=True, min=1, not_empty=True))

        password = All(UnicodeString(strip=True, min=6, not_empty=True))

        password_confirmation = All(UnicodeString(strip=True, min=6, not_empty=True))

        active = StringBoolean(if_missing=False)

        name = UnicodeString(strip=True, min=1, not_empty=True)

        lastname = UnicodeString(strip=True, min=1, not_empty=True)

        email = All(Email(not_empty=True), UniqSystemEmail(old_data))

        chained_validators = [ValidPasswordsMatch, ValidPassword]

    return _RegisterForm

def PasswordResetForm():

    class _PasswordResetForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = True

        email = All(ValidSystemEmail(), Email(not_empty=True))

    return _PasswordResetForm

def RepoForm(edit=False, old_data={}, supported_backends=BACKENDS.keys()):

    class _RepoForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = False

        repo_name = All(UnicodeString(strip=True, min=1, not_empty=True),

                        ValidRepoName(edit, old_data))

        description = UnicodeString(strip=True, min=1, not_empty=True)

        private = StringBoolean(if_missing=False)

        repo_type = OneOf(supported_backends)

        if edit:

            user = All(Int(not_empty=True), ValidRepoUser)

        chained_validators = [ValidPerms]

    return _RepoForm

def RepoForkForm(edit=False, old_data={}, supported_backends=BACKENDS.keys()):

    class _RepoForkForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = False

        fork_name = All(UnicodeString(strip=True, min=1, not_empty=True),

                        ValidRepoName(edit, old_data))

        description = UnicodeString(strip=True, min=1, not_empty=True)

        private = StringBoolean(if_missing=False)

        repo_type = All(ValidForkType(old_data), OneOf(supported_backends))

    return _RepoForkForm

def RepoSettingsForm(edit=False, old_data={}):

    class _RepoForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = False

        repo_name = All(UnicodeString(strip=True, min=1, not_empty=True),

                        ValidRepoName(edit, old_data))

        description = UnicodeString(strip=True, min=1, not_empty=True)

        private = StringBoolean(if_missing=False)

        chained_validators = [ValidPerms, ValidSettings]

    return _RepoForm

def ApplicationSettingsForm():

    class _ApplicationSettingsForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = False

        rhodecode_title = UnicodeString(strip=True, min=1, not_empty=True)

        rhodecode_realm = UnicodeString(strip=True, min=1, not_empty=True)

    return _ApplicationSettingsForm

def ApplicationUiSettingsForm():

    class _ApplicationUiSettingsForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = False

        web_push_ssl = OneOf(['true', 'false'], if_missing='false')

        paths_root_path = All(ValidPath(), UnicodeString(strip=True, min=1, not_empty=True))

        hooks_changegroup_update = OneOf(['True', 'False'], if_missing=False)

        hooks_changegroup_repo_size = OneOf(['True', 'False'], if_missing=False)

        hooks_pretxnchangegroup_push_logger = OneOf(['True', 'False'], if_missing=False)

        hooks_preoutgoing_pull_logger = OneOf(['True', 'False'], if_missing=False)

    return _ApplicationUiSettingsForm

def DefaultPermissionsForm(perms_choices, register_choices, create_choices):

    class _DefaultPermissionsForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = True

        overwrite_default = StringBoolean(if_missing=False)

        anonymous = OneOf(['True', 'False'], if_missing=False)

        default_perm = OneOf(perms_choices)

        default_register = OneOf(register_choices)

        default_create = OneOf(create_choices)

    return _DefaultPermissionsForm

def LdapSettingsForm():

    class _LdapSettingsForm(formencode.Schema):

        allow_extra_fields = True

        filter_extra_fields = True

        pre_validators = [LdapLibValidator]

        ldap_active = StringBoolean(if_missing=False)

        ldap_host = UnicodeString(strip=True,)

        ldap_port = Number(strip=True,)

        ldap_ldaps = StringBoolean(if_missing=False)

        ldap_dn_user = UnicodeString(strip=True,)

        ldap_dn_pass = UnicodeString(strip=True,)

    return _LdapSettingsForm

import sys

py_version = sys.version_info

requirements = [

        "Pylons>=1.0.0",

        "SQLAlchemy>=0.6.5",

        "Mako>=0.3.6",

        "vcs>=0.1.10",

        "pygments>=1.3.0",

        "mercurial>=1.7.1",

        "celery>=2.1.3",

        "py-bcrypt",

        "babel",

    ]

classifiers = ['Development Status :: 4 - Beta',

                   'Environment :: Web Environment',

                   'Framework :: Pylons',

                   'Intended Audience :: Developers',

                   'License :: OSI Approved :: BSD License',

                   'Operating System :: OS Independent',

                   'Programming Language :: Python', ]

if sys.version_info < (2, 6):

    requirements.append("simplejson")

    requirements.append("pysqlite")

#additional files from project that goes somewhere in the filesystem

#relative to sys.prefix

data_files = []

#additional files that goes into package itself

package_data = {'rhodecode': ['i18n/*/LC_MESSAGES/*.mo', ], }

description = ('Mercurial repository browser/management with '

               'build in push/pull server and full text search')

#long description

try:

    readme_file = 'README.rst'

    changelog_file = 'docs/changelog.rst'

    long_description = open(readme_file).read() + '/n/n' + \

        open(changelog_file).read()

except IOError, err:

    sys.stderr.write("[WARNING] Cannot find file specified as "

        "long_description (%s)\n or changelog (%s) skipping that file" \

            % (readme_file, changelog_file))

    long_description = description

try:

    from setuptools import setup, find_packages

except ImportError:

    from ez_setup import use_setuptools

    use_setuptools()

    from setuptools import setup, find_packages

#packages

packages = find_packages(exclude=['ez_setup'])

setup(

    name='RhodeCode',

    version=get_version(),

    description=description,

    long_description=long_description,

    keywords='rhodiumcode mercurial web hgwebdir gitweb git replacement serving hgweb rhodecode',

    license='BSD',

    author='Marcin Kuzminski',

    author_email='marcin@python-works.com',

    url='http://hg.python-works.com',

    install_requires=requirements,

    classifiers=classifiers,

    setup_requires=["PasteScript>=1.6.3"],

    data_files=data_files,

    packages=packages,

    include_package_data=True,

    test_suite='nose.collector',

    package_data=package_data,

    message_extractors={'rhodecode': [

            ('**.py', 'python', None),

            ('templates/**.mako', 'mako', {'input_encoding': 'utf-8'}),

            ('public/**', 'ignore', None)]},

    zip_safe=False,

    paster_plugins=['PasteScript', 'Pylons'],

    entry_points="""

    [paste.app_factory]

    main = rhodecode.config.middleware:make_app

    [paste.app_install]

    main = pylons.util:PylonsInstaller

    [paste.global_paster_command]

    make-index = rhodecode.lib.indexers:MakeIndex

    upgrade-db = rhodecode.lib.utils:UpgradeDb

    """,

)