README.md 13.6 KB
Newer Older
1 2 3
# `webis` Command

## About
Johannes Kiesel's avatar
Johannes Kiesel committed
4
The webis command is a helper utility for automating every-day tasks in our group.
5

6 7
See the wiki for [Installation Instructions](https://git.webis.de/code-generic/code-webis-cmd/wikis/webis-cmd-installation).

8
## Requirements
9 10 11
```bash
sudo apt install python3 python3-pip pep8
```
12

Johannes Kiesel's avatar
Johannes Kiesel committed
13
## Using `webis`
14
For usage help run:
15

16 17
```bash
./webis.py --help
18 19
```

Johannes Kiesel's avatar
Johannes Kiesel committed
20
Or look at the [cheatsheet](cheatsheet.txt)
21

22 23
### Installation
If you are on a Webis machine, `webis` should already be installed. If not, contact our administrators.
Johannes Kiesel's avatar
Johannes Kiesel committed
24 25

If you want to add the webis command to the PATH in your own machine (so that you don't have to specify it's full path every time, but can just use `webis`), use
Johannes Kiesel's avatar
Johannes Kiesel committed
26

27
```bash
Johannes Kiesel's avatar
Johannes Kiesel committed
28
./webis.py core install
29 30
```

Janek Bevendorff's avatar
Janek Bevendorff committed
31
### Bash and ZSH completion 
Johannes Kiesel's avatar
Johannes Kiesel committed
32
After installation, add the following to your `.bashrc`:
33 34

```bash
35
eval "$(_WEBIS_COMPLETE=source_bash webis)"
36 37
```

38 39
and run `source ~/.bashrc` or restart your shell. You should now be able to use tab-completion with `webis`.
ZSH users have to replace `source_bash` with `source_zsh` and add the snippet to their `.zshrc` instead.
40

41 42 43 44 45 46
To avoid starting the `webis` command every time you launch a shell, you can also save the output of

```bash
_WEBIS_COMPLETE=source_bash webis
```

47 48 49 50 51 52 53 54 55 56 57 58 59
directly to your `.bashrc` or to an external file which you `source` (ZSH users adapt accordingly).

If ZSH is not loading the completions automatically despite the above snippet being in your `.zshrc`, you may need to
create an extra directory `~/.zcomp` (name doesn't matter) and save the output of `_WEBIS_COMPLETE=source_zsh webis`
to a new file called `~/.zcomp/_webis`. Then add this to your `.zshrc`:

```zsh
fpath=("${HOME}/.zcomp" $fpath)
autoload -U compinit && compinit
```

The second line may be omitted if completions are already activated elsewhere (e.g., by oh-my-zsh).
After an additional `rm -f ~/.zcompdump` and a shell restart, everything should be working.
60

Johannes Kiesel's avatar
Johannes Kiesel committed
61
## Extending `webis`
62

63
The `webis` command is modular and very easy to extend with a few lines of Python or Bash code or any other shell-executable script.
64

65 66
`./webis.py --help` will show a summary of all installed top-level commands, which are either executable directly or contain other subcommands.
Each subcommand has its own help listing. For example, `./webis.py core --help` will list all subcommands int the `core` module.
67

68
### Adding new commands
69

70 71
All commands live in the `tools` directory and can either be one single file with multiple executable entry points or a directory containing multiple individual scripts.
There is no need to register new commands anywhere. As long as they follow a common standard, they will be recognized and loaded automatically.
72

73 74 75
#### Single-file commands

##### Simple Python command `./webis.py test`:
76

77 78
```python
import click
79

80 81 82 83 84 85 86 87 88 89 90 91 92

@click.command(short_help='Short help text.')
def test():
    """
    \b
    This is a test command.
    \b
    A more detailed description.
    \b
    Author: Your Name
    """
    
    click.echo('Hello World!')
93
```
94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124

Save this as `tools/test.py`. The function name must match the file name (minus the `.py`). The command is immediately executable and will be also listed int `./webis --help`.
Underscores in the file and function name will be replaced with hyphens on the command line. The original names must not contain any hyphens.

The doc string will be displayed if you call `./webis.py test --help`. The `\b` markers prevent automatic rewrapping of the text and can be omitted if not needed.
The `short_help` text is optional and used for the parent `./webis --help` listing. If an explicit `short_help` is missing, the first part of the long help description will be used.

##### Nested Python commands `./webis.py test [sub1|sub2]`:

```python
import click

@click.group()
def test():
    """This is a test command with subcommands."""

@test.command()                         # Note: it's test, not click!
@click.argument('positional-argument')  # Here it's click again
def sub1(positional_argument):
    """Subcommand 1 with positional argument."""
    
    click.echo('Hello from subcommand 1!')
    click.echo('Your argument was: {}'.format(positional_argument))
    
@test.command()
@click.option('-o', '--option', type=int, default=0, help='Some option.')
def sub2(option):
    """Subcommand 2 with integer option."""
    
    click.echo('Hello from subcommand 2!')
    click.echo('Your option was: {}'.format(option))
125 126
```

127 128
Click is a very powerful tool for building command line applications with Python and this is only the tip of the iceberg.
For more information please read the [Click user guide](https://click.palletsprojects.com/en/7.x/quickstart/).
129 130


131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149
##### Simple Shell command `./webis.py test`:

You can add any executable script to the `tools` folder as long as it has the execution bit set (`chmod +x`) and ends with `.sh`.
Add the following as `tools/test.sh` and make it executable:

```bash
#!/usr/bin/env bash
### \b
### This is a short help description.
### \b
### More detailed information follows.
### Comments starting with three hash symbols are parsed and shown in help
### listings on the command line. The first part until either the first full stop,
### the first \b (except if at the beginning), or the first double newline
### will be used as a short summary.
### \b
### The formatting rules are the same as for Python help listings.

echo "Hello World!"
150 151
```

152
##### Nested Shell commands `./webis.py test [sub1|sub2]`:
153

154 155 156 157 158
```bash
#!/usr/bin/env bash
### Your help listing here.

. "${WEBIS_LIB_PATH}/bashhelper.sh"
159

160 161 162 163
### Help for sub1.
cmd_sub1() {
    echo "Hello from subcommand 1"
}
164

165 166 167 168 169 170 171 172 173 174
### Help for sub2.
cmd_sub2() {
    echo "Hello from subcommand 2"
    
    # Return with non-zero exit code
    return 1
}

# This does all the magic
exec_sub_cmd "$@"
175 176
```

177
Any Bash function starting with `cmd_` will be parsed as a subcommand. The return code of a command function will be propagated back to the caller.
Janek Bevendorff's avatar
Janek Bevendorff committed
178 179
Any comment block starting with three hash symbols directly preceding it will be shown in help listings for this command. The first line will be used as a short summary.
As is the case for Python commands, underscores in the file name and in the command itself will be replaced with hyphens on the command line.
180

Janek Bevendorff's avatar
Janek Bevendorff committed
181 182
##### Subcommand arguments

Janek Bevendorff's avatar
Janek Bevendorff committed
183
Additional command line arguments will be passed on to the command as positional arguments. A special case are the help flags `-h` and `--help`, which are intercepted by the `webis` command itself.
184
In general, shell scripts can take any number of arguments in any form, but if the command syntax `cmd_*` is used, `webis` will take care automatically of parsing and validating arguments
Janek Bevendorff's avatar
Janek Bevendorff committed
185 186 187
and passes them to the command as environment variables prefixed `ARG_*`. Valid type specifiers for arguments are `str`, `int`, `float`, and `bool` (implicit if left empty).
For parameters that expect file or directory paths, you can use `path`, `fpath`, or `dpath`. Adding `!` after the type name means the file or directory must exist.
To configure which options a command takes, add their definition to the doc string:
188 189 190 191 192 193 194

```bash
#!/usr/bin/env bash
### Your help listing here.

. "${WEBIS_LIB_PATH}/bashhelper.sh"

195 196 197
### Example subcmd with additional options.
###
### : -o : --opt : int : 0 : This is an option that takes an int and defaults to 0
Janek Bevendorff's avatar
Janek Bevendorff committed
198
### : -p : --opt2 : int :: This is another int option without a default value
199 200
### :: --long-opt : str : foo : This is a long option, which defaults to "foo"
### : -f : --flag ::: This is a boolean flag
201
cmd_subcmd() {
202
    echo "Value of -o / --opt: ${ARG_OPT}"
203
    if [ -n "$ARG_OPT2" ]; then
204 205 206
        echo "Value of --opt2: ${ARG_OPT2}"
    fi
    echo "Value of --long-opt: ${ARG_LONG_OPT}"
207
    if [ -n "$ARG_FLAG" ]; then
208 209
        echo  "--flag was set"
    fi
210 211 212 213 214
}

exec_sub_cmd "$@"
```

215
Positional arguments can be configured in a similar fashion:
216

217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234
```bash
#!/usr/bin/env bash
### Your help listing here.

. "${WEBIS_LIB_PATH}/bashhelper.sh"

### Example subcmd with a required and an optional positional argument.
###
### : pos-opt1 : float : 2.0
### : pos-opt2 ::
cmd_subcmd() {
    echo "Value of pos-opt1: ${ARG_POS_OPT1}"
    echo "Value of pos-opt2: ${ARG_POS_OPT2}"
}

exec_sub_cmd "$@"
```

235
Colons within parameter descriptions can be escaped with a backslash.
236 237
Note that positional arguments have no help text. They are defined by their name, which should be clear and concise. A special argument definition is `:*::`, which
allows for any number of positional arguments (including zero). A type specifier is possible (e.g., `:*:int:`), but a default value will be ignored.
Janek Bevendorff's avatar
Janek Bevendorff committed
238
To provide better help listings, you can add a METAVAR name after `*` (e.g., `:*FILE:fpath!:`).
Janek Bevendorff's avatar
Janek Bevendorff committed
239 240
The `ARG_*` environment variable defined for variadic arguments is a string with tabs as separators. Either set `IFS` to `$'\t'` temporarily and split or
use the usual positional shell variables instead.
241 242

The Webis command will ensure all options and positional arguments are valid according to their specification and will throw an error if they don't pass validation.
Janek Bevendorff's avatar
Janek Bevendorff committed
243
Help listings and autocomplete entries are also generated automatically. If a shell script has no subcommands, arguments can be defined for the whole script at the beginning
Janek Bevendorff's avatar
Janek Bevendorff committed
244
of the file as part of the global doc block.
245 246

In addition to command line arguments, the following environment variables are defined by `webis` and can be used in scripts:
247 248 249 250 251 252 253

- `WEBIS_CMD_ROOT_PATH`: The `webis` command base installation directory (absolute path)
- `WEBIS_LIB_PATH`: The `webis` command library directory (absolute path)
- `WEBIS_CMD`: Name with which the command was invoked
- `WEBIS_ARGV`: Original arguments given on the command line
- `WEBIS_CMDLINE`: The full command with arguments
- `WEBIS_SUBCMD`: Name of the invoked subcommand
Janek Bevendorff's avatar
Janek Bevendorff committed
254
- `WEBIS_SUBCMD_ARGV`: Additional parameters passed on to the subcommand
255

256 257
When referring to paths within the Webis command (e.g., for including files from `WEBIS_LIB_PATH`), you must use these variables instead of hard-coded relative (or absolute) paths.

258 259 260 261 262 263 264 265 266 267 268 269 270 271 272

#### Multi-file commands

If a command becomes too complex for a single file, it can be split into multiple under a common parent directory. The general procedure is the same as above.
The only additional step required is to add an `__init__.py` file inside the command directory declaring the general properties of the command:

```python
import os
import click
from lib import LazyMultiCommand


@click.command(cls=LazyMultiCommand, tools_dir=os.path.dirname(__file__))
def multicmd():
    """Command help text."""
273 274
```

275
This will define the new multi-file command `./webis.py multicmd`. The name must match the name of the command directory. Below this directory there can be any number of Python or shell scripts or further subdirectories each with its own `__init__.py` file.
276 277


278 279 280 281 282 283 284 285 286 287 288 289 290 291
### Documenting commands

Commands are very easy to document directly from within the source code without the need for any additional resources. It is recommended you

- provide detailed doc strings for all commands or pass the `help` keyword argument to `@click.command()`,
- provide a short command summary by passing the `short_help` keyword argument to `@click.command()` or...
- ...start the long help description with a short one-line summary terminated by a full stop (or both),
- add a short description via the `help` keyword argument to all options,
- end all sentences, including short help and option descriptions, with a full stop.

After adding a new command and its documentation, update the `cheatsheet.md` file by running `./webis.py core update-cheatsheet` and then commit the changes together with the new command.


### Project guidelines and coding conventions
292
- Write concise and reusable commands.
293 294 295 296 297 298 299 300 301 302
- Start with single-file commands and only upgrade to command directories if a command becomes too complex.
- Don't write Python scripts that are primarily calling external commands.
- Don't write Bash scripts that are primarily doing data processing or manipulation.
- Gracefully degrade to noop if variadic arguments are empty (otherwise shell globs will fail if they turn out empty).
- Don't mix variadic arguments and options with nargs != 1.
- Use `STDOUT` for payload output.
- Use `STDERR` for any additional status output that is not supposed to be piped to other applications.
- Keep code lines below 120 characters.

Language-specific requirements are listed below:
303

Johannes Kiesel's avatar
Johannes Kiesel committed
304
#### Python scripts
305 306
- Scripts must adhere to the PEP8 standard (use PyCharm or at least run a linter!).
- Avoid additional dependencies not already listed in the `setup.py`.
Janek Bevendorff's avatar
Janek Bevendorff committed
307
- Do not add expensive operations to the global module scope, since they would be executed each time the command is listed. Singletons should be written like this:
308 309 310 311 312 313 314 315 316 317
```python
_singleton = None


def get_singleton():
    global _singleton
    if _singleton is None:
        _singleton = expensive_op()
    return _singleton
```
318
- Use Click and its advanced features, don't start parsing or validating arguments yourself (yes, Click can also validate paths and open files for you).
Janek Bevendorff's avatar
Janek Bevendorff committed
319
- Use `click.echo('...')` for normal `STDOUT` output and `click.echo('...', err=True)` for specific `STDERR` output.
320 321
- Don't write your own logging facility, use `libs.log` for uniform log output:
```python
Janek Bevendorff's avatar
Janek Bevendorff committed
322
from log import get_logger
323 324 325 326 327 328 329 330 331 332 333 334 335 336 337
logger = get_logger()

logger.info('Hello World!')
```
- Use `sys.exit(code)` or `ctx.exit(code)` (if using `@click.pass_context()`) for exiting cleanly with a non-zero return code if required.
- Organize imports in the following way:
    1. Python standard libraries
    2. Thirdparty imports
    3. Your own imports

#### Bash scripts
- Scripts must adhere to the Google Bash style conventions.
- Use `#!/usr/bin/env bash` as shebang, not `#!/bin/bash`.
- Use the `bashhelper` functions (`logInfo`, `logError`, `yes_no_prompt`, ...) if possible.
- Avoid dash options/flags and use the positional command syntax with `exec_sub_cmd` if possible.