Snippet 0x0B: Bash completion with sub-commands and dynamic options

March 24, 2015 · Code Snippets, Linux, Scripting

Contents

Every system administrator, most programmers and countless of command line surfing Linux/Mac users use it every day without thinking twice. Hitting the tab key twice, [TAB][TAB], has become the most common thing in the world. Bash completion is the magic behind the tab key. It’s easy to use, but it’s a pain to write. This tiny post demonstrates how to write scripts for bash completion, with sub-commands and dynamic parameters. A working script is embedded in my open source file sync software Syncany.

1. Full code

As always, the full working code (embedded in a my open source file sync software Syncany) is available on GitHub. Feel free to use it, or to leave feedback in the comments.

2. Using `sy` as an example

Bash completion isn’t complicated, but I don’t think there are many good guides/examples out there. I don’t have the know-how to write a good guide, but I can show you how I did it. The Syncany command sy is very powerful because it has several sub-commands and very dynamic options (similar to git).

2.1. Video example

To make it easier to understand how much logic a bash completion script must actually handle, the following short video shows the different possibilities that the script must be able to handle. Among others, that includes the distinction between long and short options, enums, options that require a file completion, options that trigger other sub-options, sub commands, sub sub commands, dynamic options, and so on:

2.2. Text example

Here’s the video output in text form – in case it gets lost or the video is a bit too fast:

# Long and short options
$ sy -[TAB][TAB]
-d          -h          -l          --log       --print     -vv         
--debug     --help      --localdir  --loglevel  -v          

# Long options only
$ sy --[TAB][TAB]
--debug     --help      --localdir  --log       --loglevel  --print     

# Directories
$ sy --log [TAB][TAB]
config/  db/      log/     

# Option-specific enum
$ sy --loglevel [TAB][TAB]
ALL      FINE     FINER    FINEST   INFO     OFF      SEVERE   WARNING  

# Sub-commands
$ sy [TAB][TAB]
cleanup    daemon     genlink    log        ls-remote  restore    up         watch      
connect    down       init       ls         plugin     status     update     

# Sub-command specific enums
$ sy init --plugin [TAB][TAB]
dropbox  ftp      local    raid0    s3       samba    sftp     swift    webdav   

# Sub-command specific option enum (FTP plugin specific)
$ sy init --plugin ftp --plugin-option [TAB][TAB]
hostname=  password=  path=      port=      username= 

# Option-dependent dynamic options (RAID0 plugin consists of two other plugins)
$ sy init --plugin raid0 --plugin-option storage[TAB][TAB]
storage1:type=dropbox  storage1:type=samba    storage2:type=ftp      storage2:type=sftp
storage1:type=ftp      storage1:type=sftp     storage2:type=local    storage2:type=swift
storage1:type=local    storage1:type=swift    storage2:type=raid0    storage2:type=webdav
storage1:type=raid0    storage1:type=webdav   storage2:type=s3       
storage1:type=s3       storage2:type=dropbox  storage2:type=samba

# Option-dependent dynamic sub-options (After 'storage1:type' is chosen as 'ftp', show 'ftp' options for 'storage1.')
$ sy init --plugin raid0 --plugin-option storage1:type=ftp --plugin-option storage1.[TAB][TAB]
storage1.hostname=  storage1.path=      storage1.username=  
storage1.password=  storage1.port=

# Sub-sub-commands
$ sy plugin [TAB][TAB]
install  list     remove   update   

# Sub-sub-command specific options
$ sy plugin list -[TAB][TAB]
-a              -L              -R              -s              
--api-endpoint  --local-only    --remote-only   --snapshots

3. Writing a bash completion script

You’ve certainly stumbled across the Programmable Completion guide, and probably also its second page, the Programmable Completion Builtins. While those are good resources, they don’t quite tell you the most important parts.

Let me try to do it better:

A bash completion script is a normal bash script containing one or many calls to the bash builtin command complete. This command declares certain functions to handle completions, e.g. complete -F _sy sy to declare completions for sy in the bash function _sy – as in the Syncany bash completion script.

The script is called whenever you press [TAB][TAB] and it is located at /etc/bash_completion.d/yourscript. When this file is changed, you need to open a new window/bash for the changes to take effect!

The main script input are an array called COMP_WORDS (all “words”) and an int COMP_CWORD (current “word” index). ${COMP_WORDS[COMP_CWORD]} gives you the current “word”, ${COMP_WORDS[COMP_CWORD-1]} the previous “word”. The main script output is an an array called COMPREPLY. It defines what suggestions to output to the user.

The input array COMP_WORDS contains all “words” typed into the console, including options starting with a dash. To distinguish between options and words, you must write your own helper function.

The compgen builtin command can be used to fill the COMPREPLY array from a list of words (comgen -W), or a function (compgen -F), and many others. Note that compgen takes the same options as complete. compgen is usally used like this: COMPREPLY=($(compgen -W "init connect log list ls" -- $cur)) (where $cur is ${COMP_WORDS[COMP_CWORD]}, or the actual current word) to fill COMPREPLY with the options that match. If $cur is “l”, for instance, COMPREPLY will contain “log”, “list” and “ls”.

Another useful builtin command is compopt. It can define the behavior of the completion. For instance, compopt -o nospace tells bash not to input a space after the completion. This is particularly useful for long-options with an equals-sign, e.g. --log-file=[TAB] should not add a space.

4. Example script (for Syncany)

The following is the current bash completion script for Syncany (this version on GitHub). It works very nicely for Syncany. It might evolve in the future. Please check out the current script and go from there.

#!bash
# Bash completion support for [Syncany](https://www.syncany.org/).

shopt -s progcomp
_sy() {
	local cur prev firstword lastword complete_words complete_options

	# Don't break words at : and =, see [1] and [2]
	COMP_WORDBREAKS=${COMP_WORDBREAKS//[:=]}

	cur=${COMP_WORDS[COMP_CWORD]}
	prev=${COMP_WORDS[COMP_CWORD-1]}
	firstword=$(_sy_get_firstword)
	lastword=$(_sy_get_lastword)

	GLOBAL_COMMANDS="\
		cleanup\
		connect\
		daemon\
		down\
		genlink\
		init\
		log\
		ls\
		ls-remote\
		plugin\
		restore\
		status\
		up\
		update\
		watch"

	GLOBAL_OPTIONS="\
		-l --localdir\
		-d --debug\
		-h --help\
		-v -vv\
		--log\
		--loglevel\
		--print"

	GLOBAL_LOGLEVELS="\
		OFF\
		SEVERE\
		WARNING\
		INFO\
		FINE\
		FINER\
		FINEST\
		ALL"

	CONNECT_OPTIONS="\
		-P --plugin\
		-o --plugin-option\
		-n --add-daemon\
		   --password"

	DOWN_CONFLICT_STRATEGIES="\
		ask\
		rename"

	DOWN_OPTIONS="\
		-C --conflict-strategy\
		-A --no-apply"

	DAEMON_COMMANDS="\
		force-stop\
		reload\
		restart\
		start\
		status\
		stop\
		list\
		add\
		remove"

	GENLINK_OPTIONS="\
		-s --short\
		-m --machine-readable"

	INIT_OPTIONS="\
		-P --plugin\
		-o --plugin-option\
		-E --no-encryption\
		-G --no-compression\
		-t --create-target\
		-a --advanced\
		-n --add-daemon\
		-s --short\
		   --password"

	LOG_OPTIONS="\
		-n --database-count\
		-s --database-start\		
		-f --file-count\
		-x --exclude-empty"
		
	LS_OPTIONS="\
		-V --versions\
		-t --types\
		-D --date\
		-r --recursive\
		-q --deleted\
		-f --full-checksums\
		-g --group\
		-H --file-history"

	LS_TYPES="\
		t\
		d\
		f"

	PLUGIN_COMMANDS="\
		list\
		install\
		update\
		remove"

	PLUGIN_IDS_TRANSFER="\
		dropbox\
		ftp\
		local\
		raid0\
		s3\
		samba\
		sftp\
		swift\
		webdav"
		
	PLUGIN_IDS_ALL="\
		$PLUGIN_IDS_TRANSFER\
		flickr\
		gui"		
		
	PLUGIN_LIST_OPTIONS="\
		-R --remote-only\
		-L --local-only\
		-s --snapshots\
		-a --api-endpoint"

	PLUGIN_INSTALL_OPTIONS="\
		-s --snapshot\
		-m --minimal-output"

	PLUGIN_DROPBOX_OPTIONS="\
		authToken=\
		path="

	PLUGIN_FTP_OPTIONS="\
		hostname=\
		username=\
		password=\
		path=\
		port="

	PLUGIN_LOCAL_OPTIONS="\
		path="

	PLUGIN_S3_OPTIONS="\
		accessKey=\
		secretKey=\
		bucket=\
		location="

	PLUGIN_SAMBA_OPTIONS="\
		hostname=\
		username=\
		password=\
		share=\
		path="

	PLUGIN_SFTP_OPTIONS="\
		hostname=\
		username=\
		privatekey=\
		password=\
		path=\
		port="

	PLUGIN_SWIFT_OPTIONS="\
		authUrl=\
		username=\
		password="

	PLUGIN_WEBDAV_OPTIONS="\
		url=\
		username=\
		password="

	RESTORE_OPTIONS="\
		-r --revision\
		-t --target"

	STATUS_OPTIONS="\
		-f --force-checksum\
		-D --no-delete"

	UP_OPTIONS="\
		-R --no-resume\
		$STATUS_OPTIONS"
		
	UPDATE_COMMANDS="\
		check"
				
	UPDATE_CHECK_OPTIONS="\
		-a --api-endpoint\
		-s --snapshots"		

	WATCH_OPTIONS="\
		-i --interval\
		-s --delay\
		-W --no-watcher\
		-a --announce\
		-N --no-announcements\
		$UP_OPTIONS\
		$DOWN_OPTIONS"

	CLEANUP_OPTIONS="\
		-F --force\
		-o --delete-older-than\
		-I --no-delete-interval\
		-O --no-delete-older-than\
		-T --no-temp-removal\
		$STATUS_OPTIONS"

	# Un-comment this for debug purposes:
	#   echo -e "\nprev = $prev, cur = $cur, firstword = $firstword, lastword = $lastword\n"

	case "${firstword}" in
	cleanup)
		complete_options="$CLEANUP_OPTIONS"
		;;

	connect)
		case "${prev}" in
			--plugin|-P)
				complete_words="$PLUGIN_IDS_TRANSFER"
				;;

			--plugin-option|-o)
				complete_words=$(_sy_plugin_option_words) 

				if [[ ${#COMPREPLY[@]} != 1 || $complete_words = *= ]]; then
					compopt -o nospace
				fi
				;;

			*)
				complete_options="$CONNECT_OPTIONS"
				;;
		esac
		;;

	down)
		case "${prev}" in
			--conflict-strategy|-C)
				complete_words="$DOWN_CONFLICT_STRATEGIES"
				;;

			*)
				complete_options="$DOWN_OPTIONS"
				;;
		esac
		;;

	daemon)
		case "${lastword}" in
			force-stop|reload|restart|start|status|stop|list)
				;;
				
			add|remove)
				# Special handling: return directories, no space at the end

				compopt -o nospace
				COMPREPLY=( $( compgen -d -S "/" -- $cur ) )

				return 0
				;;

			*)
				complete_words="$DAEMON_COMMANDS"
				;;
		esac
		;;

	genlink)
		complete_options="$GENLINK_OPTIONS"
		;;

	init)
		case "${prev}" in
			--plugin|-P)
				complete_words="$PLUGIN_IDS_TRANSFER"
				;;

			--plugin-option|-o)
				complete_words=$(_sy_plugin_option_words) 

				if [[ ${#COMPREPLY[@]} != 1 || $complete_words = *= ]]; then
					compopt -o nospace
				fi

				;;

			*)
				complete_options="$INIT_OPTIONS"
				;;
		esac
		;;
		
	log)
		complete_options="$LOG_OPTIONS"
		;;

	ls)
		case "${prev}" in
			--types|-t)
				complete_words="$LS_TYPES"
				;;

			*)
				case "${cur}" in
					-*)
						complete_options="$LS_OPTIONS"
						;;

					*)
						# Special handling: return directories, no space at the end

						compopt -o nospace
						COMPREPLY=( $( compgen -d -S "/" -- $cur ) )

						return 0
						;;
				esac
				;;
		esac
		;;

	ls-remote)
		;;

	plugin)
		case "${lastword}" in
		list)
			complete_options="$PLUGIN_LIST_OPTIONS"
			;;
			
		install)
			complete_words="$PLUGIN_IDS_ALL"
			complete_options="$PLUGIN_INSTALL_OPTIONS"
			;;
			
		update)
			complete_words="$PLUGIN_IDS_ALL"
			;;
			
		remove)
			complete_words="$PLUGIN_IDS_ALL"
			;;

		*)
			complete_words="$PLUGIN_COMMANDS"
			;;
		esac
		;;

	restore)
		case "${prev}" in
			--target|-t)
				# Special handling: Return file names

				COMPREPLY=( $( compgen -f -- $cur ) )
				return 0
				;;

			*)
				complete_options="$RESTORE_OPTIONS"
				;;
		esac
		;;

	status)
		complete_options="$STATUS_OPTIONS"
		;;

	up)
		complete_options="$UP_OPTIONS"
		;;
		
	update)
		case "${lastword}" in
		check)
			complete_options="$UPDATE_CHECK_OPTIONS"
			;;
		
		*)
			complete_words="$UPDATE_COMMANDS"
			;;
		esac
		;;		

	watch)
		case "${prev}" in
			--conflict-strategy|-C)
				complete_words="$DOWN_CONFLICT_STRATEGIES"
				;;

			*)
				complete_options="$WATCH_OPTIONS"
				;;
		esac
		;;

	*)
		case "${prev}" in
			--log|--localdir|-l)
				# Special handling: return directories, no space at the end

				compopt -o nospace
				COMPREPLY=( $( compgen -d -S "/" -- $cur ) )

				return 0
				;;

			--loglevel)
				complete_words="$GLOBAL_LOGLEVELS"
				;;

			*)
				complete_words="$GLOBAL_COMMANDS"
				complete_options="$GLOBAL_OPTIONS"
				;;
		esac
		;;
	esac

	# Either display words or options, depending on the user input
	if [[ $cur == -* ]]; then
		COMPREPLY=( $( compgen -W "$complete_options" -- $cur ))

	else
		COMPREPLY=( $( compgen -W "$complete_words" -- $cur ))
	fi

	return 0
}

## Helper functions ###

# Finds plugin identifier and determines possible plugin-specific settings
_sy_plugin_option_words() {
	local plugin_id i

	plugin_id=
	for ((i = 1; i < ${#COMP_WORDS[@]}; ++i)); do
		if [[ ${COMP_WORDS[i]} == "-P" ]] || [[ ${COMP_WORDS[i]} == "--plugin" ]]; then
			plugin_id=${COMP_WORDS[i+1]}
			break
		fi
	done

	# Output completions without prefix
	echo $(_sy_plugin_option_words_for_plugin "$plugin_id" "")
}

# Determines the plugin-specific settings set by --plugin-option or -o for a specific plugin
# Parameters: $1 is plugin ID (e.g. "ftp"), $2 is plugin prefix (e.g. "storage1.")
_sy_plugin_option_words_for_plugin() {
	local plugin_id="$1"
	local plugin_prefix="$2"
	local plugin_completions=""

	case "${plugin_id}" in
		dropbox)
			plugin_completions="$PLUGIN_DROPBOX_OPTIONS"
			;;

		ftp)
			plugin_completions="$PLUGIN_FTP_OPTIONS"
			;;

		local)
			plugin_completions="$PLUGIN_LOCAL_OPTIONS"
			;;

		raid0)
			plugin_completions=$(_sy_plugin_option_words_raid0)
			;;

		s3)
			plugin_completions="$PLUGIN_S3_OPTIONS"
			;;

		samba)
			plugin_completions="$PLUGIN_SAMBA_OPTIONS"
			;;

		sftp)
			plugin_completions="$PLUGIN_SFTP_OPTIONS"
			;;

		swift)
			plugin_completions="$PLUGIN_SWIFT_OPTIONS"
			;;

		webdav)
			plugin_completions="$PLUGIN_WEBDAV_OPTIONS"
			;;

		*)
			;;
	esac

	# Output with prefix
	for plugin_option in $plugin_completions; do
		echo "${plugin_prefix}${plugin_option}"
	done
}

# Determines the current plugin optons for RAID0 plugin, 
# including sub-options of other plugins
_sy_plugin_option_words_raid0() {
	local storage1_type storage2_type i

	storage1_type=
	storage2_type=
	for ((i = 1; i < ${#COMP_WORDS[@]}; ++i)); do
		if [[ $cur != ${COMP_WORDS[i]} && ${COMP_WORDS[i]} == storage1:type=* ]]; then
			storage1_type=${COMP_WORDS[i]/storage1:type=}
		fi

		if [[ $cur != ${COMP_WORDS[i]} && ${COMP_WORDS[i]} == storage2:type=* ]]; then
			storage2_type=${COMP_WORDS[i]/storage2:type=}
		fi
	done

	if [[ $storage1_type != "" && $storage1_type != "raid0" ]]; then
		echo $(_sy_plugin_option_words_for_plugin "$storage1_type" "storage1.")		
	else
		for plugin_id in $PLUGIN_IDS_TRANSFER; do
			echo "storage1:type=$plugin_id"
		done
	fi

	if [[ $storage2_type != "" && $storage2_type != "raid0" ]]; then
		echo $(_sy_plugin_option_words_for_plugin "$storage2_type" "storage2.")
	else
		for plugin_id in $PLUGIN_IDS_TRANSFER; do
			echo "storage2:type=$plugin_id"
		done
	fi
}

# Determines the first non-option word of the command line. This
# is usually the command
_sy_get_firstword() {
	local firstword i

	firstword=
	for ((i = 1; i < ${#COMP_WORDS[@]}; ++i)); do
		if [[ ${COMP_WORDS[i]} != -* ]]; then
			firstword=${COMP_WORDS[i]}
			break
		fi
	done

	echo $firstword
}

# Determines the last non-option word of the command line. This
# is usally a sub-command
_sy_get_lastword() {
	local lastword i

	lastword=
	for ((i = 1; i < ${#COMP_WORDS[@]}; ++i)); do
		if [[ ${COMP_WORDS[i]} != -* ]] && [[ -n ${COMP_WORDS[i]} ]] && [[ ${COMP_WORDS[i]} != $cur ]]; then
			lastword=${COMP_WORDS[i]}
		fi
	done

	echo $lastword
}

## Define bash completions ###

complete -F _sy sy
complete -F _sy syncany

Let me know if you have any questions or corrections. I’d love to hear your feedback!

A. About this post

I’m trying a new section for my blog. I call it Code Snippets. It’ll be very short, code-focused posts of things I recently discovered or find fascinating or helpful. I hope this helps.