0

We can add our own types to a parser's registry, for example:

from argparse import ArgumentParser
from distutils.util import strtobool

parser = ArgumentParser('flats')
parser.register('type', 'boolean', strtobool)
parser.add_argument('--conveyor-belt', type='boolean')

This works so far, i.e. args = parser.parse_args(['--conveyor-belt', 'on']) and then args.conveyor_belt == True. But when trying to add subparser(s):

subparsers = parser.add_subparsers()
subparser = subparsers.add_parser('slaughterhouse')
subparser.add_argument('--rotating-knives', type='boolean')  # crash

We get error here: ValueError 'boolean' is not callable.

I want all subparsers to automatically inherit the types already registered by parent. Possible?

wim
  • 338,267
  • 99
  • 616
  • 750
  • 2
    Umm, I can't find any documentation about the `register` function in [the argparse docs](https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser). Are you sure you're not working with implementation details here? – Aran-Fey Apr 13 '18 at 22:03
  • Also, the [`type` argument docs](https://docs.python.org/3/library/argparse.html#type) state that you may pass "any callable that takes a single string argument", but it never says that you can pass a string like `'boolean'` there. – Aran-Fey Apr 13 '18 at 22:07
  • I assume `ArgumentParser.register` is public interface and `ArgumentParser._registry_get` is non-public, because of the usual Python conventions. Undocumented != implementation detail. Happy to be convinced otherwise here, though. – wim Apr 13 '18 at 22:16
  • I demonstrated the use of `register` here (in 2013): [Parsing boolean values with argparse](https://stackoverflow.com/questions/15008758/parsing-boolean-values-with-argparse/19233287#19233287) – hpaulj Apr 13 '18 at 22:30
  • @hpaulj I know you are very familiar with argparse. Why do you think this thing is undocumented? Non-public API, or documentation bug? – wim Apr 13 '18 at 22:43
  • @wim, the documentation does not try to be a full, formal API. It's more of a common usage manual, a step beyond a tutorial. It doesn't record all public classes and their methods. – hpaulj Apr 13 '18 at 22:55
  • Meh, there is also a usage manual/tutorial [here](https://docs.python.org/3/howto/argparse.html). I think it's more a case that `argparse.py` is just a poorly maintained and poorly documented module. – wim Apr 13 '18 at 22:56
  • When looking up some past bug/issues I realized that the `register` may need to be used when customizing the `_SubParsersAction` class. – hpaulj Apr 15 '18 at 00:38

1 Answers1

0

I demonstrated the use of register here, several years ago.

Parsing boolean values with argparse

It isn't a hidden feature (i.e. no '_') but also not documented.

Each parser has a registry that matches strings with classes and functions. If action='store_true', the parser looks up 'store_true' in the registry and finds the corresponding Action subclass.

print(parser._registries)

type strings are also looked up, though as default only 'None' is registered. Other common type values such as int and float are Python functions, and don't need to be registered.

Subparsers are new parsers, using the main parser class. I don't think they use any of its __init__ parameters, and certainly they don't use any attributes that might have been changed after creation.

It might be possible to define a ArgumentParser subclass that adds these items to the registry.

Some relevant parts of the argparse.py code:

def add_subparsers(self, **kwargs):

        # add the parser class to the arguments if it's not present
        kwargs.setdefault('parser_class', type(self))
        ...
        # create the parsers action and add it to the positionals list
        parsers_class = self._pop_action_class(kwargs, 'parsers')
        action = parsers_class(option_strings=[], **kwargs)
        ....

class _SubParsersAction(Action):
    def __init__(self,
                 option_strings,
                 prog,
                 parser_class,
                 dest=SUPPRESS,
                 help=None,
                 metavar=None):
        ...
        self._parser_class = parser_class
    def add_parser(self, name, **kwargs):
       ...
        parser = self._parser_class(**kwargs)

So the class of the main parser is recorded with the subparsers Action object., and that is used when creating the subparser. But parameters like description, and help_formatter are taken from the add_parser command, not inherited. There was a bug/issue asking for inheritance of the formatter_class, but no action.


From your code:

In [22]: parser._registries['type']
Out[22]: 
{None: <function argparse.ArgumentParser.__init__.<locals>.identity>,
 'boolean': <function distutils.util.strtobool>}

and

In [24]: subparser._registries['type']
Out[24]: {None: <function argparse.ArgumentParser.__init__.<locals>.identity>}
In [25]: subparser._registries['type'].update(parser._registries['type'])
In [27]: subparser.add_argument('--rotating-knives', type='boolean')  # crash
Out[27]: _StoreAction(option_strings=['--rotating-knives'], dest='rotating_knives', nargs=None, const=None, default=None, type='boolean', choices=None, help=None, metavar=None)

or instead of update, simply share the dictionary:

subparser._registries = parser._registries

argparse does a lot of attribute sharing. For example, the _actions list is shared between parser and all of its action_groups.

So if I were to add this to argparse, or my own version, I'd probably modify the _SubParsersAction class rather than the ArgumentParser class. Either that, or just define a helper function:

   def new_parser(subparsers, *args, **kwargs):
       # parser - take from global
       sp = subparsers.add_parser(*args, **kwargs)
       sp._registries = parser._registries
       sp.help.formatter_class = parser.formatter_class
       etc

As best I can tell, subparsers does not have any reference to the main parser. That is, subparsers is in the parser._actions list, but there's no link in the other direction.

Changing subparsers Action class

parser._defaults is another attribute that isn't passed from main parser to subparsers. It is set with set_defaults, which is documented:

https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.set_defaults

Setting a different value in the subparsers is documented as a way of linking a function to the subparser:

https://docs.python.org/3/library/argparse.html#argparse.ArgumentParser.add_subparsers

Judging from questions here about subparsers, it is clear that the feature is quite useful. But it is also a bit tricky to use, and not to everyone's liking.

One of the last patches that the original author, Steven J. Bethard, wrote had to do with passing the Namespace between main parser and subparsers.

argparse set_defaults on subcommands should override top level set_defaults

But that wasn't to everyone's liking.

argparse - subparsers does not retain namespace

In light of the current question, it interesting that I suggest using the register to use a customized _SubParsersAction.

It uses a custom Action class (like your test case). It subclasses ._SubParsersAction, and replaces the 9351 namespace use with the original one. I use the registry to change the class that parser.add_subparsers() uses.

p.register('action', 'parsers', MyParserAction)

That's required because as the add_subparsers quote (above) shows, argparse uses the registry to determine what subclass to use.

One might argue that this use of parser.register needs to be documented. But I don't see how it can be done without adding confusion to users who don't need the feature.

hpaulj
  • 221,503
  • 14
  • 230
  • 353