I have a script where I'm importing a module that internally uses abseil logging. The rest of our code base is using standard python logging. This is leading to a situation where importing that module is interfering with the rest of the code base using standard logging and causing lost log messages.

See here for a description of the issue: https://github.com/tensorflow/hub/issues/263

Have others run into this issue before?

I came across this issue when using the save_main_session in Apache Beam 2.4.0 with absl.flags (absl-py version 0.2.1). save_main_session pickles then unpickles the variables in the main session, flags.FLAGS included.

The following error is raised:

Traceback (most recent call last):
  File "test.py", line 46, in <module>
    app.run(main)
  File "venv/local/lib/python2.7/site-packages/absl/app.py", line 274, in run
    _run_main(main, args)
  File "venv/local/lib/python2.7/site-packages/absl/app.py", line 238, in _run_main
    sys.exit(main(argv))
  File "test.py", line 43, in main
    _ = output | beam.io.WriteToText(FLAGS.output_file)
  File "venv/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 389, in __exit__
    self.run().wait_until_finish()
  File "venv/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 369, in run
    self.to_runner_api(), self.runner, self._options).run(False)
  File "venv/local/lib/python2.7/site-packages/apache_beam/pipeline.py", line 379, in run
    pickler.dump_session(os.path.join(tmpdir, 'main_session.pickle'))
  File "venv/local/lib/python2.7/site-packages/apache_beam/internal/pickler.py", line 242, in dump_session
    dill.load_session(file_path)
  File "venv/local/lib/python2.7/site-packages/dill/dill.py", line 363, in load_session
    module = unpickler.load()
  File "/usr/lib/python2.7/pickle.py", line 864, in load
    dispatch[key](self)
  File "/usr/lib/python2.7/pickle.py", line 1221, in load_build
    setstate = getattr(inst, "__setstate__", None)
  File "venv/local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 468, in __getattr__
    fl = self._flags()
  File "venv/local/lib/python2.7/site-packages/absl/flags/_flagvalues.py", line 141, in _flags
    return self.__dict__['__flags']
KeyError: '__flags'

Example code:

"""
Demonstrating incompatibility between absl.flags and
save_main_session in Apache Beam's Python SDK.
"""
import re
import six
import apache_beam as beam

from absl import app
from absl import flags


FLAGS = flags.FLAGS

flags.DEFINE_string(
    'output_file',
    'output.txt',
    help='Output filename.')

def main(argv):
  del argv # Unused.

  pipeline_options = beam.pipeline.PipelineOptions()
  # Uncomment the next line to break things.
  #pipeline_options.view_as(beam.pipeline.SetupOptions).save_main_session = True

  with beam.Pipeline(options=pipeline_options) as p:
    lines = p | beam.Create(
      ['This is a test of compatibility issues between absl and beam.',
       'Since absl.flags cannot be pickled correctly, it cannot be used',
       'with the save_main_session pipeline option.'])

    counts = (
        lines
        | 'Split' >> (beam.FlatMap(lambda x: re.findall(r'[A-Za-z\']+', x))
                      .with_output_types(six.text_type))
        | 'PairWithOne' >> beam.Map(lambda x: (x, 1))
        | 'GroupAndSum' >> beam.CombinePerKey(sum))

    output = counts | 'Format' >> beam.Map(
      lambda (word, count): '%s: %s' % (word, count))

    _ = output | beam.io.WriteToText(FLAGS.output_file)

if __name__ == '__main__':
  app.run(main)

python 3.6.7, install tensorflow, here is log:

Collecting absl-py>=0.1.6 (from tensorflow->-r pip_requirenents.txt (line 21)) Using cached https://files.pythonhosted.org/packages/fa/ef/1fa0376563b1e0495301ada8e881d88b7ebfda433f6b31f44447fe6ef795/absl-py-0.6.0.tar.gz Complete output from command python setup.py egg_info: /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/distutils/dist.py:261: UserWarning: Unknown distribution option: 'long_description_content_type' warnings.warn(msg) error in absl-py setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in enum34; python_version<='3.4' at ; python_version<='3.4'

Hi, I am trying to write log into file directly instead of showing in command window. I have seen following issues about it https://github.com/abseil/abseil-py/issues/83 https://github.com/abseil/abseil-py/issues/111

I tried them but I cant see anything in the file.

My code is

import os
from absl import logging
if not os.path.exists('/opt/log/'):
     os.makedirs('/opt/log/')
logging.get_absl_handler().use_absl_log_file('absl_logging', '/opt/log/')

logging.info('test')
logging.debug('test debug')

I cant see anything in that file though file was created

Hi there,

I added the WORKSPACE file from GitHub into the package published on pypi, but when I run

bazel test absl

It says:

ERROR: no targets found beneath 'absl'.
INFO: Elapsed time: 0.105s
ERROR: Couldn't start the build. Unable to run tests.

Thanks

When trying to use assertSameStructure to compar a dict to a defaultdict, it throws an error. This is due to the direct type comparison here https://github.com/abseil/abseil-py/blob/master/absl/testing/absltest.py#L1219

Not sure if it's worth adding another exception similar to int vs long. Feel free to close if this is WAI

Hello dear friends,

A few weeks ago, Tensorflow have moved to this library to manage flags using tf.app.flags. Link to file: https://github.com/tensorflow/tensorflow/blob/r1.5/tensorflow/python/platform/flags.py

I can't manage to find any way to "reset" the flags state with this library. Is there any way to do this ?

I would like to remove every flags created and return to a blank state as if it was just created.

Thanks a lot.

Traceback (most recent call last): File "DeepSpeech.py", line 11, in import absl.app File "/home/sehar/venv/lib/python3.6/site-packages/absl/app.py", line 40, in from absl import flags File "/home/sehar/venv/lib/python3.6/site-packages/absl/flags/init.py", line 41, in from absl.flags import _defines File "/home/sehar/venv/lib/python3.6/site-packages/absl/flags/_defines.py", line 31, in from absl.flags import _flagvalues File "/home/sehar/venv/lib/python3.6/site-packages/absl/flags/_flagvalues.py", line 27, in import logging File "/home/sehar/DeepSpeech/logging.py", line 6, in from util.flags import FLAGS File "/home/sehar/DeepSpeech/util/flags.py", line 6, in FLAGS = absl.flags.FLAGS AttributeError: module 'absl' has no attribute 'flags' please help me resolve this issue I am using tensorflow 1.14 gpu based

Was running into error on Python3.6 and pip 18.1

error in absl-py setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Expected version spec in enum34;python_version<='3.4' at ;python_version<='3.4'

Using the install_requires and extra_requires fixed my problems, and is ostensibly a better way to do this see reference here.

There are two logging methods in our code:

import logging
logging.info('my logging msg')

or

import logging
logger = logging.getLogger(__name__)
logger.setLevel(logging.DEBUG)
logger.info('my logger msg')

When we first used app.run(main), everything is fine. All the log lines are output in absl.logging format.

Then we needed to combine absl.flags with argparse, and used your solution in #27 (Let one parser parse "known flags", and the other parse the rest. Example 2) It works great except that we need to run main() instead of app.run(main) And all the logging were gone. So we tried to add absl.logging.use_absl_handler() in __init__.py following #148 Now we are able to see content from logger.xxx in absl.logging format, but not those from logging.xxx no matter it is from python logging or absl logging.

Did I miss anything from app.run() or not using logging correctly?

I am seeing unexpected behaviors when using absl-py with a python library that uses python's native logging library.

Here is a minimal reproduce script:

from absl import app
import logging


def main(argv):
  logger = logging.getLogger('test_logger')
  logger.setLevel(logging.DEBUG)
  ch = logging.StreamHandler()
  ch.setLevel(logging.DEBUG)
  logger.addHandler(ch)

  logger.info('My test message.')


if __name__ == '__main__':
  app.run(main)

Here is the output from the above script:

My test message.
I0806 13:54:58.868444 140734895885760 python_test.py:11] My test message.

The message is somehow duplicated, one with timestamp, and one without timestamp.

I don't have control on the third_party python library (that uses python's native logging library), so I cannot switch that library to use absl-py logging.

Is there anyway to fix or workaround the problem without updating the logging library in the third party python library?

This section of the guide has been "forthcoming" for some time. Is there any timeline on this?

The definition of class Flag, there appear to be some Google internals that "leak" out but are unused.

• allow_override_cpp
• allow_hide_cpp

They seem to be used in absltest.py but is otherwise not well documented.

Minimum repro:

# foo.py
import absl.flags.argparse_flags
import argparse
# absl.flags.DEFINE_string("do_not_use_flagfile_hack", "", "") # A
parser = absl.flags.argparse_flags.ArgumentParser()
parser.add_argument("--foo")
parser.parse_args()

echo "--foo" > flags.txt
foo.py --flagfile=flags.txt

The error is:

foo.py: error: unrecognized arguments: --flagfile=flags.txt

The command would work if I uncomment line A.

It appears that ArgumentParser only accepts --flagfile if there are some DEFINE_'d strings. This is because of this line:

https://github.com/abseil/abseil-py/blob/main/absl/flags/argparse_flags.py#L156

    if self._inherited_absl_flags: # <---
      # Handle --flagfile.
      # Explicitly specify force_gnu=True, since argparse behaves like
      # gnu_getopt: flags can be specified after positional arguments.
      args = self._inherited_absl_flags.read_flags_from_files(
          args, force_gnu=True)

--flagfile is only accepted if bool(self._inherited_absl_flags), which is False if there are no DEFINE_d strings.

There's something odd going on with absl.flags and interacting very badly with multiprocessing when using 'spawn' as a start method. This is on MacOS 11.4, using homebrew's version of Python 3.9.6, although it also fails on the system python 3, 3.8.2.

Given the following code (I'll also atttach it), multifail.py.txt

""" import absl.flags import absl.app import multiprocessing import time

absl.flags.DEFINE_integer("delay", 2, "sleep delay") FLAGS = absl.flags.FLAGS

def worker(n): time.sleep(FLAGS.delay) print(n)

def main(argv): # "fork" works fine. multiprocessing.set_start_method("spawn") with multiprocessing.Pool(20) as pool: pool.map(worker, range(1000))

if name == "main": absl.app.run(main) """

it will fail (inconsistently) with

multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/homebrew/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 125, in worker result = (True, func(*args, **kwds)) File "/opt/homebrew/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 48, in mapstar return list(map(*args)) File "/Users/anthonybaxter/multifail.py", line 10, in worker time.sleep(FLAGS.delay) File "/opt/homebrew/lib/python3.9/site-packages/absl/flags/_flagvalues.py", line 499, in getattr raise _exceptions.UnparsedFlagAccessError(error_message) absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --delay before flags were parsed.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/Users/anthonybaxter/multifail.py", line 22, in absl.app.run(main) File "/opt/homebrew/lib/python3.9/site-packages/absl/app.py", line 312, in run _run_main(main, args) File "/opt/homebrew/lib/python3.9/site-packages/absl/app.py", line 258, in _run_main sys.exit(main(argv)) File "/Users/anthonybaxter/multifail.py", line 18, in main pool.map(worker, range(1000)) File "/opt/homebrew/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 364, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/homebrew/Cellar/[email protected]/3.9.6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/multiprocessing/pool.py", line 771, in get raise self._value absl.flags._exceptions.UnparsedFlagAccessError: Trying to access flag --delay before flags were parsed. """

Sometimes we have some interface and we would like to write a generic test suite for it, so that concrete implementations just reuse it.

The problem with Python's unittest (and transitively absltest) is that the TestCase class serves as a "marker" telling the test executor that it should be executed. This is unfortunate, because abstract test suites should not run. Consider this example:

class FooTest(absltest.TestCase):

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest):

    def new_foo(self) -> Foo:
        return Norf()

Here, the test executor will instantiate FooTest, QuuxTest and NorfTest. However, FooTest is abstract and it is not possible to create an instance of it. There are three workarounds for this that I am aware of.

The first one is to configure the test executor to ignore specific test classes or prefixes (possible in pytest). However, this is awkward and requires modifying external configuration files.

The second one is described here. Basically, we del the base class once child classes are defined. This feels very wrong and works only if all the concrete test cases are defined in the same module.

The last one is to use a mixin approach. Instead of making FooTest derive from absltest.TestCase, we "mark" only concrete classes with it:

class FooTest:

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Norf()

The problem here is that it doesn't work with static type checkers: FooTest now doesn't inherit from TestCase but uses methods like self.assertEqual.

This last solution seems like the only "correct" one except for the mentioned issue. Instead, Abseil could define an abstract test class and make the normal test case class implement it:

class AbstractTestCase(ABC):

    @abstractmethod
    def assertEqual(self, this, that):
      ...

    ...

class TestCase(AbstractTestCase):
    ...

Then, it would be possible to inherit from absltest.AbstractTestCase in the abstract test case, making the type checker happy:

class FooTest(absltest.AbstractTestCase):

    @abstractmethod
    def new_foo(self) -> Foo:
        pass

    def test_bar(self) -> None:
        foo = self.new_foo()
        self.assertEqual(foo.bar(), "bar")


class QuuxTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Quux()


class NorfTest(FooTest, absltest.TestCase):

    def new_foo(self) -> Foo:
        return Norf()

Are there any plans to support a flags.DEFINE_path or something similar that utilizes python's pathlib module to support declaring os.PathLike operations for the default option?

For context, I often find myself doing the following:

from absl import app, flags
from pathlib import Path

FLAGS = flags.FLAGS
flags.DEFINE_string("filepath", "path/to/some.txt", "Input filepath for program")

def main(argv_):
    # Override FLAGS.filepath with a pathlib.Path version
    FLAGS.filepath = Path(FLAGS.filepath)
    
    # Rest of program uses pathlib for path operations

if __name__ == "__main__":
    app.run(main)

It would be awesome if there was an easy way to define a Path object from the DEFINE_ definition in the first place. If adding such a feature is not feasible, I'm curious how other people are using pathlib with absl.flags as well.

Thanks!