Some API questions are how to handle force= and the default output name for patching; I decided to ignore force for files. For the latter issue, I raise ValueError when nml_fname is a file since there is no sane default.

(personally, if it was my library, I would rip out the default name functionality entirely, and make it so that when there is no output file or path, it will still build and return the patched Namelist object without writing a patched namelist file. But that's just me)

Addresses #25 (although there is no string API here, at least yet)

File file contains

&block00
i=1
k=2
&end
&block01
m=0
n=1
&end
&block00
j=3
z=4
&end

The statement f90nml.write(f90nml.read(file),outfile) will result in outfile containing

&block00
i=1
k=2
/
&block00
j=3
z=4
/
&block01
m=0
n=1
/

where order of namelist blocks has not been preserved.

I have a file formatted this way

&iostuff
output_folder = +out
/

When parsed, ['iostuff']['output_folder'] equals ['+', 'out']. I guess this is done so that if the variable out is defined somewhere else, this list is replaced by the corresponding numerical value. But if it isn't defined, the string '+out' should be returned instead. Maybe a simple solution would be having a way to tell f90nml that some variables are strings and should be considered as such.

Trying to convert a namelist with repeated groups to a string seems to fail. If I do the following:

>>> import f90nml
>>> nml = f90nml.reads("""
... &foo
...  a = 1
... /
... &foo
...  b = 2
... /
... """)
>>> str(nml)
"Namelist([('foo', [Namelist([('a', 1)]), Namelist([('b', 2)])])])"

I was expecting str(nml) to return "&foo\n a = 1\n/\n&foo\n b = 2\n/\n", as is the case when the group names do not repeat. Interestingly, if I use nml.write('out.nml') the the output file is formatted correctly.

I'm getting an odd bug when trying the write the following namelist:

 &a
 b%c_d_E(1)%ID = 1,
 b%c_d_E(2)%ID = 2,
 /

Code is:

    import f90nml
    data = f90nml.read('test.nml')
    with open('test_RESAVED.nml','w') as f:
        data.write(f, force=True)

Error message is:

Traceback (most recent call last):
  File "test.py", line 309, in <module>
    data.write(f, force=True)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 232, in write
    self.write_nmlgrp(grp_name, grp_vars, nml_file)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 253, in write_nmlgrp
    for v_str in self.var_strings(v_name, v_val, v_start=v_start):
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 302, in var_strings
    v_strs = self.var_strings(v_title, f_vals, v_start=v_start_new)
  File "C:\Anaconda3\lib\site-packages\f90nml\namelist.py", line 313, in var_strings
    i_s = v_start[::-1][len(v_idx)]
TypeError: 'NoneType' object is not subscriptable

Oddly, when I change c_d_E to c_d_e, it works fine.

I just updated to the latest release using pip. I've been able to read variables like this before, so something has changed.

Trying to use f90nml after installing on macOS via pip3 install f90nml with the -f yaml flag results in:

f90nml: error: YAML module could not be found.

I think you should just add an explicit dependency on pyyaml. Or, better yet, if there's a way to add a recommended (installed by default) dependency on pyyaml that someone could opt out of. (I'm not familiar enough with python packaging to know for certain if this is possible one way or the other.)

Hi there! We've been using f90nml in Pyrokinetics to read/write input files to plasma physics codes, and we ran into an issue where NumPy 0D arrays would propagate unnoticed into a Namelist and cause an exception to be thrown on calling Namelist.write. I've added a small change to Namelist._f90repr to detect 0D arrays and convert them to their equivalent primitive type. Nothing changes if NumPy is not installed.

Let me know if you'd like me to add further test cases, or if you'd prefer to avoid including these sorts of edge cases in your library.

I was wondering if there were possibilities for optimizing performance for very large namelist files. I'm in the process of doing some benchmarks for some large files (say around 1500 lines, with multiple namelists and pretty much all variable types). I can only get around 1.7 calls of f90nml.read() per second. I assume that it's the parsing and/or creating of the structures that is the bottleneck (actually reading in the lines should only take a fraction of a second), but I'm going to investigate further.

One thing I was wondering if parsing of different namelists in the same file could take place in parallel? I don't know if such a thing is possible or not, but if it is that might be something to explore. Maybe this is something I can try and contribute to, rather than just reporting bugs and asking for features!

Hi Marshall, Thanks for this great tool!

The bug occurs for the following variable input line: AXFCLN = 3.0, 3.0, 48*, The 48* denotes a repetition of 48 null values Upon .write(), it is being turned into:
AXFCLN = 3.0, 3.0, ','*,

One clue is that everything is fine (output = input) if the trailing comma is removed prior to processing.

Processing attached files as follows: nml=f90nml.read('repetition_of_null_win.nml') f90nml.patch('repetition_of_null_win.nml', nml, 'patched_data_win.nml') and likewise for the unix line-endings version.

I'm using latest master commit 151438f on Oct 31, 2019 (ver 1.1.2) My sys.version is: 3.7.5 (tags/v3.7.5:5c02a39a0b, Oct 15 2019, 00:11:34) [MSC v.1916 64 bit (AMD64)] I'm running Win10/64/Pro.

patched_data_unix.txt patched_data_win.txt repetition_of_null_win.txt repetition_of_null_unix.txt

When writing namelists containing nested structures that need to be indexed, e.g.

idx_nml
   v(1, 1) = 5
   v(2, 1) = 5
/

the resulting namelist file cannot be parsed by a program compiled with nag (tested here with version 6.1) , if there is a whitespace between the indices. I was wondering, if that whitespace can be omitted?

I'm trying to transform a list of configuration files into a list of f90nml.Namelist objects. StopIteration is being raised by Tokenizer.update_chars().

For example, let's say I have two nml files, foo.nml and bar.nml, this bug can be triggered by :

[f90nml.read(s) for s in ['foo.nml','bar.nml']]

Related to esm-tools/esm_tools#843

Problem description

When loading a namelist with repeated sections, and then I try to patch one or several repeated sections, I get the following error:

AttributeError: 'Cogroup' object has no attribute 'update'

I understand it is not possible to update repeated sections, as it is not clear which one f90nml should patch. However, I find the error message misleading.

Example

To reproduce the error you can write the following fort.4 namelist:

&NAMDIM
    NPROMA = -8, 
/

&NAMPAR0
    LSTATS = true,
    LDETAILED_STATS = false,
    LSYNCSTATS = false,
    MP_TYPE = 2,
    MBX_SIZE = 32000000,
    NPROC = 144,      ! Number of MPI ranks (cores of no OpenMP)
    NOUTPUT = 1,
/

&NAEPHY
    LEPHYS = true,
    LERADI = true,
    LELAIV = false,
/

&NAERAD
    NRPROMA = -8, 
    CRTABLEDIR = "./rtables/",  ! Modify this for your installation, note trailing /
/

&NAMPAR0
/
&NAEPHY
/
&NAERAD
/

Then you can use the following python3 script to reproduce the error:

import f90nml

nml = f90nml.read("fort.4")

changes = { 
    "namdim": {
        "nproma": 0
    }   
}

nml.patch(changes)

print(nml)
print("Changing variables in non-repeated sections works!")
print()
print("Here comes the problem:")

changes = { 
    "nampar0": {
        "lstats": False
    },  
    "naephy": {
        "leradi": False
    },
    "naerad": {
        "nrproma": 0
    },
}

nml.patch(changes)

Suggested solution

Ideally, the user should be reported which variables are problematic in this context. I have sketched a very rough solution with the desired behaviour for these lines: https://github.com/marshallward/f90nml/blob/2a8663aa7dfc06a446fbab6aae20d928f694f3fb/f90nml/namelist.py#L649-L658

The rough solution:

    def patch(self, nml_patch):
        """Update the namelist from another partial or full namelist.

        This is different from the intrinsic `update()` method, which replaces
        a namelist section.  Rather, it updates the values within a section.
        """
+       conflicting_repeated_secs = [] 
        for sec in nml_patch:
            if sec not in self:
                self[sec] = Namelist()
-           self[sec].update(nml_patch[sec])
+           try: 
+               self[sec].update(nml_patch[sec])
+           except AttributeError:
+               conflicting_repeated_secs.append(sec)
+       if conflicting_repeated_secs:
+           raise AttributeError(
+               "The following sections are repeated in the namelist and cannot be "
+               f"updated: {conflicting_repeated_secs}"
+           )  

Given the namelist file setup.nml

&setup
 test%var1 = 1
 test%var2 = 2
/

and the patch

patch={'setup': {'test':{'var2': 3}}}

the call

f90nml.patch('setup.nml', patch, 'patched_setup.nml')

does not patch anything.

However, patching the first occurrence of test% in setup.nml works

patch={'setup': {'test':{'var1': 3}}}

Hi I am trying to load a namefile from the simulation code Ramses (name = namelist.txt). It works most of the time but I have a run when this fails and gives the following message (see below). This may be just that my namefile is wrongly formatted. Is there a way for me to easily read the namelist and see where the formatting may be wrong?

Apologies for the naive question.

Thanks for any tips that may help.

Eric

====================================== [within ipython]

nm = "namelist.txt" f90nml.read(nm)

AssertionError                            Traceback (most recent call last)
Input In [10], in <cell line: 1>()
----> 1 f90nml.read(nm)

File ~/.local/lib/python3.9/site-packages/f90nml/__init__.py:31, in read(nml_path)
     13 """Parse a Fortran namelist file and return its contents.
     14 
     15 File object usage:
   (...)
     28 >>> nml = parser.read(nml_file)
     29 """
     30 parser = Parser()
---> 31 return parser.read(nml_path)

File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:281, in Parser.read(self, nml_fname, nml_patch_in, patch_fname)
    279 nml_file = open(nml_fname, 'r') if nml_is_path else nml_fname
    280 try:
--> 281     return self._readstream(nml_file, nml_patch)
    282 except StopIteration:
    283     raise ValueError('End-of-file reached before end of namelist.')

File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:388, in Parser._readstream(self, nml_file, nml_patch_in)
    385 # Set the next active variable
    386 if self.token in ('=', '(', '%'):
--> 388     v_name, v_values = self._parse_variable(
    389         g_vars,
    390         patch_nml=grp_patch
    391     )
    393     if v_name in g_vars:
    394         v_prior_values = g_vars[v_name]

File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:554, in Parser._parse_variable(self, parent, patch_nml)
    551 self._update_tokens()
    552 self._update_tokens()
--> 554 v_att, v_att_vals = self._parse_variable(
    555     v_parent,
    556     patch_nml=v_patch_nml
    557 )
    559 next_value = Namelist()
    560 next_value[v_att] = v_att_vals

File ~/.local/lib/python3.9/site-packages/f90nml/parser.py:566, in Parser._parse_variable(self, parent, patch_nml)
    561     self._append_value(v_values, next_value, v_idx)
    563 else:
    564     # Construct the variable array
--> 566     assert self.token == '='
    567     n_vals = None
    569     self._update_tokens()

AssertionError: 

I have a namelist like this ( q is an eight dimensional array, q(8) ) :

&example
q(1) = 1, 2, 3
q(4) = 4, 5, 6
/

Fortan can parse correctly and get q = 1, 2, 3, 4, 5, 6, but f90nml cannot parse correctly: Namelist([('example', Namelist([('q', [1, None, None, 4])]))])

One of the namelists I am trying to parse using f90nml contains an array assignment of the following syntax

&example
  arr(1:2)%foo =   1.0,   2.0
  arr(1:2)%bar =   3.0,   4.0
/

If I try to read/parse this namelist

import f90nml
f90nml.reads('''
&example
  arr(1:2)%foo =   1.0,   2.0
  arr(1:2)%bar =   3.0,   4.0
/
''')

I will get the following error

Traceback (most recent call last):
  File "<pyshell#1>", line 5, in <module>
    ''')
  File "C:\Python\Python37-64\lib\site-packages\f90nml\__init__.py", line 47, in reads
    return parser.reads(nml_string)
  File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 280, in reads
    return self._readstream(iter(nml_string.splitlines()))
  File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 359, in _readstream
    patch_nml=grp_patch
  File "C:\Python\Python37-64\lib\site-packages\f90nml\parser.py", line 459, in _parse_variable
    assert v_idx_bounds[0][1] - v_idx_bounds[0][0] == 1
AssertionError

It doesn't seem that the CLI supports the use of '/'.

Using

f90nml -v DIR='./path/to/dir' -v ODIR='/path/to/dir' input

will make DIR have the value '.', and ODIR empty. Escaping doesn't work either.