Silences output of gdal calls in favor of tqdm, making logs much more readable.

In the logs there was a constant warning:

Warning 1: General options of gdal_translate make the COPY_SRC_OVERVIEWS creation option ineffective as they hide the overviews

I have removed this option in this MR, but it should be investigated if that is really the way to go. Link to cogification docs

Question

I have successfully run the batch download pipeline and would like to convert the batch tiles to eopatches. After locally fixing #12 I've managed to run the batch_to_eopatch pipeline, but I get the following exception in the logs:

Summary of exceptions

    LoadUserDataTask (LoadUserDataTask-29825b248e7b11ecbc3b-f57730fc0853):
        14 times:

        TypeError: execute() missing 1 required positional argument: 'eopatch'

Which is weird, because the LoadUserDataTask is the first Task and no eopatch arguments should be expected.

Here is my config:

{
  "pipeline": "eogrow.pipelines.batch_to_eopatch.BatchToEOPatchPipeline",
  "folder_key": "data",
  "mapping": [
    {"batch_files": ["B01.tif"], "feature_type": "data", "feature_name": "B01", "multiply_factor": 1e-4},
    {"batch_files": ["B02.tif"], "feature_type": "data", "feature_name": "B02", "multiply_factor": 1e-4},
    {"batch_files": ["B03.tif"], "feature_type": "data", "feature_name": "B03", "multiply_factor": 1e-4},
    {"batch_files": ["B04.tif"], "feature_type": "data", "feature_name": "B04", "multiply_factor": 1e-4},
    {"batch_files": ["B05.tif"], "feature_type": "data", "feature_name": "B05", "multiply_factor": 1e-4},
    {"batch_files": ["B06.tif"], "feature_type": "data", "feature_name": "B06", "multiply_factor": 1e-4},
    {"batch_files": ["B07.tif"], "feature_type": "data", "feature_name": "B07", "multiply_factor": 1e-4},
    {"batch_files": ["B08.tif"], "feature_type": "data", "feature_name": "B08", "multiply_factor": 1e-4},
    {"batch_files": ["B8A.tif"], "feature_type": "data", "feature_name": "B8A", "multiply_factor": 1e-4},
    {"batch_files": ["B09.tif"], "feature_type": "data", "feature_name": "B09", "multiply_factor": 1e-4},
    {"batch_files": ["B10.tif"], "feature_type": "data", "feature_name": "B10", "multiply_factor": 1e-4},
    {"batch_files": ["B11.tif"], "feature_type": "data", "feature_name": "B11", "multiply_factor": 1e-4},
    {"batch_files": ["B12.tif"], "feature_type": "data", "feature_name": "B12", "multiply_factor": 1e-4},
    {"batch_files": ["CLP.tif"], "feature_type": "data", "feature_name": "CLP", "multiply_factor": 0.00392156862745098},
    {"batch_files": ["CLM.tif"], "feature_type": "mask", "feature_name": "CLM"},
    {"batch_files": ["dataMask.tif"], "feature_type": "mask", "feature_name": "dataMask"}
  ],
  "userdata_feature_name": "BATCH_INFO",
  "userdata_timestamp_reader": "eogrow.utils.batch.read_timestamps_from_orbits",
  "**global_settings": "${config_path}/sentinel2_l1c_batch_config.json"
}

Let me know if you need to see what sentinel2_l1c_batch_config.json looks like.

Exposes another parameter of the rasterization task.

I also noticed a common pattern of validators, which i managed to extract into ensure_exactly_one_defined function. It was tested locally.

The parameter never set to anything other than 1 except by mistake. With this we get rid of some code complexity and inconsistency.

But i'm not entirely sure if this is a step in the right direction :/

By switching from gdal_merge to gdalwarp we can now specify how to resample tiffs that are warped. This is a possible improvement for pixel misalignment.

Another benefit is that gdal_merge loads all files into memory while gdalwarp runs much more memory-conservative.

1. Extracts dealing with an AOI into a BaseSplitterAreaManager (since it's common to both splitter based managers)
2. Implements new Batch area manager
3. adds tests for it that (with some mocking)
4. I learned some new tricks so I fixed some old tests as well before i forgot about it

What is the problem? Please describe.

Imagine a scenario where you are researching a workflow of nodes which are acyclic in nature. You write a task and add it to the node. You mess around, change things, explore, like researchers do. In the end you use the nodes to construct the workflow and run the workflow.

What can happen (speaking from experience):

• you create a task, but forget to use the same task in the node related to that task (old one is used)
• you link the tasks wrong, potentially missing out on a branch in a workflow
• hard to keep track of a list of all the nodes, first you have to defined the node objects and then add them to a list

Alternatives

It would be helpful if this was somehow better managed, to offer the user an easier way to construct a list of nodes with less potential mistakes.

First idea I had was perhaps an additional method of an EOTask, where you call

nodes_list = []
my_created_task = MyCreatedTask(*args, **kwargs)
my_created_node = my_created_task.get_node(input_nodes = [], nodes_list = nodes_list)

my_next_created_task = MyNextCreatedTask(*args, **kwargs)
my_next_created_node = my_next_created_task.get_node(input_nodes = my_created_node, nodes_list = nodes_list)

...

the node my_*_created_node get created and filled automatically into the nodes_list object

For simple linear graphs the input_nodes could default to [nodes_list[-1]], which points to the last node added to the list.

Again, this is just the first thing that came to mind. Not sure if it's the best. I also thought about using some decorators, but didn't manage to find a way where this would could be used.

Adds a new ZipMapPipeline and deprecates the MappingPipeline since it is subsumed by it's successor.

The MappingPipeline is not yet removed, it just emits a warning on use, but is no longer part of the test suite.

Adds an option to temporally split maps when exporting. This will make it much simpler to do byoc ingestion.

I also resolved some path-juggling by enforcing the rule that all paths are relative to the filesystem used (storage or tempfs) and system paths are used only when calling the gdal functions

Each map is suffixed with the timestamp in a near-iso format (the : character is problematic on windows so I tried to avoid it).

The output result of the test is rather huge (3k+ lines) due to around 30 timestamps... I would want it to be run as part of the test-chain that is done on github, and for that I cannot give it less data. But perhaps we could just run a smaller pipeline in the chain-test and have this only as part of the large test suite? Then i can switch it over to some smaller data as well.

ExportToTiff pipeline didn't behave as expected. When merging tiffs, I wanted to set the empty space to the no-data value, turns out there were issues because of:

• not setting the -init param
• not using the values as strings (- was understood as a parameter???)

Thanks @batic for the help.

This PR changes that in case given AWS profile doesn't exist a warning is shown instead of an error. It also adds a test that checks this.

The reason for this is purely practical - if you run a pipeline on AWS instance with a role that already allows accessing S3 buckets you don't need AWS profiles. But if you run a pipeline locally then you need to specify aws_profile in order to access S3 buckets. So to avoid constantly changing aws_profile parameter it seems easier to just give a warning instead of an error. Although I'm not 100% sure if this really justifies this change. :thinking:

Ported all the pipelines to the new managers. Removed only what needed to be removed from the old ones (utm_area and eopatch were causing tests to fail, so they were removed already) and the rest will be removed later.

Went surprisingly smoothly. We really managed to follow:

1. Make the change easy (this is hard)
2. Make the easy change