If the hash for the mra_alternatives summary file doesn't match the newest, mra_alternatives_summary.json.zip gets downloaded and the hashes are updated in the store (summary and files) but the files themselves aren't updated.

When I try to run aupdate from the MiSTer menu the part where it updates the CA-certs work. But after that it fails.

The script will run normally now:
Running MiSTer Downloader

bash: line 122: BASH_SOURCE[0]: unbound variable
Press any key to continue

I started receiving errors when trying to update today. Unzip is throwing an exception because it is recognizing the downloaded file as malicious. I am running the latest version of download.sh. The same issue occurs with any update script which downloads zipped files. Previously had no issues updating.

/root# cat /media/fat/Scripts/.config/downloader/downloader.log START!

Reading file: /media/fat/downloader.ini Reading 'distribution_mister' db section Reading 'jtcores' db section env: { "DOWNLOADER_LAUNCHER_PATH": "/media/fat/Scripts/downloader.sh", "DOWNLOADER_INI_PATH": null, "CURL_SSL": "--cacert /etc/ssl/certs/cacert.pem", "COMMIT": "856b6d5", "ALLOW_REBOOT": null, "UPDATE_LINUX": "true", "DEFAULT_DB_URL": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "DEFAULT_DB_ID": "distribution_mister", "DEFAULT_BASE_PATH": null, "DEBUG": "false", "FAIL_ON_FILE_ERROR": "false" } config: { "databases": { "distribution_mister": { "db_url": "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip", "section": "distribution_mister" }, "jtcores": { "db_url": "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip", "section": "jtcores" } }, "config_path": "/media/fat/downloader.ini", "base_path": "/media/fat", "base_system_path": "/media/fat", "storage_priority": "prefer_sd", "allow_delete": 1, "allow_reboot": 1, "update_linux": true, "parallel_update": true, "downloader_size_mb_limit": 100, "downloader_process_limit": 300, "downloader_timeout": 300, "downloader_retries": 3, "zip_file_count_threshold": 60, "zip_accumulated_mb_threshold": 100, "filter": null, "verbose": false, "debug": false, "default_db_id": "distribution_mister", "start_time": 1654702379.9457989, "user_defined_options": [], "curl_ssl": "--cacert /etc/ssl/certs/cacert.pem", "update_linux_environment": 0, "fail_on_file_error": false, "commit": "856b6d5" } cacert file at "/etc/ssl/certs/cacert.pem" seems to be fine. Loading db from url: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip Loading db from url: https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip Downloading 2 files: /tmp/temp_filet7etzsji /tmp/temp_filex45tz6id *..

Checking hashes... ++ subprocess.run unzip -p /tmp/temp_filet7etzsji Return Code was '12'

error: invalid zip file with overlapped components (possible zip bomb)

Could not load json from "https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip" subprocess.run unzip -p /tmp/temp_filex45tz6id Return Code was '12'

error: invalid zip file with overlapped components (possible zip bomb)

Could not load json from "https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip" Skipping local_store saving...

=========================== Downloader 1.5 (856b6d5) by theypsilon. Run time: 0:00:02.29s Log: /media/fat/Scripts/.config/downloader/downloader.log

Installed: none.

Errors: https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip, https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

linux_descriptions_count: 0 Length of failed_dbs: 2

Hi there,

Is there any way to set the downloader to update and install the files I want straight into the /media folder instead of adding the associated _Console folder or any other similar folder? I don't use a ton of cores so I'd love to just run the script and keep everything clean. This way, I can just boot the MiSTer, and immediately see the cores in the list without entering additional directories.

(Unless this is already possible)

Hello,

I am attempting to run update_all.sh script with the downloader enabled. The latest jtbeta.zip file is in my 'mame' folder and all of the necessary options are enabled, as far as I can tell, in the update_all options menu (i.e. Install Premium Cores, Install Unofficial Cores etc.) I am recieving the following error:

Reading file: /media/fat/download.ini Reading 'distribution_mister' db section Reading 'jtcores' db section Reading 'theypsilon_unofficial_distribution" db section Could not load storage Could not load json from "https://raw.githubusercontent.com/MiSTer--devel/Distribution_MiSTer/main/db.json.zip" Could not load json from "https://raw.githubusercontent.com/jotego/jtpremium/main/jtbindb.json.zip" Could not load json from "https://raw.githubusercontent.com/theypsilon/Distribution_Unofficial_MiSTer/unofficialdb.json.zip"

There were some errors in the Updaters. Therefore, MiSTer hasn't been fully updated.

Check these logs from the Updaters that failed:

• /media/fat/Scripts/ .config/downloader/downloader1.log
• BIOS-GETTER
• /media/fat/Scripts/ .config/downloader/downloader2.log

I receive the same error when running just the downloader.sh script. All official cores were downloaded and installed correctly using update_all.

Thank you!

Hi,

I get the following error from https://github.com/MiSTer-devel/Downloader_MiSTer/releases/download/latest/downloader.zip

{code} Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in run_code File "c:\TEMP\roka\2\tmp2bdrzqig.zip_main.py", line 49, in NameError: name 'exit' is not defined {code}

sys.exit instead of exit is needed

While I was reading the scripts (to understand how they work) I've started putting some comments at some places. Later I've started introducing some "echo" statement as well. Also, I've noticed /etc/ssl/certs/cacert.pem is used in many places as a string literal, so I've created a constant to pass this around. In the end, I've tried to run this outside of MiSTer and I got hit by some missing binaries, so I added a function to check for those.

Nothing significant here, mostly cosmetic changes I would say. Feel free to approve or reject this.

I'm on a fresh install and I am getting this error during the BiosDB step (in update_all):

Extracting NeoGeo UniBios from http://unibios.free.fr
checkdir:  cannot create extraction directory: /tmp/q48as6j6/neogeo_unibios
           No such file or directory
Traceback (most recent call last):
  File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
    exit_code = execute_full_run(
  File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
    exit_code = runner.full_run()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
    result = self._full_run_impl()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
    self._online_importer.download_dbs_contents(importer_command, full_resync)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
    db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 586, in process_changed_files
    self._import_zip_contents(needed_zips, filtered_zip_data, file_downloader)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 672, in _import_zip_contents
    self._file_system.unzip_contents(temp_zip, tmp_path, list(zipped_files['files']))
  File "/tmp/dont_download.zip/downloader/file_system.py", line 335, in unzip_contents
    raise Exception("Could not unzip %s: %s" % (file, result.returncode))
Exception: Could not unzip /tmp/hir0qr7h_neogeo_unibios_contents.zip: 2

I think the parent tmp directory isn't being created after the name is generated?

https://github.com/MiSTer-devel/Downloader_MiSTer/blob/0060984366a335546a7152ff83065c729d962d83/src/downloader/online_importer.py#L671

I opened an issue for this over in the retronas repo, but it might be better suited here.

Copy/paste from the original issue:

Hi!

Just getting started with retronas, and connected my MiSTer setup to it over CIFS. When I run downloader.sh, it tries to remove the empty folder for ATARI2600, but ends up failing:

Deleting empty folder /media/fat/games/ATARI2600
Traceback (most recent call last):
  File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
    exit_code = execute_full_run(
  File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
    exit_code = runner.full_run()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
    result = self._full_run_impl()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
    self._online_importer.download_dbs_contents(importer_command, full_resync)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 108, in download_dbs_contents
    self._remove_folders(importer_command)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 210, in _remove_folders
    system_file_system.remove_folder(full_folder_path)
  File "/tmp/dont_download.zip/downloader/file_system.py", line 239, in remove_folder
    os.rmdir(self._path(path))
NotADirectoryError: [Errno 20] Not a directory: '/media/fat/games/ATARI2600'

This makes the rest of the update/download fail as well. Creating an empty file in the ATARI2600 folder removes this failure point.

Readme says to use this ini to make changes, but it never reads from it. It's only looking at this one according to my log: Reading file: /media/fat/Update/MiSTer_Downloader/downloader.ini

My ini: https://pastebin.com/DNPrxXEJ

Log: https://pastebin.com/1HwS6gh8

I set verbose to true and log says it's false Also tried moving the ini to "/media/fat/Update/MiSTer_Downloader/downloader.ini" and it still doesn't read it.

So I created downloader.ini, copy/pasted the defaults and saved it into my root directory (where MiSTer.ini) lives. I noticed that when I run the downloader script now, nothing updates. After doing some research, I found that the downloader.ini needs to point to the default MiSTer [distribution_mister] database.

If this is the case, then shouldn't this be documented and added to the list of default ini settings?

Copying new MiSTer binary:
Traceback (most recent call last):
  File "/tmp/dont_download.zip/downloader/main.py", line 37, in main
    exit_code = execute_full_run(
  File "/tmp/dont_download.zip/downloader/main.py", line 57, in execute_full_run
    exit_code = runner.full_run()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 59, in full_run
    result = self._full_run_impl()
  File "/tmp/dont_download.zip/downloader/full_run_service.py", line 121, in _full_run_impl
    self._online_importer.download_dbs_contents(importer_command, full_resync)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 95, in download_dbs_contents
    db_importer.process_changed_files(changed_files, needed_zips, filtered_zip_data)
  File "/tmp/dont_download.zip/downloader/online_importer.py", line 588, in process_changed_files
    file_downloader.download_files(self._is_first_run())
  File "/tmp/dont_download.zip/downloader/file_downloader.py", line 121, in download_files
    self._file_system.move(FILE_MiSTer_new, FILE_MiSTer)
  File "/tmp/dont_download.zip/downloader/file_system.py", line 189, in move
    os.replace(self._path(source), self._path(target))
OSError: [Errno 18] Invalid cross-device link: '/media/usb0/MiSTer.new' -> '/media/fat/MiSTer'

Downloader failed!

Here is my download.ini

[MiSTer]
base_path = '/media/usb0'

[distribution_mister]
db_url = https://raw.githubusercontent.com/MiSTer-devel/Distribution_MiSTer/main/db.json.zip

[jtcores]
db_url = https://raw.githubusercontent.com/jotego/jtcores_mister/main/jtbindb.json.zip

[bios_db]
db_url = https://raw.githubusercontent.com/theypsilon/BiosDB_MiSTer/db/bios_db.json

A common source of problems is when the core logic changes leaving the saved config/DIP files unusable and possibly breaking some functionality if used with old files.

For DIP switches, it is easy to detect a change by comparing the dip switch section inside the MRA with the previous MRA file. If a change in the DIP sw. section is detected, the downloader should delete the old file in the config/dips folder. This would apply to any MRA of any arcade core.

For config file changes, I don't think it's easy to detect automatically because the configuration information shown in the OSD is embedded in the RBF file and has gone through some transformations. So it isn't plain ASCII. Maybe the downloader could read some metadata in the repositories. I could, for instance, make the config string available as plain text in the version.log file of each core. Currently, this file points to the source code git commit used to produce a specific RBF in JTBIN.

Please consider implementing these improvements.