A Python client for the Softcite software mention recognizer server

Overview

Softcite software mention recognizer client

Python client for using the Softcite software mention recognition service. It can be applied to

  • individual PDF files

  • recursively to a local directory, processing all the encountered PDF

  • to a collection of documents harvested by biblio-glutton-harvester and article-dataset-builder, with the benefit of re-using the collection manifest for injectng metadata and keeping track of progress. The collection can be stored locally or on a S3 storage.

Requirements

The client has been tested with Python 3.5-3.7.

The client requires a working Softcite software mention recognition service. Service host and port can be changed in the config.json file of the client.

Install

cd software_mention_client/

It is advised to setup first a virtual environment to avoid falling into one of these gloomy python dependency marshlands:

virtualenv --system-site-packages -p python3 env

source env/bin/activate

Install the dependencies, use:

pip3 install -r requirements.txt

Usage and options

usage: software_mention_client.py [-h] [--repo-in REPO_IN] [--file-in FILE_IN]
                                  [--file-out FILE_OUT]
                                  [--data-path DATA_PATH] [--config CONFIG]
                                  [--reprocess] [--reset] [--load]
                                  [--diagnostic] [--scorched-earth]

Softcite software mention recognizer client

optional arguments:
  -h, --help            show this help message and exit
  --repo-in REPO_IN     path to a directory of PDF files to be processed by
                        the Softcite software mention recognizer
  --file-in FILE_IN     a single PDF input file to be processed by the
                        Softcite software mention recognizer
  --file-out FILE_OUT   path to a single output the software mentions in JSON
                        format, extracted from the PDF file-in
  --data-path DATA_PATH
                        path to the resource files created/harvested by
                        biblio-glutton-harvester
  --config CONFIG       path to the config file, default is ./config.json
  --reprocess           reprocessed failed PDF
  --reset               ignore previous processing states and re-init the
                        annotation process from the beginning
  --load                load json files into the MongoDB instance, the --repo-
                        in parameter must indicate the path to the directory
                        of resulting json files to be loaded
  --diagnostic          perform a full count of annotations and diagnostic
                        using MongoDB regarding the harvesting and
                        transformation process
  --scorched-earth      remove a PDF file after its successful processing in
                        order to save storage space, careful with this!

The logs are written by default in a file ./client.log, but the location of the logs can be changed in the configuration file (default ./config.json).

Processing local PDF files

For processing a single file., the resulting json being written as file at the indicated output path:

python3 software_mention_client.py --file-in toto.pdf --file-out toto.json

For processing recursively a directory of PDF files, the results will be:

  • written to a mongodb server and database indicated in the config file

  • and in the directory of PDF files, as json files, together with each processed PDF

python3 software_mention_client.py --repo-in /mnt/data/biblio/pmc_oa_dir/

The default config file is ./config.json, but could also be specified via the parameter --config:

python3 software_mention_client.py --repo-in /mnt/data/biblio/pmc_oa_dir/ --config ./my_config.json

Processing a collection of PDF harvested by biblio-glutton-harvester

biblio-glutton-harvester and article-dataset-builder creates a collection manifest as a LMDB database to keep track of the harvesting of large collection of files. Storage of the resource can be located on a local file system or on a AWS S3 storage. The software-mention client will use the collection manifest to process these harvested documents.

  • locally:

python3 software_mention_client.py --data-path /mnt/data/biblio-glutton-harvester/data/

--data-path indicates the path to the repository of data harvested by biblio-glutton-harvester.

The resulting JSON files will be enriched by the metadata records of the processed PDF and will be stored together with each processed PDF in the data repository.

If the harvested collection is located on a S3 storage, the access information must be indicated in the configuration file of the client config.json. The extracted software mention will be written in a file with extension .software.json, for example:

-rw-rw-r-- 1 lopez lopez 1.1M Aug  8 03:26 0100a44b-6f3f-4cf7-86f9-8ef5e8401567.pdf
-rw-rw-r-- 1 lopez lopez  485 Aug  8 03:41 0100a44b-6f3f-4cf7-86f9-8ef5e8401567.software.json

If a MongoDB server access information is indicated in the configuration file config.json, the extracted information will additionally be written in MongoDB.

License and contact

Distributed under Apache 2.0 license. The dependencies used in the project are either themselves also distributed under Apache 2.0 license or distributed under a compatible license.

Main author and contact: Patrice Lopez ([email protected])

Orca is an extensive and extendable Python 3.x library for the Discord API.

Orca is an extensive and extendable Python 3.x library for the Discord API.

RPS 4 Apr 03, 2022
Free Game Download Client

XGames Free Game Download Client В проекте была использована библиотека igruha а также PyQt5 WARN ⚠️ Возможно потребуется скачать и установить vc_redi

LORD_CODE 3 Jun 25, 2022
Visualização de dados do TaxiGov.

Visualização de dados do TaxiGov Este repositório apresenta uma visualização das corridas de táxi do programa TaxiGov do governo federal, realizadas n

Ministério da Economia do Brasil 5 Dec 20, 2022
A self-hosted Discord music bot.

Cassette A self-hosted Discord music bot. Requirements py-cord pynacl pytube Setup Intended to be hosted on Heroku. Fork or clone this repo. Create a

Lohan 8 Apr 28, 2022
Async ready API wrapper for Revolt API written in Python.

Mutiny Async ready API wrapper for Revolt API written in Python. Installation Python 3.9 or higher is required To install the library, you can just ru

16 Mar 29, 2022
Hydrathallies'in istegi uzerine yapildi :)

Telegram-Doviz-Bot Telegram Döviz Botu, Pyrogram ile yapıldı. Deploy Deploy on Heroku Deploy on local git clone https://github.com/lambda-stock/Telegr

2 Dec 08, 2021
Telegram Bot to save Posts or Files that can be Accessed via Special Links

OKAERI-FILE Bot Telegram untuk menyimpan Posting atau File yang dapat Diakses melalui Link Khusus. Jika Anda memerlukan tambahan module lagi dalam rep

Wahyusaputra 5 Aug 04, 2022
WhatsAppCrashingToolv1.1 - WhatsApp Crashing Tool v1.1

WhatsAppCrashingTool v1.1 This is just for Educational Purpose WhatsApp Crashing

E4crypt3d 3 Dec 20, 2022
NoChannelBot - Bot bans users, that send messages like channels

No Channel Bot Say "STOP" to users who send messages as channels! Bot prevents u

Andrew 10 Oct 05, 2022
Python script to replace BTC adresses in the clipboard with similar looking ones, whose private key can be retrieved by a netcat listener or similar.

BTCStealer Python script to replace BTC adresses in the clipboard with similar looking ones, whose private key can be retrieved by a netcat listener o

Some Person 6 Jun 07, 2022
Most Simple & Powefull web3 Trade Bot (WINDOWS LINUX) Suport BSC ETH

Most Simple & Powefull Trade Bot (WINDOWS LINUX) What Are Some Pros And Cons Of Owning A Sniper Bot? While having a sniper bot is typically an advanta

GUI BOT 6 Jan 30, 2022
Create Multiple CF entry for multiple websites

AWS-CloudFront Problem: Deploy multiple CloudFront for account with multiple domains. Functionality: Running this script in loop and deploy CloudFront

Giten Mitra 5 Nov 18, 2022
pokemon-colorscripts compatible for mac

Pokemon colorscripts some scripts to print out images of pokemons to terminal. Inspired by DT's colorscripts compilation Description Prints out colore

43 Jan 06, 2023
Python wrapper for JeyyAPI

Async python wrapper for JeyyAPI

7 Dec 10, 2022
A PowerFull Telegram Mirror Bot.......

- [ DEAD REPO AND NO MORE UPDATE ] Slam Mirror Bot Slam Mirror Bot is a multipurpose Telegram Bot written in Python for mirroring files on the Interne

αвιנтн 2 Nov 09, 2021
A Wrapper for ScarletAPI

ScarletAPI A Wrapper for ScarletAPI still a work in progress Docs these are the

Amashi 0 Mar 24, 2022
Um bot para contar quantas vezes o meu amigo troca de pfp/nick/tag essas coisas ae pq aquele mlk n para quieto

EkiBot Um bot que tem apenas as suas funções de audit log com as PFP's (avatares) dos usuários Pode ser usado para um usuário em específico, ou até me

Samuel 3 Aug 11, 2021
A FORKED AND Modded version of TL:GD for 🅱️3R0K🧲support

for support join here working example group Leech Here For Any Issues/Imrovements or Discussions go here or here Please Leave A star And Fork this Rep

XcodersHub 165 Mar 12, 2022
Revolt account generator. Bypassing Hcaptcha using AI solver.

RevoltGenerator Revolt account generator. Bypassing Hcaptcha using AI solver. Config settings in config.json then put your usernames / proxies. If you

&! Ѵιchy.#0110 27 Nov 01, 2022
🤖 Telegram UserBot Untuk Memutar Lagu Dan Video Di Obrolan Suara Telegram.

🤖 Telegram UserBot Untuk Memutar Lagu Dan Video Di Obrolan Suara Telegram.

Fariz 2 Nov 13, 2021