API to parse tibia.com content into python objects.

Overview

Tibia.py

An API to parse Tibia.com content into object oriented data.

No fetching is done by this module, you must provide the html content.

GitHub Workflow Status codecov PyPI PyPI - Python Version PyPI - License

Features:

  • Converts data into well-structured Python objects.
  • Type consistent attributes.
  • All objects can be converted to JSON strings.
  • Can be used with any networking library.
  • Support for characters, guilds, houses and worlds, tournaments, forums, etc.

Installing

Install and update using pip

pip install tibia.py

Installing the latest version form GitHub

pip install git+https://github.com/Galarzaa90/tibia.py.git -U

Usage

This library is composed of two parts, parsers and an asynchronous request client.

The asynchronous client (tibiapy.Client) contains methods to obtain information from Tibia.com.

The parsing methods allow you to get Python objects given the html content of a page.

import tibiapy

# Asynchronously
import aiohttp

async def get_character(name):
    url = tibiapy.Character.get_url(name)

    async with aiohttp.ClientSession() as session:
        async with session.get(url) as resp:
            content = await resp.text()
    character = tibiapy.Character.from_content(content)
    return character

# Synchronously
import requests

def get_character_sync(name):
    url = tibiapy.Character.get_url(name)
    
    r = requests.get(url)
    content = r.text
    character = tibiapy.Character.from_content(content)
    return character

Documentation

https://tibiapy.readthedocs.io/

Comments
  • DisplayItems attribute name is returning None

    DisplayItems attribute name is returning None

    I was querying the items of a character at the bazaar and the DisplayItem attribute name is returning None

    Here is my code snippet:

    response : TibiaResponse = self.client.loop.run_until_complete(self.client.fetch_auction(character_auction_id, 
            fetch_items=True))
            
    details : AuctionDetails = response.data
    
    items : List[DisplayItem] = details.items.entries
    
    print(items)
    

    The print of the returned list:

    [<DisplayItem name=None count=22 item_id=238>, <DisplayItem name=None count=347 item_id=239>, <DisplayItem name=None count=11 item_id=266>]

    Yesterday everything was working fine, the problem emerged this morning. Maybe the Tibia Bazaar website changed something in their HTML indexing?

    opened by guriosam 6
  • Problem with the comma when there are more than 1000 players online.

    Problem with the comma when there are more than 1000 players online.

    Hello! Cipsoft added a comma in the world list when there are more than 1000 players online.

    File "\site-packages\tibiapy\world.py", line 267, in from_content world._parse_world_info(tables.get("World Information", [])) File "\site-packages\tibiapy\world.py", line 344, in _parse_world_info self.online_count = int(world_info.pop("players_online")) ValueError: invalid literal for int() with base 10: '1,024'

    opened by Pixelao 2
  • tibiapy.HousesSection.from_content is broken

    tibiapy.HousesSection.from_content is broken

    Problem Method tibiapy.HousesSection.from_content(link to code) throws tibiapy.errors.InvalidContent error instead of returning a HousesSection object.

    The reason this unintended behavior happens is because we try to access a Nonetype object inside HouseSection._parse_filters method when we call it house_results._parse_filters(tables["House Search"]) in line 172.

    Quick fix Commenting line 172 solves the issue for me (at least it does for my use case).

    opened by aaronga97 1
  • v2.4.0

    v2.4.0

    Changes

    • Added support for multiple houses per character. Accessible on Character.houses field.
    • Character.house is now deprecated. It will contain the character's first house or None.
    opened by Galarzaa90 1
  • v1.1.0

    v1.1.0

    Changes

    • Parsing Highscores from Tibia.com and TibiaData.
    • Some strings from TibiaData had unpredictable trailing whitespaces, all leading and trailing whitespaces are removed.
    • Added type hints to many variables and methods.
    opened by Galarzaa90 1
  • Parse character position

    Parse character position

    Character's position is not being parsed, for example, Steve has CipSoft Member position. This seems to be the only available position.

    Not to be confused with the position field in Account Information.

    Also, these characters show Cipsoft Member in their status when listed as "Other Characters".

    enhancement 
    opened by Galarzaa90 1
  • Add HouseStatus and Sex enums

    Add HouseStatus and Sex enums

    Testing the waters to maybe improve code. Those do not seem to be used in many places, so they are both good testing subjects.

    Tests kept successful so it seems to be ok. This might have some limitations with json parsing constructors, so let's discuss what you think later :)

    opened by Tschis 1
  • Refactor guild.py and unit tests

    Refactor guild.py and unit tests

    • Refactor guild.py to split different sections of the parsing method. This makes it easier to test and analyze/maintain the code in the future;
    • Add new resource and unit tests for best case scenario on each section;
    opened by Tschis 1
  • Parse character titles and main character information

    Parse character titles and main character information

    Summer update 2019 introduced character titles, as well as a small icon that indicates which is the main character of the account in the other characters section.

    opened by Galarzaa90 0
  • v1.0.0

    v1.0.0

    Changes

    • Added support for TibiaData JSON parsing. To have interoperability between Tibia.com and TibiaData.
    • Added support for parsing Houses, House lists, World and World list
    • Added support for many missing attributes in Character and Guilds.
    • All objects are now serializable to JSON strings.
    opened by Galarzaa90 0
  • Add /tests as a package

    Add /tests as a package

    As seen here, this might help running the tests from the root folder.

    Also added some changes that might facilitate changes into the current test case or new test cases to be added in the future (relative resources path).

    opened by Tschis 0
Releases(v5.5.2)
  • v5.5.2(Sep 3, 2022)

    Changes

    • Fixed Houses section not parsing due to a change in the filters table.
    • Fixed status parameter not generating the correct URL in the houses section.
    Source code(tar.gz)
    Source code(zip)
  • v5.5.1(Aug 2, 2022)

    Changes

    • Adjusted parsing to support the changes related to mobile devices introduced on the day of this release. The following sections were affected:
      • Highscores
      • News
      • Forums
      • Spells
    Source code(tar.gz)
    Source code(zip)
  • v5.5.0(Jul 27, 2022)

  • v5.4.0(Jul 23, 2022)

  • v5.3.0(Jul 23, 2022)

  • v5.2.1(Mar 1, 2022)

  • v5.2.0(Dec 31, 2021)

  • v5.1.0(Sep 16, 2021)

    Changes

    • Added traded attribute to death killers, to indicate that the killer was traded after the death occurred.
    • Properly handle deaths caused by summons of traded characters.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1.post0(Aug 26, 2021)

  • v5.0.1(Aug 26, 2021)

    Changes

    • Fixed many sections not being parsed correctly due to changes to Tibia.com.
      • Houses list
      • News list
      • Spells section
      • Forums section
    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    Source code(tar.gz)
    Source code(zip)
  • v5.0.1a1(Aug 24, 2021)

    Prelease to address part of the Tibia.com changes.

    Changes

    • Fixed character's houses failing to parse due to a bug in the display in Tibia.com.
      • Temporarily disabling this attribute.
    • Fixed house list not parsing correctly due to Tibia.com changes
    • Fixed news list not parsing correctly due to Tibia.com changes

    Pending

    • Forums section
    • Spells section
    Source code(tar.gz)
    Source code(zip)
  • v5.0.0(Aug 7, 2021)

    Changes

    • Added parsing for Tibia Drome leaderboards, new Leaderboard class.
      • Auxiliary classes LeaderboardEntry and LeaderboardRotation were added as well.
      • New Client method: fetch_leaderboards.
    • Added parsing for Spells library, new SpellsSection class.
      • Auxiliary classes Spell and SpellEntry.
      • New Client methods: fetch_spell and fetch_spells
    • Fix last page of highscores having 0 as page value.
    • Using the Client class, you can now fetch and parse content from the test version of www.tibia.com when available.
      • Note that if the test website has changes, parsing might not be possible.
      • Internal URL attributes might still point to the regular website.
    • New HousesSection class, including the house filtering attributes.
    • New NewsArchive class, including the news filtering attributes.
    • New GuildsSection class, to replace the lists of GuildEntry
    • Many "ListedObject" classes were renamed to "ObjectEntry", for details check the breaking changes below.
    • Fixed bug with other characters not being parsed.
    • Added traded attribute to LastPost class.
    • Added thread_starter_traded attribute to ThreadEntry

    Breaking Changes:

    • Python 3.7 or higher is now required.
    • House classes no longer have get_list_url and list_from_content methods.
    • Client.fetch_world_houses now returns a HousesSection instance in its data attribute, instead of a list of ListedHouses.
    • ListedHouse.highest_bid attribute now may be None if the house's auction has not yet started.
    • ListedHouse class renamed to HouseEntry.
    • Removed deprecated property AuctionFilters.item.
    • Client.fetch_news_archive, Client.fetch_recent_news now returns an instance of NewsArchive in the data attribute.
    • ListedNews class renamed to NewsEntry.
    • News and NewsEntry no longer have a get_list_url method.
    • ListedBoard class renamed to BoardEntry.
    • ListedThread class renamed to ThreadEntry.
    • ListedAnnouncement class renamed to AnnouncementEntry.
    • ListedWorld class renamed to WorldEntry.
    • ListedAuction class renamed to AuctionEntry.
    • AuctionDetails class renamed to Auction.
    • ListedGuild class renamed to GuildEntry.
    • ListedTournament class renamed to TournamentEntry.
    • Creature class renamed to CreatureEntry.
    • CreatureDetail class renamed to Creature.
    • Guild and GuildEntry class no longer have a get_list_url method.
    • Renamed begin_date parameter to start_date in fetch_news_archive.
    • Renamed race attribute of CreatureEntry and Creature to identifier, method parameters renamed as well.
    • CreaturesSection.from_boosted_creature_header renamed to CreaturesSection.boosted_creature_from_header.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.7(Jun 30, 2021)

  • v4.1.6(Jun 28, 2021)

    Changes

    • Fixed worlds not being parsed correctly again due to tournament worlds order changing. After this fix, the order should not matter anymore.
    Source code(tar.gz)
    Source code(zip)
  • v4.1.5(Jun 26, 2021)

  • v4.1.4(Jun 17, 2021)

  • v4.1.3(May 12, 2021)

  • v4.1.2(Apr 27, 2021)

  • v4.1.1(Apr 19, 2021)

  • v4.1.0(Mar 30, 2021)

    Changes

    • Added prey_wildcards attribute to AuctionDetails.
    • Added filters parameter to CharacterBazaar.get_auctions_history_url and Client.fetch_auction_history.
    Source code(tar.gz)
    Source code(zip)
  • v4.0.0(Mar 11, 2021)

    Changes

    • Breaking change: Removed BoostedCreature class, replaced by Creature class.
      • Attributes should be compatible, image_url is a property of Creature, calculated from its race attribute.
    • Added parsing and fetching for the Creature library section.
      • Added CreatureSection, Creature, and CreatureDetail classes.
    • Added traded attribute to ForumAuthor. Indicates if the author was a traded character.
      • Previously, it would mark the author as a deleted character and its name would include (traded).
    • Fixed a bug with ForumBoards not parsing due to the cookies dialog that was added.
    • Added battleye_type attribute to ListedWorld and World classes. Indicates the type of BattlEye protection the world has.
      • battleye_protected is now a property instead of an attribute.
    • Added YELLOW and GREEN aliases to all BattlEye related enums.
    • Fixed wrong timezone being used for forum related dates.
    Source code(tar.gz)
    Source code(zip)
  • v3.7.1(Feb 16, 2021)

  • v.3.7.0(Feb 9, 2021)

    Changes

    • Parse familiars from auctions
    • Updated the way tooltips in auctions are parsed, the format changed, resulting in the previous code not working anymore.
    • Results count in bazaar pages are now properly parsed when there are comma thousand separators.
    • Item amounts are now more accurate instead of being based from their indicator (which was grouping them in thousands)
    Source code(tar.gz)
    Source code(zip)
  • v3.6.5(Jan 28, 2021)

  • v3.6.4(Jan 26, 2021)

  • v3.6.3(Jan 15, 2021)

  • v3.6.2(Jan 2, 2021)

  • v3.6.1(Dec 28, 2020)

  • v3.6.0(Dec 12, 2020)

    Changes

    • Added support for the new filtering options in Highscores
      • Added battleye_filter and pvp_types_filter attributes.
    • Added get_page_url() instance method to Highscores class.
    • Added previous_page_url and next_page_url properties.
    Source code(tar.gz)
    Source code(zip)
  • v3.5.7(Dec 4, 2020)

Owner
Allan Galarza
Mechatronic Engineer
Allan Galarza
An helper library to scrape data from Instagram effortlessly, using the Influencer Hunters APIs.

Instagram Scraper An utility library to scrape data from Instagram hassle-free Go to the website » View Demo · Report Bug · Request Feature About The

2 Jul 06, 2022
Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages.

Video Games Web Scraper Video Games Web Scraper is a project that crawls websites and APIs and extracts video game related data from their pages. This

Albert Marrero 1 Jan 12, 2022
Crawl BookCorpus

These are scripts to reproduce BookCorpus by yourself.

Sosuke Kobayashi 590 Jan 03, 2023
Subscrape - A Python scraper for substrate chains

subscrape A Python scraper for substrate chains that uses Subscan. Usage copy co

ChaosDAO 14 Dec 15, 2022
Bulk download tool for the MyMedia platform

MyMedia Bulk Content Downloader This is a bulk download tool for the MyMedia platform. USE ONLY WHERE ALLOWED BY THE COPYRIGHT OWNER. NOT AFFILIATED W

Ege Feyzioglu 3 Oct 14, 2022
A web crawler script that crawls the target website and lists its links

A web crawler script that crawls the target website and lists its links || A web crawler script that lists links by scanning the target website.

2 Apr 29, 2022
Dailyiptvlist.com Scraper With Python

Dailyiptvlist.com scraper Info Made in python Linux only script Script requires to have wget installed Running script Clone repository with: git clone

1 Oct 16, 2021
京东茅台抢购

截止 2021/2/1 日,该项目已无法使用! 京东:约满即止,仅限京东实名认证用户APP端抢购,2月1日10:00开始预约,2月1日12:00开始抢购(京东APP需升级至8.5.6版本及以上) 写在前面 本项目来自 huanghyw - jd_seckill,作者的项目地址我找不到了,找到了再贴上

abee 73 Dec 03, 2022
Dictionary - Application focused on word search through web scraping

Dictionary - Application focused on word search through web scraping, in addition to other functions such as dictation, spell and conjugation of syllables.

Juan Manuel 2 May 09, 2022
A simple flask application to scrape gogoanime website.

gogoanime-api-flask A simple flask application to scrape gogoanime website. Used for demo and learning purposes only. How to use the API The base api

1 Oct 29, 2021
Github scraper app is used to scrape data for a specific user profile created using streamlit and BeautifulSoup python packages

Github Scraper Github scraper app is used to scrape data for a specific user profile. Github scraper app gets a github profile name and check whether

Siva Prakash 6 Apr 05, 2022
A simple reddit scraper to get memes (only images) from r/ProgrammerHumor.

memey A simple reddit scraper to get memes (only images) from r/ProgrammerHumor. Note Only works if you have firefox installed (yet). Instructions foo

2 Nov 16, 2021
Pelican plugin that adds site search capability

Search: A Plugin for Pelican This plugin generates an index for searching content on a Pelican-powered site. Why would you want this? Static sites are

22 Nov 21, 2022
抖音批量下载用户所有无水印视频

Douyincrawler 抖音批量下载用户所有无水印视频 Run 安装python3, 安装依赖

28 Dec 08, 2022
Scraping and visualising India's real-time COVID-19 data from the MOHFW dataset.

COVID19-WEB-SCRAPER Open Source Tech Lab - Project [SEMESTER IV] OSTL Assignments OSTL Assignments - 1 OSTL Assignments - 2 Project COVID19 India Data

AMEY THAKUR 8 Apr 28, 2022
基于Github Action的定时HITsz疫情上报脚本,开箱即用

HITsz Daily Report 基于 GitHub Actions 的「HITsz 疫情系统」访问入口 定时自动上报脚本,开箱即用。 感谢 @JellyBeanXiewh 提供原始脚本和 idea。 感谢 @bugstop 对脚本进行重构并新增 Easy Connect 校内代理访问。

Ter 56 Nov 27, 2022
A web scraper that exports your entire WhatsApp chat history.

WhatSoup 🍲 A web scraper that exports your entire WhatsApp chat history. Table of Contents Overview Demo Prerequisites Instructions Frequen

Eddy Harrington 87 Jan 06, 2023
Introduction to WebScraping Workshop - Semcomp 24 Beta

Extrair informações da internet de forma automatizada. Existem diversas maneiras de fazer isso, nesse tutorial vamos ver algumas delas, por meio de bibliotecas de python.

Luísa Moura 19 Sep 11, 2022
Web Scraping images using Selenium and Python

Web Scraping images using Selenium and Python A propos de ce document This is a markdown document about Web scraping images and videos using Selenium

Nafaa BOUGRAINE 3 Jul 01, 2022
The open-source web scrapers that feed the Los Angeles Times California coronavirus tracker.

The open-source web scrapers that feed the Los Angeles Times' California coronavirus tracker. Processed data ready for analysis is available at datade

Los Angeles Times Data and Graphics Department 51 Dec 14, 2022