About Solve CTF offline disconnection problem - based on python3's small crawler

Last update: Oct 25, 2022

Related tags

Machine Learning safe_message_spider

Overview

前言

某次CTF线下比赛断网,想着搞点资料,而后参考了书鱼大佬的代码,搞了一套获取资料的小工具。

在原作者的基础上补充了安全客,freebuf,seebug完善了先知的部分代码

原作者项目地址 https://github.com/bkfish/html2markdown_Spider

参数说明

-s 需要查找的关键字，空代表按时间顺序爬取
-c 需要文章的数目，默认30
-p 是否需要建立本地图床，默认0 0代表不需要 1代表需要

安装使用

安装如下

git clone https://github.com/qiannianshuangxue/safe_message_spider
cd xianzhiSpider
pip install -r requirements.txt

先知

使用前注意设置headers的cookie

python3 xianzhiSpider.py -s CTF -c 5 -p 1

seebug

无需使用其他参数

python3 seebug_Spider.py

freebuf

无需使用其他参数

python3 freebufSpider.py

anquanke

无需使用其他参数

python3 anquankeSpider.py

查看下载的文档

由于我个人习惯使用VSCODE来查看以及搜索关键词,因此并没有开发一套查看工具。

结果

包括图床一共也只有26个G左右

效果图如下

仅供学习，禁止商用行为

Owner

天河

GitHub Repository

Simple structured learning framework for python

PyStruct PyStruct aims at being an easy-to-use structured learning and prediction library. Currently it implements only max-margin methods and a perce

666 Jan 03, 2023

The easy way to combine mlflow, hydra and optuna into one machine learning pipeline.

mlflow_hydra_optuna_the_easy_way The easy way to combine mlflow, hydra and optuna into one machine learning pipeline. Objective TODO Usage 1. build do

9 Sep 09, 2022

Forecast dynamically at scale with this unique package. pip install scalecast

🌄 Scalecast: Dynamic Forecasting at Scale About This package uses a scaleable forecasting approach in Python with common scikit-learn and statsmodels

158 Jan 03, 2023

Short PhD seminar on Machine Learning Security (Adversarial Machine Learning)

Short PhD seminar on Machine Learning Security (Adversarial Machine Learning)

141 Dec 27, 2022

ParaMonte is a serial/parallel library of Monte Carlo routines for sampling mathematical objective functions of arbitrary-dimensions

ParaMonte is a serial/parallel library of Monte Carlo routines for sampling mathematical objective functions of arbitrary-dimensions, in particular, the posterior distributions of Bayesian models in

182 Dec 31, 2022

Anomaly Detection and Correlation library

luminol Overview Luminol is a light weight python library for time series data analysis. The two major functionalities it supports are anomaly detecti

1.1k Jan 01, 2023

To design and implement the Identification of Iris Flower species using machine learning using Python and the tool Scikit-Learn.

To design and implement the Identification of Iris Flower species using machine learning using Python and the tool Scikit-Learn.

1 Jan 11, 2022

Real-time stream processing for python

Streamz Streamz helps you build pipelines to manage continuous streams of data. It is simple to use in simple cases, but also supports complex pipelin

1.1k Dec 28, 2022

A library to generate synthetic time series data by easy-to-use factors and generator

timeseries-generator This repository consists of a python packages that generates synthetic time series dataset in a generic way (under /timeseries_ge

87 Dec 20, 2022

Dive into Machine Learning

Dive into Machine Learning Hi there! You might find this guide helpful if: You know Python or you're learning it 🐍 You're new to Machine Learning You

11.1k Jan 03, 2023

🌊 River is a Python library for online machine learning.

River is a Python library for online machine learning. It is the result of a merger between creme and scikit-multiflow. River's ambition is to be the go-to library for doing machine learning on strea

4k Jan 03, 2023

Toolkit for building machine learning models that generalize to unseen domains and are robust to privacy and other attacks.

Toolkit for Building Robust ML models that generalize to unseen domains (RobustDG) Divyat Mahajan, Shruti Tople, Amit Sharma Privacy & Causal Learning

149 Jan 06, 2023

QML: A Python Toolkit for Quantum Machine Learning

QML is a Python2/3-compatible toolkit for representation learning of properties of molecules and solids.

176 Dec 09, 2022

Katana project is a template for ASAP 🚀 ML application deployment

Katana project is a FastAPI template for ASAP 🚀 ML API deployment

100 Dec 26, 2022

TensorFlow Decision Forests (TF-DF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models.

TensorFlow Decision Forests (TF-DF) is a collection of state-of-the-art algorithms for the training, serving and interpretation of Decision Forest models. The library is a collection of Keras models

538 Jan 01, 2023

Greykite: A flexible, intuitive and fast forecasting library

The Greykite library provides flexible, intuitive and fast forecasts through its flagship algorithm, Silverkite.

1.7k Jan 04, 2023

pywFM is a Python wrapper for Steffen Rendle's factorization machines library libFM

pywFM pywFM is a Python wrapper for Steffen Rendle's libFM. libFM is a Factorization Machine library: Factorization machines (FM) are a generic approa

251 Sep 23, 2022

K-means clustering is a method used for clustering analysis, especially in data mining and statistics.

K Means Algorithm What is K Means This algorithm is an iterative algorithm that partitions the dataset according to their features into K number of pr

1 Nov 01, 2021

Generate music from midi files using BPE and markov model

Generate music from midi files using BPE and markov model

37 Oct 24, 2022

whylogs: A Data and Machine Learning Logging Standard

whylogs: A Data and Machine Learning Logging Standard whylogs is an open source standard for data and ML logging whylogs logging agent is the easiest

2k Jan 06, 2023