Pyspark sam - Analyze Big Sequence Alignments with PySpark in AWS EMR

Related tags

Third-party APIs Wrapperspyspark_sam
Overview

pyspark_sam

This repo hosts my code for the article "Analyze Big Sequence Alignments with PySpark in AWS EMR".

Prerequisite

  1. Spark

  2. AWS CLI

  3. AWS Account

Run

Follow the instruction in the article. Once you have uploaded the files into your S3 bucket, run

aws emr create-cluster --name "Spark_step_pip" \
    --release-label emr-6.5.0 \
    --applications Name=Spark \
    --log-uri s3://[your_S3_bucket]/logs/ \
    --instance-type m5.xlarge \
    --instance-count 3 \
    --bootstrap-actions Path=s3://[your_S3_bucket]/emr_bootstrap.sh \
    --use-default-roles --auto-terminate \
    --steps "Type=Spark,Name=SparkProgram,ActionOnFailure=CONTINUE,Args=[--deploy-mode,cluster,--master,yarn,--py-files,s3://[your_S3_bucket]/helper_function.py,s3://[your_S3_bucket]/spark_3mer.py,s3://[your_S3_bucket]/test.sam,[your_S3_bucket],sankey.json]"

When the job finishes, download the sankey.json. And run this command to visualize:

python sankey.py sankey.json

Authors

  • Sixing Huang - Concept and Coding

License

This project is licensed under the MIT License - see the LICENSE file for details

Owner
Sixing Huang
A triple Neo4j certified data scientist. I am currently working at BGI in Shenzhen.
Sixing Huang
GitHub Repository
This Wrapper is a Discum Copy With Addons, original one is made by Merubokkusu

Remaded Discum Its not Official Discum Wrapper ! This Wrapper is a Discum Copy With Addons, original one is made by Merubokkusu Authors @merubokkusu (

discum-remaded 8 Aug 09, 2022
NiceHash Python Library and Command Line Rest API

NiceHash Python Library and Command Line Rest API Requirements / Modules pip install requests Required data and where to get it Following data is nee

Ashlin Darius Govindasamy 2 Jan 02, 2022
Passive income method via SerpClix. Uses a bot to accept clicks.

SerpClixBotSearcher This bot allows you to get passive income from SerpClix. Each click is usually $0.10 (sometimes $0.05 if offer isnt from the US).

Jason Mei 3 Sep 01, 2021
StringSessionGenerator - A Telegram bot to generate pyrogram and telethon string session

⭐️ String Session Generator ⭐️ Genrate String Session Using this bot. Made by TeamUltronX 🔥 String Session Demo Bot: Environment Variables Mandatory

TheUltronX 1 Dec 31, 2021
Python wrapper for Coinex APIs

coinexpy - Python wrapper for Coinex APIs Through coinexpy you can simply buy or sell crypto in your Coinex account Features place limit order place m

Iman Mousaei 16 Jan 02, 2023
Tamil Voicechat UserBot. Powerd By TamilBots. Https://T.me/TamilSupport

Tamil Voicechat UserBot A Telegram UserBot to Play music 🎶 in Voice Chats. It's recommended to use an USA number.(if your real number is suspended I'

Tamil Bots 78 Nov 01, 2022
Python3 script to dump employee information from XING API

XingDumper Python 3 script to dump company employees from XING API. Perfect OSINT tool ;-) The results contain firstname, lastname, position, gender,

LRVT 11 Dec 26, 2022
Group Management Bot

❤️ 𝗦𝗛𝗔𝗗𝗜𝗬𝗢 ❤️ A Powerful, Smart And Advance Group Manager ... Written with AioGram , Pyrogram and Telethon... ⭐️ Thanks to everyone who starred

Abdisamad Omar Mohamed 4 Dec 01, 2021
Unofficial Python API client for Notion.so

notion-py Unofficial Python 3 client for Notion.so API v3. Object-oriented interface (mapping database tables to Python classes/attributes) Automatic

Jamie Alexandre 3.9k Jan 03, 2023
♻️ API to run evaluations of the FAIR principles (Findable, Accessible, Interoperable, Reusable) on online resources

♻️ FAIR enough 🎯 An OpenAPI where anyone can run evaluations to assess how compliant to the FAIR principles is a resource, given the resource identif

Maastricht University IDS 4 Oct 20, 2022
A delivery protection and notification system

DeliveryProtect This project builds a delivery protection and notification system, based on integration of Arduino Uno and Raspberry Pi 4. The codes a

2 Dec 13, 2021
PTV is a useful widget for trading view for doing paper trading when bar reply is enabled

PTV is a useful widget for trading view for doing paper trading when bar reply is enabled.(this feature did not implement in trading view)

Ali Moradi 39 Dec 26, 2022
An API wrapper around the pythonanywhere's API.

pyaww An API wrapper around the pythonanywhere's API. The name stands for pythonanywherewrapper. 100% api coverage most of the codebase is documented

7 Dec 11, 2022
an API to check if a url or IP address is safe or phishing

an API to check if a url or IP address is safe or phishing. Using a ML model. The API created using FastAPI.

Adel Dahani 1 Feb 16, 2022
A modern,feature-rich, and async ready API wrapper for Discord written in Python

discord.io A modern, easy to use, feature-rich, and async ready API wrapper for Discord written in Python. Key Features Modern Pythonic API using asyn

Vincent 18 Jan 02, 2023
An API which returns random AOT quote everytime it's invoked

An API which returns random AOT quote everytime it's invoked

Nishant Sapkota 1 Feb 07, 2022
Defi PancakeSwap bot is programmed in Python to buy and sell tokens in seconds once the target is hit.

Defi PancakeSwap BOT A BOT that will make easy your life in Trading. Watch tutorial on Youtube Table of Contents About The Project Built With Getting

Zain Ullah 208 Jan 05, 2023
The Easy-to-use Dialogue Response Selection Toolkit for Researchers

Easy-to-use toolkit for retrieval-based Chatbot Our released data can be found at this link. Make sure the following steps are adopted to use our code

GMFTBY 32 Nov 13, 2022
VoiceMaster-Discord-Bot - Fork from original Discord bot with max channel limit, staff role and more

VoiceMaster VoiceMaster is a discord bot created to change the way servers work,

2 Feb 28, 2022
A fully responsive interface to manage all your favorite software on your HTPC.

Python 3 port of Hellowlol's HTPC Manager fork We made this an organization repository to be more independent from single developers. If you want to j

26 Jan 04, 2023