rapidnlp-datasets
Data pipelines for both TensorFlow and PyTorch !
If you want to load public datasets, try:
If you want to load local, personal dataset with minimized boilerplate, use rapidnlp-datasets!
installation
pip install -U rapidnlp-datasets
If you work with PyTorch, you should install PyTorch first.
If you work with TensorFlow, you should install TensorFlow first.
Usage
Here are few examples to show you how to use this library.
- QuickStart: Sequence Classification Task
- QuickStart: Question Answering Task
- QuickStart: Token Classification Task
- QuickStart: Masked Language Model Task
- QuickStart: SimCSE(Sentence Embedding)
sequence-classification-quickstart
In PyTorch,
>>> import torch
>>> from rapidnlp_datasets.pt import DatasetForSequenceClassification
>>> dataset = DatasetForSequenceClassification.from_jsonl_files(
input_files=["testdata/sequence_classification.jsonl"],
vocab_file="testdata/vocab.txt",
)
>>> dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=32, collate_fn=dataset.batch_padding_collate)
>>> for idx, batch in enumerate(dataloader):
... print("No.{} batch: \n{}".format(idx, batch))
...
In TensorFlow,
>>> from rapidnlp_datasets.tf import TFDatasetForSequenceClassifiation
>>> dataset, d = TFDatasetForSequenceClassifiation.from_jsonl_files(
input_files=["testdata/sequence_classification.jsonl"],
vocab_file="testdata/vocab.txt",
return_self=True,
)
>>> for idx, batch in enumerate(iter(dataset)):
... print("No.{} batch: \n{}".format(idx, batch))
...
Especially, you can save dataset to tfrecord format when working with TensorFlow, and then build dataset from tfrecord files directly!
>>> d.save_tfrecord("testdata/sequence_classification.tfrecord")
2021-12-08 14:52:41,295 INFO utils.py 128] Finished to write 2 examples to tfrecords.
>>> dataset = TFDatasetForSequenceClassifiation.from_tfrecord_files("testdata/sequence_classification.tfrecord")
>>> for idx, batch in enumerate(iter(dataset)):
... print("No.{} batch: \n{}".format(idx, batch))
...
question-answering-quickstart
In PyTorch:
>>> import torch
>>> from rapidnlp_datasets.pt import DatasetForQuestionAnswering
>>>
>>> dataset = DatasetForQuestionAnswering.from_jsonl_files(
input_files="testdata/qa.jsonl",
vocab_file="testdata/vocab.txt",
)
>>> dataloader = torch.utils.data.DataLoader(dataset, shuffle=True, batch_size=32, collate_fn=dataset.batch_padding_collate)
>>> for idx, batch in enumerate(dataloader):
... print("No.{} batch: \n{}".format(idx, batch))
...
In TensorFlow,
>>> from rapidnlp_datasets.tf import TFDatasetForQuestionAnswering
>>> dataset, d = TFDatasetForQuestionAnswering.from_jsonl_files(
input_files="testdata/qa.jsonl",
vocab_file="testdata/vocab.txt",
return_self=True,
)
2021-12-08 15:09:06,747 INFO question_answering_dataset.py 101] Read 3 examples in total.
>>> for idx, batch in enumerate(iter(dataset)):
print()
print("NO.{} batch: \n{}".format(idx, batch))
...
Especially, you can save dataset to tfrecord format when working with TensorFlow, and then build dataset from tfrecord files directly!
>>> d.save_tfrecord("testdata/qa.tfrecord")
2021-12-08 15:09:31,329 INFO utils.py 128] Finished to write 3 examples to tfrecords.
>>> dataset = TFDatasetForQuestionAnswering.from_tfrecord_files(
"testdata/qa.tfrecord",
batch_size=32,
padding="batch",
)
>>> for idx, batch in enumerate(iter(dataset)):
print()
print("NO.{} batch: \n{}".format(idx, batch))
...
3 Nov 23, 2022
8.9k Dec 30, 2022
30 Oct 9, 2022
96 Dec 23, 2022
68 Nov 24, 2022
704 Dec 27, 2022
341 Dec 29, 2022
1 Mar 5, 2022
53 Nov 22, 2022
3.7k Jan 03, 2023
90 Dec 22, 2022
7 Mar 07, 2022
194 Dec 22, 2022
44 Dec 30, 2022
1 Dec 25, 2021
77 Dec 02, 2022
676 Jan 03, 2023
274 Dec 06, 2022
30 Dec 22, 2022
1 Nov 15, 2021
610 Dec 28, 2022
28 Dec 28, 2022
22 Aug 27, 2022
15 Nov 04, 2022
7 Sep 26, 2022
532 Jan 02, 2023
62 Jan 02, 2023
8 Dec 04, 2022
24 Dec 23, 2022