类库
› DataFlow
OpenDCAI/DataFlow
DataFlow是一个基于Python的数据准备工具，利用最新的大语言模型算子和管道生成、清洗和预处理LLM训练数据。它提供一键式解决方案，简化高质量指令数据的构建流程，支持通过PyPI安装及Docker部署，旨在提升大模型数据工程效率。
标签
技术栈
dataflow/statics/scaffold/{{cookiecutter.repo_name}} python
查看全部依赖 (1)
依赖
open-dataflow
根目录 python
框架
FastAPI
测试
pytest
网络
Requests
查看全部依赖 (67)
依赖
NumPy
Pandas
Pydantic
accelerate
addict
appdirs
chonkie
cloudpickle
colorama
colorlog
contractions
cookiecutter
datasets
datasketch
db-dtypes
distflow
docstring_parser
evaluate
fasttext
fasttext-wheel
func_timeout
gcsfs
google-api-core
google-api-python-client
google-cloud-aiplatform
google-cloud-bigquery
google-cloud-bigquery-storage
google-genai
gradio
httpx
json5
langkit
librosa
lxml_html_clean
math_verify
modelscope
nltk
openai
presidio_analyzer
presidio_anonymizer
psutil
pyarrow
pyfiglet
pymupdf
pymysql
pypdf
pyyaml
rapidfuzz
rich
scipy
sentencepiece
simhash
soundfile
sqlglot
sseclient-py
symspellpy
termcolor
tiktoken
torch
torchaudio
torchvision
tqdm
trafilatura
transformers
uvicorn
vendi-score
word2number
