类库 › DataflowTemplates
GoogleCloudPlatform

GoogleCloudPlatform/DataflowTemplates

Google Cloud Dataflow 提供的预置模板库,用于解决云上大数据处理任务(如数据导入/导出/备份/恢复、批量API操作),无需开发环境。基于 Apache Beam SDK,支持快速部署和定制。

1,289 1,078 1,289 99
在 GitHub 上查看

技术栈

ORM

SQLAlchemy

测试

pytest

网络

Requests
查看全部依赖 (461)

依赖

NumPy Pandas Pydantic aiofiles aiohappyeyeballs aiohttp aiosignal annotated-types anyio apache-beam asn1crypto attrs backports-tarfile beartype betterproto cachetools certifi cffi ch.qos.logback:logback-classic ch.qos.logback:logback-core charset-normalizer cloud-sql-python-connector com.amazonaws:aws-java-sdk-bom com.amazonaws:aws-java-sdk-cloudwatch com.amazonaws:aws-java-sdk-core com.amazonaws:aws-java-sdk-kinesis com.amazonaws:aws-java-sdk-kms com.amazonaws:aws-java-sdk-s3 com.amazonaws:aws-java-sdk-sns com.amazonaws:aws-java-sdk-sqs com.amazonaws:aws-java-sdk-sts com.azure:azure-messaging-eventhubs com.codahale.metrics:metrics-core com.datastax.astra:astra-sdk-devops com.datastax.astra:beam-sdks-java-io-astra com.datastax.cassandra:cassandra-driver-core com.datastax.oss:java-driver-query-builder com.fasterxml.jackson.dataformat:jackson-dataformat-cbor com.fasterxml.jackson.module:jackson-module-jaxb-annotations com.fasterxml.jackson:jackson-bom com.github.ben-manes.caffeine:caffeine com.github.fppt:jedis-mock com.github.jsqlparser:jsqlparser com.github.nosan:embedded-cassandra com.github.spotbugs:spotbugs-annotations com.github.stefanbirkner:system-rules com.github.vincentrussell:json-data-generator com.google.api-ads:google-ads com.google.api-client:google-api-client com.google.api.grpc:proto-google-cloud-datastream-v1 com.google.api.grpc:proto-google-cloud-firestore-admin-v1 com.google.api.grpc:proto-google-cloud-firestore-v1 com.google.api.grpc:proto-google-cloud-kms-v1 com.google.api:gax-grpc com.google.apis:google-api-services-dataflow com.google.apis:google-api-services-dataplex com.google.apis:google-api-services-datastream com.google.auto.service:auto-service com.google.auto.service:auto-service-annotations com.google.auto.value:auto-value com.google.auto.value:auto-value-annotations com.google.cloud.bigtable:bigtable-beam-import com.google.cloud.bigtable:bigtable-hbase-beam com.google.cloud.dataflow.cdc:cdc-common com.google.cloud.dataflow.cdc:cdc-embedded-connector com.google.cloud.hosted.kafka:managed-kafka-auth-login-handler com.google.cloud.sql:mysql-socket-factory-connector-j-8 com.google.cloud.sql:postgres-socket-factory com.google.cloud.teleport.metadata:metadata com.google.cloud.teleport.v2:bigtable-common com.google.cloud.teleport.v2:clickhouse-common com.google.cloud.teleport.v2:common com.google.cloud.teleport.v2:datastream-common com.google.cloud.teleport.v2:datastream-to-spanner com.google.cloud.teleport.v2:elasticsearch-common com.google.cloud.teleport.v2:failure-injected-spanner-service com.google.cloud.teleport.v2:failure-injection-policies com.google.cloud.teleport.v2:jdbc-and-googlecloud com.google.cloud.teleport.v2:jdbc-common com.google.cloud.teleport.v2:kafka-common com.google.cloud.teleport.v2:real-spanner-service com.google.cloud.teleport.v2:spanner-common com.google.cloud.teleport.v2:spanner-custom-shard com.google.cloud.teleport.v2:spanner-migrations-sdk com.google.cloud.teleport:core-plugin com.google.cloud.teleport:it-cassandra com.google.cloud.teleport:it-conditions com.google.cloud.teleport:it-datadog com.google.cloud.teleport:it-google-cloud-platform com.google.cloud.teleport:it-iceberg com.google.cloud.teleport:it-jdbc com.google.cloud.teleport:it-splunk com.google.cloud.teleport:plaintext-logging com.google.cloud.teleport:structured-logging com.google.cloud:google-cloud-aiplatform com.google.cloud:google-cloud-bigquery com.google.cloud:google-cloud-bigtable com.google.cloud:google-cloud-compute com.google.cloud:google-cloud-core com.google.cloud:google-cloud-datacatalog com.google.cloud:google-cloud-dataflow com.google.cloud:google-cloud-datastore com.google.cloud:google-cloud-datastream com.google.cloud:google-cloud-dlp com.google.cloud:google-cloud-firestore com.google.cloud:google-cloud-firestore-admin com.google.cloud:google-cloud-kms com.google.cloud:google-cloud-logging com.google.cloud:google-cloud-monitoring com.google.cloud:google-cloud-nio com.google.cloud:google-cloud-pubsub com.google.cloud:google-cloud-secretmanager com.google.cloud:google-cloud-spanner com.google.cloud:google-cloud-storage com.google.code.gson:gson com.google.guava:guava com.google.http-client:google-http-client com.google.http-client:google-http-client-apache-v2 com.google.protobuf:protobuf-java com.google.re2j:re2j com.google.truth.extensions:truth-proto-extension com.google.truth:truth com.hivemq:hivemq-mqtt-client com.microsoft.sqlserver:mssql-jdbc com.oracle.database.jdbc:ojdbc8 com.puppycrawl.tools:checkstyle com.squareup.okio:okio com.zaxxer:HikariCP commons-beanutils:commons-beanutils commons-codec:commons-codec commons-io:commons-io cryptography dev.failsafe:failsafe distro dnsjava:dnsjava dnspython docker docstring-parser envoy-data-plane execnet fastavro fasteners filelock freezegun frozenlist fsspec github.com/google/go-cmp v0.7.0 github.com/hashicorp/go-version v1.8.0 github.com/hashicorp/hc-install v0.9.3 github.com/hashicorp/hcl/v2 v2.24.0 github.com/hashicorp/terraform-exec v0.25.0 github.com/hashicorp/terraform-json v0.27.2 github.com/spf13/cobra v1.10.2 github.com/zclconf/go-cty v1.18.0 google-api-core google-apitools google-auth google-auth-httplib2 google-cloud-aiplatform google-cloud-bigquery google-cloud-bigquery-storage google-cloud-bigtable google-cloud-core google-cloud-datastore google-cloud-datastream google-cloud-dlp google-cloud-kms google-cloud-language google-cloud-monitoring google-cloud-pubsub google-cloud-pubsublite google-cloud-recommendations-ai google-cloud-resource-manager google-cloud-secret-manager google-cloud-spanner google-cloud-storage google-cloud-videointelligence google-cloud-vision google-crc32c google-genai google-resumable-media googleapis-common-protos greenlet grpc-google-iam-v1 grpc-interceptor grpcio grpcio-status grpclib h11 h2 hf-xet hpack httpcore httplib2 httpx huggingface-hub hyperframe hypothesis idna importlib-metadata iniconfig io.airlift:aircompressor io.debezium:debezium-connector-mysql io.debezium:debezium-connector-postgres io.debezium:debezium-connector-sqlserver io.debezium:debezium-embedded io.netty:netty-bom io.opencensus:opencensus-api jaraco-classes jaraco-context jaraco-functools jeepney jinja2 joblib js2py jsonpickle jsonschema jsonschema-specifications junit:junit keyring keyrings-google-artifactregistry-auth markupsafe milvus-lite mmh3 mock more-itertools mpmath multidict mysql:mysql-connector-java networkx nvidia-cublas-cu12 nvidia-cuda-cupti-cu12 nvidia-cuda-nvrtc-cu12 nvidia-cuda-runtime-cu12 nvidia-cudnn-cu12 nvidia-cufft-cu12 nvidia-curand-cu12 nvidia-cusolver-cu12 nvidia-cusparse-cu12 nvidia-nccl-cu12 nvidia-nvjitlink-cu12 nvidia-nvtx-cu12 oauth2client objsize opentelemetry-api opentelemetry-resourcedetector-gcp opentelemetry-sdk opentelemetry-semantic-conventions oracledb org.apache.activemq:activemq-client org.apache.avro:avro org.apache.beam:beam-it-cassandra org.apache.beam:beam-it-clickhouse org.apache.beam:beam-it-common org.apache.beam:beam-it-conditions org.apache.beam:beam-it-elasticsearch org.apache.beam:beam-it-jdbc org.apache.beam:beam-it-kafka org.apache.beam:beam-it-mongodb org.apache.beam:beam-it-neo4j org.apache.beam:beam-it-splunk org.apache.beam:beam-it-testcontainers org.apache.beam:beam-it-truthmatchers org.apache.beam:beam-runners-direct-java org.apache.beam:beam-runners-google-cloud-dataflow-java org.apache.beam:beam-sdks-java-bom org.apache.beam:beam-sdks-java-core org.apache.beam:beam-sdks-java-extensions-avro org.apache.beam:beam-sdks-java-extensions-google-cloud-platform-core org.apache.beam:beam-sdks-java-extensions-protobuf org.apache.beam:beam-sdks-java-extensions-python org.apache.beam:beam-sdks-java-extensions-sql org.apache.beam:beam-sdks-java-google-cloud-platform-bom org.apache.beam:beam-sdks-java-io-amazon-web-services2 org.apache.beam:beam-sdks-java-io-cassandra org.apache.beam:beam-sdks-java-io-clickhouse org.apache.beam:beam-sdks-java-io-common org.apache.beam:beam-sdks-java-io-elasticsearch org.apache.beam:beam-sdks-java-io-google-ads org.apache.beam:beam-sdks-java-io-google-cloud-platform org.apache.beam:beam-sdks-java-io-hadoop-common org.apache.beam:beam-sdks-java-io-hbase org.apache.beam:beam-sdks-java-io-jdbc org.apache.beam:beam-sdks-java-io-jms org.apache.beam:beam-sdks-java-io-kafka org.apache.beam:beam-sdks-java-io-kudu org.apache.beam:beam-sdks-java-io-mongodb org.apache.beam:beam-sdks-java-io-mqtt org.apache.beam:beam-sdks-java-io-neo4j org.apache.beam:beam-sdks-java-io-parquet org.apache.beam:beam-sdks-java-io-redis org.apache.beam:beam-sdks-java-io-splunk org.apache.beam:beam-sdks-java-io-synthetic org.apache.beam:beam-sdks-java-managed org.apache.beam:beam-sdks-java-test-utils org.apache.cassandra:java-driver-core org.apache.commons:commons-collections4 org.apache.commons:commons-compress org.apache.commons:commons-configuration2 org.apache.commons:commons-csv org.apache.commons:commons-dbcp2 org.apache.commons:commons-lang3 org.apache.commons:commons-math3 org.apache.commons:commons-pool2 org.apache.commons:commons-text org.apache.derby:derby org.apache.derby:derbyclient org.apache.derby:derbynet org.apache.hadoop:hadoop-client org.apache.hadoop:hadoop-client-runtime org.apache.hadoop:hadoop-common org.apache.hadoop:hadoop-mapreduce-client-core org.apache.hbase:hbase-common org.apache.hbase:hbase-shaded-client org.apache.hbase:hbase-shaded-testing-util org.apache.iceberg:iceberg-api org.apache.iceberg:iceberg-core org.apache.iceberg:iceberg-data org.apache.iceberg:iceberg-gcp org.apache.iceberg:iceberg-parquet org.apache.kafka:kafka-clients org.apache.logging.log4j:log4j org.apache.logging.log4j:log4j-api org.apache.logging.log4j:log4j-core org.apache.maven.plugin-testing:maven-plugin-testing-harness org.apache.maven.plugin-tools:maven-plugin-annotations org.apache.maven.surefire:surefire-junit47 org.apache.maven:maven-artifact org.apache.maven:maven-compat org.apache.maven:maven-core org.apache.maven:maven-plugin-api org.apache.parquet:parquet-avro org.apache.parquet:parquet-hadoop org.assertj:assertj-core org.awaitility:awaitility org.bouncycastle:bcpkix-jdk15on org.checkerframework:checker-qual org.codehaus.jettison:jettison org.codehaus.mojo:extra-enforcer-rules org.codehaus.plexus:plexus-archiver org.conscrypt:conscrypt-openjdk-uber org.eclipse.jetty.ee8:jetty-ee8-servlet org.eclipse.jetty.ee8:jetty-ee8-webapp org.eclipse.jetty.websocket:jetty-websocket-jetty-api org.eclipse.jetty.websocket:jetty-websocket-jetty-client org.eclipse.jetty.websocket:jetty-websocket-jetty-common org.eclipse.jetty:jetty-alpn-client org.eclipse.jetty:jetty-client org.eclipse.jetty:jetty-http org.eclipse.jetty:jetty-io org.eclipse.jetty:jetty-security org.eclipse.jetty:jetty-server org.eclipse.jetty:jetty-util org.eclipse.jetty:jetty-util-ajax org.eclipse.jetty:jetty-xml org.freemarker:freemarker org.hamcrest:hamcrest org.hamcrest:hamcrest-core org.hamcrest:hamcrest-library org.json:json org.junit.jupiter:junit-jupiter-api org.junit.jupiter:junit-jupiter-engine org.mock-server:mockserver-client-java org.mock-server:mockserver-junit-rule org.mock-server:mockserver-netty org.mockito:mockito-core org.mockito:mockito-inline org.mongodb:mongodb-driver-core org.mongodb:mongodb-driver-sync org.neo4j.driver:neo4j-java-driver org.neo4j.importer:import-spec org.neo4j:neo4j-cypher-dsl-schema-name-support org.openjdk.nashorn:nashorn-core org.postgresql:postgresql org.reflections:reflections org.scassandra:cql-antlr org.scassandra:java-client org.slf4j:slf4j-api org.slf4j:slf4j-jdk14 org.slf4j:slf4j-nop org.slf4j:slf4j-simple org.springframework:spring-expression org.syncope.identityconnectors:framework org.syncope.identityconnectors:framework-internal org.tensorflow:tensorflow-core-native org.testcontainers:hivemq org.testcontainers:mockserver org.testcontainers:testcontainers org.threeten:threetenbp org.twdata.maven:mojo-executor org.xerial.snappy:snappy-java org.yaml:snakeyaml orjson overrides packaging parameterized pg8000 pillow pluggy propcache proto-plus protobuf psycopg2-binary pyarrow pyarrow-hotfix pyasn1 pyasn1-modules pycparser pydantic-core pygments pyhamcrest pyjsparser pymilvus pymongo pymysql pyparsing pytest-timeout pytest-xdist python-dateutil python-dotenv python-tds pytz pyyaml referencing regex requests-mock rpds-py rsa safetensors scikit-learn scipy scramp secretstorage setuptools six sniffio sortedcontainers sqlalchemy-pytds sqlparse sympy tenacity testcontainers threadpoolctl tokenizers torch torchvision tqdm transformers triton typing-extensions typing-inspection tzdata tzlocal ujson urllib3 virtualenv-clone websockets wrapt yarl zipp zstandard

评论

ホーム - Wiki
Copyright © 2011-2026 iteam. Current version is 2.155.1. UTC+08:00, 2026-04-03 02:30
浙ICP备14020137号-1 $お客様$