Proofread Dataflow README.md Change-Id: Iaf098cfbcd53f2d16a0b4f02aadbe95595883fee Reviewed-on: https://chromium-review.googlesource.com/c/infra/infra/+/1910600 Auto-Submit: Quinten Yearsley <qyearsley@chromium.org> Reviewed-by: Andrii Shyshkalov <tandrii@google.com> Commit-Queue: Andrii Shyshkalov <tandrii@google.com> Cr-Original-Commit-Position: refs/heads/master@{#27024} Cr-Mirrored-From: https://chromium.googlesource.com/infra/infra Cr-Mirrored-Commit: 98c02ca99fbf13b0311f4d9436676932efa8be79
diff --git a/README.md b/README.md index 735e368..5179635 100644 --- a/README.md +++ b/README.md
@@ -7,8 +7,9 @@ abstractions. You'll notice that workflows are included in this package. Workflows are located -here to simplify job execution, as all non-[standard](https://beam.apache.org/documentation/) -and non-[beam](https://beam.apache.org/documentation/) modules must be packaged +here to simplify job execution, as all +non-[standard](https://beam.apache.org/documentation/) and +non-[beam](https://beam.apache.org/documentation/) modules must be packaged together for job execution. See @@ -29,10 +30,8 @@ # Unit Testing -From the root of the infra repository, run the command -``` -./test.py test packages/dataflow -``` +From the root of the infra repository, run the command `./test.py test +packages/dataflow`. # Workflow Testing @@ -43,28 +42,25 @@ checkout. If you need to set up or update your environment, see [bootstrap/README](../../bootstrap/README.md) for more info. -Next, you must have Google Storage buckets to pass with the --staging_location -and --temp_location options. The name is not important, but for example you -could use `gs://my-dataflow-job/staging`. [Create these](https://cloud.google.com/storage/docs/creating-buckets) -if you don't have them already. +Next, you must have Google Storage buckets to pass with the `--staging_location` +and `--temp_location` options. The name is not important, but for example you +could use `gs://my-dataflow-job/staging`. +[Create these](https://cloud.google.com/storage/docs/creating-buckets) if you +don't have them already. Next, you must have permission within the project to schedule a Dataflow job, and be authenticated to do so. -To check your -authentication status, ensure that you have the Cloud SDK -[installed](https://cloud.google.com/sdk/docs/quickstarts), then run: +To check your authentication status, ensure that you have the Cloud SDK +[installed](https://cloud.google.com/sdk/docs/quickstarts), then run `gcloud +info`. -``` -gcloud info -``` - -If you don't see the correct project id, reach out to an +If you don't see the correct project ID, reach out to an [editor](https://pantheon.corp.google.com/iam-admin/iam) of that project to request access. Finally, run the command below to test your workflow as a remote job. Note: Job -names should match the regular expression [a-z]\([-a-z0-9]{0,38}[a-z0-9]). +names should match the regular expression `[a-z]\([-a-z0-9]{0,38}[a-z0-9])`. ``` python <path-to-dataflow-job> --job_name <pick-a-job-name> \ @@ -73,42 +69,41 @@ --staging_location <staging bucket> \ --temp_location <temp bucket> --save_main_session ``` + Navigate to the [Dataflow console](https://console.cloud.google.com/project) in your browser and you should see your job running. Wait until it succeeds. Running the test will leave behind a directory, `packages/dataflow/dataflow.egg-info`, that you must manually clean up. -To run the workflow locally, first set credentials using -``` -export GOOGLE_APPLICATION_CREDENTIALS=<path_to_credentials> -``` +To run the workflow locally, first set credentials using `export +GOOGLE_APPLICATION_CREDENTIALS=<path_to_credentials>` -Then -``` -python cq_attempts.py --output <dummy_path> --project <name_of_test_project> -``` +Then `python cq_attempts.py --output <dummy_path> --project +<name_of_test_project>` # Updating the package Changes to this directory are automatically mirrored in a synthesized [repo](https://chromium.googlesource.com/infra/infra/packages/dataflow/). To deploy changes to this repository: -* Land the changes. -* Submit a separate CL that updates the version in `setup.py`. -* Build and upload a new wheel. -* Submit a single CL that updates the remote execution recipe and deps.pyl. + + * Land the changes. + * Submit a separate CL that updates the version in `setup.py`. + * Build and upload a new wheel. + * Submit a single CL that updates the remote execution recipe and deps.pyl. Jobs scheduled with the -[remote_execute_dataflow_workflow](../../recipes/recipes/remote_execute_dataflow_workflow.py) +[`remote_execute_dataflow_workflow`](../../recipes/recipes/remote_execute_dataflow_workflow.py) recipe use the version of the job at HEAD but the version of the package pinned in [bootstrap/deps.pyl](../../bootstrap/deps.pyl). So, if you make a breaking change to the package, submit the update first (which will automatically be -picked up by the [package -mirror](https://chromium.googlesource.com/infra/infra/packages/dataflow/)), -then submit the change to the job along with the ref update in deps.pyl together -in one commit. Be sure to follow the instructions in [bootstrap/README.md](../../bootstrap/README.md) -to build and upload the new wheel before submitting the change to deps.pyl. +picked up by the +[package mirror](https://chromium.googlesource.com/infra/infra/packages/dataflow/)), +then submit the change to the job along with the ref update in `deps.pyl` together +in one commit. Be sure to follow the instructions in +[bootstrap/README.md](../../bootstrap/README.md) to build and upload the new +wheel before submitting the change to `deps.pyl`. # Limits @@ -116,7 +111,7 @@ most up to date information on quotas and limits. At the time of writing, there are limits on Dataflow requests per minute, number -of GCE instances (--numWorkers), number of concurrent jobs, monitoring requests, +of GCE instances (`--numWorkers`), number of concurrent jobs, monitoring requests, job creation request size, and number of side input shards. Some of these limits are per user, others are per project, others are per
diff --git a/common/README.md b/common/README.md index 1b76c6d..56b1c6b 100644 --- a/common/README.md +++ b/common/README.md
@@ -1,6 +1,6 @@ The `common` module provides reusable classes for writing Dataflow workflows. -## chops_beam +## chops\_beam For easily constructing readable pipelines with standard defaults. @@ -24,7 +24,7 @@ event = objects.CQEvent.from_bigquery_row(row) ``` -## combine_fns +## combine\_fns Generally useful [Combine Functions](https://beam.apache.org/documentation/programming-guide/#transforms-combine).
diff --git a/common/chops_beam.py b/common/chops_beam.py index c0c3250..4a9036c 100644 --- a/common/chops_beam.py +++ b/common/chops_beam.py
@@ -34,8 +34,8 @@ class BQWrite(beam.io.Write): """Write transform created from a BigQuerySink with convenient defaults. - beam.io.BigQuerySink will automatically add unique insert ids to rows, - which BigQuery uses to prevent duplicate inserts. + beam.io.BigQuerySink will automatically add unique insert IDs to rows, + which BigQuery uses to prevent duplicate inserts. """ def __init__(self, project, table, dataset='aggregated', write_disposition=beam.io.BigQueryDisposition.WRITE_TRUNCATE):
diff --git a/common/combine_fns.py b/common/combine_fns.py index e0a48b7..2ecd6fc 100644 --- a/common/combine_fns.py +++ b/common/combine_fns.py
@@ -6,7 +6,7 @@ class ConvertToCSV(beam.CombineFn): - """Convert elements to CSV format to be written out + """Convert elements to CSV format to be written out. Transform for writing elements out in a CSV format. Can process elements of type dictonary or list. This transform only supports consistent elements, @@ -21,7 +21,7 @@ return [] def iterable(self, obj): - """Returns an iterable for a dictionary or list + """Returns an iterable for a dictionary or list. Sorting dictionary keys assures that the CSV is in the same order for all dictionaries. If given a header, use the header fields as