add microservices demo #509

travisoneill · 2016-09-08T21:42:29Z

PR from issue #499

googlebot · 2016-09-08T21:42:31Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
If you signed the CLA as a corporation, please let us know the company's name.

travisoneill · 2016-09-08T22:04:11Z

Signed CLA

googlebot · 2016-09-08T22:04:12Z

CLAs look good, thanks!

travisoneill · 2016-09-23T19:22:02Z

Any changes needed here?

theacodes · 2016-09-23T19:46:11Z

Hey @travisoneill apologies for the long response here - I'm at a team summit but will be able to review & merge this next week.

theacodes

Finished a pass, let me know when you're ready for another review.

theacodes · 2016-09-26T16:16:20Z

appengine/flexible/api_gateway/.gitignore

@@ -0,0 +1,23 @@
+# Ignore bundler config.


This file shouldn't be needed anymore, right?

theacodes · 2016-09-26T16:16:54Z

appengine/flexible/api_gateway/README.md

@@ -0,0 +1,80 @@
+# Python Google Cloud Microservices Example - API Gateway


I'd actually prefer if we put this sample in a subdirectory called multiple_services.

theacodes · 2016-09-26T16:17:13Z

appengine/flexible/api_gateway/README.md

@@ -0,0 +1,80 @@
+# Python Google Cloud Microservices Example - API Gateway
+
+This example demonstrates how to deploy multiple python services to [App Engine Flexible Environment](https://cloud.google.com/appengine/docs/flexible/)


Just 'flexible environment` (no caps).

theacodes · 2016-09-26T16:18:36Z

appengine/flexible/api_gateway/README.md

+5. To run each server locally:
+```Bash
+$ source env/bin/activate
+$ export FLASK_DEBUG=1 #optional to run in debug mode


You shouldn't need this.

theacodes · 2016-09-26T16:19:08Z

appengine/flexible/api_gateway/README.md

+
+Each directory contains an `app.yaml file`.  This file will be the same as if
+each service were its own project except for the `service` line. For the gateway:
+```YAML


Just link to the files, no need to include them here. (They'll inevitably get out of sync)

theacodes · 2016-09-26T16:19:47Z

appengine/flexible/api_gateway/gateway/api_gateway.py

+@app.route('/hello/<service>')
+def say_hello(service):
+    services = {
+        'flask1': { 'url': 'https://flask1-dot-flask-algo.appspot.com/', 'send': False },


Hard-coded urls?

@jonparrott The URLs will depend on the project ID of when the user deploys the demo, so he will have to fill those in. (Unless there is another way to handle that?) I could just leave a comment there as a placeholder and add something in the readme to instruct the user to enter his urls there. Would that be good?

There's an environment variable I think you can use: https://cloud.google.com/appengine/docs/flexible/python/runtime#environment_variables

@jonparrott It doesn't seem to like the environment variable. I'm using os.environ['GAE_APPENGINE_HOSTNAME']. Getting 500's and I have now added print(os.environ) to see what I am working with but can't seem to find the logs. I am getting the logged requests, but not able to see the actual error messages or logs from my print command. Do you know where I would go to access this?

@travisoneill they should be logged under the stdout log in the cloud console.

If I understand correctly, we're making some changes to the environment variables. It might be better to just do something like this:

PROJECT_ID = os.environ.get('GAE_LONG_APP_ID') DEFAULT_MODULE_URL = '{}.appspot.com'.format(PROJECT_ID)

@jonparrott Already done. I constructed the urls with GAE_LONG_APP_ID. Made your other changes as well and ready for review.

theacodes

Seems like we could still get the point across of this sample without having 4 separate services. Can we reduce it to 2 or 3?

theacodes · 2016-10-04T16:38:35Z

appengine/flexible/multiple_services/api_gateway/.gitignore

@@ -0,0 +1,10 @@
+# Ignore python virtual environment


hmm, are these not all already a part of the top-level .gitignore?

theacodes · 2016-10-04T16:42:15Z

appengine/flexible/multiple_services/api_gateway/README.md

+
+This example demonstrates how to deploy multiple python services to [App Engine flexible environment](https://cloud.google.com/appengine/docs/flexible/)
+
+## To Run Locally


This local run story is pretty rough, and some of that blame of course falls on Google. Do you have any ideas on how we can improve this for this sample (procfile + honcho)? Overall?

@jonparrott I've got it running now so that it will work as previously in production, and the user can run locally by using a --development flag from the command line without having to do anything with the code. Is that sufficient?

Sounds reasonable, push the code up?

OK. Needs the other changes though, and testing with deployed code. Will push up now so you can see and again when changes complete.

theacodes · 2016-10-04T16:43:11Z

appengine/flexible/multiple_services/api_gateway/README.md

+### YAML Files
+
+Each directory contains an `app.yaml file`.  This file will be the same as if
+each service were its own project except for the `service` line. For the gateway:


I would prefer to remove the line "This file will be the same as if
+each service were its own project except for the service line.", perhaps "These files all describe a separate App Engine service within the same project".

theacodes · 2016-10-04T16:43:33Z

appengine/flexible/multiple_services/api_gateway/flask/app.yaml

+
+runtime_config:
+  python_version: 3
+manual_scaling:


blank newline above this, please.

theacodes · 2016-10-04T16:43:46Z

appengine/flexible/multiple_services/api_gateway/flask/app.yaml

@@ -0,0 +1,11 @@
+# [START runtime]


No need for these fences (We'll add them when/if we add this to the docs)

travisoneill · 2016-10-04T19:17:58Z

@jonparrott I was basing it off my actual app structure, but there is no need for it to be that many. I could cut it down to 3, or even 2 if I only had the static file server behind the gateway.

theacodes · 2016-10-04T19:22:04Z

@travisoneill let's do that, simple is better. Our mantra is "do the simplest thing that works".

theacodes · 2016-10-06T16:49:14Z

appengine/flexible/multiple_services/api_gateway/flask/flask_server.py

@@ -12,7 +12,7 @@ def root():
    return 'This is flask server 1.'

 if __name__  == "__main__":
-    if len(sys.argv) > 1 and sys.argv[1] == 'development':
+    if len(sys.argv) > 1 and sys.argv[1] == '--development':
        app.run(port=int(8001))


debug=True?

theacodes · 2016-10-06T16:49:28Z

appengine/flexible/multiple_services/api_gateway/flask/flask_server.py

@@ -12,7 +12,7 @@ def root():
    return 'This is flask server 1.'

 if __name__  == "__main__":
-    if len(sys.argv) > 1 and sys.argv[1] == 'development':
+    if len(sys.argv) > 1 and sys.argv[1] == '--development':


How awful would it be to introduce argparse here?

You could also make the port an arg. :)

theacodes · 2016-10-06T16:50:07Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

 @app.route('/environment')
 def get_env():
-    return str(os.environ)
+    configuration = 'production'


unused variable?

theacodes · 2016-10-06T16:50:27Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

+        'production': 'https://static-dot-' + os.environ.get('GAE_LONG_APP_ID', '') + '.appspot.com'
+    }
+    environment = app.config['MODE']
+    print(environment)


leftover prints?

theacodes · 2016-10-06T16:50:42Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

-    print('https://static-dot-' + os.environ['GAE_LONG_APP_ID'] + '.appspot.com')
-    # return 'https://static-dot-' + os.environ['GAE_LONG_APP_ID'] + '.appspot.com'
-    res = requests.get('https://static-dot-' + os.environ['GAE_LONG_APP_ID'] + '.appspot.com')
+    urls = {


You could make this a constant?

travisoneill · 2016-10-06T20:26:49Z

@jonparrott Made this before I saw your comments. Would be happy to add argparse if you think its necessary.
Changes:
-Reduced number of services to 2
-App runs locally with the --development flag
-Added services_config.py to create a map that is passed into the flask app's config. New services can now be added easily to the gateway by adding a key value pair with the format [name] : [port] to the object at the top of the file.

theacodes · 2016-10-06T21:25:31Z

Would be happy to add argparse if you think its necessary.

Please do. :)

theacodes · 2016-10-06T21:23:55Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

-@app.route('/test')
-def test():
-    return "GATEWAY OPERATIONAL"
+    print(app.config['SERVICE_MAP'])


stray print?

theacodes · 2016-10-06T21:24:16Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

-def test():
-    return "GATEWAY OPERATIONAL"
+    print(app.config['SERVICE_MAP'])
+    return app.config


returning the entire app.config can be a security issue, can you limit the scope here?

Yeah. Meant to take that out that was just for me to play around with!

travisoneill · 2016-10-06T23:11:44Z

argparser added and readme updated to reflect changes in the structure of the app

theacodes

This is looking close!

theacodes · 2016-10-07T19:38:50Z

appengine/flexible/multiple_services/api_gateway/README.md

+Each directory contains an `app.yaml` file.  These files all describe a
+separate App Engine service within the same project.
+
+For the gateway:


You can use gcloud to deploy multiple services at once:

gcloud app deploy gateway/app.yml static/app.yml

theacodes · 2016-10-07T19:39:31Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

+#create service map for production urls unless --development flag added
+app.config['SERVICE_MAP'] = services_config.map_services('production')
+
+#setup arg parser to handle development flag


stick this down in the if __main__ section.

theacodes · 2016-10-07T19:41:04Z

appengine/flexible/multiple_services/api_gateway/gateway/services_config.py

+def production_url(service_name):
+    project_id = os.environ.get('GAE_LONG_APP_ID') or 'flask-algo'
+    project_url = project_id + '.appspot.com'
+    template = 'https://{}{}'


use named interpolation if you use a template more than once, e.g. https://{service_name_prefix}{project_url}.

Named interpolation was really verbose so I removed the template.

if service_name == 'default': return 'https://{}'.format(project_url) else: return 'https://{}-dot-{}'.format(service_name, project_url)

theacodes · 2016-10-07T19:41:18Z

appengine/flexible/multiple_services/api_gateway/gateway/services_config.py

+        return template.format(service_name + '-dot-', project_url)
+
+def local_url(port):
+    return 'http://localhost:' + str(port)


Use format instead of concatenation.

theacodes · 2016-10-07T19:41:47Z

appengine/flexible/multiple_services/api_gateway/static/static_server.py

+    return app.send_static_file(path), 200, {'Content-Type': mimetype}
+
+if __name__  == "__main__":
+    if len(sys.argv) > 1 and sys.argv[1] == '--development':


travisoneill · 2016-10-09T02:18:26Z

Can't put the production config in the if __name__ block because gunicorn doesn't hit that entrypoint. Tried to move the configuration logic to services_config.py but this breaks because argeparse interferes with gunicorn. app is now created with a factory method services_config.make_app in order to remove the line you wanted gone. I have now removed argparse altogether. The app knows whether or not to run in production or development mode based on the presence of the GAE_LONG_APP_ID environment variable. This seems non-ideal, but I'm all out of ideas.

Have you guys thought of adding an environment variable that specifies production like the NODE_ENV variable in the node flexible environment runtime? Would be really helpful here.

theacodes

@travisoneill I need you to add license headers to all of the .py .js, and .html files before I can merge this.

theacodes · 2016-10-10T19:10:23Z

appengine/flexible/multiple_services/api_gateway/gateway/api_gateway.py

-    if args.development:
-        app.config['SERVICE_MAP'] = services_config.map_services('development')
-    app.run()
+    port = os.environ.get('PORT') or 8000


won't this error because PORT is a string?

theacodes · 2016-10-10T19:12:58Z

appengine/flexible/multiple_services/api_gateway/gateway/services_config.py

+
+def make_app(name):
+    app = Flask(name)
+    environment = 'production' if os.environ.get('GAE_LONG_APP_ID') else 'development'


For the time being, I recommend using os.environ.get('GAE_INSTANCE', os.environ.get('GAE_MODULE_INSTANCE')).

travisoneill · 2016-10-10T22:01:26Z

@jonparrott Changes made. Thanks for all the code reviews! Do you want a similar sample app for the Node environment?

theacodes · 2016-10-11T17:12:32Z

@travisoneill that's up to @jmdobry. :)

Thanks for your contribution!

add code line to set metric label in addition to resource label. make the code sample align with similar samples in other languages.

add microservices demo

f795487

googlebot added the cla: no This human has *not* signed the Contributor License Agreement. label Sep 8, 2016

googlebot added cla: yes This human has signed the Contributor License Agreement. and removed cla: no This human has *not* signed the Contributor License Agreement. labels Sep 8, 2016

switch express to flask and add README

d460cc7

theacodes suggested changes Sep 26, 2016

View reviewed changes

travisoneill added 2 commits September 28, 2016 15:14

make suggested changes

eac614f

make suggested changes

8d5d4fc

theacodes suggested changes Oct 4, 2016

View reviewed changes

add config for running locally

5b8c19c

theacodes reviewed Oct 6, 2016

View reviewed changes

reduce to 2 services and add services config file

7094561

get rid of debug mode

d7cc056

theacodes reviewed Oct 6, 2016

View reviewed changes

add argparser and update readme

010fbcc

theacodes suggested changes Oct 7, 2016

View reviewed changes

make changes

ab1c495

theacodes reviewed Oct 10, 2016

View reviewed changes

make small changes and add licenses

e6ac821

travisoneill added 2 commits October 10, 2016 14:52

fix typo

7b8e8ab

fix typo

33373cf

theacodes merged commit 2e862fe into GoogleCloudPlatform:master Oct 11, 2016

travisoneill mentioned this pull request Oct 11, 2016

Microservices Sample GoogleCloudPlatform/nodejs-docs-samples#233

Closed

arbrown pushed a commit that referenced this pull request Nov 7, 2022

fix: add metric label example to the snippet (#509)

0286f46

add code line to set metric label in addition to resource label. make the code sample align with similar samples in other languages.

dandhlee pushed a commit that referenced this pull request Nov 9, 2022

fix: add metric label example to the snippet (#509)

6ccd5bb

add code line to set metric label in addition to resource label. make the code sample align with similar samples in other languages.

		@@ -0,0 +1,80 @@
		# Python Google Cloud Microservices Example - API Gateway

		@@ -0,0 +1,80 @@
		# Python Google Cloud Microservices Example - API Gateway

		This example demonstrates how to deploy multiple python services to [App Engine Flexible Environment](https://cloud.google.com/appengine/docs/flexible/)


		This example demonstrates how to deploy multiple python services to [App Engine flexible environment](https://cloud.google.com/appengine/docs/flexible/)

		## To Run Locally

add microservices demo #509

add microservices demo #509

Uh oh!

Conversation

travisoneill commented Sep 8, 2016

Uh oh!

googlebot commented Sep 8, 2016

Uh oh!

travisoneill commented Sep 8, 2016

Uh oh!

googlebot commented Sep 8, 2016

Uh oh!

travisoneill commented Sep 23, 2016

Uh oh!

theacodes commented Sep 23, 2016

Uh oh!

theacodes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

theacodes left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

travisoneill commented Oct 4, 2016

Uh oh!

theacodes commented Oct 4, 2016

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

travisoneill commented Oct 6, 2016

Uh oh!

theacodes commented Oct 6, 2016

Uh oh!