Mandatory User-Agent¶
This is a guide on how to implement Kiwi.com RFC #22: Mandatory User Agents for service-to-service communication in Python applications and libraries.
The RFC requires that all HTTP requests made to an internally developed
service must include a User-Agent
header in the following format:
<service name>/<version> (Kiwi.com <environment>) [<system info>]
In order to achieve this, the RFC also requires that all internal services must refuse requests which do not comply with this format. Essentially, the following kinds of changes must be made in all applications and libraries:
all HTTP requests made to an internally developed service must include a
User-Agent
header complying with the KW-RFC-22 formatall internally developed services must refuse requests from clients which do not comply with the KW-RFC-22 format
The kiwi-platform
library helps to resolve both requirements by
providing:
Making HTTP requests - custom client sessions for making HTTP requests and a patching mechanism
Validating client requests - custom middlewares for validating client’s
User-Agent
header
Making HTTP requests¶
The library provides custom sessions for two most used 3rd party libraries:
In order to use them, your application needs to populate the following environment variables:
APP_NAME
- name of the applicationPACKAGE_VERSION
- either the git commit hash or a version number, for example,git-a4f93
or1.0.3
APP_ENVIRONMENT
- a string that matches the environment reported, for example, to Datadog and Sentry: in most cases, this would beproduction
If you have the variables properly configured, you can use the
kw.platform.requests.session.KiwiSession
session which automatically
adds a KW-RFC-22 compliant User-Agent to all HTTP requests:
from kw.platform.requests import KiwiSession
session = KiwiSession()
session.get("https://api.example.com")
For async applications using aiohttp
as an HTTP client, you can use the
kw.platform.aiohttp.session.KiwiClientSession
session:
from kw.platform.aiohttp import KiwiClientSession
async with KiwiClientSession() as client:
async with client.get("https://api.example.com") as resp:
html = await resp.text()
print(html)
In case you are not using requests
or aiohttp
, you can still use
our kw.platform.utils.construct_user_agent()
function to construct the
User-Agent:
import urllib.request
from kw.platform.utils import construct_user_agent
request = urllib.request.Request(
url,
headers={'User-Agent': construct_user_agent()}
)
f = urllib.request.urlopen(request)
print(f.read().decode('utf-8'))
Monkey patching¶
In case you are using requests
or aiohttp
and it would be too
much work to change all HTTP requests calls to use the proper User-Agent
header, you can also monkey patch requests
:
import requests
from kw.platform.requests import patch_with_user_agent
patch_with_user_agent()
requests.get("https://api.example.com")
or with aiohttp
:
import aiohttp
from kw.platform.aiohttp import patch_with_user_agent
patch_with_user_agent()
with aiohttp.ClientSession() as client:
...
You can also use the kw.platform.requests.patch()
or
kw.platform.aiohttp.patch()
functions which provide some additional
patching of the modules like automatic logging of Sunset
HTTP header in
the response body.
HTTP requests in libraries¶
The correct way to handle KW-RFC-22 in internal libraries
such as thief
is to make it possible for the developer
to prepend their app’s User-Agent
header. For example,
this is one way to do it:
from kw.platform.utils import construct_user_agent
from kw.python_library import Client
client = Client(append_user_agent=construct_user_agent())
The client
should make HTTP requests while constructing the User-Agent
similar to this:
app/1.0 (Kiwi.com production) python_library/1.2 python-requests/2.22.1
The library can also directly use the construct_user_agent()
provided by
kiwi-platform
library.
Another way how to handle the KW-RFC-22 in libraries is to make it possible
to pass custom requests.Session
or aiohttp.ClientSession
.
The developer could than use the library like this:
from kw.session.requests import KiwiSession
from kw.python_library import Client
client = Client(session=KiwiSession())
Or in the case of aiohttp
:
from kw.session.aiohttp import KiwiClientSession
from kw.python_library import Client
client = Client(session=KiwiClientSession)
Warning
Libraries making HTTP requests to internal services should never use
KW-RFC-22 compliant User-Agent
header by default, each library should
expect to be provided with a compliant header by the application (either
via arguments or via environment variables). Otherwise all requests made
by the library would end up having the same User-Agent
header in the
logs.
Validating client requests¶
Internal services must validate requests from other internal applications.
This validation is just checking that the HTTP request to the service contains
KW-RFC-22 compliant User-Agent
header. All non-complying requests must be
refused with the HTTP 400 Bad Request
response, the message of the response
must also explain why the request to the service was denied.
For WSGI applications, the kw.platform.wsgi.user_agent_middleware()
middleware can be used for validating headers of incoming requests. In Flask,
applying the middleware can be done like this:
from flask import Flask
from kw.platform.wsgi import user_agent_middleware
app = Flask(__name__)
app.wsgi_app = user_agent_middleware(app.wsgi_app)
app.run()
Or similarly in Django:
from django.core.wsgi import get_wsgi_application
from kw.platform.wsgi import user_agent_middleware
application = user_agent_middleware(get_wsgi_application())
For async applications built with aiohttp
, you can use the
kw.platform.aiohttp.middlewares.user_agent_middleware()
middleware:
from aiohttp import web
from kw.platform.aiohttp.middlewares import user_agent_middleware
app = web.Application(middlewares=[user_agent_middleware])
And you can also use a decorator:
from kw.platform.aiohttp.uitls import mandatory_user_agent
@mandatory_user_agent
def handle(request):
# do stuff
return web.json_response(text="Hello World!")
In case you need to write your own middleware for the validation, you can use
the kw.platform.utils.UserAgentValidator
validator, like this:
from kw.platform.utils import UserAgentValidator
if not UserAgentValidator("generic user-agent").is_valid:
return 400
Note that the middlewares start restricting requests only after reaching
the date configured by settings.KIWI_REQUESTS_RESTRICT_DATETIME
.
Warning
The middlewares also start slowing down requests when the date reaches
settings.KIWI_REQUESTS_SLOWDOWN_DATETIME
and if the date is less then
settings.KIWI_REQUESTS_RESTRICT_DATETIME
(the default).
This can increase busyness and overload a service.
Note
If you want to disable the validation of requests, e.g. for development,
you can set KIWI_ENABLE_RESTRICTION_OF_REQUESTS
environment variable to
false
.