Log streaming: Kafka
Last updated November 06, 2019
IMPORTANT: This information is part of a limited availability release. For more information, see our product and feature lifecycle descriptions.
NOTE: This logging endpoint is disabled by default. To enable this endpoint for your account, contact email@example.com and request it.
NOTE: Fastly does not provide direct support for third-party services. See Fastly's Terms of Service for more information.
Before adding Apache Kafka as a logging endpoint for Fastly services, ensure Kafka is running on a remote server. You’ll need to know the hostname or IP address of one or more servers (Brokers) and the category or feed name to which messages will be stored (Topic). For more information on setting up Kafka see the Apache Kafka Quickstart guide.
Adding Kafka as a logging endpoint
Follow these instructions to add Kafka as a logging endpoint:
- Review the information in our Setting Up Remote Log Streaming guide.
Click the Apache Kafka Create endpoint button. The Create an Apache Kafka endpoint page appears.
- Fill out the Create an Apache Kafka endpoint fields as follows:
- In the Name field, type a human-readable name for the endpoint.
- In the Log format field, optionally type an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. See our guidance on format strings for more information.
- In the Brokers field, type the hostname or IP address of one or more servers (Kafka brokers). Specify multiple servers using a comma-separated string.
- In the Topic field, type the name of the topic to send logs to.
- In the Write acknowledgement area, select the appropriate write acknowledgement a leader must receive before a write is considered successful.
- In the Compression codec area, select the appropriate codec to use for compression of your logs.
- From the TLS menu, select No to disable encryption for the Kafka endpoint, or Yes to enable it. When you select Yes, additional TLS fields appear.
- In the TLS Hostname field, optionally type the hostname used to verify the server's certificate. This can be either the Common Name (CN) or Subject Alternate Name (SAN). If the hostname is not specified, the hostname of the first broker in the Brokers field will be used. This field only appears when you select Yes from the Use TLS menu.
- In the TLS CA certificate field, optionally copy and paste the certification authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certification authority. This value is not required if your TLS certificate is signed by a well-known authority. This field only appears when you select Yes from the Use TLS menu.
- In the TLS client certificate field, optionally copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection. This field only appears when you select Yes from the Use TLS menu.
- In the TLS client key field, optionally copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection. This field only appears when you select Yes from the Use TLS menu.
Click the Advanced options link of the Create an Apache Kafka endpoint page. The Advanced options appear.
- In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about
- Click the Create button to create the new logging endpoint.
- Click the Activate button to deploy your configuration changes.