Getting started
Basics
Domains & Origins
Performance

Configuration
Basics
Conditions
Dictionaries
Domains & Origins
Request settings
Cache settings
Headers
Responses
Performance
Custom VCL
Image optimization
Video

Security
Access Control Lists
Monitoring and testing
Securing communications
Security measures
TLS
Web Application Firewall

Integrations
Logging endpoints
Non-Fastly services

Diagnostics
Streaming logs
Debugging techniques
Common errors

Account info
Account management
Billing
User access and control

Reference

    Log streaming: Kafka

      Last updated June 18, 2019

    Fastly's Real-Time Log Streaming feature can send log files to Apache Kafka. Kafka is an open-source, high-throughput, low-latency platform for handling real-time data feeds.

    Prerequisites

    Before adding Apache Kafka as a logging endpoint for Fastly services, ensure Kafka is running on a remote server. You’ll need to know the hostname or IP address of one or more servers (Brokers) and the category or feed name to which messages will be stored (Topic). For more information on setting up Kafka see the Apache Kafka Quickstart guide.

    Adding Kafka as a logging endpoint

    Follow these instructions to add Kafka as a logging endpoint:

    1. Review the information in our Setting Up Remote Log Streaming guide.
    2. Click the Apache Kafka logo. The Create an Apache Kafka endpoint page appears.

      the create an Apache Kafka endpoint page

    3. Fill out the Create an Apache Kafka endpoint fields as follows:
      • In the Name field, type a human-readable name for the endpoint.
      • In the Log format field, optionally type an Apache-style string or VCL variables to use for log formatting. The Apache Common Log format string appears in this field by default. See our guidance on format strings for more information.
      • In the Brokers field, type the hostname or IP address of one or more servers (Kafka brokers). Specify multiple servers using a comma-separated string.
      • In the Topic field, type the name of the topic to send logs to.
      • In the Write acknowledgement area, select the appropriate write acknowledgement a leader must receive before a write is considered successful.
      • In the Compression codec area, select the appropriate codec to use for compression of your logs.
      • From the TLS menu, select No to disable encryption for the Kafka endpoint, or Yes to enable it. When you select Yes, additional TLS fields appear.
      • In the TLS Hostname field, optionally type the hostname used to verify the server's certificate. This can be either the Common Name (CN) or Subject Alternate Name (SAN). If the hostname is not specified, the hostname of the first broker in the Brokers field will be used. This field only appears when you select Yes from the Use TLS menu.
      • In the TLS CA certificate field, optionally copy and paste the Certificate Authority (CA) certificate used to verify that the origin server's certificate is valid. The certificate you upload must be in PEM format. Consider uploading the certificate if it's not signed by a well-known certificate authority. This value is not required if your TLS certificate is signed by a well-known authority. This field only appears when you select Yes from the Use TLS menu.
      • In the TLS client certificate field, optionally copy and paste the TLS client certificate used to authenticate to the origin server. The TLS client certificate you upload must be in PEM format and must be accompanied by a client certificate. A TLS client certificate allows your server to authenticate that Fastly is performing the connection. This field only appears when you select Yes from the Use TLS menu.
      • In the TLS client key field, optionally copy and paste the TLS client key used to authenticate to the backend server. The TLS client key you upload must be in PEM format and must be accompanied by a TLS client certificate. A TLS client key allows your server to authenticate that Fastly is performing the connection. This field only appears when you select Yes from the Use TLS menu.
    4. Click the Advanced options link of the Create an Apache Kafka endpoint page. The Advanced options appear.

      the advanced options on the  create an Apache Kafka endpoint page

    5. In the Placement area, select where the logging call should be placed in the generated VCL. Valid values are Format Version Default, None, and waf_debug (waf_debug_log). Selecting None creates a logging object that can only be used in custom VCL. See our guide on WAF logging for more information about waf_debug_log.
    6. Click the Create button to create the new logging endpoint.
    7. Click the Activate button to deploy your configuration changes.
    Back to Top