Go: Getting Alerts into the Slack Channel

Hi, Go devs! When we work in a complex system, especially in microservices, it's very hard to trace the logs of errors, warnings etc. If we can build a package like that if an error or warning occurs in any of the services, we can get an alert in a slack channel with the necessary information. It will make our life easier and save our time to trace the bugs. I have developed such a package with Go in one of our projects. Now I am going to share my experience with you.

Defining structs

First, we need to generate some structs of the slack message body according to the slack documentation.

type SlackRequestBody struct {
    Attachments []Attachments `json:"attachments"`
}
type Text struct {
    Type  string `json:"type"`
    Text  string `json:"text"`
    Emoji *bool  `json:"emoji,omitempty"`
}
type Fields struct {
    Type string `json:"type,omitempty"`
    Text string `json:"text,omitempty"`
}
type Blocks struct {
    Type   string    `json:"type,omitempty"`
    Text   *Text     `json:"text,omitempty"`
    Fields []*Fields `json:"fields,omitempty"`
}
type Attachments struct {
    Color  string   `json:"color,omitempty"`
    Blocks []Blocks `json:"blocks,omitempty"`
}

Defining functions of components

Now it's time to design the message body. Slack messages may contain several components like Single Block, Section Block, Text, Fields etc. So we have defined some functions which will generate the components of the messages.

// addSingleBlock will add a single column block
func addSingleBlock(typ string, text *Text) Blocks {
    block := Blocks{
        Type: typ,
        Text: text,
    }
    return block
}

// addSectionBlock will add a multi-column block
func addSectionBlock(fields []*Fields) Blocks {
    block := Blocks{
        Type:   "section",
        Fields: fields,
    }
    return block
}

// addText will generate a text of type plain_text/mrkdwn
func addText(typ string, txt string, emoji *bool) *Text {
    text := Text{
        Type:  typ,
        Text:  txt,
        Emoji: emoji,
    }
    return &text
}

// addField will add a field of type mrkdwn
func addField(typ string, txt string) *Fields {
    field := Fields{
        Type: typ,
        Text: txt,
    }
    return &field
}

Block: We can consider the block as a row. The full message is the combination of some blocks.

Section Block: To add multiple columns in a row, we need to create fields first where each field is a column and then place the fields in a Section Block.

Single Block: If we want to add a single column row, we can use the single block there.

Text: Text can be designed in multiple ways like markdown, plain text, header etc.

Client Request

The client request will be like this.

type ClientRequest struct {
    Header      string   `json:"header"`
    ServiceName string   `json:"service_name"`
    Summary     string   `json:"summary"`
    Metadata    string   `json:"metadata"`
    Details     string   `json:"details"`
    Status      int      `json:"status"`
    Mentions    []string `json:"mentions"`
}

Header: It will show at the top as the type of message. The header depends on the status.

ServiceName: The name of the Service name will be used to know from which service the alert has come.

Summary: Summary represents the title of the alert.

Metadata: If any additional information is provided for the alert, it will be received through Metadata.

Details: Details information will be provided through the Details key.

Mentions: We can mention someone in the message through the mentions key. For example, if we mention all the members of the channel, we will send @here.

Status: We have considered 3 types of statuses here. Success(1), Warning(2) and Alert(3).

Request processing and preparing request body for slack

Now it's time to process the request and generate the actual message body through the functions we have written previously.

headerText := addText("plain_text", headerTitle, &emoji)
headerBlock := addSingleBlock("header", headerText)
serviceNameField := addField("mrkdwn", "*Service:*\n"+serviceName)
serviceLogTimeField := addField("mrkdwn", "*Created At:*\n"+currentTimeStr)
serviceInfoBlock := addSectionBlock([]*Fields{serviceNameField, serviceLogTimeField})
summaryField := addField("mrkdwn", "*Summary:*\n"+summary)
summaryBlock := addSectionBlock([]*Fields{summaryField})
metadataText := addText("mrkdwn", "*Metadata:*\n"+metadata, nil)
metadataBlock := addSingleBlock("section", metadataText)

Here, we have passed type and text to our pre-defined functions. Types can be plain_text, header, mrkdwn etc. We need to place the text in separate blocks. The header should be a single-column row. So we have placed it in Single Block. Service name and Log event time should be placed in a single row. This row will be a multi-column row so that we can use Section Block here.

var detailsBlocks []Blocks
detailsArr := methods.Chunks(details, 2000)
for ind, detail := range detailsArr {
    if ind == 0 {
        detailsField := addField("mrkdwn", "*Details:*\n")
        detailsBlock := addSectionBlock([]*Fields{detailsField})
        detailsBlocks = append(detailsBlocks, detailsBlock)
    }
    detailsText := addText("mrkdwn", "```"+detail+"```", nil)
    detailsBlock := addSingleBlock("section", detailsText)
    detailsBlocks = append(detailsBlocks, detailsBlock)
}

Slack doesn't support more than 3000 characters in a single block. We have chunked our details text and kept it separate blocks containing 2000 characters in each block.

blocks := []Blocks{headerBlock, serviceInfoBlock, summaryBlock, metadataBlock}
for ind, block := range detailsBlocks {
    if ind < 46 {
        blocks = append(blocks, block)
    }
}

When all blocks are prepared, we have added them to the blocks array. According to the slack documentation, we can add maximum of 50 blocks in a single message. So we have handled this.

const (
    Success = iota + 1
    Warning
    Alert
)

var StatusMap = map[int]string{
    Success: "#00663a",
    Warning: "#eda200",
    Alert:   "#9e0505",
}

color := StatusMap[Warning]

if v, ok := StatusMap[status]; ok {
    color = v
}

We have set the color of the message block according to the status of the log. We have worked with three status types Success, Warning and Alert here.

attachment := Attachments{
    Color:  color,
    Blocks: blocks,
}

return []Attachments{attachment}

At the last stage of design customization, we put the Blocks and Colors in the Attachments and returned the array of Attachments.

Sending the request to Slack

Now it's time for sending the message to a specific slack channel.

slackBody, _ := json.Marshal(SlackRequestBody{Attachments: attachments})
req, err := http.NewRequest(http.MethodPost, webhookUrl, bytes.NewBuffer(slackBody))
if err != nil {
    return err
}
req.Header.Add("Content-Type", "application/json")
client := &http.Client{Timeout: 20 * time.Second}
resp, err := client.Do(req)
if err != nil {
    return err
}
buf := new(bytes.Buffer)
_, err = buf.ReadFrom(resp.Body)
if err != nil {
    return err
}
if buf.String() != "ok" {
    fmt.Print(resp.Status, "\n")
    return errors.New("non-ok response returned from slack")
}

Here, we have marshaled our request body. Then we converted it to buffer and sent the request body to the slack channel through the webhook URL. Finally, we have checked whether the response string is ok or not.

Client Initialisation:

We have created our connection here.

var client GoSlackClient

type GoSlackClient struct {
    webhookUrl string
}
func NewGoSlackClient(webhookUrl string) SlackGoClient {
    client = GoSlackClient{
        webhookUrl: webhookUrl,
    }
    return client
}

Here, we have created a client request struct GoSlackClient by which we receive the request from the client side. Webhook URL should be passed through the method NewGoSlackClient. Then we initialize our Slack Client with the webhook URL.

Logger

We are now very close to our final destination. Our slack message service is ready. Now we can use it to send the log messages. Here, we have used logrus package to log the events. We have customized the package functions as per our needs. You can use any other logger packages and customize as your need and process them for sending information to the desired channel.

var goSlackClient *goslack.GoSlackClient
var serviceName string

func SetSlackLogger(webhookUrl, service string) {
    client := goslack.NewGoSlackClient(webhookUrl)
    goSlackClient = &client
    serviceName = service
}

We have initialized our slack client here. We pass the webhook URL and service name to initialize the client.

func ProcessAndSend(slackLogReq SlackLogRequest, status int, logType string) error{
    if slackgoClient != nil {
        msg, err := json.MarshalIndent(&slackLogReq, "", "\t")
        if err != nil {
            return err
        }
        if msg != nil {
            clientReq := slackgo.ClientRequest{
                Header:      slackLogReq.Level,
                ServiceName: serviceName,
                Summary:     logType + " Log from " + serviceName,
                Details:     string(msg),
                Status:      status,
            }
            err = slackgoClient.Send(clientReq)
            if err != nil {
                return err
            }
        }
    }
    return nil
}

The ProcessAndSend function will generate the ClientRequest for our slack service and finally send it to the slack channel. Previously, we have used json.MarshalIndent to convert the struct into formatted JSON.

var logger = logrus.New()

// Error logs a message at level Error on the standard logger.
func Error(args ...interface{}) {
    entry := logger.WithFields(logrus.Fields{})
    entry.Data["file"] = fileInfo(2)
    entry.Error(args...)
    slackLogReq := SlackLogRequest{
        Message: fmt.Sprint(args...),
        File:    fileAddressInfo(2),
        Level:   "error",
    }
    _ = ProcessAndSend(slackLogReq, goslack.Alert, "Error")
}

func fileAddressInfo(skip int) string {
    _, file, line, _ := runtime.Caller(skip)
    return fmt.Sprintf("%s:%d", file, line)
}

Here, the Error() function will receive the logs as an interface. Then we trace the location of the error that occurred through fileInfo function. We have used fmt.Sprint() to convert the full log into string. After that, we send the request to our ProcessAndSend function.

Then we need to configure the logger from our service.

webhookUrl := "webhook url"
service := "service name"
logger.SetSlackLogger(webhookUrl, service)

We have configured the logger with webhook URL and the service name. Finally, we will call it from the desired parts of the code.

logger.Error("Error occurred ", e.Error(), nil)

Output

Finally, we have got output in our slack channel like this.

The source code is available here.