March 1, 2023

Writing an Analytics Server using Vapor Part 3 - Setting up UserEvent

In the previous post, I created a new Vapor project for my analytics server, added a controller, and created a new test module for it. Now it's time for the endpoint to actually receive and interpret some data. After I have the endpoint receiving and sending back a real object, then I can add database support in the next post.

Review: What do I want in my data

This is an analytics engine. I want to know when a user performs certain actions, and maybe some information about that action. In my real use case, I have slightly different requirements, but for the purposes fo this blog series, I need to know:

  • what action was performed
  • when was it performed
  • an id to uniquely identify this event (for the purposes of the database)
  • which user performed this event (not anything about them, just so I can group actions by user)
  • an extra Boolean flag, just because

adding a blank UserEvent

Vapor provides a protocol called Content (docs) which represents a type that can be received and sent in a route. I create a new type that conforms to Content called UserEvent:

struct UserEvent: Content, Equatable {}

at the moment, it contains, no properties, but I just want to make sure that Vapor can receive and return it. So I change UserEventController's create(req:) method to read a UserEvent from the request and return it instead of a String.

func create(req: Request) async throws -> UserEvent {        
    let event = try req.content.decode(UserEvent.self)
    return event
}

and I replace the one test I had with 2 separate tests:

func test_post_responds_with_200() async throws {

    try await testPOST(UserEvent(action: .start, userID: exampleUserID).toByteBuffer()) { response in
        XCTAssertEqual(response.status, .ok)
    }
}

func test_post_responds_with_UserEvent_that_was_passed_in() async throws {

    let expected = UserEvent()
    try await testPOST(expected.toByteBuffer()) { response in
        let received = try JSONDecoder().decode(UserEvent.self, from: response.body)
        XCTAssertEqual(received, expected)
    }
}

You'll notice that these tests use some helpers that I wrote because I expect to be testing this endpoint a lot. Here they are:

private func testPOST(_ byteBuffer: ByteBuffer,
                     tests: (XCTHTTPResponse) async throws ->(),
                     file: StaticString = #filePath, line: UInt = #line) async throws {
    try await sut.test(.POST, UserEventController.userevents, body: byteBuffer, afterResponse: tests)
}

and

fileprivate extension UserEvent {
    func toByteBuffer() -> ByteBuffer {
        try! JSONEncoder().encodeAsByteBuffer(self, allocator: .init())
    }
}

But the tests won't pass because Vapor expects a content-type in the header in order to decode the request's body, so I adjust the helper method accordingly:

private var defaultHeaders: HTTPHeaders { HTTPHeaders(dictionaryLiteral: ("content-type", "application/json")) }

private func testPOST(_ byteBuffer: ByteBuffer,
                     headers: HTTPHeaders? = nil,
                     tests: (XCTHTTPResponse) async throws ->(),
                     file: StaticString = #filePath, line: UInt = #line) async throws {
    try await sut.test(.POST, UserEventController.userevents, headers: headers ?? defaultHeaders, body: byteBuffer, afterResponse: tests)
}

and I add a test to verify that the headers are needed:

func test_post_responds_with_415_if_given_no_headers() async throws {

    let expected = UserEvent(action: .start, userID: exampleUserID)
    try await testPOST(expected.toByteBuffer(), headers: HTTPHeaders()) { response in
        XCTAssertEqual(response.status, .unsupportedMediaType)
    }
}

And now all tests pass.

adding a timestamp property

It's time to add properties to the UserEvent type, and probably the most important property is the date and time when the event occured. It also turns out to be the msot troublesome to get right.

You see, there are many ways to represent a Date, and we have to be explicit about which one we want to use. The first time I wrote this blog post (😮‍💨) I went through all the steps of trying one approach, then finding that it gave a certain error, then trying a different approach, and on and on. Suffice it to say that:

  • Swift's Date object likes to store dates as a TimeInterval (Double) representing the number of seconds since a certain reference date.
  • Most web applications like to pass dates as an ISO8601-formatted string
  • Vapor tries to respect either approach
  • I ran into some edge cases 🥺

So I ended up writing a separate Date wrapper type that encodes and decodes a date as a TimeInterval. The time interval works for my needs since I don't intend for the content to be human-readable and all processing both in the server and in the client app will be written in Swift.

Along the way, there was an issue with the millisecond part of the date being lost in transit. This is totally expectable. When moving to the custom date wrapper, the issue disappeared, but it still makes sense to me to not worry about milliseconds in an analytics app, so I make sure that the date is always rounded to the nearest second.

struct InternalDate: Codable, Equatable {
    
    let value: Date
    
    init(_ date: Date) {
        self.value = Date(timeIntervalSinceReferenceDate: date.timeIntervalSinceReferenceDate.rounded())
    }
    
    init(from decoder: Decoder) throws {
        
        let container = try decoder.singleValueContainer()
        let dateDouble = try container.decode(Double.self)
        self.value = Date(timeIntervalSinceReferenceDate: dateDouble.rounded())
    }
    
    func encode(to encoder: Encoder) throws {
        try value.timeIntervalSinceReferenceDate.encode(to: encoder)
    }
}

I then added this as a property on UserEvent:

struct UserEvent: Content, Equatable {

    let timestamp: InternalDate
    
	init() {
        self.timestamp = InternalDate(Date())
    }
}

The tests then pass.

I'm really yada-yada-yada-ing this. It took a long time and much testing to get things to work as intended. I'm sure that there's a better way, but this works and I have confidence it won't conflict with anything else I plan to do later on. So I kept it.

adding other simple properties

Adding the id and flag properties was simple and had no effect on the tests:

struct UserEvent: Content, Equatable {
    let id: UUID
        
    let flag: Bool
    let timestamp: InternalDate
        
    init(flag: Bool = false) {
        self.id = UUID()
                
        self.flag = flag
    }
}

though it requried one more test to be added just to be safe:

func test_post_responds_with_UserEvent_with_same_flag_as_what_was_passed_in() async throws {

    let sent = UserEvent(flag: true)
    try await testPOST(sent.toByteBuffer()) { response in
        let received = try JSONDecoder().decode(UserEvent.self, from: response.body)
        XCTAssert(received.flag)
    }
}

Adding the userID property required slightly more work since it required changing the init:

struct UserEvent: Content, Equatable {
    let id: UUID
    
    let userID: UUID
    
    let flag: Bool
    let timestamp: InternalDate
        
    init(userID: UUID, flag: Bool = false) {
        self.id = UUID()
        
        self.userID = userID
        
        self.timestamp = InternalDate(Date())
        self.flag = flag
        self.action = action
    }
}

which required altering the tests to include a userID:

private var exampleUserID: UUID { UUID() }

func test_post_responds_with_200() async throws {

    try await testPOST(UserEvent(action: .start, userID: exampleUserID).toByteBuffer()) { response in
        XCTAssertEqual(response.status, .ok)
    }
}

...

Adding the Action property

The action is slightly more involved. I want to only accept a certain subset of actions, yet I want the action to be sent to the endpoint as a string. So I create an Action enum inside the namespace of the UserEvent struct.

    enum Action: String, Codable {
        case start, pause, stop
    }

and add a property and a parameter to the init:

struct UserEvent: Content, Equatable {
    let id: UUID
    
    let userID: UUID
    
    let flag: Bool
    let timestamp: InternalDate
    
    enum Action: String, Codable {
        case start, pause, stop
    }
    let action: Action
    
    init(action: Action, userID: UUID, flag: Bool = false) {
        self.id = UUID()
        
        self.userID = userID
        
        self.timestamp = InternalDate(Date())
        self.flag = flag
        self.action = action
    }
}

I of course update the tests to make sure that the action is being sent:

func test_post_responds_with_UserEvent_with_same_action_as_what_was_passed_in() async throws 	{

    let sent = UserEvent(action: .pause, userID: exampleUserID, flag: true)
    try await testPOST(sent.toByteBuffer()) { response in
        let received = try JSONDecoder().decode(UserEvent.self, from: response.body)
        XCTAssertEqual(received.action, sent.action)
    }
}

and update the other tests as needed:

func test_post_responds_with_200() async throws {

    try await testPOST(UserEvent(action: .start, userID: exampleUserID).toByteBuffer()) { response in
        XCTAssertEqual(response.status, .ok)
    }
}

...

But what if an unexpected string is passed in for the action. In order to test this, I need to write a test that passes the raw json to see what happens.

I start by writing a test that I know will work:

private var exampleValidUserEventProperties: [String:Any] {
    [
        "id": UUID().uuidString,
        "userID": UUID().uuidString,
        "timestamp": Date().timeIntervalSinceReferenceDate.rounded(),
        "flag": true,
        "action": UserEvent.Action.start.rawValue
    ]
}

func test_post_responds_with_200_if_given_valid_json() async throws {
            
    let data = try JSONSerialization.data(withJSONObject: exampleValidUserEventProperties)
    try await testPOST(ByteBuffer(data: data)) { response in
        XCTAssertEqual(response.status, .ok)
    }
}

I then write a test that sends good data for all keys except for the action to verify that I get a 400 status code:

func test_post_responds_with_400_if_given_unexpected_action() async throws {
    
    var invalidActionProperties = exampleValidUserEventProperties
    invalidActionProperties["action"] = "something unexpected"
    let data = try JSONSerialization.data(withJSONObject: invalidActionProperties)
    
    try await testPOST(ByteBuffer(data: data)) { response in
        XCTAssertEqual(response.status, .badRequest)
    }
}

Testing Against Raw JSON

This made me wonder, though, what would happen if extra data were sent in the json:

func test_post_responds_with_200_if_given_unexpected_extra_data_in_payload() async throws {
    
    var propertiesWithExtraValues = exampleValidUserEventProperties
    propertiesWithExtraValues["some_other_key"] = "some_invalid_value"
    let data = try JSONSerialization.data(withJSONObject: propertiesWithExtraValues)
    
    try await testPOST(ByteBuffer(data: data)) { response in
        XCTAssertEqual(response.status, .ok)
    }
}

or if a needed key were left out of the json:

func test_post_responds_with_400_if_not_given_userID_in_payload() async throws {
    
    var propertiesWithMissingValues = exampleValidUserEventProperties
    propertiesWithMissingValues.removeValue(forKey: "userID")
    let data = try JSONSerialization.data(withJSONObject: propertiesWithMissingValues)
    
    try await testPOST(ByteBuffer(data: data)) { response in
        XCTAssertEqual(response.status, .badRequest)
    }
}
...

And with that the UserEvent struct is done. I don't yet save the data to a database, but I know that it is read in and returned appropriately, and I know that if good data is provided, the server will respond with a 400-type status code.

Posts in this Series:

Tagged with: